phi2-2b-absa: Fine-Tuned Aspect-Based Sentiment Analysis Model

Model Description

Fine-Tuning Details

Dataset

• Name: semeval2016-full-absa-reviews-english-translated-resampled
• Description: Annotated dataset for ABSA containing sentences, aspects, sentiments, and additional contextual text. It is split into train and test sets.

Model Architecture

• Base Model: Microsoft Phi-2
• Fine-Tuned Model: phi2-2b-absa

Fine-Tuning Parameters

• LoRA Attention Dimension (lora_r): 64
• LoRA Scaling Parameter (lora_alpha): 16
• LoRA Dropout Probability (lora_dropout): 0.1

BitsAndBytes Quantization

• Activate 4-bit Precision: True
• Compute Dtype for 4-bit Models: float16
• Quantization Type: nf4

Training Parameters

• Number of Training Epochs: 1
• Batch Size per GPU for Training: 4
• Batch Size per GPU for Evaluation: 4
• Gradient Accumulation Steps: 1
• Learning Rate: 2e-4
• Weight Decay: 0.001
• Optimizer: PagedAdamW (32-bit)
• Learning Rate Scheduler: Cosine

SFT Parameters

• Maximum Sequence Length: None
• Packing: False

How to Use

from transformers import AutoTokenizer, pipeline
import torch

model = "Alpaca69B/llama-2-7b-absa-semeval-2016"

tokenizer = AutoTokenizer.from_pretrained(model)
pipeline = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    torch_dtype=torch.float16,
    device="auto",
)

input_sentence = "the first thing that attracts attention is the warm reception and the smiling receptionists."
sequences = pipeline(
    f'### Human: {input_sentence} ### Assistant: aspect:',
    do_sample=True,
    top_k=10,
    num_return_sequences=1,
    eos_token_id=tokenizer.eos_token_id,
    max_length=200,
)
sequences[0]['generated_text']

Acknowledgments

• The fine-tuning process and model development were performed by Ben Kampmann.