Model Card for Fine-Tuned gemma-2-2b-it on Custom Korean Sentiment Dataset
Model Summary
This model is a fine-tuned version of google/gemma-2-2b-it, trained to classify sentiment in Korean text into four categories: 무감정 (neutral), 슬픔 (sadness), 기쁨 (joy), and 분노 (anger). The model utilizes LoRA (Low-Rank Adaptation) for efficient fine-tuning and 4-bit quantization (NF4) for memory efficiency using BitsAndBytes. A custom weighted loss function was applied to handle class imbalance within the dataset.
The model is suitable for multi-class sentiment classification in Korean and is optimized for environments with limited computational resources due to the quantization.
Model Details
Developed By:
This model was fine-tuned by [Your Name or Organization] using Hugging Face's peft and transformers libraries with a custom Korean sentiment dataset.
Model Type:
This is a transformer-based model for multi-class sentiment classification in the Korean language.
Language:
License:
[Add relevant license here]
Finetuned From:
- Base Model:
google/gemma-2-2b-it
Framework Versions:
- Transformers: 4.44.2
- PEFT: 0.12.0
- Datasets: 3.0.1
- PyTorch: 2.4.1+cu121
Intended Uses & Limitations
Intended Use:
This model is suitable for applications requiring multi-class sentiment classification in Korean, such as chatbots, social media monitoring, or customer feedback analysis.
Out-of-Scope Use:
The model may not perform optimally for tasks requiring multi-language support, sentiment classification with additional classes, or outside the specific context of Korean language data.
Limitations:
- Bias: As the model is trained on a custom dataset, it may reflect specific biases inherent in that data.
- Generalization: Performance may vary when applied to datasets outside the scope of the initial training data, such as other forms of sentiment classification.
Model Architecture
Quantization:
The model uses 4-bit quantization via BitsAndBytes for efficient memory usage, which enables it to run on lower-resource hardware.
LoRA Configuration:
LoRA (Low-Rank Adaptation) was applied to specific transformer layers, allowing for parameter-efficient fine-tuning. The target modules include:
down_proj, gate_proj, q_proj, o_proj, up_proj, v_proj, k_proj
LoRA parameters are:
r = 16, lora_alpha = 32, lora_dropout = 0.05
Custom Weighted Loss:
A custom weighted loss function was implemented to handle class imbalance, using the following weights:
[
\text{weights} = [0.2032, 0.2704, 0.2529, 0.2735]
]
These weights correspond to the classes: 무감정, 슬픔, 기쁨, 분노, respectively.
Training Details
Dataset:
The model was trained on a custom Korean sentiment analysis dataset. This dataset consists of text samples labeled with one of four sentiment classes: 무감정, 슬픔, 기쁨, and 분노.
- Train Set Size: Custom dataset
- Test Set Size: Custom dataset
- Classes: 4 (무감정, 슬픔, 기쁨, 분노)
Preprocessing:
Data was tokenized using the google/gemma-2-2b-it tokenizer with a maximum sequence length of 128. The preprocessing steps included padding and truncation to ensure consistent input lengths.
Hyperparameters:
- Learning Rate: 2e-4
- Batch Size (train): 8
- Batch Size (eval): 8
- Epochs: 4
- Optimizer: AdamW (with 8-bit optimization)
- Weight Decay: 0.01
- Gradient Accumulation Steps: 2
- Evaluation Steps: 500
- Logging Steps: 500
- Metric for Best Model: F1 (weighted)
Evaluation
Metrics:
The model was evaluated using the following metrics:
- Accuracy
- F1 Score (weighted)
- Precision (weighted)
- Recall (weighted)
The evaluation provides a detailed view of the model's performance across multiple metrics, which helps in understanding its strengths and areas for improvement.
Code Example:
You can load the fine-tuned model and use it for inference on your own data as follows:
1from transformers import AutoTokenizer, AutoModelForSequenceClassification
2
3# Load model and tokenizer
4model = AutoModelForSequenceClassification.from_pretrained("your-model-directory")
5tokenizer = AutoTokenizer.from_pretrained("your-model-directory")
6
7# Tokenize input text
8text = "이 영화는 정말 슬퍼요."
9inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True)
10
11# Get predictions
12outputs = model(**inputs)
13logits = outputs.logits
14predicted_class = logits.argmax(-1).item()
15
16# Map prediction to label
17id2label = {0: "무감정", 1: "슬픔", 2: "기쁨", 3: "분노"}
18print(f"Predicted sentiment: {id2label[predicted_class]}")
