This model is a Parameter-Efficient Fine-Tuned (PEFT) version of distilbert-base-uncased using LoRA (Low-Rank Adaptation) for 5-class star rating prediction (1-5 stars) on Singapore digital bank reviews.

Github: https://github.com/ajiayi-debug/SFTvsPEFT

🎯 Model Performance

Metric	Score
Test Accuracy	0.7619
Precision (Macro)	0.3064
Recall (Macro)	0.3597
F1-Score (Macro)	0.3307
MAE (Mean Absolute Error)	0.6000 stars

🚀 Quick Start

Loading the Model

python

from peft import PeftModel, PeftConfig
from transformers import AutoModelForSequenceClassification, AutoTokenizer

# Load configuration
config = PeftConfig.from_pretrained("ajiayi/bert-peft-singapore-digi-banks")

# Load base model
base_model = AutoModelForSequenceClassification.from_pretrained(
    config.base_model_name_or_path,
    num_labels=5
)

# Load PEFT adapter
model = PeftModel.from_pretrained(base_model, "ajiayi/bert-peft-singapore-digi-banks")
tokenizer = AutoTokenizer.from_pretrained(config.base_model_name_or_path)

# Make predictions
text = "Love this bank! Best interest rates in Singapore."
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128)
outputs = model(**inputs)
predicted_stars = outputs.logits.argmax(dim=1).item() + 1  # Convert 0-4 to 1-5
print(f"Predicted rating: {predicted_stars} stars")

📊 Model Details

Model Architecture

Base Model: distilbert-base-uncased (66M parameters)
Fine-tuning Method: PEFT with LoRA
Task: Multi-class text classification (5 classes)
Classes: 1-star, 2-star, 3-star, 4-star, 5-star

LoRA Configuration

Rank (r): 8
Alpha: 16
Dropout: 0.1
Target Modules: q_lin, v_lin (attention query and value layers)

Efficiency Metrics

Trainable Parameters: N/A
Total Parameters: 67,846,666
Trainable Percentage: N/A
Adapter Size: ~5MB (vs ~250MB for full fine-tuning)
Training Time: 0.96 minutes

🎓 Training Details

Training Data

Domain: Singapore digital bank reviews (GXS, Trust Bank, MariBank, etc.)
Training Samples: 489
Validation Samples: 105
Test Samples: 105
Augmentation: LLM-based data augmentation using Google Gemini

Training Hyperparameters

Learning Rate: 0.0002
Batch Size: 8
Gradient Accumulation Steps: 2
Epochs: 3
Weight Decay: 0.01
Warmup Ratio: 0.0
Optimizer: AdamW
Loss Function: Cross-Entropy with class weights

Training Infrastructure

Framework: HuggingFace Transformers + PEFT
Hardware: Apple Silicon (MPS) / CPU
Mixed Precision: FP16 (when GPU available)

🎯 Intended Use

Primary Use Cases

Sentiment analysis of Singapore digi bank reviews
Star rating prediction for financial services
Customer feedback classification
Banking app review analysis

Out-of-Scope Use

Reviews for non-Singapore banks (model trained specifically on Singapore digital banks)
General sentiment analysis (domain-specific model)

⚖️ Limitations and Bias

Known Limitations

Trained on Singapore digital bank reviews only
May not generalize well to traditional banks
Performance may vary on very short or very long reviews

Potential Biases

Data collected from Apple App Store only
May reflect platform-specific sentiment patterns
Class imbalance handled via weighted loss and augmentation (LLM-generated data augmented minority classes)

📈 Evaluation Results

The model was evaluated on a held-out test set with the following results:

Accuracy: 76.19%
Average prediction error: 0.60 stars

🔧 How to Use

For Inference Only

python

from peft import PeftModel
from transformers import AutoModelForSequenceClassification, AutoTokenizer
import torch

# Load model and tokenizer
model = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased", num_labels=5)
model = PeftModel.from_pretrained(model, "ajiayi/bert-peft-singapore-digi-banks")
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")

# Predict
def predict_rating(text):
    inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128)
    model.eval()
    with torch.no_grad():
        outputs = model(**inputs)
        prediction = outputs.logits.argmax(dim=1).item()
    return prediction + 1  # Convert 0-indexed to 1-5 stars

# Example
review = "Great banking app! Very user-friendly interface."
stars = predict_rating(review)
print(f"Predicted: {stars} stars")

📚 Citation

bibtex

@misc{peft-singapore-banks,
  author = {Your Name},
  title = {PEFT DistilBERT for Singapore Bank Review Classification},
  year = {2025},
  publisher = {HuggingFace},
  howpublished = {\url{https://huggingface.co/ajiayi/bert-peft-singapore-digi-banks}}
}

📄 License

This model is released under the Apache 2.0 License.

🙏 Acknowledgments

Base model: DistilBERT by HuggingFace
PEFT library by HuggingFace
Training framework: HuggingFace Transformers

Generated: 2025-10-14 23:08:12