Label	ID	Description	F1 Score
SAT	0	User is satisfied	100.0%
NEED_CLARIFICATION	1	User needs more information	99.7%
WRONG_ANSWER	2	System gave incorrect response	96.2%
WANT_DIFFERENT	3	User wants something different	95.9%

Performance

Metric	Value
Accuracy	98.63%
F1 Macro	97.94%

LoRA Configuration

Parameter	Value
Rank (r)	32
Alpha	64
Target Modules	query, key, value, dense
Trainable Parameters	6.7M (2.15% of total)

Training

Dataset: 51,694 examples from feedback-detector-dataset
Epochs: 5
Batch Size: 64
Learning Rate: 2e-5

Hardware

Component	Specification
GPU	AMD Instinct MI300X
VRAM	192 GB HBM3
Framework	PyTorch with ROCm
Training Time	~2 minutes

Usage

With PEFT

python

from transformers import AutoTokenizer, AutoModelForSequenceClassification
from peft import PeftModel

# Load base model
base_model = AutoModelForSequenceClassification.from_pretrained(
    "jhu-clsp/mmBERT-base",
    num_labels=4
)

# Load LoRA adapter
model = PeftModel.from_pretrained(
    base_model,
    "llm-semantic-router/mmbert-feedback-detector-lora"
)
tokenizer = AutoTokenizer.from_pretrained("llm-semantic-router/mmbert-feedback-detector-lora")

# Classify
labels = ["SAT", "NEED_CLARIFICATION", "WRONG_ANSWER", "WANT_DIFFERENT"]
inputs = tokenizer("Thank you, that was helpful!", return_tensors="pt")
outputs = model(**inputs)
pred = outputs.logits.argmax(-1).item()
print(f"Label: {labels[pred]}")

Merge LoRA Weights

python

from peft import PeftModel

# Merge for faster inference
merged_model = model.merge_and_unload()
merged_model.save_pretrained("merged_model")

Advantages of LoRA

Small Size: 27 MB vs 1.2 GB (full model)
Fast Training: Only 2.15% of parameters trained
Easy Deployment: Can swap adapters without reloading base model
Further Fine-tuning: Can continue training on domain-specific data

Related Models

Merged Model: llm-semantic-router/mmbert-feedback-detector (full weights, ready to use)
Base Model: jhu-clsp/mmBERT-base

Citation

bibtex

@model{mmbert_feedback_detector_lora,
  title={mmBERT Feedback Detector LoRA},
  author={LLM Semantic Router Team},
  year={2025},
  url={https://huggingface.co/llm-semantic-router/mmbert-feedback-detector-lora}
}

License

Apache 2.0