A 4-class user feedback classifier based on mmbert-32k-yarn. This is the merged version with LoRA weights integrated - no PEFT library required.
Model Description
This model classifies user messages into 4 feedback categories to help conversational AI systems understand user satisfaction:
Label
ID
Description
SAT
0
User is satisfied with the response
NEED_CLARIFICATION
1
User needs more explanation or details
WRONG_ANSWER
2
User indicates the response was incorrect
WANT_DIFFERENT
3
User wants an alternative approach/answer
Performance
Validation Results (2,985 samples):
Metric
Value
Accuracy
98.83%
F1 (macro)
98.24%
F1 (weighted)
98.83%
Per-Class Performance:
Class
Precision
Recall
F1-Score
Support
SAT
1.0000
1.0000
1.0000
1,491
NEED_CLARIFICATION
0.9980
0.9980
0.9980
498
WRONG_ANSWER
0.9604
0.9739
0.9671
498
WANT_DIFFERENT
0.9715
0.9578
0.9646
498
Quick Start
python
1from transformers import AutoModelForSequenceClassification, AutoTokenizer
2import torch
34# Load model and tokenizer5model = AutoModelForSequenceClassification.from_pretrained(6"llm-semantic-router/mmbert32k-feedback-detector-merged"7)8tokenizer = AutoTokenizer.from_pretrained(9"llm-semantic-router/mmbert32k-feedback-detector-merged"10)11model.eval()1213# Label mapping14labels =["SAT","NEED_CLARIFICATION","WRONG_ANSWER","WANT_DIFFERENT"]1516# Example inference17texts =[18"Thank you, that's exactly what I needed!",19"I don't understand, can you explain more?",20"That's incorrect, the answer should be different.",21"Can you give me another approach?",22]2324for text in texts:25 inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512)26with torch.no_grad():27 outputs = model(**inputs)2829 probs = torch.softmax(outputs.logits, dim=-1)30 pred = outputs.logits.argmax(-1).item()31 conf = probs[0][pred].item()3233print(f"{labels[pred]:20} ({conf:.1%}) | {text}")
Output:
SAT (81.2%) | Thank you, that's exactly what I needed!
NEED_CLARIFICATION (100.0%) | I don't understand, can you explain more?
WRONG_ANSWER (100.0%) | That's incorrect, the answer should be different.
WANT_DIFFERENT (100.0%) | Can you give me another approach?
Batch Inference
python
1# Efficient batch processing2texts =["Your text 1","Your text 2","Your text 3"]3inputs = tokenizer(texts, return_tensors="pt", padding=True, truncation=True, max_length=512)45with torch.no_grad():6 outputs = model(**inputs)78predictions = outputs.logits.argmax(-1).tolist()9feedback_types =[labels[p]for p in predictions]
Training Details
This model was fine-tuned using LoRA (Low-Rank Adaptation) with the following configuration:
