Fine-tuned distilroberta-base (82M params) for category classification on real customer-support tweets, with labels distilled from Claude Sonnet 4.6.

Part of the TicketSense project: a tiny specialized classifier that approximates frontier-LLM judgement at a fraction of the cost and latency.

Labels

billing
technical
account
feature_request
other

Eval

Macro F1 (validation): 0.5717

Training

Base model: distilroberta-base
Method: LoRA (PEFT), then merged
LoRA rank: 16, alpha: 32
LR: 0.0002, epochs: 4
Train data: ~1400 real customer-support tweets, Claude-labeled

Use

python

from transformers import pipeline
clf = pipeline("text-classification", model="Grinding/ticketsense-category")
clf("my card was charged twice and nobody has responded")

Limitations

Trained on noisy real-world Twitter data — picks up typos and slang but may also reflect their biases.
Labels are themselves model-generated (Claude Sonnet 4.6) — treat F1 vs Claude as agreement, not gold-standard truth.