A Qwen3-1.7B fine-tuned on the Distil Labs platform to route ecommerce support tickets. It reads raw ticket text plus light order metadata and emits one JSON object a helpdesk can act on directly. On the held-out benchmark it scores 83.3 percent against 75.0 percent for its 120B-parameter teacher, and it produced schema-valid JSON on 100 percent of 94 evaluated inputs, including Spanish, German, emoji, and prompt-injection attempts.
needs_human: boolean, from explicit triggers (threats, fraud, repeat contact, explicit request for a person, anything automation cannot resolve)
The system prompt that defines the task ships in the repo and must be used verbatim at inference.
Results
Held-out test of 64 rows: 24 curated benchmark tickets plus 40 real customer messages from the public Twitter customer-support corpus, hand-labeled before any model ran on them. Platform LLM-as-a-judge scoring:
Untrained Qwen3-1.7B
Teacher (gpt-oss-120b)
Tuned Qwen3-1.7B
LLM-as-a-judge
32.8%
67.2%
67.2%
Exact match
0.0%
56.3%
54.7%
ROUGE
82.6%
90.1%
90.7%
By slice, identical field-level grader on both models:
Slice
Teacher
Tuned student
Curated benchmark, in-domain (24)
75.0%
83.3%
Real customer messages, out-of-domain (40)
60.0%
57.5%
A row passes the judge only when intent, order_id, and needs_human match exactly, urgency is within one, and the action is allowed for the intent. On the curated slice the tuned model reads intent at 95.8 percent and copies order ids verbatim at 100 percent. Local latency on the f16 GGUF on an Apple M-series laptop measured p50 1.9 seconds per ticket, with the roughly 1,000-token system prompt processed per call.
Run it locally
The repo ships a GGUF plus a Modelfile with the correct Qwen3 chat template:
Call it with the system prompt from the repo and thinking disabled (think: false via the Ollama API). Temperature 0.1.
Training
Seed data: 49 hand-authored tickets across all 12 intents plus 24 held-out, committed to the repo. The platform generated 10,149 synthetic tickets from them. 409 generated labels violating the contract's hard rules and 1,114 teaching unnecessary escalation were repaired mechanically before training; 60 hand-labeled real customer messages and 6 manipulation-handling examples were folded in. Training the student on repaired labels rather than raw teacher output is the reason it passes the teacher in-domain: the teacher's own rule violations stop being the ceiling.
The bundled tokenizer_config.json removes an extra_special_tokens list entry that crashes current transformers tokenizer loading; behavior is unchanged because those tokens are already declared in tokenizer.json.
Limitations
English-first; non-English input degrades accuracy while output stays schema-valid. Raw social-media messages without order metadata are out-of-domain (57.5 percent vs the teacher's 60.0). Policy boundaries the prompt leaves implicit, like return windows, follow the seed dataset's conventions. Sentiment on sarcasm remains hard. The model routes tickets; it does not write replies.
Provenance
Trained by Vish Vadlamani on the Distil Labs platform, teacher openai.gpt-oss-120b, LoRA fine-tune of Qwen/Qwen3-1.7B. Real-message eval data derives from the public Customer Support on Twitter corpus.
