Qwen2.5-0.5B Intent Router (LoRA fine-tuned)
A 500M-parameter sequence classifier fine-tuned to route user queries to one of
four tool categories for an internal GenAI assistant, achieving 95.3% accuracy
and 3.3x lower latency than a 70B-parameter LLM baseline on the same task.
Model Description
This model classifies a user query into exactly one of four categories, intended
to sit at the entry point of an AI agent and decide which downstream tool/node
should handle the request:
| Category | Description |
|---|
rag_tool | Query needs internal company knowledge (HR policy, internal docs, onboarding, internal tools) |
code_tool | Query needs code written, debugged, explained, or reviewed |
web_search | Query needs current/live/recent external information |
direct_answer | Query is answerable from general knowledge, no lookup needed |
Base model: Qwen/Qwen2.5-0.5B-Instruct
Fine-tuning method: LoRA (r=8, alpha=16, dropout=0.1) applied to attention
projections (q_proj, k_proj, v_proj, o_proj), then merged into the base
weights for standalone deployment.
Task head: Sequence classification (AutoModelForSequenceClassification),
not generative/instruction-following output, chosen specifically for
single-forward-pass inference speed and zero output-parsing risk in production.
Intended Use
Designed as the routing node in a multi-tool AI agent (e.g. a LangGraph graph),
where a fast, cheap classification decision is needed before dispatching to a
more expensive tool (RAG pipeline, code execution, web search, or direct LLM
response). Not intended as a general-purpose chat model -- it has no
generative/conversational capability; it only outputs one of the four labels.
Training Data
581 synthetically generated employee queries (Llama-3.3-70B via Groq), covering
4 balanced categories, with two deliberate design choices:
- ~20% of examples per category were generated as near-boundary/ambiguous
cases (e.g., a query mentioning a coding library but actually asking for
current release info), so the benchmark reflects realistic routing difficulty
rather than only trivially separable examples.
- Targeted augmentation after error analysis: an initial training run
(530 examples) showed the model confusing technical-vocabulary queries with
code_tool even when the query was actually asking for current/live info
(e.g. "has there been an update to the AWS CLI recently"). 60 additional
web_search examples specifically pairing technical vocabulary with
recency framing were generated and added, improving web_search recall
from 84.4% to 90.6%, with a minor, expected trade-off on two adjacent
code_tool boundary cases (a normal small-data fine-tuning dynamic, not
a regression -- net test accuracy was unchanged at 95.3%).
Data quality controls:
- Exact-hash deduplication during generation
- TF-IDF cosine-similarity near-duplicate detection and removal, run separately
for (a) train-vs-test/val leakage and (b) train-internal redundancy
- Test set generated in a separate batch/prompt pass from train, specifically
to avoid the test set being near-paraphrases of training examples
Train/val/test split: 581 / 94 / 127 examples. Test set was evaluated exactly
once, after training was finalized, to ensure a fair held-out estimate.
Evaluation Results
Accuracy comparison (vs. zero-shot LLM baseline)
Benchmarked against an independent LLM (zero-shot prompted, no examples, same
held-out 127-query test set) as a fair, methodologically-clean comparison point
-- the LLM judge was not used to generate the training data, to avoid
contaminating the comparison.
| Metric | This model (0.5B, fine-tuned) | LLM baseline (zero-shot) |
|---|
| Accuracy | 95.3% | 96.9% |
| Macro F1 | 95.2% | 96.6% |
Latency comparison
Measured as real wall-clock time per query: local GPU forward pass (this model)
vs. a real API call to a 70B LLM via Groq (chosen specifically because Groq is
among the fastest LLM inference providers available -- making this a
conservative, harder-to-dispute speed comparison rather than picking an easy
target).
| Metric | This model | LLM baseline (Groq, 70B) |
|---|
| Mean latency | 110 ms | 365 ms |
| Speedup | 3.3x faster | -- |
| Parameter count | 0.5B | 70B (140x larger) |
Per-class breakdown (this model, final test run)
| Category | Precision | Recall | F1 |
|---|
| code_tool | 0.941 | 0.941 | 0.941 |
| direct_answer | 0.926 | 0.962 | 0.943 |
| rag_tool | 0.946 | 1.000 | 0.972 |
| web_search | 1.000 | 0.906 | 0.951 |
Honest Limitations
- The LLM baseline scores slightly higher on raw accuracy (96.9% vs 95.3%).
This model's value proposition is not "beats the LLM outright" -- it is
near-parity accuracy at 140x fewer parameters and 3.3x lower latency, which
is the relevant trade-off for a high-throughput routing node that runs on
every single user message.
- Two of the four LLM-baseline misclassifications, and several of this model's
misclassifications, cluster on the same two genuinely ambiguous test
examples (queries that blend web_search framing with general-knowledge
framing), suggesting these specific examples may sit at a real boundary in
the category definitions themselves, not purely a model weakness.
- Trained on synthetic, LLM-generated data -- phrasing diversity, while
deliberately varied, reflects the generating model's style and may not fully
capture real employee phrasing patterns in a live deployment.
- Dataset size (581 train examples) is small; performance on categories or
phrasing patterns not represented in training is untested.
Training Procedure
- Hardware: Google Colab T4 GPU
- LoRA: r=8, alpha=16, dropout=0.1, targeting q_proj/k_proj/v_proj/o_proj
- Trainable parameters: 1,084,928 / 495,121,280 (0.22%)
- Epochs: 8, learning rate 2e-4, batch size 16
- Tracked via MLflow
Citation
If you use this model, please reference the base model:
Qwen2.5 Technical Report, Qwen Team, Alibaba Group
