A lightweight GLiClass-based model router that scores which of 8 candidate LLMs should handle a request, its task type, difficulty, reasoning need, expected output length, and whether a response looks hallucinated. Additionally, the model supports zero-shot classification and works naturally with chat-like inputs.

Why use it

No generation, one pass — a calibrated probability per label, not a decoded token stream.
Zero-shot labels — swap the model roster, task taxonomy, or add a custom label set at inference time, no retraining required.
Streaming-native — a decoder KV cache per session means a multi-turn conversation is re-routed every turn at the cost of encoding only the new tokens, not the whole history.
One checkpoint, many signals — model routing, 28-way task type, difficulty, reasoning mode, output length, and hallucination detection all come from the same ~0.6B model.
Small footprint — cheap enough to co-locate with a gateway or run alongside the models it's routing to.

Installation

pip install gliclass -U

Usage

Single input

One text, scored against a label set in a single forward pass — the same call pattern for any label family (model routing, task type, difficulty, reasoning mode, output length, hallucination, or a custom label set).

python

from gliclass import GLiClassModel, ZeroShotClassificationPipeline
from transformers import AutoTokenizer

model_path = "scx-admin/scx-router-v0.1"  # or a local path to this checkpoint

model = GLiClassModel.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)

pipeline = ZeroShotClassificationPipeline(
    model, tokenizer, classification_type="multi-label", device="cuda:0"
)

text = "Write a Python function that merges two sorted linked lists."
labels = [
    "coder", "DeepSeek-V3.1", "gemma-4-31B-it", "gpt-oss-120b",
    "Llama-4-Maverick-17B-128E-Instruct", "MAGPiE",
    "Meta-Llama-3.3-70B-Instruct", "Qwen3-32B",
]

for r in pipeline(text, labels, threshold=0.5)[0]:
    print(r["label"], "=>", round(r["score"], 3))

Streamed input — message by message

StreamingZeroShotClassificationPipeline keeps one KV cache per session_id. Each call appends only the newest message; tokens_added and cached_length show exactly what got encoded versus reused from cache.

python

from gliclass import GLiClassModel, StreamingZeroShotClassificationPipeline
from transformers import AutoTokenizer

model = GLiClassModel.from_pretrained(model_path)
tokenizer = AutoTokenizer.from_pretrained(model_path)

router = StreamingZeroShotClassificationPipeline(
    model, tokenizer, classification_type="multi-label", device="cuda:0"
)

session_id = "chat-42"
turns = [
    "I need help refactoring some Rust code.",
    "Specifically the borrow checker keeps rejecting this function.",
    "fn parse(&mut self, buf: &[u8]) -> Result<Token, Error> { ... } -- here's the body.",
]

for turn in turns:
    out = router(turn, labels, session_ids=session_id, threshold=0.5)[0]
    print(f"+{out['tokens_added']} new tokens -> cache now {out['cached_length']} tokens")
    if out["triggered"]:
        for pred in sorted(out["predictions"], key=lambda p: -p["score"]):
            print(f"  {pred['label']:<38s} {pred['score']:.3f}")

# only each turn's own tokens are encoded -- earlier turns stay in the KV cache

More examples — task-type, difficulty, reasoning-mode, hallucination detection, general zero-shot, and the HTTP session server

Task-type classification — 28-way, single-label:

python

task_types = [
    "analysis", "classification", "clustering", "code", "comparison", "critique",
    "decision support", "evaluation", "explanation", "fact checking", "forecasting",
    "generation", "information extraction", "information retrieval", "instruction following",
    "math & reasoning", "multi-turn", "planning", "problem solving", "programming", "qa",
    "reasoning", "recommendation", "rewriting", "summarization", "translation",
    "verification", "visualization",
]

pipeline = ZeroShotClassificationPipeline(model, tokenizer, classification_type="single-label", device="cuda:0")
result = pipeline("Compare these two vendor contracts and flag the riskier clauses.", task_types)[0]
print(result[0]["label"], result[0]["score"])  # -> "comparison" 0.9xx

Difficulty & expected output length — feeds token-budget decisions:

python

difficulty_labels = ["very easy", "easy", "medium", "hard", "extra hard"]
length_labels = [
    "1-64 output tokens", "65-128 output tokens", "129-256 output tokens",
    "257-512 output tokens", "513-1024 output tokens", "1025-2048 output tokens",
    "2049-4095 output tokens",
]

text = "Derive the closed-form solution for this second-order linear recurrence and prove convergence."
difficulty = pipeline(text, difficulty_labels)[0][0]
length = pipeline(text, length_labels)[0][0]
print(difficulty["label"], length["label"])

Reasoning-mode routing — toggle extended thinking on/off:

python

reasoning_labels = ["reasoning", "nonreasoning"]
mode = pipeline("What's a good one-line commit message for this typo fix?", reasoning_labels)[0][0]
enable_thinking = mode["label"] == "reasoning"

Hallucination detection — score a candidate response, not the request. Treat this as a directional signal (F1 ≈ 65%, see benchmarks below) rather than a sole filtering gate:

python

hallucination_labels = ["correct", "hallucinated"]
answer_text = "The Eiffel Tower was completed in 1889 for the World's Fair."
verdict = pipeline(answer_text, hallucination_labels)[0][0]
print(verdict["label"], verdict["score"])

General zero-shot classification — custom labels, not just routing. The pre-training mix also covers topics, sentiment, emotion, NLI, guardrails and toxicity, so arbitrary label sets work too:

python

labels = ["travel", "finance", "health", "sports", "technology"]
result = pipeline("The new mid-range EV just posted a 480-mile range on a single charge.", labels)[0]
for r in sorted(result, key=lambda x: -x["score"]):
    print(r["label"], round(r["score"], 3))

Benchmarks

Threshold 0.5 unless noted. Per-family macro F1 is the unweighted mean of the listed per-class F1 scores.

Model routing — macro F1 ≈ 75.9%

Precision is consistently high (0.74–0.84) across all 8 candidates; recall is the swing factor, favoring frequently-correct generalists (Gemma 4 31B, Llama 4 Maverick) over narrower specialists (Qwen3 32B, MAGPiE).

Model	P	R	F1
Gemma 4 31B	0.8064	0.9501	0.8723
Llama 4 Maverick	0.7892	0.7529	0.7706
GPT-OSS 120B	0.8396	0.7063	0.7672
Coder	0.8140	0.7000	0.7527
Llama 3.3 70B	0.7638	0.7239	0.7433
DeepSeek V3.1	0.7475	0.7220	0.7346
MAGPiE	0.7394	0.7234	0.7313
Qwen3 32B	0.7505	0.6499	0.6966

End-to-end gain by LiveBench dataset

Router vs mean-of-8-models accuracy by LiveBench dataset

Reference numbers from the most recent end-to-end routing benchmark (results/report.json, 100 tasks per subset, checkpoint models/decoder_kv/post_train) — this end-to-end benchmark has not yet been re-run for lasty, so treat it as prior context rather than a lasty-specific result. Gains are largest where the 8 candidate models disagree the most (livebench/language +26.2 pts, livebench/math +18.3 pts) and smallest where they already cluster together (livebench/data_analysis +0.1 pts).

LiveBench dataset	Router	Mean of 8 models	Gain
livebench/language	0.779	0.517	+26.2 pts
livebench/math	0.738	0.555	+18.3 pts
livebench/instruction_following	0.863	0.768	+9.5 pts
livebench/reasoning	0.601	0.551	+5.0 pts
livebench/coding	0.500	0.456	+4.4 pts
livebench/data_analysis	0.540	0.539	+0.1 pts

Task-type routing — macro F1 ≈ 83.7%

28-way taxonomy, single-label, alphabetical order. Near-perfect on well-separated categories (analysis, code, IE, QA, reasoning, summarization, translation); weakest on task types that overlap conceptually with others — problem solving is frequently folded into programming/math & reasoning, and decision support/evaluation/critique overlap with each other and with verification.

Task type	P	R	F1
Analysis	1.0000	1.0000	1.0000
Classification	0.9974	1.0000	0.9987
Clustering	1.0000	0.8600	0.9247
Code	1.0000	1.0000	1.0000
Comparison	0.6385	0.8300	0.7217
Critique	0.6339	0.7100	0.6698
Decision support	0.5794	0.6200	0.5990
Evaluation	0.8154	0.5300	0.6424
Explanation	0.8017	0.9700	0.8778
Fact checking	0.9560	0.8700	0.9110
Forecasting	0.9655	0.8400	0.8984
Generation	0.9928	1.0000	0.9964
Information extraction / IE	1.0000	1.0000	1.0000
Information retrieval	0.9787	0.4600	0.6259
Instruction following	1.0000	0.9231	0.9600
Math & reasoning	0.4521	0.6600	0.5366
Multi-turn	1.0000	0.9936	0.9968
Planning	0.7213	0.8800	0.7928
Problem solving	0.3077	0.0800	0.1270
Programming	0.5774	0.9700	0.7239
QA	1.0000	1.0000	1.0000
Reasoning	0.9977	1.0000	0.9988
Recommendation	0.7660	0.7200	0.7423
Rewriting	0.9032	1.0000	0.9492
Summarization	1.0000	1.0000	1.0000
Translation	1.0000	1.0000	1.0000
Verification	0.7745	0.7900	0.7822
Visualization	1.0000	0.9300	0.9637

Difficulty routing — macro F1 ≈ 78.9%

Most top-1 misses land one ordinal step away (hard ↔ extra hard); top-k accuracy is a fairer read than exact top-1 match for an ordinal scale like this.

Difficulty	P	R	F1
Very easy	0.8246	0.9400	0.8785
Easy	0.8313	0.6900	0.7541
Medium	0.8381	0.8800	0.8585
Hard	0.7867	0.5900	0.6743
Extra hard	0.7073	0.8700	0.7803

Reasoning-mode routing — average F1: 89.73%

The strongest head in this checkpoint.

Mode	P	R	F1
Reasoning	0.9100	0.8709	0.8900
Non-reasoning	0.9300	0.8806	0.9046

Output-length routing — macro F1 ≈ 78.8%

Precision drops sharply in the 1,025–2,048 token bucket — the model over-predicts this bucket, likely because it sits between the more distinctive "short answer" and "long-form generation" regimes.

Output length	P	R	F1
1–64 tokens	0.9048	0.9500	0.9268
65–128 tokens	0.8182	0.8100	0.8141
129–256 tokens	0.7456	0.8500	0.7944
257–512 tokens	0.8438	0.8100	0.8265
513–1,024 tokens	0.8537	0.7000	0.7692
1,025–2,048 tokens	0.5437	0.8700	0.6692
2,049–4,095 tokens	0.7400	0.6866	0.7123

Details

Model families

Model routing — which of 8 candidate LLMs is likely to solve the request (multi-label): coder, DeepSeek-V3.1, gemma-4-31B-it, gpt-oss-120b, Llama-4-Maverick-17B-128E-Instruct, MAGPiE, Meta-Llama-3.3-70B-Instruct, Qwen3-32B.
Task type — a 28-way taxonomy: analysis, classification, clustering, code, comparison, critique, decision support, evaluation, explanation, fact checking, forecasting, generation, information extraction, information retrieval, instruction following, math & reasoning, multi-turn, planning, problem solving, programming, qa, reasoning, recommendation, rewriting, summarization, translation, verification, visualization.
Difficulty — 5 levels: very easy, easy, medium, hard, extra hard.
Reasoning mode — whether extended reasoning should be enabled: reasoning vs nonreasoning.
Expected output length — token-range buckets from 1-64 up to 2049-4095 output tokens.
Hallucination detection — whether a response is correct or hallucinated (new head in this checkpoint).
General zero-shot classification — the pre-training mix also covers topics, sentiment, emotion, NLI, guardrails, toxicity, and safety taxonomies, so the model remains a capable general-purpose classifier beyond routing.

A new architecture, not just a new checkpoint

Every previously released GLiClass model (gliclass-small/base/large-v1.0 and successors) uses a bidirectional uni-encoder: the whole input is re-encoded from scratch on every call. This checkpoint introduces GLiClass's first decoder-KV architecture — a causal Qwen3-0.6B backbone where the conversation lives in an ordinary decoder KV cache. In a session, each new turn only pays to encode its own tokens; labels are then re-scored against the full cached context. That single change turns a stateless classifier into a streaming router.

Component	Value
Architecture type	`decoder-kv` (`GLiClassModel`)
Backbone	`Qwen/Qwen3-0.6B` (28 layers, hidden size 1024, 16 attention heads / 8 KV heads, full attention, 40,960 max positions)
Parameters	~0.6B (backbone + lightweight scorer head)
Special tokens	`<<LABEL>>`, `<<SEP>>`, `<<EXAMPLE>>`
Class token pooling	`first`
Scorer head	16 attention heads, MLP hidden size 1024, 2 scorer-encoder layers

Labels are injected as <<LABEL>> label_text segments and scored against the text representation from the decoder. Because the text context lives in the KV cache, the model supports cheap incremental/streaming classification: in a multi-turn session, only new text tokens need to be encoded on each turn, while labels are (re-)scored against the full cached context.

How this compares to other router designs

Embedding models, encoder-only classifiers, and generative LLM routers each have a real strength. Decoder-KV is the one architecture that combines inference-time labels, a chat-aligned causal backbone, and incremental dialogue state without paying autoregressive generation cost on every routing decision.

Approach	Primary benefit	New dialogue turn	Inference-time labels	Chat alignment	Autoregressive generation	Routing output
GLiClass decoder-KV (this checkpoint)	Semantic label–text matching with a task-trained classification scorer.	Reuses the session KV cache, encodes only new tokens, and re-scores labels against the full context.	Native	Strong causal/chat-aligned backbone	Not required	One non-generative forward pass produces an independent probability for each label.
Embedding / bi-encoder	Very inexpensive semantic similarity; label embeddings can be precomputed.	No token-level dialogue cache; the updated conversation is typically re-embedded in full.	Native	Usually limited	Not required	Fast nearest-neighbor similarity score, but cosine similarity requires calibration and provides limited text–label interaction.
Encoder-only (e.g. DeBERTa)	Strong bidirectional discrimination for a fixed taxonomy or NLI formulation.	Requires a new full-sequence forward pass each turn; zero-shot use may require one pass per candidate label.	Fixed classification head or pairwise NLI	Usually limited	Not required	Produces logits or pairwise scores, but repeated full-context and sometimes per-label encoding increases multi-turn cost.
Generative LLM router	Chat-native and highly flexible; can also explain its routing decision.	Prefix KV reuse helps, but the router must still autoregressively generate route tokens.	Native through the prompt	Strong	Required	Generates a route that must be parsed, adding latency, cost, and output non-determinism.