You can deploy this model and start earning money today!

narcolepticchicken/
aco-specialists-tier-router-v2

Name: aco-specialists-tier-router-v2
Availability: InStock

safetensors

modernbert

ml-intern

Views

No views yet

narcolepticchicken/aco-specialists-tier-router-v2

Tier router for Agent Cost Optimizer — classifies agent queries into 3 difficulty tiers (easy/medium/hard) to select the cheapest adequate model.

Model

Base: ModernBERT-base (answerdotai/ModernBERT-base, 149.6M params)
Architecture: ModernBertForSequenceClassification (3-class)
Training: Focal loss (γ=2.0, α=[0.641, 1.044, 1.315]), 10 epochs, early stopping, dropout=0.1 throughout

Verified Metrics (independent evaluation, 2026-06-03)

Evaluated on 1,260 held-out RouterArena instances (class dist: easy=593, medium=359, hard=308).

Metric	v2 (ModernBERT)	v1 (DistilBERT)	Delta
Accuracy	69.2%	70.1%	-0.9pp
F1 (macro)	0.677	0.670	+0.007

Per-class F1: easy=0.768, medium=0.544, hard=0.720

Assessment: v2 offers no meaningful improvement over v1 for tier routing. The architecture upgrade (67M→149M) and focal loss did not help on this task. The model is usable but not better.

Training Data

RouteWorks/RouterArena (8,400 questions from 9 domains, 3 difficulty levels). Preprocessed split available at narcolepticchicken/aco-traces (data/tier_router/).

Usage

python

from transformers import AutoModelForSequenceClassification, AutoTokenizer

model = AutoModelForSequenceClassification.from_pretrained(
    "narcolepticchicken/aco-specialists-tier-router-v2")
tokenizer = AutoTokenizer.from_pretrained(
    "narcolepticchicken/aco-specialists-tier-router-v2")

inputs = tokenizer("Write a Python function to reverse a linked list", 
                    return_tensors="pt", truncation=True, max_length=2048)
outputs = model(**inputs)
# 0=easy, 1=medium, 2=hard

Limitations

Medium-difficulty queries have poor recall (0.555) — the model over-assigns "easy" and "hard"
No improvement over DistilBERT v1; prefer the smaller model for deployment
Trained on academic QA benchmarks; may not generalize to coding/agent tasks