This model is one of 10 models trained for the TFT benchmark, which compares two approaches to training Small Language Models (SLMs) from production traces:

TFT Pipeline: trace filtering + committee relabeling + synthetic data generation + finetuning
Direct Training: train directly on raw/corrupted traces (no filtering, no relabeling, no synth gen)

Both pipelines are evaluated on the same held-out test set of 34 multi-turn Restaurants_1 conversations (~359 per-turn evaluation pairs) using LLM-as-a-judge scoring (0-1 scale).

Scenario: S1 Baseline — Baseline

327 clean Restaurants_1 traces with no corruption. Tests the quality ceiling — how well each pipeline performs with perfect data.

Training Details

Trained using the TFT (Training from Traces) pipeline: production traces are filtered, committee-relabeled by multiple LLMs, then used as seeds for synthetic data generation. The student model is fine-tuned on the resulting synthetic dataset.

Configuration

Base model: Qwen3-1.7B
Task: multi-turn-tool-calling-closed-book
Teacher / synth gen model: zai.glm-5
Judge model: openai.gpt-oss-120b
Committee (TFT relabeling): openai.gpt-oss-120b + zai.glm-5
Training: LoRA fine-tuning, merged weights

Target Tools

Based on the Schema-Guided Dialogue (SGD) dataset — restaurant search and reservation:

respond_to_user — send text messages to the user
FindRestaurants — search restaurants by cuisine, city, price range, live music, alcohol
ReserveRestaurant — reserve a table (restaurant name, city, time, date, party size)

Full Benchmark Results

Scenario	TFT	Direct	Delta
S1 Baseline	0.866	0.864	+0.2pp
S2 Noisy Labels	0.844	0.721	+12.3pp
S3 Schema Drift	0.844	0.585	+25.9pp
S4 Low Data	0.852	0.649	+20.3pp
S5 Trace Mixing	0.858	0.694	+16.4pp

TFT matches Direct Training on clean data (S1) and outperforms it on every corrupted scenario by 12-26 percentage points.

distillabs/tft-benchmark-s1-tft-Qwen3-1.7B

tft-benchmark-s1-tft-Qwen3-1.7B

Benchmark Overview

Scenario: S1 Baseline — Baseline

Training Details

Configuration

Target Tools

Full Benchmark Results

Links

distillabs/
tft-benchmark-s1-tft-Qwen3-1.7B