tft-benchmark-s5-direct-Qwen3-1.7B
A
Qwen3-1.7B model fine-tuned for multi-turn tool calling as part of the
TFT (Training from Traces) Benchmark.
- Pipeline: Direct Training
- Scenario: S5 Trace Mixing — Trace Mixing
- LLM-as-a-judge score: 0.694
- staged_tool_call score: 0.74
Benchmark Overview
This model is one of 10 models trained for the TFT benchmark, which compares two approaches to training Small Language Models (SLMs) from production traces:
- TFT Pipeline: trace filtering + committee relabeling + synthetic data generation + finetuning
- Direct Training: train directly on raw/corrupted traces (no filtering, no relabeling, no synth gen)
Both pipelines are evaluated on the same held-out test set of 34 multi-turn Restaurants_1 conversations (~359 per-turn evaluation pairs) using LLM-as-a-judge scoring (0-1 scale).
Scenario: S5 Trace Mixing — Trace Mixing
80% Hotels_1 (142 traces) + 20% Restaurants_1 (36 traces). Hotels traces have function names renamed to R1 equivalents and message order shuffled randomly, so the model sees R1 function names with hotel content in random order.
Training Details
Trained using Direct Training: the student model is fine-tuned directly on the raw production traces (expanded into per-turn training examples) with no filtering, relabeling, or synthetic data generation.
Configuration
- Base model: Qwen3-1.7B
- Task: multi-turn-tool-calling-closed-book
- Teacher / synth gen model: zai.glm-5
- Judge model: openai.gpt-oss-120b
- Committee (TFT relabeling): openai.gpt-oss-120b + zai.glm-5
- Training: LoRA fine-tuning, merged weights
Target Tools
Based on the
Schema-Guided Dialogue (SGD) dataset — restaurant search and reservation:
respond_to_user — send text messages to the userFindRestaurants — search restaurants by cuisine, city, price range, live music, alcoholReserveRestaurant — reserve a table (restaurant name, city, time, date, party size)
Full Benchmark Results
| Scenario | TFT | Direct | Delta |
|---|
| S1 Baseline | 0.866 | 0.864 | +0.2pp |
| S2 Noisy Labels | 0.844 | 0.721 | +12.3pp |
| S3 Schema Drift | 0.844 | 0.585 | +25.9pp |
| S4 Low Data | 0.852 | 0.649 | +20.3pp |
| S5 Trace Mixing | 0.858 | 0.694 | +16.4pp |
TFT matches Direct Training on clean data (S1) and outperforms it on every corrupted scenario by 12-26 percentage points.
Links
