This model was fine-tuned for experimental purposes, as part of an investigation into reducing over-triggered tool calls (i.e., getting the model to correctly abstain when no tool is relevant).

nvidia/When2Call (train_sft) was included in training to improve this behavior. However, this had an unintended side effect: the model's core tool-calling ability degraded significantly. It now frequently asks the user for clarification/input even for simple, complete, unambiguous tool-call requests where all required parameters are already provided.

Do not use this model in production or for tool-calling tasks. It is kept public only as a record of this experiment (see BFCL eval results above — simple_python_acc and multiple_acc collapsed while irrelevance_acc hit ceiling, indicating over-correction rather than genuine improvement).

Qwen2.5-3B-Instruct — Tool Call (EN, Mixed + When2Call 3.3k)

LoRA fine-tune of Qwen/Qwen2.5-3B-Instruct for function/tool calling.

Data

Salesforce/xlam-function-calling-60k — 54,000 (tool-call)
HuggingFaceH4/no_robots — 8,000, filtered ≤2,000 chars (no-tool SFT)
nvidia/When2Call (train_sft) — 3,300 (irrelevance/negative signal)
Concatenated, shuffled, formatted via apply_chat_template

LoRA

r=32, lora_alpha=64, lora_dropout=0.05, target_modules=all-linear, bias=none, task_type=CAUSAL_LM

Training

lr=2e-4, epochs=1, per_device_batch=8, grad_accum=16 (effective 128), warmup_ratio=0.03, max_grad_norm=0.3, bf16=True, attn_implementation=sdpa, loss_type=nll

Training Results (step 505, final)

train_loss=0.354, val_loss=0.173, mean_token_accuracy=0.956, runtime≈4h02m (A100)

BFCL Eval Results (inspect_evals/bfcl)

Metric	Score
accuracy (all)	0.429
irrelevance_acc	1.000
multiple_acc	0.045
parallel_acc	0.595
parallel_multiple_acc	0.640
simple_python_acc	0.090

Note: irrelevance_acc reached 1.000 (perfect abstention), but simple_python_acc (0.090) and multiple_acc (0.045) collapsed even further. Even with a reduced When2Call subset (3.3k), the negative signal causes heavy over-abstention from tool calling.

Comparison to prior runs

Run	accuracy	irrelevance_acc	simple_python_acc	multiple_acc
Base Qwen2.5-3B	0.815	0.608	0.953	0.910
EN-only (xlam 60k)	0.705	0.213	0.893	0.865
EN + no_robots	0.730	0.225	0.910	0.875
EN + no_robots + When2Call (13.5k)	0.534	0.996	0.207	0.190
EN + no_robots + When2Call (3.3k) (this model)	0.429	1.000	0.090	0.045

canbingol/qwen2.5-3B-Instruct-tool-call-en-mixed-when2call-v2

⚠️ Warning