Source: Devstral-2512 rollouts on agent-diff-bench, filtered for reward > 0.8
Processing pipeline:
1. Native formatting (0 missing content, 0 consecutive assistant issues)
2. Command flattening (multi-line curl to single-line, reduced from 44% to 6%)
3. Error turn removal (failed API call + error response pairs removed, error rate 20% to 1.8%)
4. Prompt-level train/val split (0% leakage)
Final dataset: 361 rows, 142 unique prompts, ~2.5 rollouts per prompt
Dataset: hubertmarek/mistral-large-agent-diff-sft-mixed-old-plus-devstral-r0p8-64k

Hyperparameters

SFTConfig(
    num_train_epochs=8,
    per_device_train_batch_size=1,
    gradient_accumulation_steps=6,
    learning_rate=5e-5,
    lr_scheduler_type="cosine",
    warmup_ratio=0.08,
    bf16=True,
    max_length=64000,
    optim="adamw_torch_fused",
    gradient_checkpointing=True,
    save_strategy="epoch",
    packing=False,
)

LoRA Config

Rank: 64
Alpha: 128
Target modules: all linear layers

Inference

vLLM

bash

export HF_TOKEN='your_token'
vllm serve mistralai/Ministral-3-14B-Instruct-2512 \
  --tokenizer_mode mistral --config_format mistral --load_format mistral \
  --enable-lora \
  --lora-modules agent-diff=ministral-3-14b-agent-diff-sft-lora \
  --enable-auto-tool-choice --tool-call-parser mistral \
  --max-model-len 64000 \
  --max-lora-rank 64 \
  --enforce-eager

Evaluation

bash

prime eval hubert-marek/agent-diff-bench \
  -m agent-diff \
  --api-base-url http://localhost:8000/v1 \
  -n -1 -r 3 -c 15 \
  --max-retries 20 \
  --env-args '{"agentdiff_api_key": "YOUR_KEY"}' \
  --save-results \
  --temperature 0.5

Checkpoints

This repo contains multiple epochs as commits:

Epoch 3 (checkpoint-183): Recommended starting point
Epoch 5 (checkpoint-305): Best benchmark performance
Epoch 8 (checkpoint-488): Overfitting

hubertmarek/Ministral-3-14B-Agent-Diff-SFT-LoRA

Mistral Small Agent-Diff

Results

Per-Service Breakdown

Training

Data

Hyperparameters

LoRA Config

Inference

vLLM

Evaluation

Checkpoints

hubertmarek/
Ministral-3-14B-Agent-Diff-SFT-LoRA