Qwen2.5-1.5B Email Extraction LoRA

Performance

• Field F1 (test set, n=300): 0.766 [95% CI: 0.753, 0.779] — mean across 3 seeds, 10K bootstrap resamples (seed=0)
• Strict all-correct rate: 9.4% (vs. 0.0% for gpt-5.4-mini zero-shot)
• Paired diff vs. gpt-5.4-mini zero-shot: +21.5 field-F1 points (p ≈ 0.0001)
• Valid-JSON rate: 100%
• Latency: p50 1.88 s, p95 2.16 s on RTX 4080 Super (bf16)

Training Config

• Base model: Qwen/Qwen2.5-1.5B-Instruct @ 989aa7980e4cf806f80c7fef2b1adb7bc71aa306
• LoRA rank: 16, alpha: 32, dropout: 0.05
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj (all linear layers)
• Training: 3 epochs, lr 2e-4, batch_size 4, grad_accum 4 (effective batch 16), cosine LR, bf16
• Loss: completion-only (assistant tokens only)
• Seed: 0

Reproduce

Limitations

• Trained on fully synthetic data; real customer emails may have noise, typos, and jargon not seen in training.
• One narrow task — does not generalize to other extraction schemas.
• Single base-model family — Qwen2.5-1.5B-specific.

Citation