SQLForge — Qwen2.5-Coder-7B SFT (Text-to-SQL)
A LoRA adapter that fine-tunes Qwen2.5-Coder-7B-Instruct for Text-to-SQL on a
TPC-H-derived schema with custom domain conventions. Supervised fine-tuning raised
execution accuracy on a held-out 55-query benchmark from 54.5% → 74.5%, closing 54%
of the gap to a Claude-based production agent (91.7%) — while general SQL ability stayed
intact (see Generalization).
Full project, training code, and evaluation harness:
github.com/ShahaDeven/sql-forge
This is a domain adapter, not a general Text-to-SQL model. It is trained on one
schema's conventions (categorical literals like churn_risk = 'HIGH_RISK', revenue via
total_value * (1 - promo_reduction), canonical customer-side join paths). It expects
the specific system prompt it was trained on — see Serving contract below.
Results (55-query execution-accuracy suite)
Graded by result-set comparison against gold on DuckDB (TPC-H SF=0.1) — the same grader
that scores the production Claude agent at 91.7%.
| Model | Execution accuracy | Valid SQL |
|---|
| Qwen2.5-Coder-7B (base, zero-shot) | 54.5% | 90.9% |
| Qwen2.5-Coder-7B (base, 3-shot) | 61.8% | 94.5% |
| This adapter (SFT, zero-shot) | 74.5% | 98.2% |
| Claude (full production agent) | 91.7%¹ | — |
¹ Full agent: retrieval + few-shot + a 3-attempt self-healing loop. A product comparison,
not a like-for-like model comparison.
Per-tier (SFT): simple_select 10/10 · aggregation 10/10 · single_join 8/10 ·
window_function 6/10 · multi_hop 5/10 · simulation 2/5.
Generalization (Spider zero-shot)
Run zero-shot on Spider dev (1,032 queries, 20 unseen SQLite databases) with a generic
prompt — no house-style rules — to test whether fine-tuning damaged general SQL ability:
| Model | Spider dev accuracy |
|---|
| Qwen2.5-Coder-7B (base) | 82.4% |
| This adapter (SFT) | 82.7% |
+20pp in-domain for +0.3pp (i.e. zero, within noise) out-of-domain. No catastrophic
forgetting: the adapter learned this schema's conventions, not a narrower notion of SQL.
Usage
1from peft import PeftModel
2from transformers import AutoModelForCausalLM, AutoTokenizer
3
4base = "Qwen/Qwen2.5-Coder-7B-Instruct"
5model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="bfloat16", device_map="auto")
6model = PeftModel.from_pretrained(model, "devenshah21/sqlforge-qwen7b-sft")
7tok = AutoTokenizer.from_pretrained(base)
8
9messages = [
10 {"role": "system", "content": HOUSE_STYLE_SYSTEM_PROMPT}, # see Serving contract
11 {"role": "user", "content": "Which region has the lowest revenue?"},
12]
13prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
14out = model.generate(**tok(prompt, return_tensors="pt").to(model.device),
15 max_new_tokens=512, do_sample=False)
16print(tok.decode(out[0], skip_special_tokens=True))
Serve with vLLM (as used for all evals):
1vllm serve Qwen/Qwen2.5-Coder-7B-Instruct \
2 --enable-lora --lora-modules sqlforge=devenshah21/sqlforge-qwen7b-sft \
3 --max-lora-rank 32 --port 8000
Serving contract
The adapter was trained
and benchmarked (74.5%) with
one specific prompt shape: a
house-style system prompt (schema + the categorical-literal / revenue-formula / join-path
rules) and the user question,
zero-shot, temperature 0. Sending few-shot examples or a
different prompt is an unmeasured configuration and will not reproduce the numbers above —
fine-tuning already absorbed what few-shot was teaching. The exact prompt builder lives in
phase0/run_baseline.py (
build_system_prompt).
Training
| |
|---|
| Method | bf16 LoRA (no quantization) |
| LoRA | r=32, α=64, dropout 0.05, all 7 projections (q,k,v,o,gate,up,down) |
| Data | 2,314 execution-validated synthetic pairs (distilled from Claude, gated on DuckDB execution + house-style checks) |
| Schedule | 2 epochs, lr 2e-4 cosine, effective batch 16, max_seq 2048, completion-only loss |
| Hardware | 1× A40 (48GB) |
Limitations
- Single schema. Conventions are specific to this TPC-H-derived database; the Spider
result shows general SQL is intact, but in-domain gains do not transfer to other schemas.
- Residual failures concentrate in a categorical-literal bug (
churn_risk='High' vs
'HIGH_RISK'), output-column shape, and simulation/CTE format — see the project repo's
failure-mode analysis.
- Sibling adapters (DPO, and a 1.5B SFT/GRPO track) are documented in the repo; neither
beat this SFT checkpoint, which is the deployed model.
Framework versions
