A 0.5B parameter language model fine-tuned with LoRA to convert natural language questions into SQL queries — outperforming a 70B general-purpose model on the same benchmark despite being 140x smaller.

🔥 Benchmark Results

Model	Exact-Match Accuracy
Llama-3.3-70B (zero-shot, via Groq)	32.0%
Qwen2.5-0.5B (this model, fine-tuned)	84.0%

Evaluated on a held-out set of 100 text-to-SQL questions, scored by exact match after SQL normalization.

🧠 Model Details

Base model: Qwen/Qwen2.5-0.5B-Instruct
Fine-tuning method: LoRA (r=16, alpha=32, target modules: q_proj, v_proj)
Training data: b-mc2/sql-create-context (78,477 examples)
Epochs: 3
Hardware: Free Google Colab T4 GPU

🚀 Usage

python

from peft import PeftModel
from transformers import AutoTokenizer, AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
model = PeftModel.from_pretrained(base_model, "javeria163/text-to-sql")

schema = "CREATE TABLE employees (id INT, name TEXT, department TEXT, salary INT)"
question = "What is the average salary in the Engineering department?"

prompt = (
    "Task: Convert the question to SQL using the table schema.\n\n"
    "Schema:\n" + schema + "\n\n"
    "Question:\n" + question + "\n\n"
    "SQL:\n"
)

inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=150, do_sample=False)
print(tokenizer.decode(outputs[0], skip_special_tokens=True).split("SQL:")[-1].strip())

🎮 Try It Live

Interactive demo on Hugging Face Spaces — no setup required.

📊 Training

Loss decreased from 1.73 → 0.69 over 3 epochs, with final mean token accuracy of 82.9%.

🔗 Links

GitHub Repository — full training code, eval scripts, and benchmark
Live Demo