You can deploy this model and start earning money today!

glen-louis/
llama-3.2-3b-sql-qlora

Name: llama-3.2-3b-sql-qlora
Availability: InStock

safetensors

Views

No views yet

llama-3.2-3b-sql-qlora

Fine-tuned meta-llama/Llama-3.2-3B-Instruct on b-mc2/sql-create-context using QLoRA for Text-to-SQL generation.

Results

Metric	Base Model	Fine-tuned	Delta
Perplexity (eval split, ↓)	35.0650	1.8836	-94.6%
ROUGE-L (200 samples, ↑)	0.9092	0.9856	+8.4%

Evaluated on a 5% held-out split of b-mc2/sql-create-context (seed=42, 3,929 examples).

Training Configuration

Setting	Value
Base model	meta-llama/Llama-3.2-3B-Instruct
Quantization	4-bit NF4 (QLoRA)
LoRA rank	r=16, alpha=32
LoRA target modules	q/k/v/o/gate/up/down_proj
Trainable parameters	~20M (~0.67% of model)
Dataset	b-mc2/sql-create-context (~78k rows)
Epochs	1
Effective batch size	16
Learning rate	0.0002 (cosine schedule)
Optimizer	paged_adamw_8bit

Quickstart

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-3B-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, "glen-louis/llama-3.2-3b-sql-qlora")
tokenizer = AutoTokenizer.from_pretrained("glen-louis/llama-3.2-3b-sql-qlora")

messages = [
    {"role": "system", "content": "Given a SQL table schema, write a SQL query that answers the question."},
    {"role": "user", "content": "Table: employees (id INT, name TEXT, department TEXT, salary INT)\nQuestion: What is the average salary by department?"},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=256, do_sample=False)

print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Methodology

What is QLoRA? QLoRA (Quantized Low-Rank Adaptation) combines two techniques: the base model weights are frozen and compressed to 4-bit precision using NF4 quantization (reducing memory from ~6GB to ~2GB for a 3B model), while a small set of trainable LoRA adapter matrices are added in full precision. This means we only update ~20M parameters instead of 3B, making fine-tuning feasible on a single consumer GPU.

Why sql-create-context? The sql-create-context dataset contains ~78,000 natural language question / SQL answer pairs grounded in explicit CREATE TABLE schemas. Each example provides the full table definition, a natural language question, and the correct SQL query — teaching the model to ground its output in real schema structure rather than hallucinate column names or types.

What do the numbers mean? Perplexity measures how surprised the model is by the held-out text under teacher-forcing — a drop from 35.1 to 1.9 means the fine-tuned model assigns substantially higher probability to correct SQL responses. ROUGE-L measures the longest common subsequence overlap between generated and reference SQL on 200 sampled examples — an improvement from 0.909 to 0.986 confirms the model generates more structurally accurate queries, not just lower perplexity.

Limitations

The model is trained for single-table SQL queries matching the style of the sql-create-context dataset. It may struggle with multi-table JOINs, nested subqueries, or dialects (e.g. T-SQL, PL/pgSQL) not represented in the training data. The model has not been RLHF-tuned for safety — use with the same caveats as the base meta-llama/Llama-3.2-3B-Instruct.

Citation

bibtex

@dataset{sql_create_context,
  author = {b-mc2},
  title = {sql-create-context},
  year = {2023},
  url = {https://huggingface.co/datasets/b-mc2/sql-create-context},
}

Training framework: PyTorch · Transformers · PEFT · TRL · bitsandbytes