Evaluated on the first 200 samples of the gretelai/synthetic_text_to_sql test split using greedy decoding. ROUGE F-measures reported.
Model
ROUGE-1
ROUGE-2
ROUGE-L
Base Model (unsloth/Llama-3.2-3B-Instruct-bnb-4bit)
0.2908
0.2016
0.2651
Fine-Tuned (A-Kishore/llama-3.2-3b-text2sql)
0.8486
0.7232
0.8151
Improvement
+191.82%
+258.73%
+207.47%
Metric interpretation:
ROUGE-1 (unigram overlap) reflects accurate retrieval of schema identifiers and SQL keywords.
ROUGE-2 (bigram overlap) captures structural alignment of consecutive SQL constructs (e.g. GROUP BY, ORDER BY).
ROUGE-L (longest common subsequence) tracks overall query flow including nested clauses and join ordering.
Note: ROUGE measures lexical overlap, not SQL executability. A query may score slightly lower due to stylistic differences (alias names, join ordering) while still being functionally equivalent. See Limitations.
How to Use
The model weights are fully merged in 16-bit precision and load with standard transformers or unsloth.
Prompt Format
Always use this exact template — the model was trained on it:
###TASK
Generate the SQL query to answer the following question
### Database Schema
{sql_context}
### Question
{sql_prompt}
### SQL Query
(a) Standard transformers
python
1import torch
2from transformers import AutoTokenizer, AutoModelForCausalLM
34model_name ="A-Kishore/llama-3.2-3b-text2sql"56tokenizer = AutoTokenizer.from_pretrained(model_name)7model = AutoModelForCausalLM.from_pretrained(8 model_name,9 torch_dtype=torch.float16,10 device_map="auto"11)1213prompt ="""###TASK
14Generate the SQL query to answer the following question
1516### Database Schema
17{sql_context}
1819### Question
20{sql_prompt}
2122### SQL Query
23"""2425sql_context ="CREATE TABLE employees (id INT, name TEXT, department TEXT, salary REAL);"26sql_prompt ="What is the average salary per department?"2728inputs = tokenizer(29 prompt.format(sql_context=sql_context, sql_prompt=sql_prompt),30 return_tensors="pt"31).to("cuda")3233outputs = model.generate(34**inputs,35 max_new_tokens=150,36 use_cache=True,37 pad_token_id=tokenizer.eos_token_id
38)3940result = tokenizer.decode(outputs[0], skip_special_tokens=True)41sql = result.split("### SQL Query")[-1].strip()42print(sql)43# SELECT department, AVG(salary) FROM employees GROUP BY department;
(b) Unsloth Fast Inference
python
1import torch
2from unsloth import FastLanguageModel
34model, tokenizer = FastLanguageModel.from_pretrained(5 model_name="A-Kishore/llama-3.2-3b-text2sql",6 max_seq_length=768,7 dtype=torch.float16,8 load_in_4bit=False,9)10FastLanguageModel.for_inference(model)1112prompt ="""###TASK
13Generate the SQL query to answer the following question
1415### Database Schema
16{sql_context}
1718### Question
19{sql_prompt}
2021### SQL Query
22"""2324sql_context ="CREATE TABLE employees (id INT, name TEXT, department TEXT, salary REAL);"25sql_prompt ="What is the average salary per department?"2627inputs = tokenizer(28 prompt.format(sql_context=sql_context, sql_prompt=sql_prompt),29 return_tensors="pt"30).to("cuda")3132outputs = model.generate(33**inputs,34 max_new_tokens=150,35 temperature=None,36 do_sample=False,37 pad_token_id=tokenizer.eos_token_id
38)3940result = tokenizer.decode(outputs[0], skip_special_tokens=True)41sql = result.split("### SQL Query")[-1].strip()42print(sql)
LoRA freezes the base model weights and injects trainable rank-decomposition matrices into all attention and MLP projections. Only ~0.75% of parameters are updated, dramatically reducing VRAM usage and preventing catastrophic forgetting.
Hyperparameters
Parameter
Value
Optimizer
paged_adamw_8bit
Learning Rate
2e-4
LR Scheduler
linear
Warmup Steps
5
Epochs
1
Per-Device Batch Size
8
Gradient Accumulation
1
Max Sequence Length
768
Sequence Packing
True
Mixed Precision
fp16
Experiment Tracking
Weights & Biases
Training was accelerated using the unsloth library, which provides optimized GPU kernels for 4-bit quantized training (~2× faster than standard configurations).
evaluate_model.ipynb — ROUGE evaluation comparing base vs fine-tuned
Limitations
SQL executability: ROUGE is a lexical proxy. High ROUGE does not guarantee a query will execute or return logically correct results. A query with different aliases or reordered joins may score lower despite being equivalent.
Out-of-distribution schemas: Performance degrades on high-cardinality databases, deeply nested subqueries, or DDL patterns that diverge significantly from the training distribution.
Single epoch: The model was trained for one epoch on 50k samples. Further training may improve generalization.
License
The model adapter is released under Apache 2.0. The underlying base model is governed by the Meta Llama 3 Community License Agreement. Users must comply with both.
Acknowledgements
Unsloth — optimized kernels for 4-bit training and sequence packing
