Evaluated on 50 held-out samples from sql-create-context (seed=42). Zero regressions — every query the base model got right, this model also got right.

Training Details

Parameter	Value
Method	QLoRA (4-bit NF4 + LoRA)
LoRA rank	16
LoRA alpha	32
Target modules	q_proj, v_proj
Dataset	20,000 samples from sql-create-context
Epochs	2
Learning rate	2e-4
Effective batch size	16
Hardware	Kaggle T4 x2
Training time	~7 hours

Usage

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, AutoConfig
import torch

model_name = "microsoft/phi-2"
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token

config = AutoConfig.from_pretrained(model_name, trust_remote_code=True)
config.__dict__['pad_token_id'] = tokenizer.pad_token_id

base = AutoModelForCausalLM.from_pretrained(
    model_name, config=config,
    dtype=torch.float16, device_map="auto", trust_remote_code=True
)
model = PeftModel.from_pretrained(base, "antony-bryan-3D2Y/phi2-sql-lora-lr2e4")
model.eval()

prompt = """### SQL Schema:
CREATE TABLE employees (id INT, name VARCHAR, department VARCHAR, salary INT)

### Question:
What are the names of employees in the engineering department?

### SQL Query:
"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=100, do_sample=False,
                            eos_token_id=tokenizer.eos_token_id)
n = inputs['input_ids'].shape[1]
result = tokenizer.decode(output[0][n:], skip_special_tokens=True)
result = result.replace("</s>", "").replace("<|endoftext|>", "").split('\n')[0].strip()
print(result)
# → SELECT name FROM employees WHERE department = "engineering"

antony-bryan-3D2Y/phi2-sql-lora-lr2e4

Phi-2 SQL LoRA (lr=2e-4)

Results

Training Details

Usage

Links

antony-bryan-3D2Y/
phi2-sql-lora-lr2e4