This is a LoRA (Low-Rank Adaptation) adapter trained to enhance Qwen2.5-Coder-3B-Instruct for natural language to SQL query generation. The adapter is lightweight (~20 MB) and can be merged with the base model for inference.

Base Model: Qwen/Qwen2.5-Coder-3B-Instruct
Training Dataset: BIRD Benchmark (7,542 training examples)
Training Method: LoRA with 4-bit quantization (QLoRA)
Parameters: 3B base + ~20M trainable (LoRA)

Performance

Metric	Value
Validation Loss (Initial)	5.210
Validation Loss (Final)	0.061
Loss Reduction	98.9%
Dummy DB Test Accuracy	100% (4/4)
Training Time	~13 hours (RTX 3060)

Quick Start

Installation

pip install transformers peft torch bitsandbytes

Loading the Model

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

# Load base model with 4-bit quantization
base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-Coder-3B-Instruct",
    device_map="auto",
    trust_remote_code=True,
    torch_dtype=torch.float16,
    load_in_4bit=True
)

# Load LoRA adapter
model = PeftModel.from_pretrained(
    base_model,
    "YOUR_USERNAME/qwen-text-to-sql-lora"
)

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(
    "Qwen/Qwen2.5-Coder-3B-Instruct",
    trust_remote_code=True
)

Inference Example

python

def generate_sql(question: str, schema: str) -> str:
    """Generate SQL from natural language question."""
    prompt = f"""You are an expert SQL generator. Given a database schema and a question, generate a SQL query.

Database Schema:
{schema}

Question: {question}

SQL Query:"""

    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    
    outputs = model.generate(
        **inputs,
        max_new_tokens=1024,
        temperature=0.1,
        top_p=0.9,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id
    )
    
    sql = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
    return sql.strip()

# Example usage
schema = """
CREATE TABLE customers (
    customer_id INT PRIMARY KEY,
    name TEXT,
    email TEXT,
    signup_date DATE
);

CREATE TABLE orders (
    order_id INT PRIMARY KEY,
    customer_id INT,
    order_date DATE,
    total_amount DECIMAL,
    FOREIGN KEY (customer_id) REFERENCES customers(customer_id)
);
"""

question = "What is the total revenue by month in 2024?"

sql = generate_sql(question, schema)
print(f"Generated SQL:\n{sql}")

Example Output:

sql

SELECT 
    strftime('%Y-%m', order_date) AS month,
    SUM(total_amount) AS total_revenue
FROM orders
WHERE strftime('%Y', order_date) = '2024'
GROUP BY month
ORDER BY month;

Training Details

Dataset

Source: BIRD (BIg Bench for LaRge-scale Database Grounded Text-to-SQLs)
Train Split: 7,542 examples
Validation Split: 1,886 examples
Domains: Cross-domain (e-commerce, healthcare, finance, etc.)

Training Configuration

python

# LoRA Configuration
lora_config = LoraConfig(
    r=16,                    # Rank
    lora_alpha=32,           # Scaling factor
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj"
    ],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

# Training Arguments
training_args = TrainingArguments(
    num_train_epochs=3,
    per_device_train_batch_size=8,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    fp16=True,
    optim="paged_adamw_8bit",
    lr_scheduler_type="cosine",
    warmup_ratio=0.1,
)

Hardware Requirements

Training:

GPU: NVIDIA RTX 3060 (12 GB VRAM) minimum
RAM: 32 GB recommended
Storage: 50 GB

Inference:

GPU: 6 GB VRAM (with 4-bit quantization)
RAM: 16 GB
CPU inference possible but slow

Limitations

Trained primarily on English text-to-SQL pairs
Performance may vary on databases with complex schemas (100+ tables)
Requires schema context for accurate generation
May struggle with ambiguous natural language queries
Not tested on all SQL dialects (focused on standard SQL)

Intended Use

✅ Recommended Use Cases

Text-to-SQL prototyping and development
Educational purposes for learning SQL generation
Research on instruction-tuned language models
Building SQL query assistants with human oversight

⚠️ Not Recommended

Production systems without human verification
Sensitive data queries without validation
Financial or medical applications without review
Direct execution of generated queries on production databases

Safety & Best Practices

Always validate generated SQL before execution
Use read-only database connections for generated queries
Implement query timeouts to prevent long-running queries
Sanitize inputs to prevent injection attempts
Test on development databases first

Bias & Fairness

The model inherits biases from:

Base Qwen2.5-Coder-3B model
BIRD benchmark dataset
SQL syntax conventions

Users should be aware of potential biases in:

Database naming conventions
Query patterns
Domain-specific terminology

Citation

If you use this model in your research, please cite:

bibtex

@software{qwen_text_to_sql_lora_2026,
  author = Jireh Fessenden,
  title = {Qwen2.5-Coder-3B Text-to-SQL LoRA Adapter},
  year = {2026},
  url = {https://huggingface.co/YOUR_USERNAME/qwen-text-to-sql-lora},
  note = {LoRA adapter for Text-to-SQL generation}
}

Also cite the base model and dataset:

bibtex

@article{qwen2.5coder,
  title={Qwen2.5-Coder Technical Report},
  author={Hui, Binyuan and others},
  year={2024}
}

@article{li2024bird,
  title={Can LLM Already Serve as a Database Interface?},
  author={Li, Jinyang and others},
  journal={arXiv preprint arXiv:2305.03111},
  year={2024}
}

License

MIT License - See LICENSE file

The base model (Qwen2.5-Coder-3B-Instruct) is under Apache 2.0 license.

Acknowledgments

Alibaba Qwen Team for the excellent base model
BIRD Benchmark Team for the high-quality dataset
HuggingFace for PEFT and Transformers libraries
Microsoft Research for LoRA methodology

Contact

For questions or issues:

GitHub Issues: https://github.com/evlogia-kyriou/text-to-sql-fine-tuning
Email: fessenden.jf@gmail.com
LinkedIn: www.linkedin.com/in/fessenden

Model Version: 1.0.0
Last Updated: January 2026
Status: ✅ Production Ready (with safety guidelines)

evlogia-kyriou/qwen-text-to-sql-lora

Model Card for Model ID

Qwen2.5-Coder-3B Text-to-SQL LoRA Adapter

Model Description

Performance

Quick Start

Installation

Loading the Model

Inference Example

Training Details

Dataset

Training Configuration

Hardware Requirements

Limitations

Intended Use

✅ Recommended Use Cases

⚠️ Not Recommended

Safety & Best Practices

Bias & Fairness

Citation

License

Acknowledgments

Links

Contact

evlogia-kyriou/
qwen-text-to-sql-lora