A fine-tuned Qwen3-4B model for converting natural language questions into SQL queries. Trained using knowledge distillation from DeepSeek-V3, this 4B parameter model matches teacher-level accuracy while being small enough to run locally.

Results

Metric	DeepSeek-V3 (Teacher)	Qwen3-4B (Base)	This Model
LLM-as-a-Judge	80%	62%	80%
Exact Match	48%	16%	60%
ROUGE	87.6%	84.2%	89.5%
METEOR	85.1%	87.3%	86.1%

The fine-tuned model matches the 685B parameter teacher on LLM-as-a-Judge accuracy and exceeds it on exact match and ROUGE scores.

Quick Start

Using Transformers

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("distil-labs/distil-qwen3-4b-text2sql")
tokenizer = AutoTokenizer.from_pretrained("distil-labs/distil-qwen3-4b-text2sql")

schema = """CREATE TABLE employees (
  id INTEGER PRIMARY KEY,
  name TEXT NOT NULL,
  department TEXT,
  salary INTEGER
);"""

question = "How many employees earn more than 50000?"

messages = [
    {
        "role": "system",
        "content": """You are a problem solving model working on task_description XML block:
<task_description>You are given a database schema and a natural language question. Generate the SQL query that answers the question.

Input:
- Schema: One or two table definitions in SQL DDL format
- Question: Natural language question about the data

Output:
- A single SQL query that answers the question
- No explanations, comments, or additional text

Rules:
- Use only tables and columns from the provided schema
- Use uppercase SQL keywords (SELECT, FROM, WHERE, etc.)
- Use SQLite-compatible syntax</task_description>
You will be given a single task in the question XML block
Solve only the task in question block.
Generate only the answer, do not generate anything else"""
    },
    {
        "role": "user",
        "content": f"""Now for the real task, solve the task in question block.
Generate only the solution, do not generate anything else
<question>Schema:
{schema}

Question: {question}</question>"""
    }
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Using Ollama (GGUF version)

For local inference, use the quantized GGUF versions:

distil-qwen3-4b-text2sql-gguf - Full precision GGUF
distil-qwen3-4b-text2sql-gguf-4bit - 4-bit quantized (~2.5GB)

bash

# Download and create Ollama model
ollama create distil-qwen3-4b-text2sql -f Modelfile

# Run inference
ollama run distil-qwen3-4b-text2sql

Model Details

Property	Value
Base Model	Qwen/Qwen3-4B
Parameters	4 billion
Architecture	Qwen3ForCausalLM
Context Length	262,144 tokens
Precision	bfloat16
Training Data	~10,000 synthetic examples
Teacher Model	DeepSeek-V3

Training

This model was trained using the Distil Labs platform:

Seed Data: 50 hand-validated Text2SQL examples covering various SQL complexities
Synthetic Generation: Expanded to ~10,000 examples using DeepSeek-V3
Fine-tuning: 4 epochs on the synthetic dataset
Evaluation: LLM-as-a-Judge with semantic equivalence checking

Training Hyperparameters

Epochs: 4
Learning Rate: 5e-5 (cosine schedule)
Batch Size: 1 (with gradient accumulation)
Total Steps: ~40,000

Task Format

Input Format

Schema:
CREATE TABLE table_name (
  column_name DATA_TYPE [CONSTRAINTS],
  ...
);

Question: Natural language question about the data

Output Format

A single SQL query with:

Uppercase SQL keywords (SELECT, FROM, WHERE, etc.)
SQLite-compatible syntax
No explanations or additional text

Supported SQL Features

Simple: SELECT, WHERE, COUNT, SUM, AVG, MAX, MIN
Medium: JOIN, GROUP BY, HAVING, ORDER BY, LIMIT
Complex: Subqueries, multiple JOINs, UNION

Use Cases

Natural language interfaces to databases
SQL query assistance and autocompletion
Database chatbots and conversational BI
Educational tools for learning SQL

Limitations

Optimized for SQLite syntax
Best with 1-2 table schemas
May struggle with highly complex nested subqueries
Trained on English questions only

License

This model is released under the Apache 2.0 license.

Citation

bibtex

@misc{distil-qwen3-4b-text2sql,
  author = {Distil Labs},
  title = {Distil-Qwen3-4B-Text2SQL: A Fine-tuned Model for Natural Language to SQL},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/distil-labs/distil-qwen3-4b-text2sql}
}

distil-labs/distil-qwen3-4b-text2sql

Distil-Qwen3-4B-Text2SQL