A Q4_K_M quantized GGUF version of my fine-tuned Llama 3.2 3B Text-to-SQL model, designed for efficient local inference with llama.cpp and other GGUF-compatible runtimes.

The model converts natural-language questions and database schemas into SQL queries.

Model Details

Parameter	Value
Base Model	`meta-llama/Llama-3.2-3B`
Fine-tuned Model	`farehaheha/llama3.2-3B-text-to-sql`
Quantization	`Q4_K_M`
Format	GGUF
Model Size	~2 GB
Task	Text-to-SQL
Model Type	Completion model

Original Model

This GGUF model is a quantized version of:

farehaheha/llama3.2-3B-text-to-sql

The original model was fine-tuned using QLoRA on approximately 5,850 Text-to-SQL examples from the Gretel AI Synthetic Text-to-SQL dataset.

Important: This Is a Completion Model

This model was fine-tuned as a completion model, not as an instruction or chat model.

For best results, use the same prompt format used during fine-tuning:

text

### Database Schema:
{your_database_schema}

### Request:
{your_natural_language_request}

### SQL Query:

The model should continue directly from ### SQL Query:.

Avoid wrapping the prompt in chat templates or system/user/assistant messages.

Run with llama.cpp

Start llama-server:

bash

llama-server \
  -m llama3.2-3b-text-to-sql-q4_k_m.gguf \
  --port 8080 \
  -ngl 99 \
  -c 4096

On Windows:

.\llama-server.exe -m ".\llama3.2-3b-text-to-sql-q4_k_m.gguf" --port 8080 -ngl 99 -c 4096

Inference with Python

Once llama-server is running on port 8080, send the prompt directly to the completion endpoint:

python

import requests

schema = """
CREATE TABLE salesperson (
    salesperson_id INT,
    name TEXT,
    region TEXT
);

CREATE TABLE timber_sales (
    sales_id INT,
    salesperson_id INT,
    volume REAL,
    sale_date DATE
);
"""

request = """
What is the total volume of timber sold by each salesperson?
"""

prompt = f"""### Database Schema:
{schema}

### Request:
{request}

### SQL Query:
"""

response = requests.post(
    "http://localhost:8080/completion",
    json={
        "prompt": prompt,
        "n_predict": 256,
        "temperature": 0.0,
        "stop": ["###"]
    },
    timeout=60
)

response.raise_for_status()

sql = response.json()["content"].strip()
print(sql)

Example output:

sql

SELECT salesperson_id, SUM(volume) AS total_volume
FROM timber_sales
GROUP BY salesperson_id;

Inference Settings

Recommended settings for deterministic Text-to-SQL generation:

Parameter	Recommended Value
Temperature	`0.0`
Max Tokens	`128–256`
Stop Sequence	`###`
Endpoint	`/completion`

For Text-to-SQL, greedy decoding is generally preferred over creative sampling.

Python Inference Script

A simple Python example using llama-server is available in the project repository:

llama_cpp_tryout.py

The script:

starts from the model's completion-style prompt format
sends requests to a local llama-server
uses the /completion endpoint
generates SQL directly from the provided schema and natural-language request

Why Q4_K_M?

Q4_K_M provides a practical balance between:

reduced model size
lower memory usage
faster local inference
preservation of model quality

At approximately 2 GB, this version is intended to make the model easier to run on consumer hardware.

Limitations

The model may generate syntactically valid but logically incorrect SQL.
Performance depends on schema complexity and how clearly the request is written.
Generated SQL should be validated before execution in production environments.
The model was fine-tuned using examples from the test split of the source dataset, so that split should not be used for unbiased evaluation.

Acknowledgements

Built using:

Meta Llama 3.2 3B
Unsloth
Gretel AI Synthetic Text-to-SQL dataset
llama.cpp

License

Apache 2.0

farehaheha/llama3.2-3B-text-to-sql-Q4_K_M-GGUF

Llama 3.2 3B Text-to-SQL — Q4_K_M GGUF