Mistral-7B-Instruct-v0.3 text-to-SQL LoRA
The rank-32 LoRA adapter that produced
VikramPal/mistral-7b-instruct-v0.3-bf16,
which is that adapter merged into the base model. Published separately so the fine-tune can be
re-merged, inspected, or stacked onto a differently-quantized base without downloading 13.5 GiB.
This adapter is 335 611 085 B against the merge's 14 499 764 397 B, and it is the whole of the
difference between them. Everything the fine-tune learned is here; nothing else was changed.
What it was trained on
| |
|---|
| Base | mistralai/Mistral-7B-Instruct-v0.3 |
| Mixture | gretelai/synthetic_text_to_sql + Salesforce/wikisql + b-mc2/sql-create-context, 13 334 / 13 333 / 13 333 conversations |
| Kept | 39 531 of 40 000; 469 dropped for exceeding 2048 tokens |
| Loss on | completion only (mask_mode: template), 1 426 125 supervised tokens of 15 858 075 |
| Regime | LoRA r=32, alpha=64, dropout 0.05, on q,k,v,o,gate,up,down |
| Schedule | 2 epochs, lr 1e-4, effective batch 32, 2472 steps |
| Final train loss | 0.0540 |
| Wall clock | 3 h 49 m on one RTX PRO 6000 Blackwell |
Decontaminated against the eval split before training: 4 gretel, 16 wikisql and 3342
create-context conversations removed for overlapping an evaluation problem. create-context
is a training-only source -- it contributes to the adapter and is scored on nothing.
What it scores
Evaluated on 2454 held-out problems drawn equally from gretel, wikisql and spider, 2-shot,
greedy, execution-free logic match:
78.16% overall (1918/2454), 0 unparseable, 0 truncated. By source:
| source | accuracy | in the training mixture? |
|---|
| wikisql | 93.89% (768/818) | yes |
| gretel | 77.02% (630/818) | yes |
| spider | 63.57% (520/818) | no |
Spider is a third of the evaluation and none of the training mixture, so the 30-point gap
between it and wikisql is what this adapter does not transfer. 1144 of the 1918 correct
answers match the gold SQL as text; the other 774 are correct by execution equivalence.
Using it
1from peft import PeftModel
2from transformers import AutoModelForCausalLM, AutoTokenizer
3
4base = AutoModelForCausalLM.from_pretrained(
5 "mistralai/Mistral-7B-Instruct-v0.3", dtype="bfloat16", device_map="auto"
6)
7model = PeftModel.from_pretrained(base, "VikramPal/mistral-7b-instruct-v0.3-lora")
8tok = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.3")
Call model.merge_and_unload() to get the bf16 checkpoint published above, byte-for-byte
modulo the merge's own float arithmetic.
The rest of the campaign
This adapter is one artifact of a quantization study. The others:
The signals DynQuant allocates from -- per-module activation saliency and gradient plasticity --
were harvested during this LoRA run, by forward and backward hooks on 226 modules, and
written alongside the adapter. That is the only reason the adapter and the quantization are the
same campaign: the fine-tune is where the allocation's inputs come from.
Limitations
Trained to emit a single SQL statement for a schema and a question, and nothing else. It is not
a general assistant any more, and the 2-epoch schedule at loss 0.054 is well into the regime
where it will answer off-task prompts in SQL. Outputs are not validated against a database and
have not been checked for injection-safe parameterization -- do not execute them against
anything you care about without review.
