A 14B parameter model for Ada 2022 and SPARK code generation. QLoRA fine-tune of Qwen2.5-Coder-14B on 6,110 compiler-verified instruction pairs across 9 training categories, evaluated on a 754-prompt benchmark spanning 10 task categories. Named after the 1978 DoD Steelman requirements that defined the Ada language.
*Grok 4 and Gemini 3.1 Pro results are partial (537/754 and 547/754 respectively). Final numbers will be updated when complete.
A 14B QLoRA adapter outperforms every frontier model tested on strict Ada code generation by 49+ percentage points over the best complete frontier result (GPT-5.4 at 12.9%).
Per-Category Breakdown (Steelman Eval v4)
Category
Count
Steelman R7
Opus 4.6
GPT-5.4
spark
53
79.2%
9.4%
15.1%
error_fix
78
73.1%
14.1%
0.0%
ada2022
47
70.2%
8.5%
8.5%
tasking
54
70.4%
7.4%
29.6%
spec_to_body
88
67.0%
54.5%
51.1%
multi_file
57
59.6%
0.0%
0.0%
spec_impl
91
59.1%
3.8%
4.4%
generics
55
54.5%
0.0%
1.8%
modification
124
53.8%
5.8%
4.8%
standard
107
53.3%
12.1%
12.1%
Steelman leads in all 10 categories. The largest gaps are on error_fix (+59pp vs Opus), multi_file (+59.6pp vs both), and generics (+54.5pp vs Opus).
MultiPL-E HumanEval-Ada (157 problems, pass@1)
Standard code generation benchmark translated to Ada. Measures functional correctness: code must compile, run, and produce correct output.
Model
Pass@1
Compile Rate
Claude Opus 4.6
81.5% (128/157)
83.4%
Grok 4
77.1% (121/157)
81.5%
GPT-5.4
70.7% (111/157)
80.2%
Steelman R7 v0.3
45.9% (72/157)
85.4% (134/157)
Gemini 3.1 Pro
35.7% (56/157)
37.6%
Steelman has the highest Ada compile rate of any model tested (85.4%), including all frontier models. Frontier models win on algorithmic correctness because HumanEval tests general CS algorithms, not Ada-specific skills. Our eval (above) tests what matters for Ada development.
Reproducibility note: HumanEval is a code completion benchmark -- it provides a code prefix and expects the model to continue it. This model uses an Alpaca instruction template for inference. When we ran HumanEval through the Alpaca template (wrapping the code prefix in ### Instruction: / ### Response:), the model scored 8.9% because it treated the code as an instruction rather than code to complete. The 45.9% result above uses a raw completion template ({{ .Prompt }}{{ .Response }} with no system prompt and no instruction wrapping). If you are reproducing these results, you must use a raw template for HumanEval, not the Alpaca template used for instruction-following tasks.
What Changed in v0.3
v0.2 (R6)
v0.3 (R7)
Base model
Qwen2.5-Coder-14B-Instruct
Qwen2.5-Coder-14B (base)
Training data
3,018 pairs
6,110 pairs (all compile-verified)
LoRA rank / alpha
32 / 64
64 / 128 (rsLoRA)
Template
ChatML
Alpaca
Eval
v3 (500 prompts, compile-only)
v4 (754 prompts, compile + functional scoring)
Task categories
8
10 (+modification, +spec_impl)
Decontamination
Cosine similarity 0.80
Multi-stage: exact + 10-gram + Levenshtein
Why the base variant? With 6,110 training examples, the base model learns more effectively than starting from an instruction-tuned checkpoint. Qwen2.5-Coder has Ada in its 92-language training set, making it the strongest available foundation for Ada fine-tuning.
Why the eval numbers look lower than v0.2: Eval v4 is substantially harder than eval v3. It uses -gnatwe (warnings as errors) plus functional correctness scoring (modification tasks require passing test suites, not just compiling). The model is better; the bar is higher.
What This Model Does
Generates complete Ada 2022 source files from natural language instructions
Generates package bodies from .ads specifications (spec-to-body)
Modifies existing Ada code: adds features, fixes bugs, refactors (new in R7)
Handles multi-file Ada projects using [FILE: name.ext]/[/FILE] delimiters
Fixes compilation errors given GNAT diagnostics (73.1% accuracy)
Uses modern Ada 2022 features: expression functions, aspects, quantified expressions, delta aggregates, declare expressions, target name @
What This Model Does NOT Do
Not optimized for code completion or fill-in-the-middle (FIM) -- it generates complete files from instructions
Not a chatbot -- best results come from direct code generation prompts with a system prompt
SPARK contracts compile but may not prove with gnatprove -- formal verification is not part of the training loop
Algorithmic correctness is weaker than frontier models (45.9% vs 81.5% on HumanEval) -- the model excels at Ada-specific patterns, not general CS puzzles
FROM ./steelman-r7-coder-base-q8_0.gguf
TEMPLATE "Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
{{ .Prompt }}
### Response:
{{ .Response }}"
SYSTEM "You are an expert Ada 2022 and SPARK programmer."
PARAMETER stop "### Instruction:"
PARAMETER temperature 0.0
PARAMETER num_ctx 32768
bash
1ollama create steelman -f Modelfile
2ollama run steelman "Write an Ada procedure implementing a producer-consumer pattern with protected objects"
Python (transformers + peft)
python
1from peft import PeftModel
2from transformers import AutoModelForCausalLM, AutoTokenizer
34base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-14B", device_map="auto")5model = PeftModel.from_pretrained(base,"the-clanker-lover/steelman-14b-ada")6tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-Coder-14B")78SYSTEM ="You are an expert Ada 2022 and SPARK programmer."9TEMPLATE =(10"Below is an instruction that describes a task. "11"Write a response that appropriately completes the request.\n\n"12"### Instruction:\n{system}\n\n{instruction}\n\n### Response:\n"13)1415prompt = TEMPLATE.format(16 system=SYSTEM,17 instruction="Write an Ada function with SPARK contracts that performs binary search on a sorted array.",18)19inputs = tokenizer(prompt, return_tensors="pt").to(model.device)20outputs = model.generate(**inputs, max_new_tokens=2048, temperature=0.0)21print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Effective 32 (4 per device, 8 gradient accumulation)
Early stopping
Patience 2 (based on eval loss)
Hardware
NVIDIA H100 80GB
Every training example compiles cleanly with gnatmake -gnat2022 -gnatwa. The model never trains on broken code.
Training History
Round
Score
Eval
Dataset
Base Model
Notes
Baseline
~35%
v2 (compile)
--
Qwen2.5-Coder-14B-Instruct
Untuned
R1
53.8%
v2 (compile)
1,800
Qwen2.5-Coder-14B-Instruct
First release
R5 (v0.1)
68.6%
v2 (compile)
3,430
Qwen2.5-Coder-14B-Instruct
+task types
R6 (v0.2)
72.0%
v3 (strict compile)
3,018
Qwen2.5-Coder-14B-Instruct
Strict-triaged data
R7 (v0.3)
62.4%
v4 (compile + functional)
6,110
Qwen2.5-Coder-14B
New base, 2x data, 10 categories
Note: R7's 62.4% on eval v4 and R6's 72.0% on eval v3 use different scoring methodologies. Eval v4 includes functional correctness scoring (modification tasks must pass test suites) and two new categories (modification, spec_impl). The model improved; the eval got harder.
Evaluation Methodology
Eval v4 is a 754-prompt benchmark across 10 Ada task categories with three scoring modes:
Scoring note: Steelman's eval was run via two independent methods (local Ollama Q8_0 and RunPod fp16 transformers) with results within 0.5pp. The 62.4% headline uses the fp16 run. Frontier models were evaluated via OpenRouter API with identical prompts and local compilation scoring.
Decontamination: Multi-stage pipeline following published methodology:
Exact substring match after whitespace normalization (Lozhkov et al. 2024 / StarCoder2)
10-gram overlap (Guo et al. 2024 / DeepSeek-Coder)
Levenshtein similarity on extracted task content (Riddell et al. 2024 / ACL)
Instruction template boilerplate stripped before comparison (Lee et al. 2023 / Open-Platypus)
Result: 754 prompts, 0 contaminated against 6,110 training examples.
Ada/SPARK only -- not a general-purpose coding model
Algorithmic reasoning is weaker than frontier models. HumanEval-Ada pass@1 is 45.9% vs Opus 4.6's 81.5%. The model excels at Ada-specific patterns (specs, SPARK, error fixing, multi-file) but lags on general CS algorithms.
SPARK contracts compile but may not prove.gnatprove verification is not part of the training loop.
Synthetically generated training data. All 6,110 examples were produced by language models, not written by Ada developers. Every example is compiler-verified, but there may be non-idiomatic patterns.
Alpaca template only. This model uses an Alpaca-style instruction template, not ChatML. Using the wrong template will significantly degrade output quality (we measured 8.9% vs 45.9% on HumanEval with the wrong template).
14B parameter model. It will miss things a larger model would catch.
About
This project exists because Ada developers working on safety-critical systems -- avionics, defense, rail, medical devices, space -- couldn't use LLM-assisted tooling. Every major model struggles with Ada's strict type system, SPARK's verification contracts, and the GNAT compiler's error messages.
Steelman is a solo project, built by a self-taught developer using AI assistance at every step -- dataset generation, training pipeline, evaluation harness, and this model card. The entire project was built in about three weeks.
Related
GGUF quantization -- Q8_0 for local inference via Ollama or llama.cpp
Fer (Irvise) on forum.ada-lang.io -- recommended the runtime verification flags (-gnata, -gnateE, -gnateF, -gnatVa, -fstack-check) that became the foundation of strict evaluation. The single most impactful technical input.
K_Kolomeitsev on r/LocalLLaMA -- identified generics and tasking coverage as priorities, directly shaping the eval categories.
License
Apache 2.0 (same as base model).
Citation
bibtex
1@misc{steelman-14b-ada,
2 title={Steelman-14B-Ada: QLoRA Fine-Tuning for Ada 2022/SPARK Code Generation},
3 author={the-clanker-lover},
4 year={2026},
5 url={https://huggingface.co/the-clanker-lover/steelman-14b-ada}
6}
