Show Your Work: Verified Math and Code (Gemma 4 31B LoRA)
A PEFT LoRA adapter trained on 17,586 math and code problems where the answers were independently checked: 9,104 math rows carry a gold answer computed separately, 4,156 code rows were executed against unit tests, and 3,740 general rows were deliberately left unverified as an anti-forgetting slice.
It lost. The head-to-head result is 46 wins for the adapted model against 54 for the base. That is a regression, not a tie and not a win. Training loss fell from 3.48 to 1.24 and held-out eval loss fell from 2.37 to 1.22, and the model still came out behind.
This card gives the diagnosis from the weights and the training state rather than dressing the number up. There are two causes and they compound: the base was already strong on this domain, and the run was numerically unstable, with 29 of 59 optimizer steps hitting the gradient clip.
What it is: a rank-8 LoRA adapter on google/gemma-4-31B-it, 10,956,800 trainable parameters, 43.9 MB.
What it was for: step by step derivations for math and code, with the working shown rather than just the answer.
The number: 46 for the adapted model, 54 for the base. A regression of 8 points.
The reasons: the base scored 8.0 out of 10 on the source corpus before any adaptation, the highest starting point of the four datasets in this project, so there was almost nothing to add. On top of that, 29 of 59 steps exceeded the clip threshold of 2, peaking at 677.
Good news on loading: unlike the sibling adapters in this project, the declared base model string is correct and resolves. google/gemma-4-31B-it is ungated and Apache-2.0.
Built by: MANIFESTA (Aivaras Navardauskas) for the Adaption AutoScientist Challenge, Math and Code.
Quickstart
The base string in adapter_config.json is google/gemma-4-31B-it, which resolves, is ungated, and is Apache-2.0 licensed. No workaround needed here. (The two chart adapters in this project both shipped broken base strings, so this is worth stating explicitly.)
Gemma 4 is a conditional-generation checkpoint with a vision tower, so use the image-text-to-text auto class even when you only want text.
python
1import torch
2from transformers import AutoModelForImageTextToText, AutoProcessor
3from peft import PeftModel
45BASE ="google/gemma-4-31B-it"6ADAPTER ="manifesta/adaption_verified_math_code_instruct"78processor = AutoProcessor.from_pretrained(BASE)9base = AutoModelForImageTextToText.from_pretrained(10 BASE, torch_dtype=torch.bfloat16, device_map="auto"11)12model = PeftModel.from_pretrained(base, ADAPTER)13model.eval()1415messages =[{16"role":"user",17"content":[{"type":"text","text":18"A factory makes 1,440 pencils in 8 hours. Two machines are down, cutting output "19"by 40%. How many pencils are made in a 6 hour shift? Show your work."}],20}]2122inputs = processor.apply_chat_template(23 messages, add_generation_prompt=True, tokenize=True,24 return_dict=True, return_tensors="pt",25).to(model.device)2627with torch.inference_mode():28 out = model.generate(**inputs, max_new_tokens=512, do_sample=False)29print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
Because the base string is correct, the one-line loader works too:
Given the evaluation result, the most useful thing to do with this snippet is run the same prompt through the base with and without the adapter attached and compare. That is what the judge did, and the base won.
What is actually in these weights
Read straight out of adapter_model.safetensors and adapter_config.json.
Property
Value
PEFT type
LoRA, task_type: CAUSAL_LM
Rank r
8
lora_alpha
8 (scaling factor alpha/r is exactly 1.0)
lora_dropout
0.0
bias
none
use_rslora / use_dora
false / false
Target modules
q_proj, v_proj
exclude_modules
empty
Tensors in the file
220
Decoder layers touched
60 of 60 for q_proj, 50 of 60 for v_proj
Vision tensors
0
Trainable parameters
10,956,800 (float32, 43.9 MB)
Share of the 31B base
roughly 0.035%
The v_proj gap is architecture, not a bug
v_proj is missing on exactly ten layers: 5, 11, 17, 23, 29, 35, 41, 47, 53 and 59. Every sixth layer.
Those are precisely the layers listed as full_attention in the base model's layer_types. The other fifty are sliding_attention. Checking the base model's own weight map confirms it: model.layers.5.self_attn contains q_proj, k_proj, o_proj, q_norm and k_norm, and no v_proj.weight at all, while model.layers.4.self_attn does have one.
The reason is in the base config: attention_k_eq_v: true, with num_global_key_value_heads: 4 and global_head_dim: 512 on the global layers. On full-attention layers the key and value projections are shared, so there is no separate v_proj module for PEFT to wrap. It asked for v_proj on all 60 layers and silently got it on 50.
Nothing failed. But it does mean the adapter covers less of the attention stack than target_modules: [q_proj, v_proj] suggests, and the layers it covers least are the global-attention layers, which are the ones carrying long-range context. For multi-step derivations, long-range context is the part that matters most.
No vision tensors
Gemma 4 31B is multimodal, with a 27-layer vision tower. This adapter touched none of it. For a text-only math and code task that is the right call, so it is recorded here for completeness rather than as a criticism. It is a different story on the chart QA adapter, where the frozen vision tower is the whole explanation for that run's null result.
Evaluation
Judged by Adaption's AutoScientist evaluator on a held-out split, against the same base model it was trained from. Head-to-head, blind.
Metric
Base
Adapted
Change
Win rate
54
46
-8 points
Held-out eval loss (first eval)
2.3657
Held-out eval loss (final)
1.2200
-1.146
Training loss
3.482 (step 1)
1.238 (step 59)
-2.244
Eval loss across the five checkpoints: 2.3657, 1.5618, 1.3206, 1.2417, 1.2200.
Win rates, adapted versus base
Read those two rows together, because they disagree. Loss dropped by 64% and the model still lost the head-to-head by 8 points. A falling loss curve says the adapter learned to reproduce the training distribution. It says nothing about whether reproducing that distribution is an improvement over what the base already did. Here it was not.
This is the single most useful thing in this repo: eval loss is not the headline metric, and if you optimise against it you can ship a regression with a beautiful chart attached.
Why it lost
Two causes, and they compound.
1. The base was already good, so there was no room. Adaptive Data scored the source corpus at 8.0 out of 10 before adaptation, grade B, the highest starting quality of the four datasets built in this project. Adaptation moved it to 8.2, a +2.5% lift, and the grade did not change. Compare the agronomy corpus at 5.0 to 8.4 (+68.0%) and the chart corpus at 6.0 to 7.1 (+18.3%).
Gemma 4 31B is instruction-tuned and already writes competent step by step derivations for exactly this kind of problem. A rank-8 adapter on two projections, roughly 0.035% of parameters, cannot add mathematical capability the base lacks. What it can do is change style and formatting. When the base's existing style is already good, changing it is as likely to hurt as help, and a blind judge comparing two competent derivations will punish any drift toward the training corpus's quirks.
2. Two thirds of the updates were clipped. See below.
3. The global-attention layers got the least adaptation. As covered above, the ten full_attention layers received q_proj only. Those are the layers handling long-range dependency, which is what a multi-step derivation depends on to keep an intermediate result consistent from line four to line twelve.
What would need to change: a lower learning rate or a longer warmup to stop the gradient spikes, a higher rank if the goal is genuinely new capability rather than style transfer, and a harder corpus. Problems the base already solves at 8.0 out of 10 are the wrong training signal. The value should sit in problems it gets wrong.
Numerical stability
This run was not stable.
Gradient norm statistic
Value
Clipping threshold (max_grad_norm)
2
Peak gradient norm
677.21 (step 38)
Median gradient norm
1.97
Minimum
0.389
Steps above 1
39 of 59 (66%)
Steps above the clip threshold of 2
29 of 59 (49%)
Steps above 10
9 of 59 (15%)
Steps above 100
5 of 59 (8%)
Largest spikes: step 38 at 677.2, step 15 at 645.0, step 40 at 389.4, step 14 at 172.5, step 26 at 138.6. The final step of the run, step 59, was still at 40.1.
Training metrics
What clipping at 2 does when the true norm is 677 is keep the direction and throw away the magnitude. The update becomes a fixed-size step along whatever direction that batch happened to point in, and the relative weighting between batches is gone. When that happens on half the steps of a 59-step run, the optimizer is not really following the loss surface any more, it is taking a sequence of equal-length steps in noisy directions.
The loss still came down, which is worth sitting with. A falling loss curve does not certify a healthy run.
For contrast, the chart QA adapter trained on the same platform with the same clip threshold peaked at 0.72 and clipped on zero of its 34 steps. Same settings, completely different numerical behaviour, which points at the data and the base rather than the configuration.
(The Adaption run dashboard reports the peak as 744.9. The 677.21 above is the maximum in trainer_state.json in this repo, which is the number you can verify yourself. Both are two to three orders of magnitude over the threshold and the conclusion is the same.)
The cross-run finding
Three completed AutoScientist runs, three different base models. Win rate tracked how weak the base already was on the domain, and did not track dataset size.
Run
Base
Params
Rows ingested
Steps
Rows per step
Win rate (adapted vs base)
Chart QA, first build
Qwen3.5 9B
9B
6,976
21
332
51 vs 49
Chart QA 17k
Gemma 3 27B
27B
17,070
34
502
50 vs 50
Verified math and code (this model)
Gemma 4 31B
31B
17,586
59
298
46 vs 54
The bigger the base, the smaller the win. 9B scored 51, 27B scored 50, 31B scored 46. This model sits at the bottom of that trend and it is the only one that went backwards.
Scaling the corpus did nothing. The chart corpus was rerun at 2.4x the rows, 6,976 to 17,070, and the win rate went 51 to 50. Because batch_size is set to max, extra rows widen the batch rather than adding updates: step count went 21 to 34 while rows per step went 332 to 502. More rows per step is a smoother gradient estimate for the same small number of updates, not more learning.
Base model size against win rate across three runs
Caveat, stated plainly: three data points, three different base models, three different corpora, three different domains. Base size, base family and task difficulty all move together, so nothing here isolates a cause. It is not a controlled experiment and should not be read as one. Testing it properly means holding the corpus fixed and varying only the base.
Training
Trained with Adaption AutoScientist, SFT with LoRA, on Adaptive Data output.
Hyperparameter
Value
Base model
google/gemma-4-31B-it (resolves correctly)
Method
SFT, LoRA (PEFT), chat format
LoRA rank / alpha / dropout
8 / 8 / 0.0
Trainable modules
q_proj, v_proj (requested on 60 layers, attached on 60 and 50)
Learning rate
1e-4
Scheduler
cosine, 0.5 cycles, warmup ratio 0.1, min LR ratio 0.1
Weight decay
0
Max gradient norm
2
Epochs
1
Batch size
max (per-device train batch size 1 with accumulation)
Optimizer steps
59
Evaluations
5, every 11 steps
Train on inputs
false
Precision
bfloat16 base, float32 adapter weights
Total FLOPs
4.073e18
Adaption job ID
9f46e2b9-ac73-4b97-ad13-cc6ccd1501cf
Training experiment ID
3ce14a29-6d5d-48ea-849f-372a6102c360
Adaption dataset ID
6ad1a83f-b806-4a20-ba6d-239029d30711
Intended use and limits
Use it for
Reproducing and inspecting this regression. That is the honest primary use.
Studying the gap between a falling loss curve and a falling win rate, which this run demonstrates about as cleanly as it can be demonstrated.
Studying gradient instability in short LoRA runs, using the 59 logged steps in trainer_state.json.
A baseline for a rerun with a lower learning rate, a longer warmup, or a harder corpus.
Do not use it for
Production math or code assistance. Stock google/gemma-4-31B-it beat it 54 to 46 on the held-out split. Use the base.
Anything where a wrong derivation carries cost: engineering, finance, dosing, safety calculations. Verified training data does not make outputs verified.
Executing generated code without a sandbox. The training data was executed against unit tests; the model's output at inference time was not.
Technical limits
The adapter is behind its own base model on the only head-to-head evaluation that was run. Treat the weights as a research artefact.
Rank 8 on two projections, roughly 0.035% of base parameters, one epoch, 59 optimizer steps.
v_proj is absent on the ten global-attention layers. Not a defect, but the coverage is thinner than the config implies.
Half the optimizer steps were clipped. The weights reflect a run that did not converge cleanly.
English only.
Adapter weights are float32 while the base is bfloat16. PEFT casts on load, but the adapter file is double the size a bf16 export would be.
One judge, one held-out split. Reported as-is, no reruns, no best-of selection.
17,586 rows ingested, expanded to 21,063 rows in the training split after adaptation
9,104 math rows with an independently computed gold answer
4,156 code rows executed against unit tests, 100% of them run
3,740 general instruction rows deliberately left unverified, as an anti-forgetting slice
Domain mix after adaptation: math 56%, code 25%, science 3%, with a long tail across roughly 40 further domains at 1% or below
What was verified inside the 17,000 rows
Adaptive Data lifted the corpus quality before training:
Adaptive Data metric
Before
After
Quality score
8.0
8.2 (+2.5%)
Grade
B
B
Percentile
15.3
17.8
That +2.5% is the smallest lift of the four datasets built in this project, and it is the clue. The corpus started at 8.0, so there was very little for adaptation to fix, and by extension very little for the model to learn that it did not already know. +2.5% is an improvement in the data, not in the model. The model's own number went the other way.
Live interface
The full written case for this entry, every claim paired with the command that checks it, is at
The Verdict.
Show Your Work runs at manifesta.adaptionlabs.app. Give it a problem, get a step by step derivation with a verification panel, and use the "Change a number" button to check that the derivation actually recomputes rather than pattern matching to a memorised answer.
Show Your Work solving a multi-step word problem with a verification panel
Related models
manifesta/adaption_scientific_chart_qa_17k, scientific chart QA on Gemma 3 27B. 50 vs 50, a null result, because the LoRA covered zero vision tensors while the task is entirely visual. Its declared base string 404s.
manifesta/scientific-chart-qa-lora-qwen3.5-9b, the first chart build on a 9B base. 51 vs 49, 21 optimizer steps, LoRA on only 8 of 32 decoder layers and again zero vision tensors. Its declared base togethercomputer/Qwen3.5-9B also 404s.
manifesta/brandvoice-marketing-model, the Part 1 marketing adapter that did work, 56% win rate on Llama 3.3 70B. Style transfer on a base with a weak default style, which is the case where a small adapter has room to move.
Everything behind these weights is public
The dataset this adapter was trained on, the scripts that built it, and a verifier that rechecks
every number claimed here against the live artifacts:
Standard library only, no install step and no account. 23 checks, and the same 23 run on a daily
schedule in GitHub Actions, so the badge above goes red if any claim on this card stops being true.
The tensor facts in this card are among the checks: layer coverage is re-derived from the published
adapter_model.safetensors by name, not copied from the config.
1@misc{showyourwork_mathcode_2026,
2 title = {Show Your Work: a verified math and code adapter for Gemma 4 31B, and why it regressed},
3 author = {Navardauskas, Aivaras},
4 year = {2026},
5 note = {MANIFESTA. Adapted with Adaptive Data by Adaption, AutoScientist Challenge, Math and Code. Win rate 46 vs 54. Half the optimizer steps clipped.},
6 howpublished = {\url{https://huggingface.co/manifesta/adaption_verified_math_code_instruct}}
7}
Built with Adaptive Data by Adaption. Platform documentation at docs.adaptionlabs.ai. Dataset, adapter and analysis by MANIFESTA (Aivaras Navardauskas) for the AutoScientist Challenge.
