A Llama-3.2-1B fine-tune on 5k examples from MathLLMs/MathCodeInstruct, trained to solve math word problems with step-by-step natural-language reasoning interleaved with executable Python.

This is one of three sibling models trained on {5k, 10k, 20k}-example subsets of the same dataset, to study how fine-tuning data volume trades off against both math performance and general capability. See the training write-up for the full comparison across all three.

Training details


Base model	`unsloth/Llama-3.2-1B`
Method	LoRA (r=16, α=16, dropout=0) on all attention + MLP projections, merged to full weights
Dataset	MathLLMs/MathCodeInstruct, 5k training examples
Epochs	1
Effective batch size	16 (batch 1 × grad. accum. 16)
Learning rate	2e-4, cosine schedule, warmup ratio 0.03
Hardware	1× RTX 4060 (8GB)
Framework	Unsloth + TRL `SFTTrainer`

Benchmark results

All benchmarks run with lm-evaluation-harness, each at its standard published shot count, compared against the un-tuned base model.

Benchmark	Llama-3.2-1B (base)	MathCodeInstruct-5k	Change
GSM8K	5.8%	7.4%	🟢 +1.5%
ARC-Challenge	36.9%	36.8%	⚪ -0.1%
HellaSwag	64.2%	63.8%	🔴 -0.3%
WinoGrande	60.8%	62.4%	🟢 +1.7%

Speed: 40.67 tokens/sec (base model: 40.59 tokens/sec)

MMLU by category

Usage

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "OliverSundaram/Llama-3.2-1B-MathCodeInstruct-5k"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="auto")

messages = [
    {"role": "system", "content": "Below is a math problem. Please solve it step by step."},
    {"role": "user", "content": "If a train travels 60 miles in 45 minutes, what is its speed in miles per hour?"},
]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
output = model.generate(inputs, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Limitations

Trained on a single epoch of a 5k-example subset — not intended to be a general-purpose assistant.
MMLU/ARC/HellaSwag/WinoGrande scores reflect a small 1B-parameter base model and should be read relative to the base model's own scores, not against much larger models.
No safety alignment or RLHF was applied beyond what the base Llama-3.2-1B already has.