A LoRA adapter for Llama-3.3-70B-Instruct, fine-tuned to solve mathematical problems — arithmetic word problems through algebra, geometry and combinatorics — and answer short coding questions.

Trained with Adaption Labs' AutoScientist for the AutoScientist Challenge (Math & Code category).

Result

Evaluation	Base	Adapted
Math category	28	72
In-distribution test set	52	48

Wins in a paired comparison, not accuracy percentages.

The rows disagree, which is worth explaining. On the narrow in-distribution set a judge slightly prefers the base model's phrasing. Across the wider category — including problems well outside the training distribution — the adapted model wins decisively. The mathematical substance generalised further than the answer style.

What produced the 12-point gain

An earlier version of this model scored 60–28 on the same category evaluation. The difference was a single filter in the training data.

v1 capped every solution at 18–75 words. In the upstream corpus, MATH-level solutions have a median length of 121–156 words, while grade-school word problems sit at 89–101. The cap therefore kept only the shortest, easiest examples from the hard sources — the model trained almost entirely on arithmetic and was then evaluated across the full difficulty range.

Setting the word budget per source (30–150 for algebra and geometry, 18–80 for word problems) raised solution p90 from 77 words to 133, and the category win rate from 60 to 72.

Usage

The adapter is stored unpacked and loads directly.

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE = "meta-llama/Llama-3.3-70B-Instruct"
ADAPTER = "flamiinngo/adaption_math_word_problems_solutions"

tokenizer = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(
    BASE, torch_dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

messages = [{"role": "user", "content":
    "Mrs Thompson has 7 Harry Potter books, 6 Twilight books and 5 Hunger Games "
    "books. Each series must stay together on the shelf. How many orderings are "
    "there?"}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt").to(model.device)

out = model.generate(inputs, max_new_tokens=400, do_sample=False)
print(tokenizer.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))

Hardware: the 70B base needs roughly 140 GB in bf16, or about 40 GB with 4-bit quantisation. The adapter is 3.3 GB.

Output style: brief worked steps, then the result stated explicitly as "The answer is X."

Note on the base model name. adapter_config.json records togethercomputer/Meta-Llama-3.3-70B-Instruct-Reference, the base as served during training. Same architecture — load against meta-llama/Llama-3.3-70B-Instruct.

Training

Parameter	Value
Base	`meta-llama/Llama-3.3-70B-Instruct`
Rank (`r`)	64
`lora_alpha`	128
Target modules	all-linear
Epochs	3
Peak learning rate	1e-4, cosine

Dataset

flamiinngo/math-code-qa-v2 — 5,297 rows (4,197 math, 1,100 code), every math answer ending in a result verified against the upstream expected_answer column. Derived from nvidia/OpenMathInstruct-2 and sahil2801/CodeAlpaca-20k, both CC-BY-4.0.

Also on Kaggle: model · dataset

Limitations

It can produce confident wrong reasoning. The training solutions are model-generated upstream; only their final answers were verified. Errors in algebraic reasoning exist in the data and this model reproduces that style of mistake. Check any result that matters.
Not a calculator. Fine-tuning improved the working, not arithmetic guarantees.
Scope is school through early-undergraduate. Not olympiad or research mathematics.
Code output is untested. The code training data was filtered for length, not executed. Treat generated code as a draft.
Win rate is not accuracy. It measures preference against one base model on one evaluation.
English only.

License

The adapter is a derivative of Llama-3.3-70B-Instruct and is subject to the Llama 3.3 Community License. The training data is CC-BY-4.0.

Acknowledgements

Adaption Labs — AutoScientist platform and the challenge
NVIDIA and sahil2801 — upstream open datasets
Meta — Llama 3.3 base model