A LoRA fine-tune of Qwen3-4B-Thinking-2507 for grade-school math word problems. Trained on a 2,000-example filtered subset of OpenR1-Math-220k on a single M1 Max 32GB Mac Studio using mlx-lm.

The fine-tune teaches the model to produce reliable #### <number> final answer formatting and improves both accuracy and output efficiency on GSM8K-style problems.

Results

Evaluated on the first 200 problems of the GSM8K test set, identical prompt and answer-extraction pipeline for both models.

Model	Accuracy	Avg output tokens
Qwen3-4B-Thinking-2507 (base, 4-bit MLX)	47.5% (95/200)	878
This model	69.0% (138/200)	775
Delta	+21.5 pp	-104 tokens

Note on the base score: the base model often produces correct reasoning without ending in the #### <number> convention, so a regex-based extractor under-counts its true accuracy. Both models ran through the exact same extraction pipeline, so the comparison is fair, but the base number is lower than published GSM8K results for Qwen3-4B-Thinking (~85%+ with format-aware extractors). Part of what this fine-tune does well is enforce the answer format reliably.

Training details

Base model: mlx-community/Qwen3-4B-Thinking-2507-4bit
Method: LoRA SFT
LoRA layers: 16 (top half of model)
Dataset: 2,000 examples sampled from open-r1/OpenR1-Math-220k, filtered to 100-2000 tokens per (problem + solution)
Train/valid split: 1,800 / 200
Batch size: 1
Max sequence length: 2,048
Iterations: 1,000
Learning rate: 1e-5
Final train loss: 0.522
Final val loss: 0.401
Hardware: Apple M1 Max, 32GB unified memory
Framework: mlx-lm
Training time: ~6 hours
Peak memory: ~15 GB

Intended use

Solving grade-school math word problems with explicit step-by-step reasoning and a clearly formatted final numerical answer.

Limitations

Evaluated only on GSM8K-style problems. No evaluation on competition math (MATH, AIME), coding, general chat, or non-English problems.
Trained with a single LoRA run, no hyperparameter search.
Inherits all limitations and biases of the base model.
4-bit quantization may cause minor degradation vs full-precision base.

Usage

With mlx-lm:

python

from mlx_lm import load, generate

model, tokenizer = load("morganlinton/qwen3-4b-thinking-gsm8k-sft")
prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Janet has 5 apples. She gives 2 to her brother and buys 3 more. How many apples does Janet have now?"}],
    tokenize=False,
    add_generation_prompt=True,
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=512, verbose=True))

Reproduction

Training pipeline: mlx-lm LoRA on filtered OpenR1-Math-220k. Full script and hyperparameters above. Adapter-only weights are also published at morganlinton/qwen3-4b-thinking-gsm8k-sft-lora for users who want to apply the LoRA to a different quantization of the base.

Acknowledgements

Base model: Qwen team at Alibaba
Training data: open-r1/OpenR1-Math-220k
Framework: Apple MLX