Fine-tune of HuggingFaceTB/SmolLM3-3B-Base, trained in two stages — full fine-tuning followed by LoRA — to solve grade-school math word problems with clear, step-by-step reasoning.

Highlights

87.21% mean token accuracy on held-out validation data — up from 84.75% after the initial full fine-tune, and 82.4% at the very first checkpoint
Validation loss reduced by ~19% across the full pipeline (0.678 → 0.462), with training and validation loss tracking closely throughout every stage — no overfitting observed
Trained on the complete GSM8K dataset (both main and socratic reasoning styles) across two LoRA passes, on top of an initial full fine-tune
Available in this repo as GGUF with quantization (BF16 and Q8_0) for efficient usage on CPU and a smaller size. Safetensors version available here.

Full fine-tune: Training vs Validation Loss

LoRA fine-tune: Training vs Validation Loss

Training Details

Stage 1 — Full fine-tuning


Dataset	GSM8K (`main`), 1,000 random samples, 90/10 train/val split
Steps	450 (1 epoch)
Learning rate	5e-5, cosine schedule
Final validation loss / accuracy	0.569 / 84.75%

Stage 2 — LoRA fine-tuning


Method	LoRA, r=16, all-linear target modules
Dataset	Full GSM8K — both `main` and `socratic` reasoning styles
Steps	3,550 (combined across two passes)
Learning rate	5e-5, cosine schedule
Final validation loss / accuracy	0.462 / 87.21%

Run as two consecutive passes over the dataset, with the main/socratic split swapped between them so every problem was seen in both reasoning styles.

Languages Supported

The model supports conversational input and output in 9 languages: English, Arabic, French, Spanish, German, Italian, Portuguese, Chinese, and Russian.

Limitations

Fine-tuned primarily on GSM8K-style problems (single correct numeric answer, grade-school arithmetic/word problems) — performance on more complex, multi-part, or differently-structured math problems is untested. Occasional arithmetic slips on multi-step problems can still occur, consistent with known limitations of models at this scale.

Files

Al-Khwarizmi-3B-bf16.gguf — full precision, closest to the original model's accuracy
Al-Khwarizmi-3B-Q8_0.gguf — 8-bit quantized, smaller and faster with minimal accuracy loss

Note: Based on my testing, Q4_0 quantization noticeably degrades accuracy on this model. Lower bit-widths are likely affected similarly or more.

Usage

python

from llama_cpp import Llama

# Use Al-Khwarizmi-3B-bf16.gguf or Al-Khwarizmi-3B-Q8_0.gguf
llm = Llama.from_pretrained(
    repo_id="mzoelfakar/Al-Khwarizmi-3B-GGUF",
    filename="Al-Khwarizmi-3B-Q8_0.gguf",
    n_ctx=2048,
)

messages = [
    {"role": "system", "content": "You are a math tutor. Solve problems step by step."},
    {"role": "user", "content": "If a train travels 120 miles in 2 hours, what is its average speed?"}
]
response = llm.create_chat_completion(messages=messages, max_tokens=300, temperature=0.7)
print(response["choices"][0]["message"]["content"])

Recommended system prompt

For higher efficiency, use the following system prompt:

You are a careful math tutor. Before answering:
- Use only the numbers and quantities explicitly stated in the problem. Do not introduce, assume, or carry over any value that wasn't given.
- Compute each arithmetic operation one at a time, and verify each result before using it in the next step.
- If asked to recheck or redo a calculation, ignore your previous answer entirely and recompute from the stated numbers.
- Before finalizing your answer, check whether every quantity mentioned in the problem (fees, taxes, discounts, additions) has been included in the final result — not just the main calculation.
- When asked to redo or resolve a problem "based on" a previous correction, use that corrected value as the starting point. Do not revert to an earlier, uncorrected path.

Reply in the user's last used language.

You are Al-Khwarizmi, named after Muhammad ibn Musa al-Khwarizmi, the mathematician who lived in the 9th century and whose name is the direct origin of the word 'algorithm'.

Refrain from repeating unnecessary information.

NEVER help with any topics other than math like the weather, cooking, sports, etc.

Note on raw output formatting

Because this model was fine-tuned on GSM8K (including the socratic reasoning style), raw generations may contain training artifacts not meant for direct display:

<<...>> — calculator-style intermediate annotations
** — separator between a sub-question and its calculation in Socratic-style reasoning (not Markdown bold)
#### <answer> — marker preceding the final numeric answer
* — used as a multiplication sign (e.g. 8*9); if two or more appear in the same response, Markdown may pair them as emphasis delimiters, causing text between them to render in italic with the asterisks hidden

If you're piping output through a Markdown renderer or displaying it in a UI, you'll likely want to strip or reformat these first, since ** in particular can be misread as Markdown bold syntax if left unescaped.

Training Code

The full training code is available here.

Try it online

A live chat demo is available via Colab: Al-Khwarizmi-3B.ipynb

Credits

Fine-tuned by Mohamed Zoelfakar, as part of Hugging Face's smol-course, at Nvidia Studio Cairo.

License

Released under the Apache License 2.0.