NanoMath: A 136M Parameter Math-Reasoning Language Model
🚀 Model Overview
NanoMath is a custom-built, GPT-style Large Language Model explicitly designed to solve mathematical problems using Chain-of-Thought (CoT) reasoning. Built entirely from scratch in PyTorch, the primary goal of this model is to demonstrate an end-to-end language modeling pipeline, helping to understand transformer architectures, training optimizations, and data processing bottlenecks.
🛠️ Model Details
- Architecture: Decoder-only Transformer (GPT-style) built from scratch.
- Parameters: 136 Million.
- Training Compute: Trained for 12 hours on a single NVIDIA T4 GPU via Kaggle.
- Tokenizer: Custom Byte-Pair Encoding (BPE) tokenizer trained using
sentencepiece to explicitly recognize mathematical operators and digits efficiently.
📚 Training Data
The model was trained on approximately 1 Billion tokens. The dataset is a combination of high-quality math datasets and procedural generations:
- Synthetic Basic Math: Hundreds of thousands of synthetic arithmetic problems generated via a custom Python script.
- GSM8K: Gold-standard grade school math word problems.
- NuminaMath: Large-scale math reasoning problems.
To teach the model step-by-step reasoning, the data enforces a strict <|thinking|> and <|answer|> formatting structure.
🧠 Performance and Capabilities
NanoMath was trained to "think before it speaks". It successfully generates intermediate reasoning steps and performs well on basic arithmetic tasks.
Successes:
- The model successfully executes multi-step algorithmic logic like partial products.
- It correctly utilizes long division from left to right.
⚠️ Limitations
Given its small scale (136M parameters) and heavily constrained training time (12 hours), the model exhibits standard LLM hallucinations on complex operations.
- Multi-Step Borrowing: It sometimes struggles to carry values perfectly across complex mathematical borrowing steps.
- Logic Routing: It can lose track of logical routing in word problems, leading to hallucinated results.
💻 How to Use
Because this is a custom model architecture built from scratch, you cannot use the standard HuggingFace transformers library out-of-the-box. Instead, download the weights and run inference using the codebase provided in the linked GitHub repository.
- Download the
model_weights.pth and token.model files from this repository.
- Clone the NanoMath GitHub repository and install the dependencies:
1pip install -r requirements.txt
2
- Place the downloaded
.pth and .model files into the build/ directory of the cloned repository.
- Run the interactive chat CLI:
