AxionLab-Co/

MiniAxion1.5-3M

MiniAxion1.5-3M is an ultra-compact (~2.7M parameters) GPT-style language model designed to investigate reasoning emergence at extreme small scale.

Unlike typical small models optimized for fluency, MiniAxion is explicitly trained to produce:

Structured reasoning traces Step-by-step thinking () Deterministic answer formatting

It operates primarily in Portuguese, making it a rare example of a non-English reasoning-first nano model.

Most models follow this trajectory:

Language → Knowledge → Reasoning

MiniAxion flips part of that:

Structure → Reasoning format → (still learning correctness)

The model demonstrates that reasoning structure can emerge independently of reasoning accuracy.

🧪 Evaluation Task Performance Task Accuracy Addition 10% Subtraction 10% Multiplication 0% Even/Odd 100% Comparison 5% Sequence Completion 0% Word Problems (Addition) 10% Word Problems (Subtraction) 0% Word Problems (Multiplication) 10% True/False 100% Chat/Greetings 100%

🧠 Reasoning Behavior Metrics Metric Score Thinking Rate 100% Step Format 100% Answer Completion 100%

✔ The model always thinks ✔ The model always structures reasoning ✔ The model always produces an answer

MiniAxion exhibits a clear dissociation:

✅ What it learned Reasoning format Step-by-step decomposition Logical task patterns (parity, boolean) ❌ What it did NOT learn Arithmetic correctness Numerical reasoning Multi-step computation

Reasoning ≠ Correctness

MiniAxion shows that:

Models can internalize thinking patterns Without actually learning how to solve problems

This makes it a strong candidate for studying:

Emergent reasoning Tiny Recursive Models (TRMs) Reasoning distillation

🏗️ Architecture Type: GPT-style Transformer Parameters: ~2.7M Objective: Next-token prediction Language: Portuguese (primary) Specialization: Structured reasoning traces

The model was trained with a reasoning-first approach:

Portuguese language grounding Structured reasoning data () Emphasis on: Deterministic formats Multi-step thinking Explicit reasoning tokens

🚫 No RLHF 🚫 No instruction tuning at scale 🚫 No large model distillation (yet)

⚠️ Limitations

Near-random performance in:

Addition

Subtraction

Multiplication

→ Indicates lack of numerical representation learning

Strong dependence on:

Prompt format

Token patterns

Seen reasoning templates

This model is just the beginning.

📈 Scaling

5M / 10M / 20M versions

Track emergence of correctness

🧪 Distillation

Inject reasoning from larger models

Improve accuracy without scaling params

🔁 Self-Play / Synthetic Data

Generate reasoning loops

Reinforce correct chains

🧩 Hybrid Reasoning

Combine symbolic + neural learning

Fix arithmetic weakness

🧾 Example Output

✔ Perfect reasoning structure ❌ Incorrect answer

MiniAxion1.5-3M proves something important:

Even a 2.7M model can learn to simulate thinking before it learns to actually think correctly.

Research on emergent reasoning

Tiny model experimentation (CPU-friendly)

Educational demos of:

Chain-of-Thought

Reasoning failure modes

Base model for:

Distillation

NRM experiments