This is the SFT (instruction-tuned) version of PolyChromaticLM-1.0-base-0.6B, fine-tuned on mathematical problem-solving data with chain-of-thought reasoning in ChatML format.
The core innovation is PolyGLU (Polychromatic Gated Linear Unit) — a drop-in SwiGLU replacement that implements state-conditional activation routing. Each FFN neuron dynamically selects among K=4 activation functions (ReLU, Tanh, SiLU, GELU) via a differentiable Gumbel-Softmax mechanism.
Author: Daniel Nobrega (independent research)
Key SFT Results
Training loss: 1.77 → 0.91 (48.7% reduction over 1 epoch)
Routing entropy: 1.386 (maximum) throughout all 13,067 SFT steps — the PolyGLU routing architecture is fully robust to fine-tuning
MMLU-STEM improved by +3.14 pp after SFT, with large gains on quantitative subtasks (High School Statistics +20.84 pp, College Mathematics +11.00 pp)
Moderate forgetting on general benchmarks (mean -2.89 pp across 10 tasks) — 9/10 benchmarks remain above random
SFT training dynamics: loss curve, learning rate, and throughput
Loss curve detail
SFT loss curve from 1.77 to 0.91
Step
Loss
10
1.77
500
~1.10
5,000
~0.95
10,000
~0.90
13,067
0.91
Routing Entropy Stability
The most remarkable observation: routing entropy remained at exactly 1.386 (= ln(4) = maximum entropy for K=4) throughout all 13,067 SFT steps. This means:
Static routing preferences learned during pre-training were NOT disturbed by SFT
PolyGLU neurons maintained equal activation diversity across all 4 functions
The routing architecture is robust to fine-tuning — a critical validation of the design
SFT modifies what is computed, not how: the routing mechanism (which activation function each neuron uses) remains unchanged, while the model's weights adapt to produce chain-of-thought reasoning.
Context: Qwen3-0.6B-Base was trained on ~36T tokens (3,600x our budget). On the 6 tasks with published Qwen3 scores, our SFT model achieves 47-80% of Qwen3 performance. SFT narrows the gap on reasoning tasks like ARC-Challenge (71% of Qwen3, up from 66% pre-SFT).
Base vs SFT benchmark comparison
Forgetting Analysis
Per-benchmark delta: SFT minus Base
Pattern: Tasks requiring reasoning (ARC-Challenge +1.88, MMLU-STEM +3.14) improved, while tasks measuring text fluency (LAMBADA -8.34, SciQ -8.50) regressed. Mean regression of 2.89 pp is moderate and acceptable for math-focused SFT. 9/10 benchmarks remain above random.
GSM8K
GSM8K generation-based evaluation was not completed due to compute budget constraints. Without KV cache, autoregressive generation of 1,319 test examples required ~9+ hours of A100 GPU time. Indirect evidence of SFT effectiveness includes the converged training loss (0.91) and MMLU-STEM improvement (+3.14 pp with large gains on quantitative subtasks). See the full evaluation report for details.
This model was trained from scratch in pure PyTorch (no HuggingFace model wrappers). To load and use it:
python
1import torch
2from transformers import AutoTokenizer
34# Clone the training repo for model code5# git clone https://github.com/danielxmed/PolyGLU.git6from src.model.config import ModelConfig
7from src.model.model import load_checkpoint
89# Load model10config = ModelConfig(use_flash_attn=False)11model, step, tau = load_checkpoint("path/to/model.safetensors", config, device="cuda")12model.eval()1314# Tokenize (ChatML format for instruct model)15tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B-Base")16prompt ="<|im_start|>user\nWhat is 15% of 240?<|im_end|>\n<|im_start|>assistant\n"17input_ids = tokenizer(prompt, return_tensors="pt")["input_ids"].cuda()1819# Generate (greedy, no KV cache)20with torch.no_grad():21for _ inrange(200):22 logits = model(input_ids)23 next_token = logits[:,-1,:].argmax(dim=-1, keepdim=True)24 input_ids = torch.cat([input_ids, next_token], dim=1)25if next_token.item()== tokenizer.eos_token_id:26break2728print(tokenizer.decode(input_ids[0]))
Note: This model loads from the custom PyTorch checkpoint format. The load_checkpoint function in the PolyGLU repo handles both .pt and .safetensors formats. See the GitHub repo for full details.
Limitations
No GSM8K evaluation — generation-based evaluation was too expensive without KV cache (~9h for 1,319 examples). This is the most significant evaluation gap.
Math-only SFT — fine-tuned exclusively on math problems. General instruction-following capability is limited.
10B token pre-training budget — significantly less than comparable production models.
No KV cache — inference requires the full training codebase; generation is slow.
English only — trained exclusively on English-language data.
