An iso-token experiment run on top of a Qwen3-0.6B reproduction whose logits match the official Hugging Face weights exactly in CPU fp32. Same corpus, same eval code, same step count, one thing changed: the recipe.

A 17.94 percent improvement, step-matched.

Two corrections to how this was previously stated. First, 23.52 is the in-loop evaluation at step 18,000, not at the final step 18,150; train_imu1.py runs no post-training evaluation, so there is no final-step number for this arm. Comparing 23.52 against the baseline's final-step 28.65 is not step-matched, and an earlier version of this card did that. The table above uses the baseline's own step-18,000 reading of 28.66 instead.

Second, this is iso-token, not iso-FLOP. No FLOP artifact exists in this project and the 5 percent iso-FLOP gate in its own rules was never evaluated. The 1.00043 figure sometimes quoted as an iso-FLOP ratio is a parameter-count ratio, not a FLOP measurement. The repository README was corrected on 2026-08-05 and now labels it as such; three LaTeX sources under research/papers/qwen3-0.6b-study/ still present it as a FLOPs-per-token ratio, as do two c5_evidence.json files.

I am also withdrawing the claim that this narrows the gap to the released Qwen3-0.6B-Base from 2.14x to 1.76x. Both ratios are cross-slice: the baseline's 28.65 and the released model's 13.40 were measured on different 300,000-token validation slices, so neither is a like-for-like comparison.

Status: weights upload pending. Code, configs and logs: https://github.com/yashb98/BuildFromScratch

Note on metadata: the Hugging Face model tree lists this repository as a finetune of the baseline it points to. It is not one. Both are separate training runs of the same architecture on the same token budget, sharing a codebase and an eval harness. The base_model field is set so the two stay linked, because the comparison between them is the whole point of this repository.

IMU-1 is slower, and by more than previously stated. Throughput was 5,172 tokens per second against the baseline's 7,444, a 30.5 percent drop. The wall-clock penalty is the reciprocal of that, about 43.9 percent, giving roughly 63.9 hours against the baseline's 44.4. That 63.9 is derived rather than logged; it is corroborated by checkpoint mtimes, which imply 56.28 hours for the first 16,000 steps at 5,175 tokens per second. Peak memory 66.1 GB against 52.4 GB.

An "about 30 percent more wall-clock" figure previously in the repository README was the throughput drop misread as a time penalty, and it understated the cost. It was corrected on 2026-08-05: the README now reports NorMuon at -30.5 percent throughput (5,172 vs 7,444 tok/s final), which is +43.9 percent wall-clock (63.9 h vs 44.4 h).

NorMuon at lr 0.011, weight decay 0.1, beta1 0.95, beta2 0.95, on 224 two-dimensional matrices. AdamW at lr 0.006, betas 0.9 and 0.95, eps 1e-8, weight decay 0.0, on the remaining 198 one-dimensional and embedding parameters. Warmup-stable-decay schedule rather than cosine: linear warmup over 900 steps, then stable, then linear decay to zero over the final 20 percent. z-loss at 1e-4. Gradient clipping 1.0. Seed 0.

The warmup is 900 steps. Earlier material said 50. The command line is on disk at Qwen3-0.6B/builds/phase_b_driver.sh line 24 with W=900, and the logged learning rate at step 50 is 6.11e-04, which is exactly 0.011 times 50/900. The 50 was a transcription error in the write-up of this run. It is not an error everywhere: the 640-step NorMuon ablation cells legitimately used a cosine schedule with 50 warmup steps, and the ledger entry recording that is correct. The two values belong to different experiments.

This is a full-recipe result, not a single-component result. The bundle moves about seven things at once: NorMuon, value residuals, LayerNorm scaling, head gating, warmup-stable-decay in place of cosine, z-loss, and two-dimensional weight decay at 0.1 instead of 0.01. No individual component can be credited from this number, and the bundle breaks the one-variable rule the rest of the project follows.

There is also a difference between the arms that is not part of the intended bundle and should not be there: IMU-1 accumulates its cross-entropy in bf16, while the faithful baseline upcasts cross-entropy chunks to fp32. That is a difference in the measurement path, not only in the recipe.

NorMuon versus AdamW, three seeds per arm, at 42M tokens: -0.4743 BPB on wikitext, 95 percent CI 0.443 to 0.505. Then hold parameters fixed and sweep the budget.

On wikitext it converges and the project ledger records the verdict as null. On the code corpus it plateaus rather than converging, edging up between 168M and 420M. At 420M the wikitext gap is 0.0717, CI 0.0553 to 0.0881, which is significant and small. The 42M cells also ran both arms at two-dimensional weight decay 0.1, NorMuon's tuned value, while AdamW was tuned at 0.01, so the baseline was handicapped there. Learning rates were not re-tuned per horizon.

Anyone quoting -0.474 without this ladder is quoting a small-budget artifact. That includes earlier versions of this card.

wikitext-2-raw-v1 validation perplexity at sequence length 1024, stride 512, 200 overlapping windows.

The IMU-1 entry here and the 23.52 above both come from checkpoint_imu1_2tpp_step18000.pt.

Partial RoPE lost. Rotating 25 percent of head dimensions gave 38.08 against full RoPE's 37.01, about 2.9 percent worse, over a completed 18,150-step run. The 10 percent variant reads 69.63, but that is a step-4000 checkpoint of a run that died at roughly step 5,450 of 18,150, so it is undertrained relative to its peers and should not be lined up against them.

Reporting this because a recipe search that only publishes its winners is not a recipe search.

A practical note: partial_rotary_factor is not supported by the Qwen3 implementation in transformers 5.8.0. A partial-RoPE checkpoint loaded into stock Qwen3 would silently run full RoPE.

Every build carries a verify gate that collapses bit-for-bit back to the faithful baseline when its change is disabled, and the baseline's logits match the official Hugging Face weights with max absolute error 0.0 in CPU fp32. That rules out a large class of silent implementation bugs, which is the point of building the baseline first.

It does not rule out the rest. One run per arm with no confidence interval. A validation slice that is a sequential continuation of the training stream and was not decontaminated; document-disjoint splitting and 13-gram decontamination landed in commit 86e79f3 on 2026-06-16, after these runs finished. A bundle that moves seven variables. Parity checked only on CPU in fp32, never on GPU, never at long context.

No weights are published here yet. The IMU-1 checkpoint would not load into stock Hugging Face Qwen3 in any case: 423 of its keys carry the _orig_mod. prefix left by torch.compile, and its config carries use_value_residual, use_layernorm_scaling and use_head_gating, which stock Qwen3 does not recognise. 752,091,220 elements. It also carries only model, config and step as top-level keys, with no tok_seen and no training_recipe, so unlike the faithful baseline this checkpoint does not record the recipe that produced it. That recipe survives only in phase_b_driver.sh and the run log. Checkpoints are raw torch.save pickles, not safetensors, and no converter exists for Qwen3 in this project.

Single machine, single GPU, self-directed research. Iso-token at one scale with one seed per arm does not establish that the ordering holds at larger scale, and the NorMuon ladder above is direct evidence that at least one component of this bundle does not. The remaining gap to the released model is dominated by data scale, not architecture.

Part of the BuildFromScratch collection.

Verified baseline this is measured against: https://huggingface.co/bishnoiyash/Qwen3-0.6B-reproduction

Smaller verified reproduction: https://huggingface.co/bishnoiyash/SmolLM2-135M-reproduction

This model was pre-trained on FineWeb-Edu (sample-10BT), which is released under the Open Data Commons Attribution License (ODC-By) v1.0 and is additionally subject to CommonCrawl's Terms of Use. ODC-By section 4.3 requires this notice on any Produced Work, which a model trained on the database is:

The training loader (train_qwen3.py:151) streams the dataset without pinning a revision, so the exact snapshot used is not recoverable from the code.

Yash Bishnoi, Edinburgh, UK.

GitHub: https://github.com/yashb98 Portfolio: https://yashbishnoi.io LinkedIn: https://www.linkedin.com/in/yash-bishnoi/