A Metal Soul In My Hand — A feature-decoupled Transformer architecture with native interpretability.
reFlow factorizes the embedding matrix $E \in \mathbb{R}^{V \times d}$ into a Recipe Matrix $W_{recipe} \in \mathbb{R}^{V \times S}$ and a Signal Basis Matrix $W_{basis} \in \mathbb{R}^{S \times d}$, forcing the model to maintain a set of continuous, low-redundancy signal bases in latent space. The same factored product $W_{recipe} \times W_{basis}$ serves as both the input embedding and the output projection, forming an end-to-end signal-manifold computation loop without a separate LM head.
Convergence. At matched depth and scale (36 layers, ~515M parameters), reFlow-1-Big achieves a validation loss within ~1% of GPT-2-New (514M). Three scale points — Small (46.47M), reFlow-1 (463.67M), Big (515.06M) — confirm strict scaling law compliance (val loss: 3.55 → 3.01 → 2.92).
Emergent Interpretable Structure (pure language modeling objective, no auxiliary loss):
Recipe-space semantic algebra: king + woman − man → queen (rank #1), 3/3 tests passed
Natural sparsity: each token activates ~11% of signals (mean 117/1024), Gini coefficient 0.085
Causal traceability: single-signal ablation collapses target probability from 8.31% to 0.03%
Information crystallization boundary: semantic interventions are effective at L0–L12 but inert beyond L18
This downloads the OpenWebText corpus (~54 GB) and tokenizes it with the GPT-2 BPE tokenizer. Output: data/openwebtext/train.bin (~17 GB, ~9B tokens) and val.bin.
Training
All configurations are in config/. No CLI overrides — all hyperparameters must be set in the config file.
