KOLM-Alpha — Fully-Oscillatory Kuramoto Language Model
Delon Swartz — AI Researcher | Engineer
KOLM-Alpha is, to our knowledge, the first language model with no
transformer computation anywhere — both routing between tokens and
processing within them are performed by networks of coupled Kuramoto
oscillators. It is the successor to
KOLM-Hybrid-1,
which kept standard attention and made only the feed-forward slot
oscillatory. KOLM-Alpha replaces attention itself with
synchronization
routing.
Naming note: the name KOLM-Alpha previously labelled the hybrid twin
study, now published as KOLM-Hybrid-1. Under the project naming
registry (2026-07), KOLM-Alpha denotes this fully-oscillatory line
(working name "KOLM-True").
The mechanism: KuramotoRouter A
Each layer's routing is a competitive synchronization process, not static
attention:
- Softmax competition over the causal past — sharp selection, not
blurry averaging.
- RoPE on the routing scores — position is in the routing, not just
the moved content.
- Dynamic query modulation — routing weights are recomputed at every
settling step, modulated by the evolving oscillator state. This is
genuinely dynamic routing that static attention cannot express.
The routed value drives each token's oscillators, which settle via Kuramoto
dynamics under trained couplings, natural rotations, and (optionally)
frustrated (phase-offset) coupling. Normalization is SphereNorm. Every
block is an exact identity at initialization.
The result
Under the same controlled protocol as KOLM-Hybrid-1 (identical tokenizer,
data, order, budget, seed; 11.26M TinyStories tokens; single laptop):
| Model | Params | Val loss | Perplexity |
|---|
| KOLM-Alpha (Router A) | 17.06M | 2.3170 | 10.14 |
| KOLM-Hybrid-1 (attention + osc-FFN) | 16.92M | 2.6946 | 14.80 |
| Transformer twin (TMT) | 17.54M | 2.7188 | 15.16 |
KOLM-Alpha wins decisively — 0.38 nats below the hybrid, 0.40 below the
transformer twin — while doing zero transformer computation. It crossed
the hybrid's final score at 5M tokens (41% through training).
An honest negative precedes this win: Router v1 (sigmoid-gated,
position-blind, routing frozen across settle steps) lost at 3.0273. The
three diagnosed failures motivated Router A's softmax + RoPE + dynamic
modulation.
Scaling to 42M from scratch (the flagship)
KOLM-Alpha-42M ("W42") is the same architecture at 16 layers, d=384,
H=320, frustrated coupling, randomized settling depth K ~ U[1,4], trained
from scratch on 188M tokens (32k vocabulary, TinyStories + Simple English
Wikipedia):
- Val loss 1.7141 / perplexity 5.55 — from scratch, matching
function-preserving-transfer-warmed comparables (K-sweep K1 1.7162 /
K2 1.6848 / K4 1.7044).
- Chat, via staged SFT: broad chat mixture → single-turn discipline
(Alpaca). The staged order beats Alpaca-from-base by 0.35 nats
(alpaca-val 3.19 vs 3.55). Turn discipline is clean (answers once, stops
at
<|end|>); simple facts land ("The capital of France is Paris").
Entity binding and counting remain beyond 42M — capacity limits, not
recipe bugs.
Why it matters
The hybrid showed oscillators can process as well as an MLP. KOLM-Alpha
shows they can route better than attention — the whole layer is now
synchronization dynamics, with a settling-depth compute dial that adds a
memory-free axis of scale. Randomized-depth training makes that dial
extrapolate past its trained depth.
Files (weights to follow)
| File | Purpose |
|---|
kolm_true.py | full-oscillator model + trainer (Router A) |
kuramoto_torch.py | Kuramoto block (parity-tested to 2.2e-16 vs NumPy) |
chat_true.py | REPL / sampler with the /think K settling dial |
native_true_A.pt | KOLM-Alpha 17M weights (Router A) |
native_w42.pt | KOLM-Alpha-42M pretrain weights |
native_w42chat_alp.pt | 42M chat model (staged SFT) |
tiny32k.json | 32k tokenizer |
Roadmap
- KOLM-Beta-T (repo) —
transplanting pretrained transformer weights into this architecture to
reach billion-parameter capability without pretraining.
- Larger from-scratch KOLM-Alpha rungs exposing the settling dial as a
user-facing "think-harder" control.
Status & limitations
Research preview. Headline comparisons are single-seed; the 17M Router-A run
used a settling/lr protocol that differs slightly from the twin runs (a
matched rerun is queued before external citation). Corpora are easy and all
budgets are far below Chinchilla-optimal; factual recall at 42M is limited.
License
Apache-2.0.
Acknowledgements
Builds on the Kuramoto-oscillator formulation of AKOrN (Miyato et al.,
ICLR 2025), the frustrated-synchronization principle of FSN
(arXiv:2606.18694), and the TinyStories corpus (Eldan & Li, 2023).
