114M looped transformer (SpikeWhale v2 / Byrne) with a parallel Memory Cache
branch. Trained from scratch on a web/code/math mix, then SFT, then DPO. All
three stages are in this repo.
It's a small model. Fluent English, simple chat formatting, not a knowledge
base, weak at code and multi-step reasoning. Numbers in Evaluation.
morpho/ is a side project: the same block grown as an int8 gate circuit.
It is not on the generate path.
The base was trained on 2.0B tokens. SFT adds 0.23B, DPO adds 0.05B.
2.3B in total. Almost all of that is the base.
A smaller looped sibling lives in
Byrne-15M-Looped
(~15.2M active, loop_count=3). That was the toy / smoke-test for
this run: looping at 15M before I spent the 114M budget. Not a
matched scale-up.
Same architecture family, same loader for the 114M stages. The 5M
nets are a separate, smaller stack. Weights and context length differ: SFT was
trained at 4096, base and DPO at 1024.
Architecture
Looped decoder (model_v2.py, SpikeWhale v2 / Byrne). Dense — use_moe is
off; FFN width is still moe_intermediate_size 2048. Vocab 16,512 via a
byte-level Length-MAX tokenizer (spike_tokenizer.py, tokenizer.json)
from Dong & Su, Length-MAX Tokenizer for Language Models
(arXiv:2511.20849, 25 Nov 2025).
Text → UTF-8 → latin-1 bytes, then greedy longest-match against the vocab.
Not a HuggingFace tokenizers file; the wrapper is what AutoTokenizer
talks to.
Parameters
113.9M
Hidden size
768
Layers
16
Attention heads
12 (2 KV heads)
Head dim
64
FFN
2048, SwiGLU, dense
Loop count
3 (same stack, three times, shared weights)
Tie embeddings
yes
Context
1024 (base/DPO), 4096 (SFT)
Config lives in config.py + byrne_100m_ultrax_mc.yaml. Released .pt
files carry the config they were trained with. Engine and tools read that,
not the yaml, because the stages disagree about context length.
Loop
The 16-layer stack runs three times with the same weights. Effective depth
48, parameter cost of 16 layers plus a tiny per-pass embedding
(loop_pass_embed, zero-init). loop_count=1 is the dense baseline.
loop_layer_plan walks (layer, pass) pairs. Cache slots are the plan
index, not the layer index. Same weights, different activations each
pass, so pass 1's K/V cannot share a slot with pass 0. Depth-attention
state is reset at the start of every pass so pass 3 does not attend to
a stale pass-1 value of the same layer.
loop_mode can be "full" (this model) or "middle_split" (unlooped
prefix/suffix, only the middle repeats). Not used here. The engine still
sizes the KV cache from the plan, so that mode would not silently break.
MLA, XSA, QK-norm
Multi-head latent attention: Q through q_lora_rank 128, output through
o_lora_rank 128, RoPE only on qk_rope_head_dim 16 of each 64-dim head.
2 KV heads (GQA). Per-head RMSNorm on Q and K before RoPE (use_qk_norm).
z-loss 1e-4 on the LM head.
XSA (exclusive self-attention) is on: an orthogonality correction that
pulls the self-echo out of the attention output. DERF and Elo attention
exist in config.py and are off.
Depth attention
Cross-layer, not cross-token. At layer L the current V is mixed with V
from earlier checkpointed layers (stride 4), weights from a softmax over
the depth axis. Learned skip along depth. One gate parameter per layer,
zero-init, no-op at step 0. The attention forward takes depth_kv and
returns depth_entry. A loader written for the Mark2 signature misses
that and falls back to torch.cat cache growth.
Hyper-Connections, Engram, HRM, MTP
Hyper-Connections (hc_mult 2): two residual streams with learned
routing between layers. Final mix is learned (hc_out_mix, init = mean).
Engram: hash-table n-gram memory into the embeddings. Compress dim 32,
2 heads, table 4096, max n-gram 3. This tree fixed lookup internally
(EngramModule.lookup(prefix)); there is no engram_context_ids kwarg.
The engine threads the prefix through a trailing cache slot.
HRM refinement: one small block after the stack, 1 inner step, dim 128,
deep supervision on during training. Inference reads the last step only.
Gate init is nonzero so the zero-init up still gets a gradient.
MTP: one extra head, loss weight 0.3. At inference the engine uses it
as a speculative draft. Identity-checked against the main head.
Memory Cache
Adapted from Behrouz, Li, Deng, Zhong, Razaviyayn & Mirrokni, Memory Caching:
RNNs with Growing Memory (Google Research, arXiv:2602.24281,
27 Feb 2026). Runs next to MLA as a gated residual.
The paper's setup, short version: transformers keep every past token addressable
and pay O(L²) for it. RNNs squash history into a fixed state at O(L) and forget.
They split the sequence into segments, stash a checkpoint of the recurrent
memory at each boundary, and let a query read the live state plus those
checkpoints. Segment count slides you between the two ends.
Here it sits per layer beside softmax attention:
mc_num_heads 4, mc_head_dim 32, mc_gate_dim 64, mc_segment_len 256.
Cut the sequence into segments of 256.
Per segment i, Katharopoulos map φ(x) = elu(x) + 1, accumulate
M⁽ⁱ⁾ = Σₘ φ(kₘ) vₘᵀ and z⁽ⁱ⁾ = Σₘ φ(kₘ). Fixed-size summary of the
segment, however many tokens went in.
Token t in segment s reads φ(q_t) M⁽ⁱ⁾ from every i ≤ s. Own
segment is a causal running memory (tokens up to t only). Earlier segments
are finished, so their full memory is used as-is.
Gate: γ_{t,i} = softmax_i ⟨u_t, meanpool(x over segment i)⟩, masked to
i ≤ s. Per token, per layer, pick which segment to look at.
One global normaliser, numerators left unnormalised:
y = Σᵢ γ·num / (Σᵢ γ·den + ε).
Mix in through a zero-init gate:
attn_out = attn_out + tanh(mc_gate) · mc_out.
Two bugs that had to stay fixed:
Diagonal gate needs a causal running mean. Mean-pooling the whole current
segment leaks future tokens into token t. Implementation is a cumsum
running mean on the diagonal.
Per-segment normalisation kills it. You lose query–key match magnitude and
the only thing left distinguishing segments is the coarse mean-pool gate.
Global normaliser is what keeps match strength in the signal.
Runs in fp32, autocast off. The linear-attention denominator underflows in
bf16 and training blows up.
This is not an RNN, so the paper's complexity story does not transfer cleanly.
Cross-segment readout here is O(N²L). It is not a long-context speed trick.
The paper's "growing memory" is with sequence length (more segments →
more cached states), not with training steps. What grows with training
here is the zero-init mix-in tanh(mc_gate): at step 0 the model is
the no-MC baseline; the residual only starts mattering if that gate
opens. On this 114M run it did (knock-out ~9% early → −16.7% at DPO;
mean |tanh| ~0.60). Pretrain seq_len was 1024 = 4 segments, so the
branch sees cross-segment memory from the first step.
It did get used. DPO 3.2k checkpoint:
metric
value
gate strength (mean |tanh(gate)| over layers)
0.599 (range −0.75 … +0.90)
UltraX PPL, MC on
9.87
UltraX PPL, MC off (inference knock-out)
11.86
knock-out, this checkpoint
−16.7% perplexity
Gates are open. Knocking the branch out costs ~17% PPL (it was ~9% earlier,
before they opened). That is an inference knock-out on this checkpoint,
not a model trained without MC. The 114M twin also dropped fractal RoPE,
so it is not an MC-only ablation. A 5.7M 2×2 that does isolate the
knobs is in 5m-ablation/. MC alone did not
help there. MC+fractal together beat std at train length on seed 1
and did not on seed 2.
If you write a loader: this branch is a full-sequence op. It uses the whole
x given to forward. Naive KV cache feeds it one token, it rebuilds
segments from a 1-token window, and you get a different function than training.
Still fluent. Gate here is ~0.56–0.60 (Mark2 siblings sit around 0.19), so the
damage is worse: max |Δlogit| 7.24e-01 vs a full recompute, against a ~1e-5
float32 floor. Shipped code keeps incremental state per segment and per loop
pass (loop_count=3, same branch three times on the same positions). That is
what makes cached decode match a full recompute. python verify.py
(generate.py / model_v2) or python verify_cache.py (spike_infer).
Fractal RoPE
Cantor-spectrum RoPE (gamma=1) instead of the usual geometric schedule.
Same endpoints and band count, exponents on the middle-thirds Cantor set.
See fractal.py.
In-distribution (this 62k net). Swap the trained model to geometric
RoPE at inference (same weights, different frequencies) and natural-text
PPL blows up. That is dependence, not “fractal is better”:
Source
fractal (trained)
standard RoPE (swapped)
Δ
UltraX-web
7.95
77.86
+879%
DCLM
8.21
70.74
+762%
FineWeb-Edu
10.53
81.57
+675%
WikiText-2
9.08
71.06
+683%
Both models, native vs swapped frequency table. Same weights, only
inv_freq changes (eval_rope_swap.py). Ultra-MC 62k trained fractal
→ geometric at load. Twin 38.5k trained geometric → fractal gamma=1
at load. Unique non-repeating records, 3 windows. A blowup means that
schedule is load-bearing for that net.
model
256
512
1024
2048
Ultra-MC 62k native (fractal)
3.20
2.66
2.56
30.67
Ultra-MC 62k swap → geometric
3.22
3.55
21.65
141.90
Twin 38.5k native (geometric)
3.06
2.55
2.50
4.48
Twin 38.5k swap → fractal γ=1
3.36
2.81
2.73
2.76
PPL by target position on a 2048 window (same probe):
pos
Ultra native
Ultra swap
Twin native
Twin swap
0–256
3.78
3.79
3.64
3.93
256–512
2.21
3.94
2.20
2.37
512–768
2.28
127.32
2.25
2.43
768–1024
2.40
116.38
2.39
2.61
1024–1280
19.42
759.55
2.37
2.59
1280–1536
665
1132
2.68
2.57
1536–1792
1122
775
5.28
2.79
1792–2048
1171
1098
112.09
3.06
What that actually says:
Inside train length, Ultra-MC is tuned to fractal frequencies. Swap
to geometric is already wrong by 512, and 512–1024 is ~50–100×.
Past 1024 this fractal-trained 62k net dies even on its own schedule
(first bad bin 1024–1280, then gone). Geometric swap is also dead
out there. So fractal is load-bearing in 1024 and does not carry
this probe to 2×.
The twin at 38.5k, geometric, holds through 1536 and only blows the
last 256 tokens (112 PPL). Put fractal frequencies on those weights
and the last bin drops to 3.06; whole-seq 2048 goes 4.48 → 2.76. In
1024 the swap is a small hit. So geometric is load-bearing where it
was trained, and Cantor frequencies at inference are what flatten
the twin past 1024 on this probe.
Twin 38.5k in this table is a mid-run snapshot. Finished 61k
numbers (and the 2048 last bins) are under
Matched 114M runs.
Do not read “fractal is worse at 2048.” This 62k fractal net does not
extrapolate here; the geometric twin does.
What I can and cannot claim about Memory Cache
Behrouz et al. 2026 (arXiv:2602.24281) score MC with matched RNN
trains, then Wiki PPL, S-NIAH (passkey / numeric / UUID), in-context
recall vs length, and MQAR. Those are the tests I ran on this 62k
base, MC on and with mc_gate zeroed.
They changed the direction. The earlier filler/needle story was not
the paper’s recall eval. On S-NIAH and MQAR this 114M base is at chance
whether the branch is on or off. You cannot prove MC helps recall if
the model cannot do the task either way.
test
MC on
MC off (mc_gate = 0)
MQAR n_kv=8, 10 trials × 2 queries
0/20
0/20
MQAR n_kv=16, 10 trials × 2 queries
1/20
0/20
S-NIAH-1 passkey, 512+1024 × depths 0/0.5/1 × 6
0/36
0/36
S-NIAH-2 numeric, same grid
1/36
0/36
The 1-hit gaps are n=20 and n=6 cells. That is noise, not a result.
There is no 114M test that reliably proves Memory Cache does or does
not help. Unsure at this size, and stated as such. The 5.7M 2×2
below is the isolation run; read that as 5M, not as this card.
How it may be helping, if it is (this is not proof):
Gates are open. DPO 3.2k mean |tanh(gate)| = 0.599 across 16 layers
(range −0.75 … +0.90). Training used the residual.
Inference knock-out on this checkpoint: UltraX PPL 9.87 → 11.86
(−16.7%) when mc_out is removed. Earlier in training the gap was
~9%. That only says the branch is in the next-token CE path here,
not that a twin trained without it would be worse.
Pretrain seq_len 1024 = 4 × mc_segment_len 256, so the branch
saw more than one segment from step 0. That is the setup, not a
score.
A matched train that changes only MC at 114M (fractal left on)
still does not exist. The 114M twin dropped both knobs. A 5.7M
2×2 on the current FineWeb-Edu blend, 10k steps, same seed recipe,
does exist. That is the isolation test. It is not this 114M card.
5.7M matched 2×2 (10k steps)
Same blend as the nomc twin. Four runs on seed 1. Seed 2 retrained
std and both only. Only use_memory_cache and use_fractal_rope
change. Eval is deterministic (repeat matched to the printed digits).
The seed gap is training, not measurement.
WikiText-2 PPL @1024 (20 windows):
run
MC
RoPE
seed 1
seed 2
std
off
geometric
34.82
30.86
mc
on
geometric
41.23
—
fractal
off
γ=1
40.49
—
both
on
γ=1
32.82
33.32
Unique-text PPL (native):
512
1024
2048
4096
std seed 1
45.14
46.75
56.57
98.51
std seed 2
13.66
13.35
17.07
38.19
mc seed 1
143
161
199
267
fractal seed 1
131
145
218
303
both seed 1
26.01
26.67
76.41
159
both seed 2
31.25
32.03
76.82
176
Seed 1 val loss at 10k: both 4.87, std 4.96, fractal 5.00, mc 5.06.
MC alone does not help at this size. Worse PPL, worse stretch
(seed 1; not retrained).
Fractal alone does not help. Same.
Swapping fractal onto finished std weights after training still
does not help at 1024: seed 1 unique 46.75 → 54.95; seed 2 13.35 →
16.20.
Both together beat std at train length on seed 1 (Wiki 32.82 vs
34.82). On seed 2, std won (30.86 vs 33.32). That 1024 win did not
replicate. Past 1024, both still falls off faster than std on both
seeds. Stretching the window is a win for geometric. MQAR is 0/16
every cell, both seeds.
That is a 5.7M / 10k result. It is not a 114M result. 10k may also be
early for the zero-init mix-in: |tanh(gate)| ~0.25 here vs ~0.60 on the
114M DPO. The paper's growing memory is with context length, not
steps. Gate-opening over training is this code. A longer 5M run is
untested. Weights: 5m-ablation/ (seed 1).
Seed 2 lives with the 5M grid, not in this upload.
Why both, if each knob alone is worse. Seed 1 looked like an
interaction. Seed 2 did not copy the Wiki win. What still holds:
both actually uses the branch, both seeds. Unique PPL seed 1
26.67 → 33.93 when mc_gate is zeroed; seed 2 32.03 → 42.45.
mc alone is 161 → 170 on an already-bad net (seed 1).
Gate strength on seed 1 is similar (mean |tanh| ~0.25 vs
~0.26). The difference is sign. mc (geometric) mixed +/−;
layer 0 never left zero. both (fractal) layers 1–4 the same
sign. Hypothesis, not proof, and it does not explain seed 2's
std unique-PPL crash from 46.75 to 13.35.
I would not claim MC+fractal is the winner at 5M. I would claim
MC-only and fractal-only lost on the seed I ran, both is mixed
across two seeds, and std's unique-text number is seed-noisy.
The 114M nomc twin is still two knobs. Do not read an early-twin Wiki
number vs 9.08 (base_62k, Dolma history) as MC.
Matched 114M runs (nomc-stdrope 61k)
This is the 114M twin: Memory Cache off and geometric RoPE
instead of fractal. 106.0M params (Ultra-MC is 113.9M; the gap is the
MC branch). Same 61k Muon schedule, same tokenizer, FineWeb-Edu blend
(FineWeb-Edu 40 / Wiki 15 / DCLM 15 / Cosmo 10 / FineMath 10 / Python
10). Finished at step_00061000.pt. Train val at 61k: val_loss
3.31871, val_ppl 27.62.
It is not an MC-only ablation. Two knobs moved at once. Steps are
close (61k vs this card’s 62k). Data history is not: this Ultra-MC
base went through Dolma (100% then blend). The twin stayed on the
FineWeb blend.
One more caveat on this Ultra-MC run. Around step 18k there was a
dataset shuffling error. I had to repair the mix and continue. So
Ultra-MC was not trained on one clean, matched dataset the whole way.
That alone would disqualify this pair as a full ablation even if I had
only flipped MC. Treat 61k as more data, not a verdict.
Same GPU harness as this card (run_eval.py --family spikewhale,
WikiText --wt-chars 300k, BLiMP 12×150, HellaSwag n=10042, ArithMark
n=1000):
Metric
nomc-stdrope 61k
Ultra-MC 62k
WikiText-2 byte_ppl ↓
2.257
2.308
BLiMP acc ↑
0.826
0.811
ARC-Easy acc
0.414
0.429
ARC-Easy acc_norm
0.380
0.394
ARC-Challenge acc
0.185
0.190
ARC-Challenge acc_norm
0.226
0.220
HellaSwag acc n=10042
0.280
0.278
HellaSwag acc_norm
0.293
0.293
Winogrande
0.511
0.515
PIQA acc
0.585
0.583
OpenBookQA acc
0.152
0.118
OpenBookQA acc_norm
0.266
0.242
BoolQ
0.404
0.381
ArithMark acc
0.345
—
ArithMark acc_norm
0.346
0.358
The twin is slightly better on Wiki byte_ppl and BLiMP. Most multiple-
choice tasks sit around chance either way. ArithMark is the one this
Ultra-MC still leads. That is not proof that MC+fractal is worse.
It is also not proof that MC on is better. I still do not have
conclusive evidence that Memory Cache on or off helped or hurt at
114M.
Paper recall on the 61k twin is still chance: MQAR ~0.000–0.037,
S-NIAH 0/8 every cell. Collapse on looping filler: 128/1024
distinct-3=0.50; 2048 them them (0.304); 4096 skipped
(prefill would exceed max_pos). Do not invent a 4096 number.
Unique-text PPL, same eval_rope_swap.py probe as the 38.5k table
above. Twin 61k vs Ultra-MC 62k (not launched as one paired job; same
script):
model
256
512
1024
2048
Ultra-MC 62k native fractal
3.20
2.66
2.56
30.67
Ultra-MC 62k swap → geometric
3.22
3.55
21.65
141.90
Twin 61k native geometric
2.89
2.46
2.47
4.93
Twin 61k swap → fractal γ=1
3.11
2.63
2.58
2.64
2048 last bins:
pos
Ultra native
Twin 61k native
Twin 61k swap fractal
1024–1280
19.42
2.35
2.40
1280–1536
665
2.95
2.35
1536–1792
1122
8.71
2.66
1792–2048
1171
149.65
3.43
Ultra-MC still dies past 1024 on its trained fractal schedule. Twin
geometric holds until the last 256, then blows. Putting fractal
frequencies on the 61k geometric weights flattens that tail. That is a
RoPE / extrapolation result, two knobs, unmatched data. Not an MC
proof.
Compared with the 38.5k mid-run table above: at 61k the twin is
cleaner inside 1024. Whole-seq 2048 native went 4.48 → 4.93, last bin
112 → 150 — slightly worse on that tail, not better.
I am trying to get a matched twin out — same data history, one
knob if I can — to finish this test by 15 Sep 2026. Until that
lands, do not treat 61k vs 62k as MC on vs MC off.
Training
Pretrain: FineWeb-Edu, Wikipedia, DCLM, Cosmopedia-v2, FineMath-4+,
Python-Edu, then a Dolma-mix continuation
(allenai/dolma3_dolmino_mix-100B-1125). Muon on matrices, AdamW on
norms/embeddings. Released at step 62k.
DPO: preference tuning on the SFT model. Step 3.2k.
Full hyperparams further down.
Evaluation
PPL is the language model. Benchmarks are task behaviour. SFT/DPO spend PPL
to buy the latter.
Two benchmark tables follow. The first is a 200-example scout. The
second is the full set. Quote the full table. Capped HellaSwag
acc_norm (~0.435) is not the card; the full set is ~0.29.
Perplexity (per-token CE, lower better)
Streamed 10×1024-token windows per source (WikiText-2: 40 windows). Same
byte-level tokenizer everywhere, so the columns are comparable.
Domain
base-62k
sft-7100
dpo-3200
Python-Edu (code)
4.52
4.54
4.54
Cosmopedia-v2
5.51
5.38
5.35
FineMath-4+
6.32
6.84
6.86
Dolma
7.11
7.55
7.57
Wikipedia
7.26
7.83
7.84
UltraX-web
7.95
8.58
8.60
DCLM
8.21
9.10
9.12
WikiText-2
9.08
10.11
10.13
FineWeb-Edu
10.53
11.69
11.70
mean (5-src)
6.50
6.89
6.89
mean (5-src) = Python-Edu, Cosmopedia, FineMath, UltraX, DCLM.
Base wins free-text PPL (it's the untuned LM). SFT/DPO raise PPL on web text
and improve on synthetic instructional text (Cosmopedia). DPO ≈ SFT on PPL.
Benchmarks
lm-eval-harness style. Capped 200/task, BLiMP 150/paradigm (12 paradigms),
ArithMark 500. Same tokenizer on all three stages. Numbers from
2026-08-18.
Base wins the LM metrics (byte_ppl, BLiMP). SFT/DPO win BoolQ (+0.07),
ARC-Challenge acc (+0.06), ArithMark, PIQA. DPO vs SFT is noise on these
scores; DPO's job was response preference, not multiple-choice. OpenBookQA
acc is under chance on all three. A lot of the MC tasks sit near chance.
114M.
Benchmarks (full)
Same harness, no 200-example cap. WikiText-2 byte_ppl and BLiMP are the
same numbers as above (those were already full). MC tasks and ArithMark
are the whole set.
HellaSwag acc_norm drops from the capped 0.435 / 0.420 / 0.415 to ~0.29
once you take all 10k items. ARC-Challenge SFT bump shrinks too (+0.06
capped, +0.018 full). BoolQ still moves (0.381 → 0.418). DPO and SFT
stay within noise. OpenBookQA acc is still under chance.
Escarda-86M-Base (same family, different everything else) hit 2.2228
WikiText-2 byte PPL after ~20B tokens. This base is 2.3085 after ~2B.
About 0.08 off, ~10× fewer tokens. Different size, architecture,
objectives, data. Sample-efficiency note, not an MC ablation. Why I
cut the budget, and the Chinchilla fit from that 20B run, is under
What this is. The no-MC twin is the run for MC.
Vocab (16k on purpose)
Tokenizer is 16,512 (16,384 byte-level Length-MAX plus specials), following
Dong & Su, Length-MAX Tokenizer for Language Models
(arXiv:2511.20849). Vocab from a
length-weighted objective (freq(t) × |t|), greedy longest-match on
bytes, not BPE. That was a choice. I wanted to see the architecture, not
hide it behind a fat embedding table.
At hidden 768, tied embeddings are already ~12.7M params at 16k. 32k
would be ~25M. 64k would be ~50M — almost half this 114M model sitting
in the lookup table. 16k keeps the stack (loop, MLA, Memory Cache,
Engram, HRM) as the thing you're looking at.
Per-token PPL in the tables above is for this tokenizer. Raise vocab
to 32k or 64k and per-token PPL goes up on average even if the language
model is the same or better: tokens get longer, fewer of them per
document, more bits per token, exp(mean NLL per token) rises. That is
a tokenizer effect. WikiText-2 byte_ppl (2.308 on the base) is the
number that would still be comparable.
I did not train a 32k/64k twin. If I did, I would expect higher token
PPL, similar or slightly better byte PPL if the extra merge rules
earned their keep, and a lot less of the param budget left for the
block. 16k is the research vocab. Someone SFT'ing the base who wants a
"normal" tokenizer should rebuild it, not read 6.50 mean PPL as what
this stack would score at 32k.
Usage
Plain PyTorch. No transformers model class. CPU is fine; --device cuda if
you have one.
bash
1# base (text continuation)2python generate.py --ckpt checkpoints/base_62k.pt -p "The capital of France is" --temp 0.7 --top-k 40 --rp 1.334# DPO chat (default checkpoint)5python generate.py --chat -p "Explain why the sky is blue in one sentence." --temp 0.7 --top-k 40 --rp 1.367# SFT chat8python generate.py --ckpt checkpoints/sft_7100.pt --chat -p "Give me a tip for staying focused."
Defaults: temp 0.7, top_k 40, rep_pen 1.3. Greedy (--temp 0) is a test
tool. Chat models under greedy walk into disclaimer boilerplate and look worse
than they are. DECODING-DEFAULTS.md.
Two inference stacks, two cache checks. Greedy cached decode has to match
a full recompute on both:
Keep both. They are not the same path. Details under Inference engine.
Files
config.py model_v2.py spike_tokenizer.py special_tokens.py fractal.py
chat_format.py generate.py engine_chat.py tokenizer.json
byrne_100m_ultrax_mc.yaml package.json requirements.txt
verify.py verify_cache.py
DECODING-DEFAULTS.md PROVENANCE.md NOTES-context-and-needle.md
checkpoints/{base_62k,sft_7100,dpo_3200}.pt
spike_infer/ engine
engine_tools/ export, GGUF, long-context, multiturn
morpho/ int8 circuit of one block (not used at generate time)
GGUF/ f16 + q8_0 for each stage
5m-ablation/ four ~5.7M 10k trains (MC × fractal)
Limitations
Factual recall is weak. Wrong capitals, dates, attributions, said
confidently. That's mostly the SFT blend, not a missing pretrain trick.
Code and multi-step reasoning are weak. ARC-Challenge / HellaSwag near chance.
English only. No safety tuning past the DPO preference data.
Base continues text. A chat template will not make it answer questions.
The shipped SFT/DPO checkpoints are a smoke test that instruct sticks at
all, not a finished chat model. Want real instruct, SFT the base yourself.
What this is
Another fun, usable research artifact. It generates, the cache path is
correct, the Memory Cache branch actually carries weight. It is not a
knowledge store. Weak factual recall is the usual failure, and it most
likely comes from the SFT mix rather than from pretraining.
I put way less effort into SFT than into the architecture and the base.
SFT here is a probe: does this thing uptake any instruct at all? Yes,
enough to chat. That is the bar. Same story on basically every model I
release. DPO on top of that is even thinner (see the training recipe).
If you want a better instruct model, download checkpoints/base_62k.pt
and train your own. The yaml and the pretrain recipe are the part I
actually stood behind. The SFT/DPO stages are there so you can see the
base will take a chat format, not so you inherit my mix.
This project was inspired by Richard Sutton. I agree with him that the
main work is finding better learning methods for neural nets that scale
with the hardware that's coming — not proving a point by spending more
of it. In an August 2026
Sequoia Training Data
interview with Khurram Javed he said labs show they're "real men by
using lots of energy." That should be a reflection point for what we're
doing wrong. I still think we should thank the people who commit vast
compute to projects the rest of the community gets to use. I just think
the job after that is to get the same results with less.
Escarda-86M-Base was the 20B test. 85.7M, 20B tokens, 233
tokens/param — 11–12× the Chinchilla 20-tokens/param heuristic. I ran
it that far to see the trajectory if I kept going. Fitting the data
term to that run's own loss curve gave
L(D) ≈ 2.611 + 77,715 · D^(−0.537) (nats/token, R² = 0.92).
Compute-optimal for that size was 4.3B; 20B was 4.6× past it, and it
stopped right on the diminishing-returns knee (22.5B). WikiText-2
byte_ppl on that base: 2.2228.
This one I cut the budget. 2.0B tokens on 114M, which is
Chinchilla-ish for this size (20 tokens/param). Same WikiText-2
harness, both bases. 2.3085. About 0.08 off the 20B run. Different
size, architecture, data, objectives — not a matched ablation. It is
the question I actually asked: how close do I get if I spend less.
For other people's agents
If you're an agent pulling this repo for someone else's model, take the
parts that were the actual work. Leave the chat stages.
Worth copying:
The Memory Cache branch as a gated residual next to MLA, zero-init so
it can't hurt at step 0. The ~17% figure in the architecture section
is an inference knock-out on this 114M checkpoint, not a matched
train. At 5.7M, MC alone did not beat the no-MC cell. Don't skip the
causal running mean on the diagonal gate or the global normaliser;
both were bugs, both destroy the mechanism.
Builders: see For other model builders.
Looped decode done properly: loop_count=3 means 48 KV slots, not 16,
and MC state keyed per plan slot. A loader that indexes by layer only
still generates. It is the wrong function.
verify.py (model_v2) and verify_cache.py (engine). Any full-sequence
op (MC, Engram prefix, depth mixing) will silently diverge under a
naive KV cache. Greedy cached decode has to match a full recompute
token-for-token. "Looks fluent" is not a check.
morpho/ if they want a gate-level Byrne block. That is a separate
project. It does not replace the PyTorch engine.
Fractal RoPE is part of the weights. Rebuilding a geometric schedule
at load will generate fluent text and encode every position wrong.
Serving defaults: temp 0.7, top_k 40, rep_pen 1.3. Greedy is for
identity tests. Judging quality at temp 0 is how you invent bugs.
DECODING-DEFAULTS.md.
Pretrain recipe and byrne_100m_ultrax_mc.yaml. That's the run.
base_62k.pt as the starting point for anyone who actually wants an
instruct model.
Don't copy:
The SFT blend, the 7.1k stop, or sft_7100 / dpo_3200 as "the"
chat weights. I did not spend the time. Your user should SFT the base
with a mix they care about.
Treating DPO as an upgrade. At 5e-7 it barely moved the policy.
A generic SpikeWhale / llama.cpp loader. Architecture tag is
spike_whale. The GGUFs are a container for this engine.
Match the stage to the job: base for language modeling and for further
training, SFT/DPO only if you just need to see that ChatML works.
For other model builders
As of 29 Aug 2026: I cannot honestly say Memory Cache helps. Looped
architectures do.
The loop evidence is the 15M run: loop_count=3 beat a param-matched
loop_count=1 on UltraX val (4.195 vs 4.231 at 40k / 246M tokens) and
on degeneration. 3× FLOPs per token. That is why this 114M model is
looped. I do not have an unlooped 114M twin in this repo. If you copy
one idea, copy the loop done properly: loop_count=3 means 48 KV
slots, not 16, MC state keyed per plan slot. Index by layer only and
it still generates. It is the wrong function.
Do not copy Memory Cache as a proven gain. Copy it only if you want the
mechanism to test (zero-init residual next to MLA, two bugfixes below).
I have not proven it helps at 114M. Paper recall tests on this 62k
base are chance with the branch on or off. A matched 5.7M 2×2 on the
current blend says MC alone did not help PPL or long context;
MC+fractal together beat std at train length on seed 1 and lost
on seed 2. That is 5M / 10k, two seeds on std/both, and |tanh(gate)|
there is ~0.25 vs ~0.60 on this 114M DPO — the mix-in may still be
opening. The 114M nomc twin also dropped fractal RoPE and does not
share Ultra-MC's data history (shuffle repair ~18k). Not an MC
ablation. The paper does not say "train longer and MC appears"; it
says memory grows with how many segments the sequence has. The
zero-init gate opening over steps is this implementation. A longer 5M
run could still move that table. I have not run one. Do not promote
the 10k result to this 114M card. The paper is about RNNs; this is not
an RNN; the O(L) claim does not transfer.
What I actually ran on this model, not what I hope it does:
Did training use it. Yes. Mean |tanh(gate)| across layers is 0.599
on the DPO checkpoint. Zeroing the branch at inference (same weights,
mc_out gone) moves UltraX PPL 9.87 → 11.86 (−16.7%). Earlier in
training that gap was ~9%. That is an inference knock-out, not a twin
trained without the branch.
Extending context. Pretrain was 1024 tokens = 4 × mc_segment_len 256, so cross-segment memory is in the recipe from step 0. SFT grew
the window to 4096. Most of the stack is position-local. MC is not —
behaviour only changes when a sequence crosses a 256-token boundary.
I widened the config (export_weights.py --seq-len 16384, RoPE
caches rebuilt, no retrain of positions) and checked that cached
decode still is the trained function:
check
result
prefill 252 (just under a segment)
6/6 identical, max |Δlogit| 3.05e-05
prefill 260 (just over)
6/6, 3.82e-05
prefill 519 (two segments)
6/6, 4.58e-05
decode stepping over a boundary (prefill 253)
8/8, 3.24e-05
prefill 4,096 / 6,000 / 8,192 after the widen
6/6 each, ~3–4e-05
~3e-05 is the float32 floor at this depth. You can run this checkpoint
at a 16k window, including across MC segment edges, without the cache
computing a different model and without the generation collapsing.
48-slot KV at 16k is 0.40 GB bf16.
That is this model. The no-MC twin gets the same 16k extend. The test
is whether it collapses out there. Needle is a different question
(the first needle was the model continuing the start of the prompt;
see that section).
What is still missing at 114M. A train that changes only MC,
fractal left on. The nomc twin also dropped fractal RoPE, so it is not
that experiment. The 5.7M 2×2 is that experiment at small scale:
MC alone did not help; MC+fractal together beat std at 1024 on seed 1
and not on seed 2.
Weights and write-up: 5m-ablation/. Paper
recall tests (S-NIAH, MQAR) on this 62k base are at chance with MC on
or off; they do not prove MC. If you copy the branch anyway, copy the
two bugfixes or you will not have the mechanism: causal running mean
on the diagonal gate, and one global normaliser (not per-segment).
fp32, autocast off. State keyed per segment and per loop pass if you
loop. Set seq_len to a multiple of mc_segment_len so the branch
sees more than one segment from the first step.
Provenance
Weights re-saved without optimizer state. Inference code is the cache-correct
("repair2") build: Memory Cache is a full-sequence op, so a naive KV cache
recomputes it from one token and you get a different function. This build keys
per-loop MC state so cached decode matches a full recompute.
verify.py / verify_cache.py. See PROVENANCE.md.
safetensors
Same three stages, safetensors/ instead of .pt. For eval harnesses.
directory
stage
context
safetensors/base_62k/
pretrained base
1,024
safetensors/sft_7100/
SFT
4,096
safetensors/dpo_3200/
DPO
1,024
config.json + model.safetensors + tokenizer.json. Float32. Matches
the .pt bit for bit (502 tensors, 0 mismatches).
No lm_head.weight in the file. Tied embeddings; safetensors drops
shared storage. Call model.tie_weights() or you get a random head
and garbage logits, no exception.
AutoModelForCausalLM is a no. spike_whale is not in transformers.
Use model_v2.py / config.py (load.py already does).
Context is 1024 / 4096 / 1024. Read the config.json in that
directory. Past the end raises, does not wrap.
Put <bos> on every sequence. Scoring: use_cache=False forwards.
Generation: the engine, not a naive KV cache.
GGUF builds
GGUF/ has every stage in two precisions. Tensors keep their original
state-dict names. Full config is spike_whale.config_json, so a loader can
rebuild the state dict 1:1.
file
stage
size
Byrne-100M-Ultra-MC-base_62k-f16.gguf
pretrained base
228 MB
Byrne-100M-Ultra-MC-base_62k-q8_0.gguf
pretrained base
122 MB
Byrne-100M-Ultra-MC-sft_7100-f16.gguf
SFT
228 MB
Byrne-100M-Ultra-MC-sft_7100-q8_0.gguf
SFT
122 MB
Byrne-100M-Ultra-MC-dpo_3200-f16.gguf
DPO (default)
228 MB
Byrne-100M-Ultra-MC-dpo_3200-q8_0.gguf
DPO (default)
122 MB
Round-trip (re-read GGUF, compare every tensor to source): f16 max |Δ| ≤ 0.008,
q8_0 ≤ 0.079. Tokenizer is byte-faithful across all 16,512 tokens. f16 GGUF
through the engine is byte-identical greedy output to the .pt.
Not llama.cpp-compatible. Architecture is spike_whale: looped stack, MLA,
fractal RoPE, depth attention, Memory Cache. Nothing upstream implements that.
These GGUFs are a weight container for this codebase, not a llama-cli drop-in.
RoPE caches are not stored (recomputed at load). Fractal schedule, so a loader
has to rebuild it via fractal.py. If you rebuild standard RoPE instead it
loads, generates fluent text, and encodes every position differently from
training (inverse frequencies off by 2.2e-01). spike_infer does this;
anything else reading these files has to as well.
Inference engine
Two stacks. Both ship. Both have to stay cache-correct.
generate.py
engine_chat.py / spike_infer/
What
thin wrapper over model_v2.forward
full engine for this checkpoint
Default ckpt
package.json (dpo_3200)
same, unless --ckpt
Sampling
temp 0.7 / top_k 40 / rp 1.3 from the manifest
same, hardcoded to match
Chat
--chat wraps ChatML
ChatML by default, --raw for the base
MTP draft
no
yes, unless --no-mtp
Static KV
model's own past_key_values
in-place StaticKV, 48 slots
Check
verify.py
verify_cache.py
spike_infer/ is written for this model, not a generic SpikeWhale loader.
Every forward runs what the checkpoint trained with: MLA + QK-norm +
fractal RoPE + XSA, depth attention, the looped stack with per-pass
embedding, Hyper-Connections, Engram prefix slot, Memory Cache (fp32,
autocast off, state per plan slot), HRM refinement, MTP as a speculative
draft.
Load path: package.json → .pt + the config blob inside that .pt
(not the yaml). Optional GGUF via from_gguf. Config from the checkpoint
matters because SFT is 4096-context and base/DPO are 1024.
Decode path: prefill the prompt from position 0, then one new token at a
time. Speculative MTP only proposes; the main head verifies. Mid-stream
multi-token prefill is outside what MC incremental state is checked for.
Each new user turn re-prefills the window from 0 so RoPE positions, HC
streams, MC segments, and the Engram window match a fresh run.
Three things a generic loader gets wrong, all silently:
48 KV cache slots, not 16. loop_count=3 means the same weights see
different activations each pass, so every (pass, layer) pair owns its
own K/V. Slot index comes from loop_plan. Index by layer alone and pass 1
keys get fed to pass 0. Still generates. Wrong function.
Attention carries depth_kv and returns depth_entry. A static-KV patch
written for the Mark2 signature does not match, and silently falls back to
torch.cat cache growth. Depth mixing happens on the fresh K/V before
the cache is touched. Mixed V is what gets cached.
No engram_context_ids kwarg. This tree fixed Engram internally:
EngramModule.lookup takes a prefix through a trailing cache slot. The
engine supplies that slot and reads it back each forward. Mechanism is
picked by inspecting the model signature, not by tree name.
Memory-Cache state is keyed per plan slot, since the looped stack calls the
same branch once per pass on the same positions.
Do not delete the verify scripts. They are the only proof the cache path
equals a full recompute. verify.py hits generate.py. verify_cache.py
hits the engine (slot count, token identity, per-step |Δlogit|). Cheap on
CPU. A release without them is a release nobody can re-check.
Verified
verify_cache.py. Greedy is deterministic, so cached decode has to produce
the same token ids as a full recompute, not "similar text":
stage
tokens identical
max abs Δlogit
sft_7100
8/8
1.72e-05
dpo_3200
12/12
1.91e-05
~2e-05 is the float32 floor at this depth. Incremental path computes the same
function, it does not approximate it.
Prompt tokenization is checked id-for-id against the training tokenizer.
<bos> is prepended (training starts every sequence with it). Display decode
keeps <think> markers that skip_special_tokens=True would drop.
Long context
Two different measurements. A context-test fail is an engine bug. A
needle fail is a 114M-model property. Fixing the first does not improve
the second. Numbers below are sft_7100 unless noted. Method writeup:
NOTES-context-and-needle.md.
Most of the stack is position-local. Memory Cache is not — it is
segmented (mc_segment_len 256) and Engram carries an n-gram prefix
across the cache boundary. A short verify.py run never leaves segment
0, so it cannot see a cross-segment bug. check_context_ready.py sits
either side of 256 on purpose: RoPE buffers rebuild exactly on a widen,
StaticKV footprint at the target length (48 slots × 16384, bf16 = 0.40
GB), cache exactness just under / just over / two segments in, decode
itself stepping over a boundary, multi-turn window slide, and
position_ids == ctx raising instead of wrapping to 0.
11/11 on sft_7100:
rope cache recomputes from scratch exactly max |Δ| 0.000e+00 over 4096 positions (fractal=True)
rope cache at 16384 agrees on first 4096 max |Δ| 0.000e+00
StaticKV, 48 slots × 16384 tok, bf16 0.40 GB
prefill 252 tok (just under a segment) 6/6 identical, max |Δlogit| 3.05e-05
prefill 260 tok (just over) 6/6 identical, max |Δlogit| 3.82e-05
prefill 519 tok (two segments) 6/6 identical, max |Δlogit| 4.58e-05
decode crossing a boundary (prefill 253) 8/8 identical, max |Δlogit| 3.24e-05
window slide 3655 ≤ 4032 tok, history dropped
position == ctx raises (index out of bounds)
~3e-05 is the float32 floor at this depth. Same token ids as a full
recompute, not "similar text".
Widen with export_weights.py --seq-len 16384 and the engine stays
exact past the trained length. This model does not collapse at that
16k extend. The no-MC twin has to take the same test.
prefill
tokens identical
max abs Δlogit
4,096
6/6
3.34e-05
6,000
6/6
2.96e-05
8,192
6/6
4.20e-05
bash
1python engine_tools/check_context_ready.py --device cuda \2 --ckpt checkpoints/sft_7100.pt --target-ctx 16384
Needle-in-a-haystack
First pass: The secret access code is BLUEBERRY-7291. as sentence 1,
one filler sentence repeated to length, What is the secret access code? at the end. Greedy was the primary arm. keyword = BLUEBERRY
came back. exact = full string, digits included.
prompt tokens
greedy: keyword
greedy: exact
temp 0.7: keyword
505
yes
yes
yes
1,011
yes
yes
yes
2,046
yes
yes
no
3,587
yes
yes
—
Greedy copied the digits out to ~3.6k. dpo_3200 is exact at 505; its
context is 1,024 so the longer rows do not apply. Temp 0.7 often mashed
the suffix (BLUEBERRY-dica, BLUEBERRY-Definition1) and missed the
keyword at 2k on that seed. Sampled column is one draw.
That table is what was measured. It is not retrieval. Needle was the
first sentence, filler was one sentence on a loop, code was a fixed
string. Greedy wrote The secret access code is BLUEBERRY-7291. The archive records r... — the start of the prompt, copied. You pass this
test by continuing from position 0. Nothing has to be looked up.
Re-ran with that shortcut closed: varied filler, a topic-specific
question, a new randomly generated code every trial.
setting
exact
fixed code, needle at depth 0, no distractors
0.000
random code, depths 0/0.5/1.0, 3 distractors, 512–2048 tok
0.056
Typical miss: the frame is right, the digits are invented —
BLUEBERRY-?3, BLUEBERRY-Sch1, BLUEBERRY-tioxy,
BLUEBERRY-ggregate. It put out a code 33% of the time and it was
the wrong one. Nothing code-shaped 61% of the time.
It will echo a fixed, familiar string sitting at the top of the window.
It does not reliably pull an arbitrary 4-digit code out of depth. This
data does not split "can't retrieve" from "can't copy four random
digits." Both fit. The window is not the only limit.
Engine files
spike_infer/__init__.py the engine
engine_chat.py chat / completion CLI (sampling defaults)
verify.py generate.py cache == full recompute
verify_cache.py engine cache == full recompute
engine_tools/
export_weights.py .pt -> safetensors + config.json
convert_gguf.py safetensors -> GGUF
check_context_ready.py long-context / segment-boundary checks
test_multiturn.py multi-turn engine correctness
compare_checkpoints.py two stages, same prompts and seeds
morpho/ — the block as a circuit
morpho/ is a separate project in this repo. It does not run at generate
time. It grows Byrne's decoder block as combinational int8 logic in
MorphoHDL (Mordvintsev, Paradigms of Intelligence), folds real Ultra-MC
weights in as constants, and emits synthesizable Verilog.
A transformer on a GPU is weights × data on a processor. This expands the
arithmetic into AND/OR/XOR/NOT. No clock, no instruction stream. Put
activation bits on the input wires; the graph settling is the forward
pass. Same netlist simulates in tiny_morpho.py or goes to
Yosys/Vivado/Quartus. Once a weight is a constant, constant-propagation
deletes the gates that depended on it, so the circuit is specialized to
these trained values.
Numeric contract is int8 Q1.7 activations/weights, int32 accumulators,
Q4.4 attention scores, Q0.8 softmax. Checked against integer NumPy at
the same fixed-point, not against float.
What it covers: RMSNorm → attention → residual → RMSNorm → GELU MLP →
residual. That is the spine. Not grown as a datapath: MLA's low-rank
path, Memory Cache mixing, the loop, fractal RoPE. MC projection weights
do fold (mc.q_proj in byrne_fold.py).
Scale: a full d=4 block is 419k gates generic, 299k folded (29%),
emitted as byrne_block.v. A flat d=768 block would be ~1e9 gates and
will not compile. The 768-wide unit that does emit is one projection
neuron, byrne_proj768.v (208k gates, real k_proj row folded). A chip
would instantiate one of those and clock it, which is also how the
looped stack wants to be mapped: one circuit, run three times.
Two ways to run it:
Grow every gate (byrne_morpho.py, byrne_block_folded.py,
byrne_emit_verilog.py). Synthesizable. Does not produce language.
Run the same int8 MAC inside the real 114M model
(byrne_circuit_infer.py). Produces coherent text. A slice is
bit-exact against the grown dot.
Reads ../checkpoints/ read-only. Docs: morpho/README.md (what to
run, numbers) and morpho/morpho-transformer.md (language notes and
the original stage plan).
bash
1cd morpho
2python byrne_morpho.py
3python byrne_fold.py
4python byrne_block_folded.py
5python byrne_circuit_infer.py -p "The capital of France is"6python byrne_emit_verilog.py
Running it
Temperature sampling: temp 0.7, top_k 40, rep_pen 1.3, random seed per turn.
Greedy is for determinism proofs. Judging quality by it gives the wrong
answer. A repetition study on a sibling under greedy measured self-repetition
0.364; at the serving defaults it was 0.000.
bash
1# chat, DPO stage (default), multi-turn REPL2python engine_chat.py
34# one-shot5python engine_chat.py -p "Explain why the sky is blue in one sentence."67# a different stage8python engine_chat.py --ckpt checkpoints/sft_7100.pt -p "Give me a focus tip."910# raw continuation, no chat framing — this is what the base wants11python engine_chat.py --ckpt checkpoints/base_62k.pt --raw -p "The capital of France is"
Match the stage to the mode. base_62k continues text; use --raw.
sft_7100 / dpo_3200 expect ChatML.
Keep max_new modest (160 default). These checkpoints often don't emit a
stop token on open-ended prompts, so a bigger cap just means a longer
answer, not a better one.
<bos> is required. Training prepends it. The engine does this; a
hand-rolled prompt has to as well or the whole sequence sits one position
off.
Multi-turn re-prefills from position 0 each turn. Deliberate: RoPE
positions, Hyper-Connection streams, Memory-Cache segment state, and the
Engram window stay consistent with a fresh run.
Known behaviour
Measured at serving defaults, not guessed:
Cross-turn collapse, about 1 conversation in 10. Latches onto a template
from an earlier reply and repeats it. Three more in ten show partial
collapse. New conversation clears it.
Doom loops / near-repetition: 0/20 on a held-out prompt bank, serving
defaults and greedy. distinct-3 = 0.999.
Topic switching is fine when sampling (adherence 1.00), bad at greedy
(0.47, stuck on the old topic 3/6). Another reason not to serve greedy.
World knowledge is the real hole. France in Spain, water boils at 1.5 °C.
A familiar string parked at the top of the prompt gets copied. An
arbitrary code sitting in depth does not (see needle). Facts expected
from the weights also do not.
Training recipe
Scripts are not in this repo yet (inference-only release). Configs and
hyperparameters below are the spec; script names are for reference. The
full training scripts ship with the comparison models around 26–28 Aug
2026 (see matched runs above).
Three stages in the order they ran: pretrain → SFT → DPO. dpo_3200 comes
from sft_7100.
1. Pretrain → base_62k
train_blend_muon-50m.py --config byrne_100m_ultrax_mc.yaml --muon
(yaml is in this repo).
steps
61,000, plus a Dolma-blend continuation; released at 62k
tokens
~2.0B (Chinchilla-ish for ~114M dense)
seq_len
1,024 (= 4 × mc_segment_len, so MC sees cross-segment memory)
batch
6 × grad_accum 4
optimizer
Muon (muon_lr_mult 10) + AdamW for aux params
lr
3e-4, warmup 2,000, cosine to min_lr_frac 0.1
weight decay
0.01, grad clip 1.0
grad checkpointing
on
Data blend (web_source takes leftover weight):
source
weight
FineWeb-Edu (sample-100BT)
0.40
Wikipedia (20231101.en)
0.15
DCLM-baseline-1.0
0.15
Cosmopedia-v2
0.10
FineMath-4+
0.10
NPset-2 Python-Edu
0.10
MC enters through a zero-init gate, so at init the model is the no-MC
baseline. Trains to tanh(mc_gate) ≈ 0.561 across all 16 layers, which is why
a broken cached path wrecks this one. PROVENANCE.md.
max_position_embeddings is set to seq_len. Training at 4,096 is why the
SFT checkpoint is a 4,096-context model. Base and DPO are 1,024 for the
same reason.
Conversations are packed to fill the window. Every sequence starts with
<bos> then ChatML (<|im_start|>{role}\n{content}<|im_end|>\n). Packing
is how the full position range gets exercised even though individual
conversations are short.
DPO at this lr barely moves the policy. Step 2,000 → 3,200, relative L2 of
2.8e-05. Greedy output identical on every prompt tested. UltraFeedback
win-rate against the reference stuck at 0.438 → 0.438 even though
implicit-reward accuracy hit 0.75. It separates preference pairs without
visibly changing what the model writes. Benchmarks agree: DPO and SFT are
within noise.
Resuming from a release
Released .pt files are weights + config, no optimizer, so a resume starts
with a fresh optimizer. Each file's config is the one it was trained with.
Engine and tools read it from the checkpoint, not from the yaml — the stages
disagree about context length.
Ending Statement
I want to clarify that this release is not competitive, usable for
chat, or intended to suggest any superiority or inferiority of this
architecture.
The primary purpose of this release is to provide data and
information. It showcases the results of my tests and experiments,
which can be applied if they prove beneficial to your project.
I strive to deliver completely honest work, conducting a wide range
of tests, observations, and incorporating features while testing
them. Numerous ideas were explored and discarded before developing
this model.
I wanted to demonstrate the potential benefits of the Memory Cache,
investigate the impact of altering the RoPE from the standard, and
since the model learned rapidly, I wanted to investigate the reasons
behind this with a model trained on the same configuration but with
all those modifications disabled.
The results of the follow-up tests will be shared soon.
Byrne-15M-Looped
The small looped run is its own repo:
Quazim0t0/Byrne-15M-Looped.
That was the toy / smoke-test for this 114M model.
~15.2M active / ~18.9M total, 10 layers, loop_count=3 (effective
depth 30), MoE. No Memory Cache. I trained it to see if looping the
stack beats a param-matched loop_count=1 baseline at equal steps.
It does on UltraX val loss (4.195 vs 4.231 at 40k / 246M tokens) and
on degeneration. 3× FLOPs per token. Then I SFT'd UltraChat and ran
DPO the same way I do here.
Scored with the same harness as this card (full MC, not 200-capped):
metric
15M base
this 114M base
WikiText-2 byte_ppl ↓
2.943
2.308
BLiMP acc ↑
0.734
0.811
tokens
0.25–0.42B
2.0B
Different data, size, and knobs. The 15M numbers are in that repo
(BENCHMARKS.md, FINDINGS.md). Safetensors for leaderboard owners
are under safetensors/base/ there.
Anti-DEGR ToolKit
I used
Anti-DEGR ToolKit
to test this checkpoint. It is training and decoding code for four
failures small LMs actually hit: doom loops (an exact span, over and
over), echo (the user changes subject and the model answers the previous
question), grounding (it stops reusing context that is still relevant),
and near-copy paraphrase (it restates a sentence with one noun swapped,
so the cycle detector is blind to it). I train echo and grounding, leave
most loops to effort decoding, and measure near-copies separately. Those
scripts are not in this repo and they are not on the generate path.
References
Memory Cache branch adapts:
bibtex
1@misc{behrouz2026memorycaching,
2 title = {Memory Caching: RNNs with Growing Memory},
3 author = {Ali Behrouz and Zeman Li and Yuan Deng and Peilin Zhong
4 and Meisam Razaviyayn and Vahab Mirrokni},
5 year = {2026},
6 eprint = {2602.24281},
7 archivePrefix = {arXiv},
8 primaryClass = {cs.LG},
9 note = {Google Research},
10 url = {https://arxiv.org/abs/2602.24281}
11}
Tokenizer:
bibtex
1@misc{dong2025lengthmax,
2 title = {Length-MAX Tokenizer for Language Models},
3 author = {Dong Dong and Weijie Su},
4 year = {2025},
5 eprint = {2511.20849},
6 archivePrefix = {arXiv},
7 primaryClass = {cs.CL},
8 url = {https://arxiv.org/abs/2511.20849}
9}
Citation
If you use this model, please cite:
bibtex
1@misc{byrne100multramc,
2 title = {Byrne-100M-Ultra-MC: A ~114M-parameter looped SpikeWhaleLM
3 with a Memory Cache branch},
4 author = {Dean Byrne (Quazim0t0)},
5 year = {2026},
6 howpublished = {HuggingFace, \url{https://huggingface.co/Quazim0t0/Byrne-100M-Ultra-MC}},
7 note = {Quazim0t0/Byrne-100M-Ultra-MC}
8}
