Multi-head Latent Attention (MLA + XSA) - Q and O LoRA-compressed (rank 128);
each head splits RoPE dim 16 / NoPE dim 48; 10 query heads, one KV head
(MQA); QK-norm.
Engram n-gram memory - gated table, hashes local n-grams (up to trigrams)
into 4,096 rows, mixes back into the residual.
Hash-lookup layers (×2) - content-addressable features next to the token
embeddings.
Hyper-Connections - learned width-expanded residuals, Sinkhorn routing,
instead of a plain add.
HRM refinement - extra latent pass over hidden states before the output
head.
Multi-Token Prediction (MTP) - DeepSeek-V3-style extra head, more than one
next token. Training only.
FFN is dense. The block can do MoE; MoE is off in this release.
JEPA vs HRM. Escarda trains both HRM refine and JEPA
(use_hrm_refine=True, use_jepa=True). JEPA predicts future latents during
training. Byrne drops JEPA and keeps HRM only.
SpikeTokenizer. Byte-level length-max (greedy longest-match), 16,512 vocab.
Not BPE. Text → UTF-8 → latin-1 bytes → longest vocab key that fits. ChatML-aware.
Atomic specials: <|im_start|>, <|im_end|>, <think>/</think>,
<begin_solution>/<end_solution>, tool-call markers, plus <bos>/<eos>/<pad>/<unk>.
PreTrainedTokenizer in spike_tokenizer.py. Load with
AutoTokenizer.from_pretrained(..., trust_remote_code=True).
Evaluation
Zero-shot multiple-choice, continuation log-likelihood on each task's val/test
splits. byte_ppl is exp(sum_NLL_nats / total_UTF8_bytes) on WikiText-2 test
(tokenizer-independent). BLiMP is fraction of minimal pairs with
logprob(good) > logprob(bad) (12 paradigms × 150). Stderr is binomial
sqrt(p(1-p)/n).
Distilled base. Lowest byte-perplexity in the Escarda family, weaker
downstream. PPL is not a capability ranking. Chat behaviour:
Escarda-86M. This one if you
want a low-loss base to keep training.
Training & token budget
Tokens: ~20B from-scratch SpikeWhale (~28k steps). This checkpoint is a
JEPA-distilled snapshot of that base.
Token/param: ~233 (20B / 85.7M). About 11-12× Chinchilla's
~20-tokens/param. Over-trained small model on purpose. Inference is the trade.
Fitting Chinchilla's data term to this run's pretrain loss:
Parameter-bound, not data-bound at 20B: capacity term (~0.82 nats) beats
the data term (~0.54). Extra tokens do little. Doubling to 40B is projected
~0.07 nats lower loss (~7% PPL) with basically no downstream gain. Next lever
is more params, not more tokens. Same reason this distilled base has the
lowest PPL in the family without the best downstream scores - it is already
on the data-term floor.
Caveats: single-size fit (irreducible loss + capacity floor folded into one
constant). Cosine-LR decay inflates the fitted exponent, so treat β as an
upper bound. Token counts are anchored to ~20B and scale linearly if that
figure is off.
Usage
Custom architecture. trust_remote_code=True. Modeling code is in the repo
(auto_map):
Docling (same held-out doc images, atomic DocTags): both emit well-formed
DocTags. Byrne-Docling is a bit more complete on the hardest samples (closes
</formula>, includes the <code> wrapper), which matches the slightly higher
teacher-alignment. Escarda-Docling is structurally on par and has the JEPA
representation-learning trait.
Pros/cons.Byrne (HRM): higher teacher-alignment, all capacity on
distillation fidelity; no self-supervised objective. Escarda (HRM+JEPA):
self-supervised neighbour-prediction (richer spatial structure) at zero
inference cost, trading ~1-3% teacher-alignment. Same size class.
The n-gram Engram in the original weights was degenerate: frozen LSH compressor
at init scale hashed every token to bucket 0, so only one table row ever got
gradient. This revision rescales the (frozen) compressor and broadcasts the
learned bucket-0 vector across all table rows.
Outputs are bit-identical to the previous revision (verified: max logit
difference 0.0 across a prompt battery). The only change: the Engram hash now
spreads across the full table and every bucket is independently trainable - so
if you distill or SFT on top of this base, the n-gram memory will actually learn
instead of staying a constant bias.
