Research artifact — private. This is an experiment, not a product. Do not use it for anything real unless it is trained much longer (see Intended use & limitations).

Chimera-64M is a ~64M-parameter, from-scratch decoder-only language model — the capstone of a family of models that ask whether a transformer's generalizing ability can be carried by unusual dynamical systems in place of the feed-forward block. Chimera doesn't pick one: its per-layer channel mixer is a council of three physics cores — coupled oscillators, a fungal-colony growth model, and the Wheeler–DeWitt wave equation — and a small reinforcement-learning rule (GRPO) learns which law of physics to apply to each token.

Architecture graph for Quazim0t0/Chimera-64M. Open in hfviewer

Model summary


Parameters	~64.0M
Type	Decoder-only autoregressive LM (custom `QuazimotoLM`)
Layers / hidden	10 / 768
Attention	MLA + partial RoPE + QK-Norm + GQA + Elo/Bradley–Terry ratings
Channel mixer	ChimeraBlock — GRPO council over 3 physics cores
Context	`block_size` 1024 (RoPE `max_position_embeddings` 4096)
Vocab	16,512 (custom SpikeWhale byte-merge tokenizer)
Traits	HRM refinement, MoE-SwiGLU, MTP, JEPA (train-time), fractal phase seed

Two checkpoints are included:

chkpt/quazimoto.pt — base, pretraining step 40,000 (continued at block 1024).
chkpt/quazimoto_sft.pt — chat SFT, step 4,250.

The ChimeraBlock — a GRPO council of physics

Instead of an MLP, each layer runs three physics cores in parallel, each proposing a candidate update of the hidden state, then combines them with a differentiable, in-forward-pass GRPO step:

KuramotoCore — a bank of coupled phase oscillators (Kuramoto dynamics), integrated by a few differentiable Euler steps. (from Quazimoto/Positronic)
GrowthCore — a Neighbour-Sensing fungal-colony growth model: tips grow in a bounded latent region, sensing their own density and steering away from it. (from Mycel)
WaveCore — the Wheeler–DeWitt equation: wdw_modes minisuperspace modes under a learnable Lorentzian supermetric, run as a leapfrog wave; exposes a Hamiltonian-constraint aux loss ⟨H²⟩. (from Wheeler)

The GRPO council (the meta-mixer, from grpo_lm) then decides the blend, per token:

an internal critic scores each candidate → reward r_g;
group-relative advantage A_g = (r_g − mean)/(std + ε), clamped;
selection weights softmax(A/temp), clipped around uniform (the PPO clip);
anchored back toward uniform (the KL-to-reference anchor);
output = the resulting convex combination of the three candidates.

So the network learns which mechanism to trust for each token rather than committing to one. Everything else is the shared "SpikeWhale/Byrne family": MLA attention with Elo/Bradley–Terry key ratings, HRM refinement, MoE-SwiGLU, MTP heads, a JEPA objective, and the SpikeWhale tokenizer. Each trait is a gated near-no-op at init that activates only if it helps.

Usage

Self-contained — load with the bundled code:

python

import torch
from model import QuazimotoLM, QuazimotoConfig
from spike_tokenizer import SpikeTokenizer

tok = SpikeTokenizer(vocab_file="tokenizer.json")
ck = torch.load("chkpt/quazimoto.pt", map_location="cpu", weights_only=False)
cfg = QuazimotoConfig(**ck["family_config"])
model = QuazimotoLM(cfg); model.load_state_dict(ck["model"], strict=False); model.eval()

ids = torch.tensor([tok.encode("The history of science", add_special_tokens=False)])
out = model.generate(ids, 80, temperature=0.8, top_k=40, use_cache=True)
print(tok.decode(out[0].tolist(), skip_special_tokens=True))

Also bundled: generate.py (KV cache + self-speculative decoding), chat_sft.py (ChatML REPL), and visualize.py (a live GRPO-council dashboard showing which physics law wins each token).

Intended use & limitations

This is a research artifact for studying architecture generalization — not a usable assistant.

The base produces fluent, locally-coherent free-form text, but is not factual or reliable over long spans.
The SFT checkpoint (only 4,250 steps) stays on-topic and produces structured, math-formatted answers, but hallucinates (its formulas and facts are unreliable) and does not truly follow instructions. This is a capacity-and-scale limit of a 64M model with a short SFT run.
Do not deploy this. To be actually useful it needs to be trained substantially longer (more tokens, larger, longer SFT). It is shared to demonstrate that three exotic physics mixers, GRPO-selected per token, can learn language — the point is that it generalizes, not what it knows.
Trained on public web/edu/math corpora; inherits their biases and can produce incorrect or offensive text.

Family & thesis

Chimera is the synthesis of a controlled series of mixer swaps (Kuramoto → Quazimoto/Positronic; Neighbour-Sensing growth → Mycel; Wheeler–DeWitt → Wheeler). The shared thesis: the specific mixing mechanism matters far less than the loop of attend → compress → repeat — and a network can even learn to arbitrate between several exotic mechanisms per token.

Citation

bibtex

@misc{byrne2026chimera,
  title  = {Chimera-64M: a language model whose channel mixer is a GRPO council of
            coupled oscillators, fungal-colony growth, and the Wheeler--DeWitt equation},
  author = {Byrne, Dean},
  year   = {2026},
  note   = {Research artifact; custom PyTorch architecture (Quazim0t0 / SpikeWhale family)}
}

Update: format-blended SFT

Short instruction/format SFT on a 60/25/15 blend of HuggingFaceTB/smoltalk, GSM8K-train ('#### N' reasoning), and MMLU-style ('Answer: ') examples, so chat fluency improves while the benchmark output-formats are preserved. Held-out (test-split) before->after:

held-out 2.23->1.34, MMLU acc 0.035->0.250, format 0.140->1.000

Note: fluency + output-format gains. Benchmark accuracy remains near the floor for a model this size -- the SFT does not add reasoning ability.

Quazim0t0/Chimera-64M

Chimera-64M