SpikeWhale-SNN × Brain2Qwerty decoder
A
brain2qwerty-style decoder
(
Conv encoder → causal Transformer → LM head) adapted as a readout for the
spiking language model
Quazim0t0/SpikeWhale-SNN-216M,
plus a
jointly fine-tuned SNN+decoder checkpoint that beats the SNN's native
linear readout on held-out FineWeb-Edu.
Brain2Qwerty is built to decode multichannel neural time-series (MEG/EEG) into
text. SpikeWhale's all-layer pre-reset membrane trunk (state["hidden"], shape
[B, T, num_layers*hidden] = [B, T, 5952]) is a multichannel neural signal —
so this project treats it as one and decodes it.
| Brain2Qwerty | SpikeWhale |
|---|
| MEG/EEG channels | neurons: num_layers*hidden = 5952 |
| time samples | per-token LIF timesteps |
| conv encoder | causal temporal conv over the raster |
| transformer | causal transformer over token positions |
| char LM head | linear head over the SpikeWhale vocab |
No FAIR code or weights are used — the decoder is a clean re-implementation of
the brain2qwerty pattern, so it carries no upstream license.
What was tested
All numbers are held-out bits/token on FineWeb-Edu (lower is better).
1. Frozen readout (SNN frozen, only the decoder trains)
The SNN trunk was extracted once and cached; a full sweep trained decoders on it:
capacity 4.4M–27.8M params, dropout 0.1–0.4, weight decay 0.01–0.2, data
1024→4096 windows, training 8→24 epochs, with early stopping.
| readout | params | held-out |
|---|
SNN native linear mem_readout | ~0.09M | 5.228 |
| brain2qwerty decoder (best of sweep) | 27.8M | 5.457–5.469 |
Every configuration lost to the linear head. Train loss could be driven to
~0.03 bits (the decoder memorizes the train windows) but held-out never crossed
the baseline. Varying every axis and seeing no improvement showed the ceiling is
the fixed trunk's information content, not decoder capacity, data, or
regularization: a frozen trunk exposes no extra generalizable next-token signal.
2. Joint fine-tune (unfreeze the SNN, train end-to-end)
Unfreezing the 216M SNN and training it with the decoder via backprop-through-time
(SNN lr 5e-5, decoder lr 1e-4, decoder warm-started from the frozen-readout best,
3000 steps on streamed FineWeb-Edu), then a continued fine-tune at a gentler LR
(SNN lr 3e-5, decoder lr 4e-5, ~900 more steps):
| readout | held-out | vs linear baseline |
|---|
linear mem_readout (original SNN) | 5.228 | — |
| decoder on frozen SNN (start) | 5.473 | +0.245 (loses) |
| decoder + jointly fine-tuned SNN | 5.221 | −0.007 |
| + continued gentle fine-tune | 5.187 | −0.041 |
| + segmented fine-tune chain | 5.020 | −0.208 (beats) |
Initial descent (monotonic, still falling at the end of the 3000-step run):
5.473 → 5.396 → 5.369 → 5.354 → 5.332 → 5.307 → 5.283 → 5.280 → 5.239 → 5.221.
Segmented fine-tune chain (joint_chain.py): repeatedly resume from the current
best and run a fresh 5k-step segment with a NEW seed (a different SGD path over the
same streamed pool), early-stopping each segment on patience and promoting only when
held-out improves beyond noise. Each segment first drifts up, then finds a lower
minimum around step ~1200–4800. It kept improving for four segments, then stalled:
5.187 → 5.107 → 5.094 → 5.057 → 5.020 (seg5 gained <0.004 → chain stopped).
joint_best.pt is the 5.020 checkpoint.
Takeaway. A heavier readout on a frozen trunk cannot beat the linear head.
Once the SNN's recurrent weights are allowed to move, it learns to expose signal
the brain2qwerty decoder can use — and repeated re-seeded continued fine-tuning
compounds into a 0.21-bit win over the linear readout before it saturates. The
gains still don't cross into qualitatively coherent text (the undertrained base SNN
and the one-timestep-per-token axis cap that), but the direction is unambiguous:
unfreezing the SNN and chaining fresh-seed fine-tune segments is what makes the
decoder win, and each segment's different SGD path finds further headroom until it
saturates.
Generation samples
⚠️ The base SpikeWhale SNN is explicitly undertrained research code, so both
readouts produce locally-grammatical but globally-incoherent text. These samples
are representative, not cherry-picked. Prompt in italics.
Jointly fine-tuned SNN + decoder (joint_best.pt, val 5.020, temp 0.7, top-k 20):
- In the beginning — X-ray regional structure in the United States. The United Nations General Agency is not an interpretant in the nation.
- Scientists have discovered — "No saturated it." The research results were based on the real University, but the eighth-grade ... Center for Environmental European Union.
- The history of — Johnson's "It is not to be the 'People'." They also makes an overcome it only. They were to be understood and not ...
- She opened the door and — King King Karl Park. As a Keep on July 5, 2007. The first week of the One of Kings.
- The most important thing about — "pretination" is something, not as the same as the amount of Vietnam. At the same word as a "sign" ...
Frozen-SNN decoder (for contrast):
- According to the report, — "It's important to the Atlantic Law." ... the reception of the United States
- During the war, — ... the town of the city in the northern. There are times of the atmospheric area ... the greenhouse gas
The joint model drifts toward an "encyclopedic" register (dates, institutions,
place names) — a fingerprint of the FineWeb-Edu fine-tune — but the
gain is not visible as a qualitative jump; the base SNN caps both.
Distilled variant (joint_distill.pt)
A sequence-level distillation experiment: continue-fine-tune the joint model on
permissive assistant data so it imitates that style. (True logit-KD is impossible —
the teachers use a different tokenizer than SpikeWhale's 16.5k byte vocab — so this
is sequence-level KD: train on the teacher text.) Lineage, all resumed in order:
- SmolLM2-135M-Instruct-generated corpus (educational explainer style)
- SmolTalk2
SFT/OpenHermes_2.5_no_think → chat register, learns "Ah, …" openers (6.646 → 4.773)
- SmolTalk2
SFT/smol_magpie_ultra_no_think → technical/coding-instruction register (4.597 → 3.841)
joint_distill.pt is stage 3. It clearly imitates an instruction/coding-assistant
register — "Here's a…", "help you…", markdown and python def blocks — but the
undertrained 216M SNN still can't produce coherent content, so output is
style-accurate word-salad. Samples (ChatML prompt → assistant turn, temp 0.7):
- What is a lion? → "Here's a Python program that involves a list of integers that lists the list…"
- Once upon a time → "…-driven approach to help you create a great time complexity to your story… revise your own characters…"
This variant fits assistant chat far better (3.841 vs the 5.020 model's ~6.6 on the
same data) but is worse on FineWeb by design — it is a style-transfer artifact,
not a better general model. Teachers (SmolLM2, SmolTalk2) are Apache-2.0; MobileLLM-R1
was deliberately not used (noncommercial license — can't publish a derivative).
Reproduce stage 3:
1PYTHONPATH=src python joint_finetune.py --resume joint_best.pt --out joint_distill.pt \
2 --stream --dataset HuggingFaceTB/SmolTalk2 --config SFT \
3 --split smoltalk_smollm3_smol_magpie_ultra_no_think \
4 --seq-len 128 --batch-size 4 --grad-accum 2 --snn-lr 3e-5 --dec-lr 5e-5 \
5 --max-steps 6000 --eval-every 300 --patience 6
Files
| file | what |
|---|
decoder.py | Brain2QwertyDecoder + the frozen-SNN trunk tap (extract_trunk) |
train_decoder.py | extract/cache trunk, train frozen-readout decoder, save |
joint_finetune.py | unfreeze SNN + decoder, BPTT end-to-end (--resume, patience early-stop) |
joint_chain.py | chain fresh-seed fine-tune segments, promote while improving |
distill_generate.py | generate a teacher corpus for sequence-level distillation |
generate.py / generate_joint.py | sample from the frozen / joint models |
decoder_best.pt | frozen-readout decoder weights (val 5.469) |
joint_best.pt | fine-tuned SNN + decoder (val 5.020) — the main artifact |
joint_distill.pt | SmolTalk2-distilled assistant-style variant (see above) |
Usage
Requires the base repo's
snn package on the path (clone
SpikeWhale-SNN-216M and
put its
src/ on
PYTHONPATH, and its
snn_stream_program.pth alongside).
1# sample from the jointly fine-tuned model
2PYTHONPATH=src python generate_joint.py --n 20
Limitations
- Inherits the undertrained base SNN; not a usable general LM.
- The joint checkpoint bundles the full 216M SNN, so it needs the base repo's
snn code to load.
- Held-out gain over the linear head reaches ~0.21 bits after the segmented chain,
but this is a next-token bits/token metric — it does not translate into
qualitatively coherent generation. The undertrained base SNN and the
one-timestep-per-token axis cap output quality regardless of the readout.
- Copies of
joint_finetune.py / joint_chain.py reference these locally-bundled
files; the base repo's published snn package is the only external dependency.
