Harmonic GPT Spherical KAN 128M
An experimental 128.26M-parameter causal byte language model from the Harmonic GPT
geometry A/B. This revision is a deliberately benchmark-specialized derivative of the
step-52,500 spherical pretraining checkpoint (validation BPB 1.1740).
Evaluation contamination: the post-training set directly transcribes 24 public
prompts from evals.blah.dev, with 28 assistant targets across multi-turn cases. Scores
on those prompts are training-set measurements and must not be presented as evidence of
generalization. Use an earlier Hub revision for the clean base model.
The model uses a fixed QPSK + Hamming wire encoder, 18 unified blocks, seven heads,
eight Funk-Hecke/SpectralKAN channels per head, bounded decayed recurrent state, and
projection-free spherical Lohe mean-field coupling. Its hidden width is 1,330 (seven
oscillator groups of 190 dimensions). The minimal readout exposes raw oscillator state,
alignment, and order-parameter observables to a 256-byte head.
Checkpoint and data
- Parameters: 128,260,556
- Pretraining exposure: 3,440,640,000 bytes
- Context used in training: 2,048 bytes
- Optimizer: split-decay RotationalAdamW
- Dropout: 0
- Training stream: ordered
document-corpus-v3
- Checkpoint precision in this repository: BF16
- Specialist updates: 1,550 full public-eval sweeps
- Specialist trainable scope: final normalization + byte head only (346,724 parameters)
- EOS objective: null byte, 4x primary-CE weight
- Public training-set audit: 18/28 greedy exact, 28/28 null-stopped, 92.07% teacher-forced byte accuracy
The demo formats chat turns as Q: ...\nA:. Byte value zero is an explicitly trained
end-of-response signal and is never emitted as visible text.
Native bounded-state inference
modeling_spherical_kan.py is a standalone inference implementation. Its step() path
preserves the trained SpectralKAN recurrence and spherical Lohe update while retaining
O(1) state with respect to generated sequence length. Packaged BF16 logits were checked
against the original checkpoint's BF16 step() path with maximum absolute delta 0.0.
Important limitations
This is a contaminated research artifact, not a reliable general assistant. It can
produce plausible but incorrect, repetitive, malformed, or nonsensical text. Do not use it for medical,
legal, financial, safety-critical, or production decisions.
The model is released under CC BY-NC 4.0 for research use. The training corpus is an
internally assembled, filtered, globally deduplicated research mixture of curated,
reference, long-form, mathematical, reasoning, scientific/code, and document-QA sources;
downstream users remain responsible for reviewing source-specific terms.
Research context
The checkpoint is the spherical geometry sibling of the modern Transformer and planar
Givens controls. Harmonic GPT treats benchmark scores as diagnostics of self-organizing
oscillator computation rather than as claims of production-model competitiveness.
