Harmonic GPT Spherical KAN 128M — clean final pretraining
An experimental 128.26M-parameter causal byte language model from the Harmonic GPT
geometry A/B. This repository contains the clean ordered-corpus pretraining checkpoint
at step 122,076, after one complete pass through the training stream. It received no
chat, instruction, EOS, or public-evaluation fine-tuning.
Checkpoint and data
- Parameters: 128,260,556
- Pretraining step: 122,076
- Pretraining exposure: 8,000,327,497 byte tokens
- Validation BPB: 1.115637
- Context used in training: 2,048 bytes
- Optimizer: split-decay RotationalAdamW
- Learning rate at checkpoint: 3e-4
- Dropout: 0
- Training stream: ordered
document-corpus-v3
- Corpus fraction seen: 100% of the 8.000B-byte training stream
- Checkpoint precision in this repository: BF16
- Public-eval or chat fine-tuning: none
The model uses a fixed QPSK + Hamming wire encoder, 18 unified blocks, seven heads,
eight Funk-Hecke/SpectralKAN channels per head, bounded decayed recurrent state, and
projection-free spherical Lohe mean-field coupling. Its hidden width is 1,330 (seven
oscillator groups of 190 dimensions). The minimal readout exposes raw oscillator state,
alignment, and order-parameter observables to a 256-byte head.
Native bounded-state inference
modeling_spherical_kan.py is a standalone inference implementation. Its step() path
preserves the trained SpectralKAN recurrence and spherical Lohe update while retaining
O(1) state with respect to generated sequence length.
The hosted demo maps chat messages to a Q: ...\nA: byte prompt solely as an API
adapter. That formatting was not used for fine-tuning. Byte value zero is treated as a
stop signal by the server when it occurs, but this clean checkpoint was not explicitly
trained to emit EOS and commonly reaches the requested byte limit.
Important limitations
This is a raw pretrained research artifact, not a reliable assistant. It may produce
repetitive, malformed, nonsensical, or factually incorrect text. Do not use it for
medical, legal, financial, safety-critical, or production decisions.
The model is released under CC BY-NC 4.0 for research use. The training corpus is an
internally assembled, filtered, globally deduplicated research mixture of curated,
reference, long-form, mathematical, reasoning, scientific/code, and document-QA sources;
downstream users remain responsible for reviewing source-specific terms.
Research context
This clean checkpoint is intended to test how continued pretraining changes the
spherical architecture without conflating that result with the deliberately contaminated
BENCHMAX specialist. Benchmark scores should be treated as diagnostics of oscillator
computation rather than evidence of production-model competitiveness.
