Harmonic GPT 128M Byte Chat
An experimental 127.6M-parameter causal byte transformer trained by the Harmonic GPT project.
This repository contains the retained step-300 checkpoint from the August 7 conversational-SFT
control run. It was selected for conventional chat serving after a native-format linguistic and
long-form attractor comparison against the Agent multiparty sibling.
The model uses raw UTF-8 bytes rather than a subword tokenizer. Its decoder has 18 layers,
768 hidden dimensions, 12 attention heads, RoPE, RMSNorm, SwiGLU, tied byte embedding/readout,
and PyTorch causal SDPA.
Important limitations
This is a research artifact, not a reliable general assistant. It learned prompt/response
framing and usually emits the trained end-of-response sequence, but it frequently gives incorrect
or nonsensical factual answers. Do not use it for medical, legal, financial, safety-critical, or
production decisions.
The model is released under CC BY-NC 4.0 because its SFT mixture includes
HuggingFaceH4/no_robots, which uses that license. Other SFT components include
MonumentalSystems historical chat, reasoning, and synthetic salon datasets.
Training snapshot
- Pretraining checkpoint: step 90,000
- Pretraining exposure: 5.898B ordered UTF-8 bytes
- SFT: 300 selected steps at peak LR
3e-5, completion-only loss
- SFT mixture: 40% No Robots Q&A, 30% everyday conversations, 15% No Robots
practical dialogue, 5% reasoning, and 10% pretraining replay
- Held-out assistant BPB: 1.243345 (starting value 1.292222)
- Matched base-corpus BPB after SFT: 0.955184 (starting value 0.940957)
- Held-out multiparty BPB: 1.369315 (starting value 1.411642)
- Learned response delimiter:
\n<|end|>\n
The matched linguistic probe produced a response-only composite of 74.4. In a six-prompt,
768-byte attractor probe, stochastic decoding at temperature 0.8/top-k 40 had zero detected loops
and 6/6 EOS termination; greedy decoding looped on 2/6 prompts. These shape-oriented diagnostics
do not measure factual accuracy.
Dedicated Inference Endpoint
The repository includes a custom handler.py. Deploy it with Hugging Face Inference Endpoints
using the Custom framework. The handler accepts either a plain prompt or role/content messages.
Plain prompt request:
1{
2 "inputs": "What is DNA?",
3 "parameters": {
4 "max_new_bytes": 256,
5 "temperature": 0.7,
6 "top_k": 40,
7 "seed": 42
8 }
9}
Multi-turn request:
1{
2 "inputs": [
3 {"role": "user", "content": "Hello!"},
4 {"role": "assistant", "content": "Hello. How can I help?"},
5 {"role": "user", "content": "Explain DNA briefly."}
6 ],
7 "parameters": {"max_new_bytes": 256}
8}
Response shape:
1{
2 "generated_text": "...",
3 "ended": true,
4 "generated_bytes": 123
5}
Generation currently recomputes the causal context for every byte and does not implement a KV
cache. GPU serving is strongly recommended.
Local handler usage
1from handler import EndpointHandler
2
3model = EndpointHandler(".")
4print(model({"inputs": "Ping", "parameters": {"max_new_bytes": 128}}))
Research context
This checkpoint is the conventional-transformer control used alongside spherical and planar
geometric architectures in Harmonic GPT. Benchmarks are treated as diagnostics for architecture
research rather than claims that this model is competitive with production language models.
