Binary Transformers: Learning Language from Raw Binary
Zero-tokenization transformers that learn directly from network bytes, bits, and beyond.
This repository contains four novel transformer architectures exploring the limits of minimal vocabulary learning:
Model
Vocab
Input
Weights
Description
Byte-level
256
bytes (0x00-0xFF)
real
One token per byte value
Bit-level
2
bits (0, 1)
real
Pure binary, 8 tokens per byte
Dibit
4
dibits (00,01,10,11)
real
2-bit tokens, 4 per byte
Pure Binary
2
bits (0, 1)
binary (-1/+1)
BITS ALL THE WAY DOWN
Why?
Traditional LLMs use tokenizers (BPE, SentencePiece) with 32k-256k vocabulary. This creates:
Tokenizer overhead and complexity
Language/domain bias baked into vocabulary
Preprocessing bottleneck
What if we eliminated tokenization entirely?
These models learn directly from raw binary data - no tokenizer, no preprocessing, just bytes flowing into neural networks. The ultimate goal: wire-speed learning where models absorb network traffic in real-time.
Results (Live Experiments - 16 Jan 2026)
Byte-Level (vocab=256)
Data: 350KB web crawl
BPB: 4.68 (vs 8.0 random = 41% compression)
Speed: 8.7 KB/s learning rate
Params: 0.6M
Learns HTML structure, XML tags, timestamps from raw bytes.
BITS ALL THE WAY DOWN - input bits, binary weights (-1/+1), output bits.
On specialized hardware, this enables XNOR+popcount operations instead of multiply-accumulate.
Architecture
All models use standard transformer architecture with:
Causal self-attention
GELU activation
LayerNorm
AdamW optimizer
Straight-Through Estimator (STE) for binary weight gradients
Key Innovation: Online Learning
Unlike traditional batch training, these models learn from streaming data:
Micro-batches (32-512 tokens)
Single-pass, no data curation
Real-time network stream compatible
Usage
Byte-Level
bash
1# Pipe any data source2cat data.bin | python byte_trainer.py
3curl -s http://example.com | python byte_trainer.py
4zcat crawl.jsonl.gz | python byte_trainer.py
