You can deploy this model and start earning money today!

OpenTransformer/
purebit-transformer

Views

No views yet

PureBit Transformer

A transformer that operates on raw binary bits instead of tokens.

Architecture

Vocab size: 2 (just 0 and 1!)
d_model: 256
Layers: 6
Heads: 8
Parameters: ~18M

Training

Trained on raw UTF-8 bytes converted to bits
Best loss achieved: 0.6863 (random = 0.693)
Training data: ~70MB of text = 560M bits

Key Insight

This explores whether transformers can learn at the bit level. Results show minimal learning beyond random - predicting individual bits is extremely hard without byte-level structure.

Usage

python

import torch

# Load checkpoint
ckpt = torch.load('purebit_best_70mb.pt')
print(f"Loss: {ckpt['loss']:.4f}")
print(f"Bits seen: {ckpt['bits']:,}")

# Model architecture in model.py

Files

purebit_best_70mb.pt - Best checkpoint (loss 0.6863)
model.py - Model architecture
train.py - Training script
infer.py - Inference script

Author

OpenTransformers - Experimental architecture research

OpenTransformer/purebit-transformer