You can deploy this model and start earning money today!

AxiomicLabs/
GPT-X2-TOK-32K

tokenizer

bpe

fineweb

HuggingFaceFW/fineweb-edu

apache-2.0

Views

No views yet

GPT-X2-TOK-32k

A 32,768-vocabulary BPE tokenizer trained on 50B of FineWeb-Edu using the HuggingFace tokenizers library.

Key Details

Property	Value
Vocab Size	32,768
Type	Byte-level BPE
Training Data	50GB FineWeb-Edu (sample-100BT)

Why 32K?

~9% better compression on FineWeb-Edu compared to Llama's generic 32K tokenizer (810 vs 887 tokens on representative samples)
Smaller embedding table than 50K (GPT-2) — saves ~10M parameters that can be reinvested into transformer layers
Domain-optimized — trained specifically on educational web text, capturing common patterns in the training distribution

Usage

from transformers import AutoTokenizertokenizer = AutoTokenizer.from_pretrained("YOUR_USERNAME/gpt-tok-32k-fineweb")text = "The quick brown fox jumps over the lazy dog."tokens = tokenizer.encode(text)print(f"Tokens: {len(tokens)}")print(tokenizer.decode(tokens))

Training

Trained using the HuggingFace tokenizers library with:

Byte-level BPE
No normalization
ByteLevel pre-tokenizer with regex splitting
50GB of streaming FineWeb-Edu text

Used By

GPT-X2-125M-50BT — 125M parameter Llama-style language model trained on 50B tokens

AxiomicLabs/GPT-X2-TOK-32K