EspBPE-49K is a custom, hyper-optimized 49,152-vocabulary Byte-Level Byte Pair Encoding (BPE) tokenizer built specifically for modern Spanish Large Language Models (LLMs).

Trained across a massive multi-domain Spanish dataset mixture, EspBPE-49K achieves an extraordinary compression ratio (~4.8 to 5.1 characters per token), preserving contextual length and embedding space efficiency across formal encyclopedic text, technical documentation, sports commentary, and regional internet conversations.

⚡ Quickstart

You can load EspBPE-49K directly using the Hugging Face transformers library:

python

from transformers import AutoTokenizer

# Load EspBPE-49K directly from Hugging Face Hub
tokenizer = AutoTokenizer.from_pretrained("DinoResearch/EspBPE-49K")

# Sample text
text = "Estamos entrenando un tokenizer en español super rápido y eficiente con 49,152 de vocabulario! 🚀"

# Encode text into token IDs
input_ids = tokenizer.encode(text)
tokens = tokenizer.convert_ids_to_tokens(input_ids)

print(f"Total Tokens: {len(input_ids)}")
print("Tokens:", tokens)
print("Decoded:", tokenizer.decode(input_ids))

📊 Performance & Compression Ratio

Text Type	Character Count	Token Count	Compression Ratio
Sports Commentary	142 chars	28 tokens	~5.07 chars / token
Encyclopedic History (Wikipedia)	245 chars	49 tokens	~5.00 chars / token
General Spanish Sentence	107 chars	22 tokens	~4.86 chars / token

🗃️ Training Dataset Mixture

EspBPE-49K was trained on a carefully balanced multi-source Spanish corpus:

FineWeb-2 (spa_Latn): Modern, high-quality web crawl text.
Spanish Wikipedia (20231101.es): Encyclopedic knowledge baseline.
CulturaX (es): Filtered, clean multilingual web data (OSCAR + mC4).
Spanish Tweets (pysentimiento): Informal dialogue, regional accents, and modern social media slang.
PleIAs Spanish PD-Books: Classic literature and historical Spanish prose.

🌟 Key Highlights & Features

Single-Token Complex Vocabulary: Long formal words like inconstitucionalidad, extraordinariamente, descentralización, afortunadamente, and Rehabilitación map directly to single Token IDs.
Native Accent & UTF-8 Support: Seamless byte-level decoding for accented characters (á, é, í, ó, ú, ñ) without producing <unk> errors.
Native Emoji Primitives: High-frequency multi-byte emojis (e.g., 🤪, 😌, 🇵🇷, 🟢) are learned as native single-token lookup entries.
Regional & Technical Fluency: Optimized subword roots for tech terms (javascript, tensor, Unix, coaxial) alongside regional slang across Spanish-speaking countries.

📜 Model Card Details

Vocabulary Size: 49,152 (49k)
Model Type: ByteLevel BPE
Space Prefix: Ġ (ByteLevel space representation)
Organization: DinoResearch## ⚔️ Benchmark: EspBPE-49K vs. OpenAI o200k_base (GPT-4o)

To evaluate real-world web performance, EspBPE-49K was benchmarked head-to-head against OpenAI's o200k_base tokenizer (used in GPT-4o) across 500 unfiltered Spanish web documents (2,292,109 raw characters) from the FineWeb-2 (spa_Latn) dataset stream.

🏆 500-Text FineWeb-2 Benchmark Results

Metric	🦖 EspBPE-49K	🌐 o200k_base (GPT-4o)	Advantage / Delta
Vocabulary Size	49,152	200,019	~4x Smaller Table
Total Encoded Tokens	484,581	520,509	-35,928 Tokens (-6.90%)
Compression Ratio	4.730 chars/token	4.404 chars/token	+0.326 chars/token
Document Match Wins	441 / 500 (88.2%) 🥇	50 / 500 (10.0%)	Landslide Victory
Draws / Ties	9 / 500 (1.8%)	9 / 500 (1.8%)	—

🧠 Benchmark Key Takeaways

Massive Parameter Savings: EspBPE-49K achieves superior compression using less than 1/4th of o200k_base's vocabulary size, saving tens of millions of embedding parameters during pretraining.
6.90% Token Reduction: Encodes identical Spanish text into 6.90% fewer overall tokens, directly reducing memory requirements and accelerating autoregressive pretraining speed.
Extended Context Window: Higher token efficiency means more raw text fits into a fixed context length (e.g., 2048 / 4096 tokens).