You can deploy this model and start earning money today!

andre15silva/
pt-es-hi-tokenizer

Name: pt-es-hi-tokenizer
Availability: InStock

tokenizer

byte-level-bpe

multilingual

Views

No views yet

pt-es-hi-tokenizer

A shared multilingual byte-level BPE tokenizer (GPT-2/Llama-3 style) for Portuguese, Spanish, and Hindi — one vocabulary, not per-language, intended for a single multilingual pre-training run rather than three separate models.

Vocab size: 32,000, chosen via the Kneedle algorithm (point of maximum curvature on the compression-vs-vocab-size curve) over a candidate sweep of 8k/16k/32k/50k/65,536/100k, rather than a hand-picked threshold.
Byte-level pre-tokenization: every input string is representable with no OOV/<unk> fallback, which matters since the training corpus mixes Devanagari (Hindi) and Latin (Portuguese/Spanish) scripts.
Special tokens follow the Llama-3 chat convention (<|begin_of_text|>, <|end_of_text|>, <|pad|>, <|start_header_id|>/<|end_header_id|>/<|eot_id|>), so the tokenizer is chat-fine-tuning-ready without resizing the embedding table later.

Training data

Trained on a MinHash-deduplicated corpus of pt-fineweb2, es-fineweb2, hi-fineweb2, and hi-sangraha (ai4bharat/sangraha's "verified" split), deduplicated per-language with datatrove (hi's two sources deduplicated against each other, not just internally).

Usage

python

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("andre15silva/pt-es-hi-tokenizer")
tok.encode("Olá, ¿cómo estás? नमस्ते")

Evaluation

Fertility (tokens/word, lower is better) and compression (bytes/token, higher is better) on a disjoint held-out sample:

lang	fertility	compression
pt	1.459	4.332
es	1.366	4.552
hi	3.247	3.951
overall	1.773	4.264

Hindi's fertility is notably higher than pt/es — consistent across every scale this was measured at (pilot, pre-dedup, and this final run), pointing to Hindi being comparatively data-starved in the training corpus rather than a tokenizer-vocab problem.