BPE tokenizer trained on Spanish Wikipedia using SentencePiece.
Details
Vocab size: 64,000
Model type: BPE
Training data: Spanish Wikipedia (100K articles)
Character coverage: 99.95%
Byte fallback: enabled
Usage
python
1import sentencepiece as spm
23sp = spm.SentencePieceProcessor()4sp.load("tokenizer.model")56text ="El procesamiento de lenguaje natural es fascinante."7tokens = sp.encode_as_pieces(text)8print(tokens)
Why?
English-centric tokenizers (GPT-4, Llama) use 20-40% more tokens on Spanish text.
This tokenizer is optimized for Spanish, giving better compression and cheaper inference.
