You can deploy this model and start earning money today!

Gpeik/
Sophira-tokenizer-64k-v0

Name: Sophira-tokenizer-64k-v0
Availability: InStock

Views

No views yet

Sophira-tokenizer-64k-v0

Sophira-tokenizer-64k-v0 is the first frozen tokenizer artifact for the Sophira project. It is intended for Italian decoder-only language model pretraining.

Artifact Summary

Public artifact family: Sophira
Internal tokenizer name: ItalianFM-tokenizer
Vocabulary size: 64000
Tokenizer type: huggingface-tokenizers
Training corpus version: Corpus v0.1 (80% CulturaX Italian subset, 20% PleIAs/Italian-PD)
Tokenizer sample manifest: tokenizer-production-1b.manifest.json
Byte fallback: True
Special tokens: ['<bos>', '<eos>', '<unk>', '<pad>']

Corpus Recipe

Sources: uonlp/CulturaX Italian subset and PleIAs/Italian-PD
Tokenizer-training corpus: Corpus v0.1
Character mixture: 80/20 (CulturaX / Italian-PD)
Production tokenizer sample target: 1,000,000,000 normalized characters

Evaluation

This tokenizer was evaluated on held-out slices sampled from the same corpus recipe used for tokenizer development, with no overlap with the released artifact files.

Evaluation data:

uonlp/CulturaX Italian subset
PleIAs/Italian-PD
held-out slices totaling about 10M characters per evaluation slice

Compared baselines:

sapienzanlp/Minerva-3B-base-v1.0
Musixmatch/umberto-commoncrawl-cased-v1
gsarti/it5-efficient-small-el32
sapienzanlp/modello-italia-9b

Qualitative checks covered:

accented Italian
apostrophes and punctuation
literary-register Italian
rare Unicode edge cases

Main result:

Sophira-tokenizer-64k-v0 achieved the best compression among the successfully evaluated baselines on the held-out Italian slices:
- mean tokens/character: 0.195306
- mean tokens/word: 1.264066

Benchmark Summary

Tokenizer	Mean Tokens/Char	Mean Tokens/Word	All Round-Trip Equal
Sophira-64k	0.195306	1.264066	no
it5-efficient-small-el32	0.215350	1.393791	no
UmBERTo-commoncrawl-cased-v1	0.216695	1.402497	no
modello-italia-9b	0.224812	1.455031	no
Minerva-3B-base-v1.0	0.240413	1.556008	no

Known Limitations

The tokenizer is optimized for Italian pretraining text, not arbitrary multilingual or symbolic text.
Rare Unicode symbols and out-of-domain scripts may still degrade relative to in-domain Italian text.

Reproducibility

tokenizer_metadata.json records the frozen artifact metadata.
tokenizer_training_config.json records the training configuration.
tokenizer_release_manifest.json records checksums and artifact references.
Evaluation report: tokenizer_evaluation_report.md

License

This tokenizer artifact is released under apache-2.0.

Citation

If you use this tokenizer in academic work, please cite it as follows:

bibtex

@misc{peikos2026sophiratokenizer64kv0,
  title        = {Sophira-tokenizer-64k-v0},
  author       = {Peikos, Georgios},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Gpeik/Sophira-tokenizer-64k-v0}},
}