You can deploy this model and start earning money today!

MultilingualUnigramLM/
las-nl-tokenizers-yi-1_5-6b-v64000-tur

Name: las-nl-tokenizers-yi-1_5-6b-v64000-tur
Availability: InStock

Views

No views yet

Base + Language-Specific LangMAP — yi-1_5-6b × tur_Latn

Unsupervised tokenization specialised for tur_Latn, derived from the yi-1_5-6b base BPE tokenizer using the LangMAP framework.

This repository bundles:

Inference uses base + language-specific scores together (the LangMAP variant); do not use the bare overlay or base on its own.

Trained from job smoke.tur.yi-1_5-6b.v64000 (vocab=64000, langs=[tur_Latn], iters=5, em_mode=soft, byte_fallback=True, seed-fix applied).

python

from tokenizers import Tokenizer
tok = Tokenizer.from_file("tokenizer.json")