You can deploy this model and start earning money today!

chhatramani/
nyayalm-tokenizer-64k_v3

Views

No views yet

NyayaLM 64K Tokenizer

Domain-specific bilingual (Nepali + English) SentencePiece-BPE tokenizer for the NyayaLM-200M legal-domain language model.

Vocab size: 64 000
Algorithm: SentencePiece BPE + byte_fallback
Special tokens: Qwen3 / ChatML style (high IDs)
EOS: <|im_end|>
PAD: <|endoftext|>
Chat template: Qwen3-compatible

Quick use

python

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("YOUR_HF_REPO")
print(tok.apply_chat_template(
    [{'role': "user", "content": "नेपालको संविधान के हो?"}],
    tokenize=False,
    add_generation_prompt=True,
))

chhatramani/nyayalm-tokenizer-64k_v3

NyayaLM 64K Tokenizer

Quick use

chhatramani/
nyayalm-tokenizer-64k_v3