You can deploy this model and start earning money today!

DumbsterDrekk/
hindi-tokenizer

Name: hindi-tokenizer
Availability: InStock

Views

No views yet

Hindi-English BPE Tokenizer

Load and use the pre-trained Byte-Pair Encoding (BPE) tokenizer directly from Hugging Face Hub under DumbsterDrekk/hindi-tokenizer.

pip install transformers

(If your repo is private, also install and login to huggingface_hub)

bash

pip install huggingface_hub
huggingface-cli login

Set your HF token (if private repo):

export HF_TOKEN="hf_yourPersonalAccessToken"
Load the tokenizer:

python
1import os 2from transformers import PreTrainedTokenizerFast 3 4hf_token = os.getenv("HF_TOKEN") # or None for public repo 5 6tokenizer = PreTrainedTokenizerFast.from_pretrained( 7 "DumbsterDrekk/hindi-tokenizer", 8 use_auth_token=hf_token, # omit if repo is public 9 unk_token="<unk>", 10 eos_token="</w>", 11 pad_token="<pad>", 12)
Tokenize text:

python
1text = "मैं आज बहुत खुश हूँ। I’m very happy today." 2 3# Single sequence 4enc = tokenizer( 5 text, 6 padding="max_length", 7 truncation=True, 8 max_length=512, 9 return_tensors="pt", # or omit for Python lists 10) 11 12print("Input IDs: ", enc["input_ids"]) 13print("Attention Mask:", enc["attention_mask"])
Decode tokens back to text:

python
1decoded = tokenizer.decode(enc["input_ids"][0], skip_special_tokens=True) 2print("Decoded Text: ", decoded)

Once loaded, this tokenizer can be plugged into any 🤗 Transformers training or generation pipeline exactly like a built-in model tokenizer.