Views
No views yet
pip install --upgrade huggingface_hubhuggingface-cli loginhf_tokenizer/ (tokenizer.json + tokenizer_config.json). Generate them via:python3 scripts/export_hf_tokenizer.pyexport HF_REPO=arisin/hindi-bpe-tokenizer
huggingface-cli repo create $HF_REPO --token YOUR_TOKEN --type space --private false
mkdir -p build/$HF_REPO
cp hf_tokenizer/tokenizer.json build/$HF_REPO/
cp hf_tokenizer/tokenizer_config.json build/$HF_REPO/
cp hf_tokenizer/special_tokens_map.json build/$HF_REPO/
cp docs/huggingface_guide.md build/$HF_REPO/README.md
cp notebooks/hf_tokenizer_demo.ipynb build/$HF_REPO/cd build/$HF_REPO
git init
git add .
git commit -m \"Add Hindi BPE tokenizer\"
git remote add origin https://huggingface.co/$HF_REPO
git push -u origin mainhttps://huggingface.co/$HF_REPO.1from transformers import AutoTokenizer
2
3tokenizer = AutoTokenizer.from_pretrained(\"arnabsinha/hindi-bpe-tokenizer\")
4encoded = tokenizer(\"वाराणसी की सुबह अद्भुत होती है।\")
5print(encoded)
6print(tokenizer.decode(encoded[\"input_ids\"]))python -m transformers.models.auto.tokenization_auto \
--tokenizer arnabsinha/hindi-bpe-tokenizer \
--text \"मुंबई की लोकल ट्रेनें लाखों सपने ले जाती हैं।\"notebooks/hf_tokenizer_demo.ipynb can be uploaded alongside the tokenizer or converted to a README example to guide notebook-based exploration.scripts/export_hf_tokenizer.py after retraining.git tag v1.1 before pushing for versioning.