You can deploy this model and start earning money today!

chetanpun/
devanagari-english-bpe-tokenizer

Name: devanagari-english-bpe-tokenizer
Availability: InStock

Views

No views yet

Nepali-English Tokenizer

This repository contains a custom tokenizer built to handle both Nepali (Devanagari) and English languages, facilitating seamless tokenization for NLP tasks that involve code-switching or translation between these two languages.

Features

Multilingual Support: This tokenizer is specifically designed for both Nepali and English, capable of handling complex scripts like Devanagari alongside the Roman alphabet.
Bidirectional Encoding: Efficiently encodes and decodes both languages for tasks such as translation, language modeling, and sequence tagging.
Custom Vocab: The vocabulary is tailored for common words and tokens in both Nepali and English, making it optimal for tasks involving both languages.

Installation

You can easily install and use the tokenizer with Hugging Face’s transformers library.

pip install transformers

To use the tokenizer, load it directly from Hugging Face:

bash

from transformers import PreTrainedTokenizerFast

tokenizer = PreTrainedTokenizerFast.from_pretrained('chetanpun/devanagari-english-bpe-tokenizer')

To tokenize text in either Nepali or English:

bash

nepali_text = "मेरो नाम चेतन हो।"

english_text = "My name is Chetan."

mix_text = "Hello मेरो नाम चेतन हो।"

nepali_tokens = tokenizer(nepali_text)
english_tokens = tokenizer(english_text)

print("Nepali Tokens:", nepali_tokens)
print("English Tokens:", english_tokens)

You can encode text into token IDs and decode them back:

bash

input_ids = tokenizer.encode(nepali_text, return_tensors="pt")

decoded_text = tokenizer.decode(input_ids[0], skip_special_tokens=True)

print("Decoded Text:", decoded_text)