India is a vast country that has a multi-lingual culture that covers 22 Official languages and more than 1700 languages and dialects. It has been observed
that various languages share words among themselves, sometimes even across language families. To capitalize on this observation, we trained our tokenization model
with a vocabulary size of 160,000 (160k) using the dataset of Wikipedia articles in 18 Indic languages and English by applying the Byte-Pair Encoding (BPE) algorithm.
When compared among all the popular open-source tokenizers trained on multilingual Indic languages on fertility scores, our model outperformed them in most of the languages,
with significant performance improvement in Sanskrit (sa), Kashmiri (ks), Sindhi (sd) and Konkani (gom).
Use the code below to get started with the model.
1from transformers import AutoTokenizer
2try:
3 tokenizer = tokenizer = AutoTokenizer.from_pretrained("LingoIITGN/Ansh-160k"))
4 print("Tokenizer loaded successfully!")
5except Exception as e:
6 print(f"Error loading tokenizer: {e}")
7 print("Please ensure you have the correct model name and are connected to the internet.")
8 exit()
9
10
11input_text = "Hello, world! This is an example of how to use the tokenizer."
12#input_text = 'मुझे यह presentation कल morning तक submit करना है। '
13#input_text = 'What is capital city of India?'
14
15encoded_input = tokenizer.encode(example_text)
16print("\nOriginal Text:", example_text)
17print("Encoded (Token IDs):", encoded_input)
18
19decoded_output = tokenizer.decode(encoded_input)
20print("Decoded Text:", decoded_output)
21
