This repository contains a Unigram tokenizer trained from scratch on a parallel English-Hindi corpus. It was developed as part of a research project to analyze the impact of different subword tokenization strategies (BPE, WordPiece, and Unigram) on the performance, training stability, and inference latency of a Neural Machine Translation (NMT) model.

Research Finding: This Unigram tokenizer demonstrated high efficiency and stability, performing on par with WordPiece and proving to be a vastly superior alternative to BPE for this task. It enabled stable model training and is expected to yield a model with low inference latency and high-quality translations.

This is a PreTrainedTokenizerFast object, wrapping a tokenizer from the tokenizers library.

This tokenizer is intended to be used with a Transformer-based model for English-to-Hindi machine translation. It is designed to produce efficient, semantically meaningful subword units that lead to stable training and fast inference.

You can load and use this tokenizer directly in your Python code using the transformers library.