This repository contains a Byte-Pair Encoding (BPE) tokenizer trained from scratch on a parallel English-Hindi corpus. It was developed as part of a research project to analyze the impact of different subword tokenization strategies (BPE vs. WordPiece) on the performance, training stability, and inference latency of a Neural Machine Translation (NMT) model.

Research Finding: This BPE tokenizer was found to be computationally inefficient for the task, leading to training failures and poor translation quality (BLEU score: 1.99). It serves as a benchmark to demonstrate the superiority of more advanced tokenizers like WordPiece for this specific use case.

This is a PreTrainedTokenizerFast object, wrapping a tokenizer from the tokenizers library.

This tokenizer is intended to be used with a Transformer-based model for English-to-Hindi machine translation. Its primary use in our research was to serve as a baseline to highlight the performance gains achieved with a more efficient WordPiece tokenizer.

You can load and use this tokenizer directly in your Python code using the transformers library.