Bengali BPE Tokenizer

Dataset

Steps

1. Followed the instructions from the video from Andrej Karpathy and created the notebook for experiment.
2. Experimented with the regular expression that suits bengali language. The intention was to the regular expression for splitting Bengali words instead of characters.
   • Using gpt2 regex 's|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+ was resulting in splitting of individual characters instead of words.
   • Used the regex ?\p{Bengali}+| ?[^\s\p{Bengali}]+|\s+(?!\S)|\s+ that could split the sentence "সবাই যা করতে চায় তা করতে চায়নি।" into following words 'সবাই', ' যা', ' করতে', ' চায়', ' তা', ' করতে', ' চায়নি', '।'
3. Updated BPE training process to use text chucks as output from the regular expression splits instead of the complete sentences. This helps avoid merging of tokens across different worlds. Ref
4. Updated encode and decoder function to deal with text chucks instead of complete sentences. Ref

Tokenizer training

1. Experimented with the regular expression that suits bengali language. The intention was to the regular expression for splitting Bengali words instead of characters.
   • Using gpt2 regex 's|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+ was resulting in splitting of individual characters instead of words.
   • Used the regex ?\p{Bengali}+| ?[^\s\p{Bengali}]+|\s+(?!\S)|\s+ that could split the sentence "সবাই যা করতে চায় তা করতে চায়নি।" into following words 'সবাই', ' যা', ' করতে', ' চায়', ' তা', ' করতে', ' চায়নি', '।'
2. Updated BPE training process to use text chucks as output from the regular expression splits instead of the complete sentences. This helps avoid merging of tokens across different worlds. Ref
3. Updated encode and decoder function to deal with text chucks instead of complete sentences. Ref
4. Trained the tokenizer to reach vocab size of 5001 and compression of 11X
5. Saved vocab file (contains the mapping from tokens to bengali text), merges files (contains the mapping from pair of tokens to be merged to token after merging) and the regular expression that is used for splitting the bengali sentences. All these artifacts are required to perform BPE tokenization on a new text.

The HuggingFace Spaces Gradio App

1. Numeric tokens that represent the sentence (using encode function)
2. Regenerated sentence using the tokens (using decode function)
3. A visualization for each token to Bengali text mapping as explanation for the tokenization.