Bilingual Language Model with BPE Tokenizer

What's in this project?

1. BPE Tokenizer: I made a special tool called a Byte Pair Encoding (BPE) tokenizer. It's like a smart dictionary that breaks words into smaller pieces, which helps the model understand language better.
2. LSTM Language Model: I used something called an LSTM (Long Short-Term Memory) neural network. It's a fancy way of saying the model can remember important stuff from earlier in a sentence to help predict what comes next.
3. Bilingual Support: The coolest part is that this model can work with both English and Chinese! It's like having a brain that can switch between two languages.

How it works

1. The BPE tokenizer learns to split words into subwords. For example, "playing" might become "play" + "ing".
2. The LSTM model then learns patterns in these subwords to predict what comes next in a sentence.
3. When we want to generate text, the model uses what it learned to create new sentences in either English or Chinese.

What I learned

• How to implement a BPE tokenizer from scratch (it was tricky but fun!)
• Training a neural network on a bilingual dataset
• Dealing with perplexity (a way to measure how good the model is at predicting text)
• Generating text in two very different languages

Results

Next steps