Bilingual Language Model
Overview
This project implements a bilingual (English-German) language model using LSTM (Long Short-Term Memory) neural networks. The model is trained on a combined corpus of English and German text, and can generate text in both languages.
Features
BPE (Byte Pair Encoding) tokenizer
LSTM-based language model
Training with checkpointing and resuming capability
Text generation in both English and German
Performance evaluation using perplexity
Requirements
Python 3.7+
PyTorch
NumPy
Matplotlib
tokenizers
we can install the required packages using:
pip install torch numpy matplotlib tokenizers
Usage
Data Preparation
First we combine English and German text files respectively.
Training the Model
To train the model, run this cell in the notebook:
model = LSTM(input_size, HIDDEN_SIZE, NUM_LAYERS, output_size).to(device)
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=LEARNING_RATE)
checkpoint_path = 'model_checkpoint.ckpt'
train_losses, val_losses, perplexities, training_time = train(model, train_loader, val_loader, criterion, optimizer, NUM_EPOCHS, device, checkpoint_path)
This code will:
Load and preprocess the data
Initialize the model
Train the model, saving checkpoints periodically
Plot the training curve and save it as results/training_curve.png
Save the training data as results/training_data.csv
Generating Text
To generate text using the trained model:
First run this cell to load the trained model:
def load_model(model_path, tokenizer, seed_text, num_chars, device):
model = LSTM(tokenizer.vocab_size(), HIDDEN_SIZE, NUM_LAYERS, tokenizer.vocab_size(), DROPOUT).to(device)
model.load_state_dict(torch.load(model_path), pickle_module=pickle)
model.eval()
return generate_text(model, tokenizer, seed_text, num_chars, device)
Finally run the text generation cells at the bottom of the notebook:
print("\nGenerating random English:")
random_model_e = LSTM(input_size, HIDDEN_SIZE, NUM_LAYERS, output_size).to(device)
print(generate_text(random_model_e, dataset.tokenizer, 20, device))
print("\nGenerating trained English:")
print(generate_text(model, dataset.tokenizer, 50, device, seed_text="Once upon a time"))
You can modify the seed text in this script.
Model Architecture
The model uses an LSTM architecture with the following hyperparameters:
Batch Size: 64
Sequence Length: 100
Hidden Size: 256
Number of Layers: 2
Learning Rate: 0.001
Number of Epochs: 50
Dropout: 0.2
Results
The model achieves a final perplexity of 1.1113 on the validation set, indicating excellent performance. However, this unusually low perplexity warrants further investigation to ensure it aligns with the specific nature of the task and data.
