Natural Language Processing Midterm Project

Project Overview

Contents of the Project

1. Data Handling:
   • The dataset is read and encoded for model training.
   • The dataset includes an English-Basque parallel corpus.
2. Hyperparameter Setup:
   • Hyperparameters include batch size, sequence length, number of iterations, evaluation intervals, learning rate, and embedding sizes.
3. Model Definition:
   • The project defines a simple RNN-based language model using LSTMs and fully connected layers.
   • The model is trained to predict the next character in a sequence.
4. Training and Evaluation:
   • The training loop iterates for 200,000 steps, saving checkpoints periodically.
   • Both training and validation losses are logged and saved.
5. Text Generation:
   • A function is provided to generate text from the trained model, starting from a prompt (like "Once upon a time").
   • Text is generated in both English and Basque languages.
6. Plotting and Saving Results:
   • Training and validation loss curves are plotted and saved as a PNG file.
   • Training loss data is saved to a CSV file for further analysis.

Key Steps for Running the Notebook

1. Import Required Libraries:
   • The project relies on PyTorch, Pandas, and Matplotlib.
   python

   1import torch 2import pandas as pd 3import matplotlib.pyplot as plt
2. Data Preprocessing:
   • The dataset is read and tokenized using character encoding.
   python

   1with open('/content/eng_basque.txt', 'r', encoding='utf-8') as f: 2 text = f.read() 3 4encoded_data = torch.tensor(encode(text), dtype=torch.long)
3. Model Training:
   • The SimpleRNNModel is trained using the provided dataset.
   python

   1model = SimpleRNNModel().to(device) 2optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate)
4. Model Evaluation:
   • Evaluation functions and loss estimation are defined.
   python

   1def estimate_loss(): 2 # Function to estimate train/val losses
5. Text Generation:
   • After training, text can be generated using the trained model.
   python

   1start_text = "Once upon a time" 2generated_text = generate_text(loaded_model, start_text, max_new_tokens=500) 3print(generated_text)

Generated Results

• English Text Example:
  Once upon a time...
• Basque Text Example:
  Gaur egun...

Requirements

• Python 3.7+
• PyTorch
• Pandas
• Matplotlib

How to Run the Code

1. Clone the project or open the notebook in Google Colab.
2. Install the necessary dependencies.
   pip install torch pandas matplotlib
3. Upload the dataset to the notebook and run the cells in sequence.
4. After training, generated text and results will be available.

Conclusion

Author

• Giri Chandragiri - October 12, 2024