Neural Network-Based Language Model for Next Token Prediction
Project Overview:
This project involves the development of a Neural Network-based Language Model for next token prediction. The task is to create a model that can predict the next token (word or character) given a sequence of previous tokens. The model will be trained using two languages: English and Finnish. The main goal is to explore how neural networks can be applied to natural language processing tasks without relying on transformer-based architectures.
This repository contains a Jupyter Notebook implementation of a neural network-based language model designed for next token prediction. The model is trained using both English and assigned language(Finnish), leveraging a custom tokenizer and embedding model.
Dataset Description
The datasets used for training are corpora in English and Finnish. For English
https://huggingface.co/datasets/yahma/alpaca-cleaned was used, and for Finnish
https://huggingface.co/datasets/saillab/taco-datasets/tree/main/multilingual-instruction-tuning-dataset%20/multilingual-alpaca-52k-gpt-4/Finnish was used.
Each dataset was tokenized using a custom tokenizer, and preprocessing included lowercasing, removing punctuation, and handling out-of-vocabulary (OOV) words. The data was then split into training, validation, and test sets.
Features
- Training with Multiple Languages: The model supports training in English and an additional language(Finnish).
- Custom Tokenizer: A tokenizer appropriate for the project requirements.
- Model Architecture: The implementation utilizes Neural Networks (NN), Recurrent Neural Networks (RNN), or Long Short-Term Memory Networks (LSTM).
How to Use
- Clone the repository:
1git clone <repository_url>
2cd <repository_directory>
Install the required packages: Ensure you have the necessary libraries installed. Can be done by running:
pip install -r requirements.txt
Run the Jupyter Notebook: Open the notebook in Jupyter:
jupyter notebook SuryaManjri.ipynb
Generate Text: Follow the instructions in the notebook to generate text using the trained model in both languages.
Checkpoints
The model includes checkpoint implementation to save progress during training, allowing for recovery and continued training.
Video Demo
A video demonstration of the model’s text generation capabilities can be found here
