Neural Network-based Next Token Prediction Model
Overview
This project implements a neural network-based model to predict the next token in a sequence using both English and Georgian datasets. The model utilizes LSTM layers for handling sequential text data and is designed to generate text by predicting the next word/token based on previous input.
Model Architecture
Embedding Layer: Captures word embeddings.
LSTM Layers: Two LSTM layers (150 and 100 units) to process sequential data.
Dense Layer: Fully connected layer with softmax activation for predicting the next token.
Datasets
Languages: English and Georgian text combined for multilingual prediction.
Preprocessing: Tokenization and padding applied to input sequences.
Training Details
Optimizer: RMSprop
Loss Function: Categorical Crossentropy
Training Epochs: 100
Batch Size: 16
Results
The model achieves a test accuracy of . Future work could include improving the dataset size and experimenting with different architectures to enhance performance.
