A compact implementation of an Encoder-Decoder Transformer for sequence-to-sequence translation tasks. This project implements a translation model from English to Hindi using the Samanantar dataset.
Features
Encoder-Decoder Architecture: Full transformer implementation with separate encoder and decoder
Sinusoidal Positional Embeddings: Learnable position encoding for better sequence understanding
Multi-Head Attention: Self-attention and cross-attention mechanisms
Advanced Generation: Top-K sampling and beam search for text generation
Mixed Precision Training: Automatic mixed precision for faster training
Gradient Accumulation: Support for large effective batch sizes
Comprehensive Logging: WandB integration for experiment tracking
Architecture
Model Components
Encoder:
Multi-layer encoder blocks with self-attention
Sinusoidal positional embeddings
Layer normalization and feed-forward networks
Decoder:
Multi-layer decoder blocks with masked self-attention
The model configuration can be modified in config.py:
python
1@dataclass2classModelArgs:3 block_size:int=512# Maximum sequence length4 batch_size:int=32# Training batch size5 embeddings_dims:int=512# Model embedding dimensions6 no_of_heads:int=8# Number of attention heads7 no_of_decoder_layers:int=6# Number of decoder layers8 max_lr:float=6e-4# Maximum learning rate9# ... additional parameters
Usage
Training
python trainer.py
Web Application
Launch the interactive Gradio web interface:
python launch_app.py
The app will be available at http://localhost:7860 and provides:
Interactive Translation: English to Hindi translation interface
Multiple Generation Methods: Top-K sampling and beam search
