You can deploy this model and start earning money today!

N1K0L0Z/
vanilla-transformer-georgian-english

Name: vanilla-transformer-georgian-english
Availability: InStock

Darsala/english_georgian_corpora

1706.03762

mit

Views

No views yet

Vanilla Transformer — Georgian ⇄ English Machine Translation from Scratch

A bidirectional Georgian ↔ English neural machine translation model, built by implementing the "Attention Is All You Need" (Vaswani et al., 2017) Transformer architecture from first principles in PyTorch — no torch.nn.TransformerEncoder, no pretrained weights, no HuggingFace model classes.

Trained entirely from random initialization for educational purposes: a faithful reproduction of the original 2017 Transformer, with two small, deliberate modern departures (GELU activation, pre-LayerNorm) for training stability.

Model Details

Architecture: Encoder-decoder Transformer, hand-implemented (Linear, Embedding, LayerNorm, multi-head self-/cross-attention, sinusoidal positional encoding)
Modifications from the original paper: GELU instead of ReLU; Pre-LayerNorm instead of Post-LayerNorm (the modern default since GPT-2, for more stable training)
Directionality: A single model handles both directions via a source-side tag (2en / 2ka) and shared encoder-decoder parameters — not two separate models
Training hardware: Single NVIDIA L4 GPU, no pretraining
Loss: Label-smoothed cross-entropy — final training loss ~4.5 (label-smoothed), equivalent to ~3.7 without smoothing

Dataset & Tokenization

Corpus: Darsala/english_georgian_corpora
Tokenizer: Joint SentencePiece BPE tokenizer (~32,000 vocabulary), shared across both languages, trained on the parallel corpus

Evaluation

Scored with COMET (wmt22-comet-da) on a held-out test split, greedy decoding:

Direction	COMET
Georgian → English	0.58
English → Georgian	0.54

Scores in this range reflect moderate translation quality — reliable on common, everyday sentence structures, with an expected drop-off on rare vocabulary and specialized domains, given the model was trained entirely from scratch with no pretraining.

Usage

First, install the model architecture package directly from GitHub:

pip install git+[https://github.com/N1k0l0z/Vanilla_Transformer.git](https://github.com/N1k0l0z/Vanilla_Transformer.git)

python

import torch
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer

# 1. Import model architecture and configuration
from Scripts.Transformer import TranslationModel
from Scripts.ModelConfig import Config

# 2. Download assets from Hugging Face Hub
REPO_ID = "N1k0l0z/vanilla-transformer-georgian-english"

model_path = hf_hub_download(repo_id=REPO_ID, filename="model.pt")
tokenizer_path = hf_hub_download(repo_id=REPO_ID, filename="tokenizer.json")

# 3. Load tokenizer and PyTorch checkpoint
tokenizer = Tokenizer.from_file(tokenizer_path)
checkpoint = torch.load(model_path, map_location="cpu", weights_only=False)

# 4. Initialize model and load trained weights
model = TranslationModel(checkpoint["config"])
model.load_state_dict(checkpoint["model_state_dict"])
model.eval()

# 5. Translate!
print(model.translate("ვინ ხარ?", tokenizer, direction="2en"))  
# Output: "Who are you?"

print(model.translate("Where is your house?", tokenizer, direction="2ka"))
# Output: "სად არის შენი სახლი?" 

Examples

Georgian → English

Input	Output
ვინ ხარ?	Who are you?
სად არის შენი სახლი?	Where is your house?
გამარჯობა, როგორ ხარ?	How are you?
დღეს კარგი ამინდია.	It is a good weather today.
ყავა ძალიან მიყვარს.	I love coffee.

English → Georgian

Input	Output
Where is the house?	სად არის სახლი?
Hello, nice to meet you.	კარგი, ძალიან კარგად გამოიყურები.
This project is built from scratch.	ეს პროექტი აშენებულია.
I am learning how transformers work.	მე როგორ მსურს, როგორ მუშაობს სამუშაო.
See you tomorrow morning.	ხვალ დილით წადი.

Limitations

No pretraining — quality is bounded by what a from-scratch model can learn from the training corpus alone
Weaker on rare, technical, or specialized vocabulary due to limited coverage in training data
Best suited for everyday, conversational language rather than domain-specific translation

Citation

If referencing the original architecture:

Vaswani, A. et al. (2017). Attention Is All You Need. NeurIPS.

N1K0L0Z/vanilla-transformer-georgian-english