You can deploy this model and start earning money today!

MSkilll/
Alpha55

Name: Alpha55
Rating: 2.5 (14 reviews)

safetensors

encoder-decoder-transformer

Views

No views yet

Alpha Transformer

A custom Encoder-Decoder Transformer model trained from scratch for English → French translation.

Alpha is a 51M parameter Transformer implemented entirely in PyTorch, including:

Multi-Head Attention
Rotary Positional Embeddings (RoPE)
Encoder-Decoder architecture
Cross Attention
Custom tokenizer
Mixed precision training
Checkpointing system

The model was trained on the OPUS-100 English-French translation dataset.

Model Overview

Feature	Value
Architecture	Encoder-Decoder Transformer
Parameters	51.26M
Framework	PyTorch
Task	English → French Translation
Dataset	OPUS-100
Training Samples	~1M sentences
Embedding Dimension	256
Attention Heads	8
Feed Forward Dimension	1024
Encoder Layers	3
Decoder Layers	3
Maximum Sequence Length	64 tokens
Positional Encoding	RoPE

Performance

Evaluation on the OPUS-100 test set:

Metric	Result
Token Accuracy	62.8%

The model was trained from scratch without using pretrained weights.

Architecture

Encoder

Each encoder block contains:

Decoder

Each decoder block contains:

Files

Alpha/ │ ├── config.json │ └── Model architecture configuration │ ├── model.safetensors │ └── Trained model weights │ ├── src_vocab.json │ └── English vocabulary │ ├── tgt_vocab.json │ └── French vocabulary │ ├── tokenizer.py │ └── Custom tokenizer implementation │ ├── main.py │ └── Transformer architecture │ ├── inference.py │ └── Translation script │ └── README.md

Installation

Clone the repository:

bash

git clone https://github.com/MehdiSkilll24/Alpha.git
cd Alpha

Install dependencies:

pip install torch safetensors

Usage

Run inference:

python inference.py

Example:

Input:
> hello, how are you?

Output:
> bonjour, comment allez-vous ?



Loading the Model Manually
import torch
import json
from safetensors.torch import load_file
from main import Transformer

with open("config.json") as f:
    config = json.load(f)

weights = load_file("model.safetensors")

model = Transformer(
    src_vocab_size=config["src_vocab_size"],
    tgt_vocab_size=config["tgt_vocab_size"],
    d_model=config["d_model"],
    num_heads=config["num_heads"],
    d_ff=config["d_ff"]
)

model.load_state_dict(weights)
model.eval()


Training

The model was trained using:

AdamW optimizer
Mixed precision training (FP16)
ReduceLROnPlateau scheduler
Custom PyTorch training loop

Training configuration:

Learning Rate: 3e-4
Batch Size: 32
Epochs: 18
Dataset

Training data:

OPUS-100 English-French translation dataset.

The dataset contains parallel English/French sentence pairs collected from various sources.

Limitations

This model is designed as an educational and experimental Transformer implementation.

Limitations:

Vocabulary is based on a custom word-level tokenizer
Limited context length
No pretrained initialization
Smaller dataset compared to production translation systems
Token accuracy does not directly represent BLEU score
Future Improvements

Possible improvements:

Replace word tokenizer with BPE/SentencePiece
Increase model depth
Train on larger datasets
Add beam search decoding
Evaluate with BLEU / COMET
Implement KV-cache for faster inference
Motivation

Alpha was built to understand and implement the Transformer architecture from first principles.

Rather than using existing libraries, the project implements the core components manually:

Attention calculations
Positional embeddings
Encoder/decoder blocks
Training pipeline
Model serialization

The goal is to better understand how modern language models are built internally.

Source Code

GitHub:

https://github.com/MehdiSkilll24/Alpha

Model weights:

https://huggingface.co/MehdiSkilll24/Alpha

License

MIT License


One suggestion before you upload: add **3-5 real translation examples** from your model. That will make the HF page look 10x more alive. A model card with "62.8% accuracy" is okay; a model card with:


English:
"The weather is beautiful today."

Alpha:
"Le temps est magnifique aujourd'hui."

MSkilll/Alpha55