This repository contains trained checkpoints, benchmark logs, and evaluation reports for 4 distinct ~50M parameter language model architectures trained from scratch on the Amharic Wikipedia corpus using an RL-trained subword tokenizer (3,919 tokens).

📊 Models & Architectures

Architecture	Model Class	Parameters	Distinct Features
Qwen3.5 (Transformer)	`TransformerLM`	50.79M	3:1 Gated DeltaNet / Gated Attention, Zero-Centered RMSNorm, Partial RoPE (`0.25`), Multi-Token Prediction (MTP) Head
HRM-Text	`HRMLM`	49.02M	Dual-timescale `H2L3` Recurrence, MagicNorm parameterless RMSNorm, PrefixLM masking, Warmup TBPTT ($2 \rightarrow 5$)
Mamba SSM	`MambaLM`	49.47M	Selective State Space Model, `dt_rank=32`, S4D $A_{\text{log}}$, specialized log-uniform $\Delta$ initialization
Hybrid Mamba-Transformer	`HybridMambaTransformerLM`	52.04M	Interleaved 2:1 Mamba to Qwen3.5 Attention, Unified SwiGLU FFN, Idempotent Depth-Scaling

📁 Repository Structure

├── models/                     # PyTorch architecture implementations
│   ├── transformer_lm.py       # Qwen3.5 3:1 DeltaNet + Attention + MTP
│   ├── hrm_lm.py               # HRM-Text H2L3 recurrence
│   ├── mamba_lm.py             # Mamba selective SSM
│   └── hybrid_lm.py            # Hybrid Mamba + Qwen attention
├── tokenizer/                  # Amharic subword tokenizer
│   ├── vocab.txt
│   └── config.json
├── checkpoints/                # Best model weights for each architecture
│   ├── transformer/best_model.pt
│   ├── hrm/best_model.pt
│   ├── mamba/best_model.pt
│   └── hybrid/best_model.pt
└── analysis/                   # Comparative benchmark results & plots
    ├── report.md
    ├── results_table.tex
    ├── loss_curves.png
    ├── throughput_scaling.png
    └── pareto_frontier.png

🚀 How to Load and Use

python

import torch
from models import create_model

# Load model architecture
model = create_model("transformer", vocab_size=3919)

# Load checkpoint
checkpoint = torch.load("checkpoints/transformer/best_model.pt", map_location="cpu")
model.load_state_dict(checkpoint["model_state"], strict=False)
model.eval()

# Generate tokens
tokens = torch.tensor([[2, 45, 128, 902]], dtype=torch.long)
with torch.no_grad():
    logits, _ = model(tokens)
    next_token = torch.argmax(logits[:, -1, :], dim=-1)
print("Next token ID:", next_token.item())

📜 Citation & Attribution

If you use these models or comparative benchmarks in your research, please cite:

Qwen3.5: Alibaba Qwen Team (2025/2026)
HRM-Text: Sapient Intelligence (Wang et al., 2026)
Mamba: Gu & Dao (2023)

amanuelbyte/amharic-50m-architecture-benchmark

50M-Class Comparative Architecture Benchmark (Amharic)

📊 Models & Architectures

📁 Repository Structure

🚀 How to Load and Use

📜 Citation & Attribution

amanuelbyte/
amharic-50m-architecture-benchmark