You can deploy this model and start earning money today!

oddadmix/
Emhotob-5M-MSA-Egyptian-v1

Name: Emhotob-5M-MSA-Egyptian-v1
Availability: InStock

Views

No views yet

Emhotob-5M-MSA-Egyptian-v1 — Bidirectional MSA ↔ Egyptian Arabic (~5M params)

A 5.08M-parameter model that translates both ways between Modern Standard Arabic (الفصحى) and Egyptian colloquial Arabic (المصرية العامية). A single set of weights serves both directions; a direction-specific Arabic system prompt selects which way to translate.

Finetuned from oddadmix/Emhotob-5M, a tiny Llama-architecture base (hidden size 128, 5 layers, 4 heads, tied embeddings).

Where translation "turns on." This model is part of a scaling study that runs the exact SFT + evaluation recipe of oddadmix/50M-MSA-Egyptian-v1 across shrinking base models. At 500K and 1M params the model collapses to a degenerate repeated token (BLEU ≈ 0). At 5M it genuinely translates — roughly, with drift and repetition, but with correct register-switching and real lexical choices. It is a working demonstration, not a production translator; for fluent output use the 50M sibling.

Evaluation

Evaluated on a deterministic held-out set of 3,000 pairs (seed=42), decoded greedily (do_sample=False, no repetition penalty), scored with sacreBLEU:

Direction	sacreBLEU	chrF
MSA → Egyptian	4.36	25.43
Egyptian → MSA	4.24	25.08

The saved weights are the best checkpoint by validation loss (eval_loss = 3.552, epoch 3 of 3; loss keeps improving across all three epochs, unlike the sub-2M bases which plateau immediately).

Scaling context (same data, recipe, and eval; only the base model changes)

Base	Params	eval_loss	BLEU (both dir.)	Behavior
`Emhotob-500K`	0.52M	8.42	~0.01	repeats a punctuation token
`Emhotob-1M`	1.07M	7.39	~0.00	repeats common function words
`Emhotob-5M` (this)	5.08M	3.55	~4.3	real, rough translation
`50M-2048-Emhotob`	51.8M	~1.25	~24–26	fluent

Example translations

Real greedy-decoded outputs from the held-out set (rough — this is a 5M model):

MSA → Egyptian

MSA input	Model output (Egyptian)	Reference
شكرًا جزيلًا لك، أنت لطيف للغاية.	شكرًا على اللي أنا حلو جدًا.	شكراً جداً، إنت طيب قوي.
ليحفظ الله الجميع الذين لديهم أصدقاء مخلصون.	ربنا يبارك في كل الناس اللي هيبقى في ناس كتير.	ربنا يبارك لكل واحد عنده صاحب وافي…

Egyptian → MSA

Egyptian input	Model output (MSA)	Reference
شكراً جداً، إنت طيب قوي.	شكرًا لك، أنا سعيد جدًا.	شكرًا جزيلًا لك، أنت لطيف للغاية.
انا بس بحاول اطمن نفسي. ايه أسوأ حاجة ممكن تحصل؟	أنا فقط أتفق على نفسي. ما هي مشكلة؟	أنا فقط أحاول أن أطمئن نفسي…

The model reliably switches register (Egyptian markers بس/إيه/هقولك one way, MSA فقط/ما هي the other) but frequently drifts or repeats on longer inputs. 20 samples per direction with references are in eval_bidirectional.json.

Usage

ChatML format. Pick the system prompt for the direction you want:

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "oddadmix/Emhotob-5M-MSA-Egyptian-v1"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()

SYS_TO_EGY = "أنت مترجم محترف. ترجم النص من اللغة العربية الفصحى إلى اللهجة المصرية العامية."
SYS_TO_MSA = "أنت مترجم محترف. ترجم النص من اللهجة المصرية العامية إلى اللغة العربية الفصحى."

def translate(text: str, system: str) -> str:
    prompt = (
        f"<|im_start|>system\n{system}<|im_end|>\n"
        f"<|im_start|>user\n{text.strip()}<|im_end|>\n"
        f"<|im_start|>assistant\n"
    )
    ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
    if tok.bos_token_id is not None:  # training prepends BOS
        bos = torch.tensor([[tok.bos_token_id]], device=model.device)
        ids["input_ids"] = torch.cat([bos, ids["input_ids"]], dim=1)
        ids["attention_mask"] = torch.cat([torch.ones_like(bos), ids["attention_mask"]], dim=1)
    out = model.generate(**ids, max_new_tokens=256, do_sample=False,
                         eos_token_id=tok.eos_token_id, pad_token_id=tok.pad_token_id)
    return tok.decode(out[0, ids["input_ids"].size(1):], skip_special_tokens=True).strip()

print(translate("شكرًا جزيلًا لك، أنت لطيف للغاية.", SYS_TO_EGY))

Training

Base model: oddadmix/Emhotob-5M (Llama arch, hidden 128, 5 layers, 4 heads, vocab 32000, tied embeddings; 5,080,704 params after resizing for 2 ChatML tokens)
Dataset: oddadmix/egyptian-msa-2.9-openai-bytedance-translations (132K rows, egyptian/msa columns)
Method: HuggingFace Trainer, ChatML, prompt-masked cross-entropy (loss only on the assistant turn). Each row is exploded into two training examples (one per direction). Two ChatML special tokens (<|im_start|>, <|im_end|>) were added and embeddings resized.
Hyperparameters: 3 epochs · effective batch 64 · LR 3e-4 (cosine, 5% warmup) · bf16 · max length 1024 · load_best_model_at_end on eval_loss.
Split: 129,009 train / 3,000 deterministic held-out (seed=42), scored both directions.

Limitations

A 5M model at the emergence threshold for this task: expect frequent drift, repetition, hallucinated content, and errors on anything beyond short everyday sentences.
Gender is disambiguated only from context; ambiguous inputs may default one way.
For fluent translation use oddadmix/50M-MSA-Egyptian-v1. For the degenerate smaller points in this study see oddadmix/Emhotob-500K-MSA-Egyptian-v1.

License

Apache-2.0, inherited from the base model.