Emhotob-10M-Egyptian-English-v1 — Bidirectional English ↔ Egyptian Arabic (~11M params)
An 10.9M-parameter model that translates both ways between English and
Egyptian colloquial Arabic (المصرية العامية). A single set of weights serves both
directions; a direction-specific system prompt selects which way to translate.
Finetuned from
oddadmix/Emhotob-10M,
a tiny Llama-architecture base (hidden size 256, 4 layers, 8 heads, tied embeddings).
Scaling study. This runs the exact recipe of
oddadmix/50M-Egyptian-English-v1 on
a base ~5× smaller. At 10M cross-lingual translation becomes
usable on short sentences — a
big jump from the 5M rung (BLEU <2) — with Egyptian→English notably stronger than the reverse.
Still expect drift and repetition on longer or rarer inputs.
Evaluation
Evaluated on a deterministic held-out set of 3,000 pairs (seed=42), decoded
greedily (do_sample=False, no repetition penalty), scored with sacreBLEU:
| Direction | sacreBLEU | chrF |
|---|
| English → Egyptian | 9.82 | 34.22 |
| Egyptian → English | 16.69 | 36.07 |
The saved weights are the best checkpoint by validation loss (eval_loss = 2.003, epoch 3 of 3).
For reference, the 50M sibling scores BLEU ~24 (en→egy) / ~34 (egy→en) with the same data and eval;
at 5M this pair scored <2 both ways.
Example translations
Real greedy-decoded outputs from the held-out set:
English → Egyptian
| English input | Model output (Egyptian) | Reference |
|---|
| So what should I get you? Should I get you this or that? | يعني إيه اللي ياخدك؟ أخدك كده ولا ده؟ … | طب أجيب لك إيه؟ أجيب لك دي ولا دي؟ … |
Egyptian → English
| Egyptian input | Model output (English) | Reference |
|---|
| طب أجيب لك إيه؟ أجيب لك دي ولا دي؟ | Well, what do you get to you? Just go to you, or is… | So what should I get you? Should I get you this or that? |
Short, common exchanges come out well and the model reliably picks the right language; longer
inputs still drift or repeat. 20 samples per direction with references are in
eval_bidirectional.json.
Usage
ChatML format. Pick the system prompt for the direction you want:
1import torch
2from transformers import AutoModelForCausalLM, AutoTokenizer
3
4model_id = "oddadmix/Emhotob-10M-Egyptian-English-v1"
5tok = AutoTokenizer.from_pretrained(model_id)
6model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()
7
8SYS_TO_EGY = "أنت مترجم محترف. ترجم النص الإنجليزي إلى اللهجة المصرية العامية."
9SYS_TO_EN = "You are a professional translator. Translate the Egyptian Arabic text into English."
10
11def translate(text: str, system: str) -> str:
12 prompt = (
13 f"<|im_start|>system\n{system}<|im_end|>\n"
14 f"<|im_start|>user\n{text.strip()}<|im_end|>\n"
15 f"<|im_start|>assistant\n"
16 )
17 ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
18 if tok.bos_token_id is not None: # training prepends BOS
19 bos = torch.tensor([[tok.bos_token_id]], device=model.device)
20 ids["input_ids"] = torch.cat([bos, ids["input_ids"]], dim=1)
21 ids["attention_mask"] = torch.cat([torch.ones_like(bos), ids["attention_mask"]], dim=1)
22 out = model.generate(**ids, max_new_tokens=256, do_sample=False,
23 eos_token_id=tok.eos_token_id, pad_token_id=tok.pad_token_id)
24 return tok.decode(out[0, ids["input_ids"].size(1):], skip_special_tokens=True).strip()
25
26print(translate("I will be fine, don't worry", SYS_TO_EGY))
Training
- Base model:
oddadmix/Emhotob-10M (Llama arch, hidden 256, 4 layers, 8 heads, vocab 32000,
tied embeddings; 10,947,328 params after resizing for 2 ChatML tokens)
- Dataset:
oddadmix/egyptian-translation-dataset-2.9-openai-batch (135,282 rows,
English/Arabic columns; Arabic = Egyptian colloquial)
- Method: HuggingFace
Trainer, ChatML, prompt-masked cross-entropy (loss only on the
assistant turn). Each row is exploded into two training examples (one per direction).
Two ChatML special tokens (<|im_start|>, <|im_end|>) were added and embeddings resized.
- Hyperparameters: 3 epochs · effective batch 64 · LR 3e-4 (cosine, 5% warmup) ·
bf16 · max length 1024 ·
load_best_model_at_end on eval_loss.
- Split: 132,282 train / 3,000 deterministic held-out (
seed=42), scored both directions.
Out-of-domain evaluation — Egyptian Arabic Translation Benchmark
The results above are
in-domain: a held-out split of the same corpus this model was
trained on. The numbers below are
out-of-domain — the same model scored on the
Egyptian Arabic Translation Benchmark
(
oddadmix/egyptian-arabic-translation-benchmark, 319 English→Egyptian pairs written by a different annotator with different
orthographic conventions).
Expect these to be substantially lower than the in-domain scores. That gap is the
generalization penalty, not a regression — both numbers are real, they measure different things.
| Metric | Score |
|---|
BLEU (evaluate, 0–1 — leaderboard metric) | 0.0313 |
| BLEU (sacrebleu, 0–100) | 3.13 |
| chrF | 28.52 |
| METEOR | 0.1985 |
Decoding is deterministic greedy (do_sample=False, no repetition penalty), using the
exact ChatML prompt format the model was trained with — the same protocol as every other
number in this study.
Where this rung sits
| Model | Params | BLEU (hf) | BLEU (sacre) | chrF | METEOR |
|---|
| 5M v1 | 5.2M | 0.0000 | 0.19 | 13.00 | 0.0558 |
| 5M v2 | 5.2M | 0.0108 | 1.08 | 22.19 | 0.1352 |
| 10M v1 (this model) | 11.2M | 0.0313 | 3.13 | 28.52 | 0.1985 |
| 10M v2 | 10.9M | 0.0341 | 3.41 | 31.60 | 0.2235 |
| 25M v1 | 25.3M | 0.0991 | 9.91 | 41.07 | 0.3223 |
| 25M v2 | 25.3M | 0.0824 | 8.24 | 40.29 | 0.3148 |
| 50M v1 (bidi) | 51.8M | 0.1113 | 11.13 | 44.41 | 0.3482 |
| 50M v1 (uni) | 51.8M | 0.1261 | 12.61 | 46.13 | 0.3559 |
Reading these numbers
BLEU understates quality on this set. Scoring is against a single reference, so a correct
translation that picks a different valid word is penalized — e.g. فريش vs the reference's
طازة for "fresh", or التليفون اللي ضاع vs تليفونها الضايع for "her lost phone". Both are
good Egyptian; only one matches the reference. chrF and METEOR track perceived quality more
closely here.
At 319 rows, differences of roughly 1–2 BLEU between adjacent rungs are within noise.
These are small models — 5M to 50M parameters, orders of magnitude below the large systems
typically evaluated on this benchmark. The result of interest is the scaling curve and
per-parameter efficiency, not absolute rank against models 100–1000× the size.
Limitations
- An 11M model: usable on short/common sentences but drift, repetition, and meaning loss appear
on longer or rarer inputs.
- Gender is disambiguated only from context; ambiguous inputs may default one way.
- For fluent translation use
oddadmix/50M-Egyptian-English-v1.
License
Apache-2.0, inherited from the base model.
