Views
No views yet
oddadmix/Emhotob-5M-v2, a tiny Llama-architecture
base (hidden 128, 5 layers, 4 heads, vocab 32000, tied embeddings).Scaling study. This is one rung of a from-scratch Arabic scaling study that runs an identical SFT + eval recipe across bases from 0.5M to 50M parameters to locate where translation emerges. On the headline MSA↔Egyptian pair, output is degenerate at ≤1M, becomes real-but-rough at 5M, and usable at 10M+. See the siblingoddadmix/50M-English-MSA-v1for the fluent reference.
seed=42),
decoded greedily (do_sample=False, no repetition penalty), scored with sacreBLEU:| Direction | sacreBLEU | chrF |
|---|---|---|
| English → MSA | 15.87 | 37.76 |
| MSA → English | 21.97 | 38.91 |
eval_loss = 1.802). 20 samples
per direction with references are in eval_bidirectional.json.| Source | Model output | Reference |
|---|---|---|
| I'm just trying to reassure myself. What's the worst that could happen? It's just mentally exhausting, but they're stand | أنا فقط أحاول إقناع نفسي. ما هو أسوأ أي سبب يمكن أن يحدث؟ إنه فقط مرهق تمامًا، لكنهم يقفون بشكل غير هادئين. حقًا، إنهم ي | أنا فقط أحاول أن أطمئن نفسي. ما هو الأسوأ الذي يمكن أن يحدث؟ إنه مرهق عقليًا فقط، لكنهم يقفون بحزم. حقًا، إنهم يقفون بحز |
| Thank you very much, you are so kind. | شكرًا جزيلًا، أنت لطيف جدًا. | شكرًا جزيلًا لك، أنت لطيف للغاية. |
| Source | Model output | Reference |
|---|---|---|
| أنا فقط أحاول أن أطمئن نفسي. ما هو الأسوأ الذي يمكن أن يحدث؟ إنه مرهق عقليًا فقط، لكنهم يقفون بحزم. حقًا، إنهم يقفون بحز | I'm just trying to definitely trying to definitely. What's the worst that can happen? He's just a mind, but they're stil | I'm just trying to reassure myself. What's the worst that could happen? It's just mentally exhausting, but they're stand |
| شكرًا جزيلًا لك، أنت لطيف للغاية. | Thank you so much, you're very nice. | Thank you very much, you are so kind. |
1import torch
2from transformers import AutoModelForCausalLM, AutoTokenizer
3
4model_id = "oddadmix/Emhotob-5M-English-MSA-v2"
5tok = AutoTokenizer.from_pretrained(model_id)
6model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).to("cuda").eval()
7
8SYSTEM = "أنت مترجم محترف. ترجم النص الإنجليزي إلى اللغة العربية الفصحى."
9
10def translate(text, system=SYSTEM):
11 prompt = (f"<|im_start|>system\n{system}<|im_end|>\n"
12 f"<|im_start|>user\n{text.strip()}<|im_end|>\n<|im_start|>assistant\n")
13 ids = tok(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)
14 if tok.bos_token_id is not None:
15 bos = torch.tensor([[tok.bos_token_id]], device=model.device)
16 ids["input_ids"] = torch.cat([bos, ids["input_ids"]], dim=1)
17 ids["attention_mask"] = torch.cat([torch.ones_like(bos), ids["attention_mask"]], dim=1)
18 out = model.generate(**ids, max_new_tokens=256, do_sample=False,
19 eos_token_id=tok.eos_token_id, pad_token_id=tok.pad_token_id)
20 return tok.decode(out[0, ids["input_ids"].size(1):], skip_special_tokens=True).strip()oddadmix/Emhotob-5M-v2 (Llama arch, hidden 128, 5 layers, 4 heads, vocab 32000, tied embeddings;
5,080,704 params after resizing for 2 ChatML tokens)oddadmix/egyptian-msa-2.9-openai-bytedance-translationsTrainer, ChatML, prompt-masked cross-entropy (loss only on the
assistant turn). Each row is exploded into two training examples (one per direction).load_best_model_at_end on eval_loss.seed=42), scored both directions.