
Architecture	LLaMA-style (`LlamaForCausalLM`)
Parameters	~50M
Context	2048 tokens
Hidden size	512
Layers	12
Vocab size	32,000
Precision	bfloat16
Base model	`oddadmix/50M-2048-Emhotob`
Training data	`Misraj/Sadeed_Tashkeela` (`train` split)

Prompt format

Training and inference use this prompt:

قم بتشكيل هذة الجمله : {undiacritized_text}

The model should continue with the diacritized Arabic text.

Quick start

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ahmedsamirtarjama/Tashkeel-50M"
device = "cuda" if torch.cuda.is_available() else "cpu"

tok = AutoTokenizer.from_pretrained(model_id)
tok.padding_side = "left"
if tok.pad_token is None:
    tok.pad_token = tok.eos_token

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16 if device == "cuda" else torch.float32,
).to(device)
model.eval()

text = "اللغة العربية لغة جميلة"
prompt = f"قم بتشكيل هذة الجمله : {text}\n"
inputs = tok(prompt, return_tensors="pt").to(device)

with torch.inference_mode():
    out = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=False,
        pad_token_id=tok.pad_token_id,
    )

print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Training

Full fine-tune (not LoRA) with Hugging Face Trainer:

Hyperparameter	Value
Epochs	1
Learning rate	3e-4
Scheduler	cosine
Warmup steps	500
Batch size	32
Max sequence length	768 (longer examples dropped)
Loss	next-token LM loss on the diacritized target only (prompt tokens masked with `-100`)
Precision	bfloat16

Approximate training recipe:

<source prompt> + <diacritized target> + </s>

Evaluation

Evaluated on Misraj/SadeedDiac-25 with standard Morph/Total DER & WER (missing GT diacritics skipped).

Mapping used below: Total ≈ (CE) (with case endings), Morph ≈ (w/o CE) (without case endings). Hallucinations ≈ share of examples skipped due to word-count mismatch.

Model	DER (CE)	WER (CE)	DER (w/o CE)	WER (w/o CE)	Hallucinations
Claude-3-7-Sonnet	1.39	4.67	0.77	2.31	0.82
Tashkeel-50M	3.08*	9.56*	2.26*	6.77*	~99
GPT-4	3.86	5.27	3.86	10.93	1.02
Gemini-Flash-2.0	3.19	7.99	2.38	5.50	1.17
Sadeed	7.29	13.74	5.26	9.92	7.19

*Tashkeel-50M DER/WER are computed only on examples where the prediction and reference have the same word count. Because most generations change length (truncation / repetition / insertions), they are skipped by the length-matching evaluator — hence the high hallucination rate. Treat the starred numbers as optimistic and not a full apples-to-apples comparison with systems that preserve word identity on nearly all examples.

For production tashkeel, prefer stronger constrained models or add decoding constraints that keep the undiacritized skeleton fixed.

Intended use

Research and prototyping for Arabic diacritization
Baseline for small / efficient tashkeel models
Educational demos of causal-LM fine-tuning for sequence transduction

Limitations

Small capacity (~50M); quality lags dedicated / large instruction models on hard classical Arabic
Causal generation can truncate, repeat, or insert words; DER/WER only apply when word counts match
Prompt is Arabic-instruction style; changing the prompt may degrade quality
Not a general-purpose chat model

Citation

If you use this model, please also cite the base model and dataset:

bibtex

@misc{tashkeel50m,
  title        = {Tashkeel-50M},
  author       = {Ahmed Samir},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/ahmedsamirtarjama/Tashkeel-50M}}
}

Base: oddadmix/50M-2048-Emhotob
Data: Misraj/Sadeed_Tashkeela