You can deploy this model and start earning money today!

NAMAA-Space/
alexandriax-mt5-large-balanced

Name: alexandriax-mt5-large-balanced
Rating: 2.5 (5 reviews)

Views

No views yet

AlexandriaX-2026 · Subtask 1 — mT5-large, dialect-rebalanced (partial run)

English to dialectal Arabic dialogue translation over 13 Arabic varieties, from the NAMAA Community work on AlexandriaX-2026 (ArabicNLP 2026 / EMNLP). Full fine-tune of google/mt5-large on the same temperature-rebalanced country distribution as the AraT5v2 sibling, with the target dialect selected by a text prefix.

[!WARNING] This is an unfinished, never-evaluated checkpoint. Training was configured for 8 epochs (31,568 optimiser steps) and stopped at step 2,500 — epoch 0.63, about 7.9% of the schedule. It was never decoded on the development set, so no spBLEU or chrF++ number exists for it, and none is claimed here. It is published for completeness of the release and because the recovered training configuration is exact and reusable, not as a system with known quality. If you need a validated checkpoint for this task, use alexandriax-arat5v2-base (25.12 dev / 23.26 blind spBLEU).


Task	AlexandriaX-2026 Subtask 1 (context-aware EN→DA dialogue translation)
Base model	`google/mt5-large` (mT5, 24+24 layers, d_model 1024, d_ff 2816, 250,112-token vocab)
Parameters	≈1.23B, all trained (full fine-tune, no adapters)
Dialect control	text prefix — `translate English to {Dialect} Arabic: {source}`
Context	none — each turn is translated in isolation
Training reached	2,500 / 31,568 steps (epoch 0.63 / 8); train loss 25.39 → 1.88
Evaluated	no — no dev or test score exists
Precision	bf16 — mT5 produces NaNs in fp16, never use fp16
Weights	`model.safetensors`, bf16, 2.46 GB
License	Apache-2.0, inherited from mT5

Usage

[!IMPORTANT] bf16 or fp32 only. mT5 is known to overflow to NaN in fp16 — this is a property of the pre-trained weights, not of this fine-tune. On a GPU without bf16 support, load in fp32.

python

import torch
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

REPO = "NAMAA-Space/alexandriax-mt5-large-balanced"

tok = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForSeq2SeqLM.from_pretrained(REPO, torch_dtype=torch.bfloat16)
model.eval()

DIALECT = {"EG": "Egyptian", "JO": "Jordanian", "LB": "Lebanese", "LY": "Libyan",
           "MA": "Moroccan", "MR": "Mauritanian", "OM": "Omani", "PS": "Palestinian",
           "SA": "Saudi", "SD": "Sudanese", "SY": "Syrian", "TN": "Tunisian",
           "YE": "Yemeni"}

def translate(sentences, country, num_beams=5, max_new_tokens=128):
    prompts = [f"translate English to {DIALECT[country]} Arabic: {s}" for s in sentences]
    enc = tok(prompts, return_tensors="pt", padding=True, truncation=True, max_length=256)
    enc = {k: v.to(model.device) for k, v in enc.items()}
    with torch.no_grad():
        out = model.generate(**enc, num_beams=num_beams,
                             max_new_tokens=max_new_tokens, length_penalty=1.0)
    return tok.batch_decode(out, skip_special_tokens=True)

preds = translate(["Good morning. How much for the whole quantity?"], "EG")
print(preds)

# Because this checkpoint is undertrained, check the output is Arabic before trusting a batch.
assert any("؀" <= ch <= "ۿ" for ch in preds[0]), "output is not Arabic"

The decoding settings above (beam search, 5 beams, length_penalty=1.0, max_new_tokens=128, source truncated at 256 tokens) are the ones the project's inference harness uses for every encoder–decoder system, so they are the right starting point — but note that no score was ever produced with them for this checkpoint.

Tokenizer — verified

Unlike the AraT5v2 siblings, this repo's tokenizer loads cleanly on current libraries and needed no repair. Checked directly:

Check	Result
`AutoTokenizer.from_pretrained(REPO)` on transformers 5.8	loads (vocab 250,100)
ids vs the repo's own `spiece.model` + `</s>`, on Arabic and mixed probes	identical
`<unk>` produced on those probes	zero
decode → encode round-trip	exact

One caveat to know rather than fear: mT5's SentencePiece model uses byte fallback, which the fast tokenizer does not implement, so on characters outside the vocabulary a fast tokenizer can emit <unk> where the slow one would emit a byte sequence. It did not occur on any probe tested here. Training used the fast tokenizer, so keeping the repo default is also the choice that matches training.

Intended use

Research and reproduction only: a starting point for anyone who wants to finish this run, and a record of the exact configuration used. Not a system with measured quality; not for production translation.

The shared task

AlexandriaX-2026 (ArabicNLP 2026 / EMNLP) — Context-Aware Dialectal Arabic MT and MT Evaluation. This model was built for Subtask 1: Context-Aware English-to-Dialectal Arabic Dialogue Translation.

Given one English dialogue turn together with its conversation history and metadata — target country/dialect, domain, participant roles, speaker, and speaker→addressee gender direction — the system must produce the turn in the requested country's spoken Arabic, preserving meaning while adapting lexical, morphological, pragmatic and sociolinguistic choices to that variety.

Two tracks: constrained (provided data only, ≤5B parameters) and unconstrained (any external data or model). Ranking is by spBLEU (primary) and chrF++ (secondary), each macro-averaged over countries.

Official data (`UBC-NLP/alexandria`)

Split sizes in turns, as published by the organisers:

Split	EG	JO	LB	LY	MA	MR	OM	PS	SA	SD	SY	TN	YE	Total
train	3,108	5,501	8,906	0	2,573	5,515	6,280	14,933	8,470	0	6,071	2,034	3,089	66,480
dev	1,113	1,113	1,118	0	1,110	1,114	1,109	1,110	1,110	0	1,119	1,116	1,118	12,250
public test	1,118	1,107	1,106	1,109	1,115	1,112	1,118	1,109	1,113	1,106	1,114	1,109	1,106	14,442
private (blind) test	1,113	1,109	1,110	1,309	1,111	1,119	1,107	1,111	1,114	915	1,114	1,114	1,113	14,459

Libyan (LY) and Sudanese (SD) appear only at test time — they are zero-shot for every system trained on this data.

Conversation-level counts: 21,146 train / 3,963 dev / 4,706 public-test conversations; mean 3.13 turns per conversation (range 1–5). Mean length 102 characters of English source, 74 characters of dialectal target.

Dialects (13 countries). Egyptian, Jordanian, Lebanese, Libyan, Moroccan, Mauritanian, Omani, Palestinian, Saudi, Sudanese, Syrian, Tunisian, Yemeni. Labels are country + sub-dialect, and several countries carry more than one: Palestinian 10 (Nabulsi and Albira urban, plus Falahi varieties of Surif, Kobar, Noba, Ni'lin, Shuqba, Aboud, Silwad, Ramallah), Omani 5 (Suri, Rustaqi, Al-Wafi, Ibri, Seebi), Saudi 3 (Southern, Hijazi, Khaleeji), Yemeni 3 (Taiz, San'ani, Central), Syrian 2 (Levantine Standard, Homsi). The remaining countries carry one label each (e.g. Egyptian Arabic (Cairene), Moroccan Standard Darija, Mauritanian Hassaniya, Libyan Arabic (Misrati/Central)).

Domains (11, near-uniform). Agriculture and farming, Commerce and transactions, Construction and real estate, Education and academia, Energy and resources, Everyday and social, Healthcare and medical, Legal and financial, Logistics and transportation, Professional and workplace, Tourism and hospitality.

Speaker direction (turns, train+dev+public test): female→male 30,636 · male→female 30,203 · male→male 20,465 · female→female 11,868. The corpus carries 76 distinct translator IDs and 44 reviewer IDs.

Code-switching in the gold is strongly dialect-specific — the share of gold turns containing Latin characters runs from TN 39.1% / MA 33.8% / LB 18.1% down to SY 1.2% / YE 0.8%. Systems that normalise every borrowing into Arabic script are penalised hardest on Maghrebi references (see Known limitations).

Evaluation protocol

spBLEU — sacrebleu.BLEU(tokenize="flores200"), corpus-level per country, then averaged over countries.
chrF++ — sacrebleu.CHRF(word_order=2), same averaging.
Decoding is turn-by-turn: at turn n the conversation history contains the system's own previous outputs, never the gold ones. (An early evaluation harness in this project leaked gold previous-turn Arabic into the prompt and inflated scores by ≈2.4 spBLEU; every number reported here comes from the corrected, self-conditioned harness.)

Training data — the rebalancing recipe

Same source and same resampling as the AraT5v2 sibling: the official Subtask-1 training conversations, with roughly 5% held aside internally (20,094 of 21,146 conversations were used), flattened by temperature sampling and clipped, giving 63,130 training examples — the same total as the unbalanced runs, redistributed.

python

REBALANCE = True
TEMP  = 3.0      # p_c proportional to (n_c / N) ** (1 / TEMP)
FLOOR = 4000     # minimum examples per country
CAP   = 12000    # maximum examples per country
SEED  = 42

Country	raw turns	raw %	rebalanced turns	rebalanced %
PS	14,183	22.5	8,044	12.7
LB	8,464	13.4	6,772	10.7
SA	8,035	12.7	6,656	10.5
OM	5,965	9.4	6,026	9.5
SY	5,760	9.1	5,957	9.4
MR	5,234	8.3	5,769	9.1
JO	5,224	8.3	5,766	9.1
EG	2,943	4.7	4,762	7.5
YE	2,946	4.7	4,764	7.5
MA	2,443	3.9	4,475	7.1
TN	1,933	3.1	4,139	6.6

LY and SD are test-only in this task and contribute no training data; they would be produced zero-shot from the prefix alone. No conversation context, no auxiliary corpus, no back-translation.

Training procedure — every hyperparameter

These values are recovered from the checkpoint itself — training_args.bin and trainer_state.json in the repo's last-checkpoint/ — not reconstructed from memory, so they are exactly what ran. The runnable single-file version is in this repo as train_mt5_large_balanced.py.

Model and data

Setting	Value	Note
Base model	`google/mt5-large`	`MT5ForConditionalGeneration`
Regime	full fine-tune	all ≈1.23B parameters; no LoRA, no quantisation
Tokenizer	`AutoTokenizer.from_pretrained("google/mt5-large")`	fast tokenizer; verified to give ids identical to `spiece.model`, zero `<unk>` on Arabic probes
Vocabulary	250,112 (config) / 250,100 (tokenizer)	`tie_word_embeddings=False`
`max_length` source / target	256 / 256 tokens	truncation, no padding at map time
Label padding	`-100`	`DataCollatorForSeq2Seq(label_pad_token_id=-100)`
Training examples	63,130	rebalanced; from 20,094 conversations
Shuffle seed	42
`model.config.use_cache`	`False` during training

Optimisation

Setting	Value	Note
Optimiser	`adafactor`	recovered from `training_args.bin`
Learning rate	1e-3
LR scheduler	`linear`	confirmed arithmetically: the logged LR at step 2,500 (9.2668e-4) is exactly `1e-3 × (31568−2500)/(31568−200)`
Warmup	200 steps	logged LR at step 25 is 1.25e-4 = `1e-3 × 25/200`
Epochs configured	8
`per_device_train_batch_size`	16
`gradient_accumulation_steps`	1
Effective batch	16	the smallest in the family — the AraT5 runs used 32
Steps per epoch	3,946	63,130 / 16
Steps planned	31,568	3,946 × 8
Steps actually run	2,500	epoch 0.634 — the run did not finish
Weight decay	0.0
Gradient clipping (`max_grad_norm`)	1.0
Adam β₁ / β₂ / ε	0.9 / 0.999 / 1e-8	recorded but unused under Adafactor
Label smoothing	none
Seed	42

Precision and hardware

Setting	Value	Note
Hardware	1 × A100-SXM4-40GB	Colab
`bf16`	True
`fp16`	False	deliberate — mT5 overflows to NaN in fp16
TF32	not set
Model load dtype	`bfloat16`
`gradient_checkpointing`	False
`group_by_length`	True	length-bucketed batches
Total FLOPs logged	8.71e15	for the 2,500 steps that ran

Bookkeeping

Setting	Value
`logging_steps` / `save_steps` / `save_total_limit`	25 / 500 / 2
`eval_strategy`	`"no"` — no evaluation ran during training
`save_safetensors`	True
`push_to_hub`	True, `hub_strategy="checkpoint"` — why this repo contains a `last-checkpoint/` folder
Stack	`torch` 2.11.0+cu128, `transformers` 4.46.3, `tokenizers` 0.20.x, `datasets` 2.x

Repository layout

Path	Size	What it is
`model.safetensors` + `config.json` + tokenizer files	2.46 GB	the model — what you load
`last-checkpoint/`	2.46 GB	a full Trainer checkpoint at step 2,500: identical weights plus `optimizer.pt`, `scheduler.pt`, `rng_state.pth`, `trainer_state.json`, `training_args.bin`

The duplication is a side effect of hub_strategy="checkpoint". Keep last-checkpoint/ if you intend to resume this run — that is exactly what it is for, and trainer_state.json plus optimizer.pt are what make a faithful resume possible. Delete it if you only want to run inference and would rather not pull 5 GB.

Training loss

The run was learning normally when it stopped; these are the logged values, not a quality claim.

step	25	275	525	775	1,025	1,275	1,525	1,775	2,025	2,275	2,500
loss	25.39	3.57	3.07	2.80	2.63	2.46	2.43	2.36	2.29	2.21	1.88
epoch	0.006	0.07	0.13	0.20	0.26	0.32	0.39	0.45	0.51	0.58	0.63

Gradient norm fell from 528 at step 25 to 1.11 at step 2,500. Loss was still descending at the stop, and the learning rate (9.27e-4) had barely begun its linear decay from the 1e-3 peak.

Results

None. No development or test score exists for this checkpoint.

Evidence	State
Training notebook	ran to step 2,500, produced no score output
`trainer_state.json`	`eval_strategy: "no"`, `best_metric: null` — no evaluation ever ran
Inference notebook (`Task1_Infer_mt5_large_balanced.ipynb`)	exists, never executed — zero output cells
Prediction files	none for this system anywhere in the project
Project score table (`dev_scores.json`)	has no mT5 entry

For orientation, the systems that were scored on the official 12,250-turn dev set: AraT5v2 full fine-tune 25.12 spBLEU, Qwen2.5-1.5B LoRA 23.71, NileChat-3B QLoRA 23.54, Gemma-3-1B LoRA 22.71, NLLB-200-1.3B QLoRA 21.83. At ≈1.23B parameters this mT5 would have sat in the middle of that size range — and the family's headline finding is that parameter count does not predict rank there, so its size implies nothing about where it would have landed.

Known limitations

Undertrained by design of the stop, not of the recipe. 2,500 of 31,568 steps. Whatever the recipe is worth, this checkpoint has not realised it.
Never evaluated. Any statement about its translation quality would be invention. If you finish or score it, the honest comparison is against 25.12 dev spBLEU on the official 12,250-turn development set with beam-5 decoding.
No fp16. mT5 NaNs in fp16; use bf16 or fp32.
Byte fallback means a fast tokenizer can in principle emit <unk> where mT5's SentencePiece would emit bytes. It did not happen on any probe tested here, and the shipped tokenizer is the one training used — but keep the default rather than switching to use_fast=False.
5 GB of repo for a 2.46 GB model — the last-checkpoint/ duplicate is only useful for resuming.
Resampling adds no information. Upsampling repeats existing Tunisian and Moroccan turns; it cannot supply vocabulary those varieties are missing.
Inherits the family's other limits: no conversational context, one prefix per country (sub-dialects unaddressed), LY/SD zero-shot, and — had it been measured — metric-only evaluation via spBLEU/chrF++.

Where this model sits in the NAMAA system

All Subtask-1 systems built by the team, scored on the official 12,250-turn dev set (11 countries) and, where they were run, on the 14,459-turn private blind test (13 countries). Country-macro spBLEU / chrF++.

System	Params / arch.	dev spBLEU	dev chrF++	blind spBLEU	blind chrF++	Released
Gemma, beam search (submitted, constrained)	~3.1B, dec-only	—	—	27.413	42.58	no
Routed ensemble (submitted, unconstrained)	—	—	—	27.412	43.05	n/a
Gemini 2.5 Flash, 5-shot	API	—	—	26.68	42.49	n/a
Claude Sonnet 4.5, 5-shot	API	—	—	26.36	42.26	n/a
AraT5v2 full fine-tune	368M, enc–dec	25.12	40.66	23.26	39.03	`alexandriax-arat5v2-base`
Qwen2.5-1.5B LoRA	1.5B, dec-only	23.71	40.51	21.24	38.06	no
NileChat-3B QLoRA, context-free	3B, dec-only	23.54	39.68	—	—	`alexandriax-nilechat-lora`
NileChat-3B QLoRA, +context	3B, dec-only	22.87	39.11	—	—	no
NileChat-3B QLoRA, +context +back-translation	3B, dec-only	22.77	38.71	—	—	`alexandriax-nilechat-ctx-aux`
Gemma-3-1B LoRA	1B, dec-only	22.71	38.86	20.09	36.20	no
NLLB-200-1.3B QLoRA	1.3B, enc–dec	21.83	38.13	—	—	`alexandriax-nllb-1.3b-lora`
AraT5v2, dialect-rebalanced	368M, enc–dec	void run¹		—	—	`alexandriax-arat5v2-balanced`
mT5-large, dialect-rebalanced	1.23B, enc–dec	not evaluated²		—	—	`alexandriax-mt5-large-balanced`
MBR over 3 NileChat variants	—	23.57	39.86	—	—	n/a
MBR over 5 samples, one model	—	20.09	37.50	—	—	n/a
Linear adapter merge	—	19.90	35.33	—	—	n/a

¹ That run was trained against destroyed targets — a tokenizer fallback substituted t5-base (32,100 English tokens) for AraT5v2's 110,208-token vocabulary, so every Arabic character became <unk>. It scored 0.00 spBLEU and cannot be recovered without retraining; the post-mortem and a fixed training script are in its card.

² That run stopped at step 2,500 of a planned 31,568 (epoch 0.63 of 8) and was never decoded on the development set, so no score exists for it. Its card carries the full recovered configuration.

Two findings from this bank of models are worth carrying elsewhere.

Parameter count does not predict rank below the cap. The 368M encoder–decoder AraT5v2 beats every larger decoder-only fine-tune on identical data, and among the decoder-only models spBLEU falls Qwen2.5-1.5B > NileChat-3B > Gemma-3-1B — the reverse of their size order. A reading consistent with this: the metric rewards fidelity to the annotators' conventions over generative fluency. A translator fine-tuned on the provided targets acquires those conventions; a decoder-only model several times its size contributes fluency n-gram overlap does not credit.
Combination is not free. Fitted and evaluated on disjoint halves of the dev conversations: routing per country +0.07, per country + sub-dialect +0.28, per country + domain −0.32, MBR consensus over 5 systems −0.57, MBR over the top-2 per dialect −0.81 — against a best single system of 24.98. The per-turn oracle reaches 32.25 (+7.27), so the right output is usually in the pool and the failure is in selection: three NileChat variants agree with one another and outvote the single strongest system, so consensus weights model-family size rather than quality. The submitted system therefore routes per dialect under a ±0.40 spBLEU margin guard instead of voting.

The collection

All released artefacts live in NAMAA at AlexandriaX-2026:

Repo	What it is
`alexandriax-arat5v2-base`	AraT5v2-base full fine-tune — best small fine-tune, 25.12 dev / 23.26 blind spBLEU
`alexandriax-arat5v2-balanced`	the same recipe on a temperature-rebalanced dialect mixture — void run, released for the post-mortem and the fixed script
`alexandriax-nilechat-lora`	NileChat-3B QLoRA, context-free — best of the three NileChat variants, 23.54 dev spBLEU
`alexandriax-nilechat-ctx-aux`	NileChat-3B QLoRA, context + back-translation — the augmentation ablation, 22.77 dev spBLEU
`alexandriax-nllb-1.3b-lora`	NLLB-200-1.3B QLoRA with per-dialect language codes, 21.83 dev spBLEU
`alexandriax-mt5-large-balanced`	mT5-large on the rebalanced mixture — partial run (2,500/31,568 steps), never evaluated
`alexandria-backtranslated-pairs`	348,787 synthetic EN→dialect pairs over 14 varieties

Every model repo above carries a single-file train_*.py reproduction script with the exact hyperparameters that produced its checkpoint; the dataset repo carries build_backtranslated_pairs.py.

Official task data: UBC-NLP/alexandria. Base models: UBC-NLP/AraT5v2-base-1024, UBC-NLP/NileChat-3B-Base, facebook/nllb-200-1.3B, google/mt5-large.

Team

NAMAA Community — Fatimah Emad Eldin (Cairo University) · Omer Nacar (Tuwaiq Academy) · Khloud Al Jallad (Arab International University) · Mona Abdelazim (Ain Shams University).

Citation

Coming soon. The NAMAA system-description paper for AlexandriaX-2026 is under review for the ArabicNLP 2026 (EMNLP) proceedings; this card will be updated with the final ACL Anthology reference and DOI when the proceedings are published. Until then, please cite as:

bibtex

@inproceedings{namaa-alexandriax-2026,
  title     = {{NAMAA} Community at {AlexandriaX-2026}: Prompting, Fine-Tuning and Agreement
               Voting for Dialectal Arabic Translation and Evaluation},
  author    = {Emad Eldin, Fatimah and Nacar, Omer and Al Jallad, Khloud and Abdelazim, Mona},
  booktitle = {Proceedings of the Fourth Arabic Natural Language Processing Conference
               (ArabicNLP 2026)},
  year      = {2026},
  note      = {To appear. Citation coming soon.}
}

Please also cite the shared task and the base model: