Nawah-50M-RAG-Chat-8K-GRPO

A 51.8M-parameter Arabic (MSA) retrieval-augmented customer-support model that holds a conversation over an 8,192-token knowledge base — roughly 50 passages — and answers from those passages only.

It is the multi-turn, long-context sibling of Nawah-50M-RAG-Support-2K. Where that model answers one question against ~8 passages, this one tracks a 5-round dialogue across ~50, resolves pronouns and ellipsis between turns, handles a customer correcting themselves mid-conversation, and refuses when a follow-up asks for something the passages do not contain.


Parameters	51,801,088
Architecture	`Gemma3ForCausalLM` — 12 layers, hidden 512
Attention	10 sliding-window layers (1,024) + 2 global at positions 5 and 11
Context	8,192 tokens (~50 passages)
Vocabulary	32,002 — Arabic-first byte-level BPE + 2 ChatML tokens
Conversation	Multi-turn, 1–5 rounds, with optional per-customer account block
Post-training	SFT → GRPO

Quick numbers

Gold-forced multi-turn replay over the held-out split — 1,651 turns across 632 conversations, company-disjoint from training. Judged by Gemma-4-31B; score is 0–2 over answerable turns.

	SFT base	this model (GRPO)
score — all	0.72	0.86
score — short / mid / long	0.73 / 0.72 / 0.71	0.84 / 0.93 / 0.81
hallucination	38.1%	29.1%
part-coverage	50.1%	58.5%
aggregation-recall	53.0%	53.7%
distractor-number rate	22.4%	20.7%
chrF++	43.75	46.93
mid-conversation refusal	95.3%	93.9%

Quality holds across the conversation, which is the thing a 51.8M model was least likely to manage:

round	0	1	2	3	4
score	0.90	0.85	0.77	0.85	0.84
hallucination	31.6%	27.3%	27.3%	26.6%	33.9%

The hallucination result is paired, not a difference of headline means: against the previous reward set this model is better on 266 turns and worse on 134 (exact two-sided sign test, p = 3.9e-11), holding independently in the short bucket (86/44, p = 2.9e-04) and the long bucket (96/53, p = 5.4e-04). Score moves too, but far more weakly (262/214, p = 0.031) — the gold-coverage reward moved specifically what it was written to move.

Cost: replies grew from 37.2 to 41.2 words on average, and mid-conversation refusal gave up 1.4pp.

Usage

ChatML. The system turn carries the support preamble, an optional customer account block, then the numbered passages. Then the conversation alternates.

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

REPO = "oddadmix/Nawah-50M-RAG-Chat-8K-GRPO"
tok = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForCausalLM.from_pretrained(REPO, dtype=torch.float32).eval()

chunks = [
    "الباقة الذهبية بسعر 150 ريالاً شهرياً وتشمل توصيلاً مجانياً.",
    "رسوم التأخر في السداد 25 ريالاً بعد مرور 7 أيام من تاريخ الاستحقاق.",
]
system = (
    "أنت مساعد خدمة عملاء. أجب عن سؤال العميل بالفصحى اعتماداً فقط على المعلومات "
    "التالية. إذا لم تكن الإجابة موجودة في المعلومات، فقل ذلك بأدب واعرض تحويل "
    "العميل إلى أحد موظفي خدمة العملاء."
    "\n\n" + "\n".join(f"[{i}] {c}" for i, c in enumerate(chunks, 1))
)

messages = [
    {"role": "system",    "content": system},
    {"role": "user",      "content": "كم سعر الباقة الذهبية؟"},
    {"role": "assistant", "content": "الباقة الذهبية بسعر 150 ريالاً شهرياً."},
    {"role": "user",      "content": "وكم لو تأخرت أسبوع؟"},   # ellipsis: needs turn 1
]
enc = tok.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True)
out = model.generate(**enc, max_new_tokens=160, do_sample=False, pad_token_id=1)
print(tok.decode(out[0][enc["input_ids"].shape[-1]:], skip_special_tokens=True))

The system string above is verbatim the training format — preamble, blank line, then passages numbered [1], [2], … one per line. Paraphrasing the preamble or dropping the numbering degrades grounding.

The account block

An optional per-customer record goes between the preamble and the passages, separated by a blank line, and stays outside the [n] numbering:

بيانات حساب العميل:
رقم الحساب: 8821-4403
الباقة الحالية: الباقة الذهبية
الرصيد المستحق: 0 ليرة لبنانية
تاريخ الاستحقاق: 2024-12-15

It sits outside the numbering on purpose: in production that record arrives from a billing system, not from retrieval, and the model is trained to let a per-customer figure override the generic policy passage. 55% of training conversations carry one.

How it was built

Pretraining. oddadmix/50M-8192-Nawah-gemma — a Gemma-3-architecture Arabic base trained from scratch at 8,192 context on 20B tokens. The interleaved 10-sliding/2-global layout is what makes an 8K window affordable at this size: attention FLOPs per token drop from 453M (all-global) to ~13M.
Multi-turn SFT → Nawah-50M-RAG-Chat-8K on arabic-rag-chat-30K: 30,662 conversations of 1–5 rounds over knowledge bases stratified short (~1K) / mid (3–5K) / long (6–7K). Loss lands on every assistant turn.
GRPO — this checkpoint — on arabic-rag-chat-grpo-5K, a company-disjoint pool. Prompts are gold-forced replays of round k with rounds 0..k−1 filled in from reference answers, capped at 4,096 tokens: 3,046 prompts, 725 multi-part and 1,270 multi-gold.

The SFT model's measured failure was never fluency or format — it answers in the right shape. It was passage discrimination: it cited a distractor figure in 22.4% of answerable turns and covered only 50.1% of the key facts a question asked for. GRPO targeted that directly, with seven programmatic rewards and no judge in the loop:

reward	what it scores
number grounding	every number in the reply must occur in the gold chunks, the account block, or the question; numbers found only in distractor chunks — or nowhere — are penalised
gold coverage	n-gram overlap with the gold passages specifically, so quoting the right passage beats quoting a plausible neighbour
part coverage	fraction of a multi-part question's key facts actually present in the reply
aggregation recall	for turns needing ≥2 gold chunks, the fraction contributing at least one key fact
refusal correctness	refuse iff the answer is not in the context
chrF to the gold answer	anchors content and MSA fluency
sanity	Arabic-script prose of reasonable length

Training ran on a single consumer GPU.

Limitations

Ten of twelve layers use a 1,024-token sliding window. A passage 6,000 tokens back reaches the answer through two global layers only. This is why the long bucket (0.81) still trails the mid bucket (0.93). Better data and better rewards improve passage selection; they do not change the architecture.
29.1% of answered turns still contain an ungrounded number. Much better than the 38.1% it started from, and far from solved. This is a 51.8M model — do not put it in front of customers without a human path.
Passage discrimination is the core weakness. In practice it usually finds the right topic and sometimes the wrong specific passage.
Modern Standard Arabic only. Dialectal input is out of distribution.
It quotes, it does not compute. Trained to repeat figures, never to derive them. Do not ask it to total a bill.
8,192 tokens is a hard ceiling, and retrieval quality is the real one — this model does not retrieve. Everything above assumes you supply the gold passage among the candidates.
The judge is one model's opinion. Treat the 0–2 score as a comparison between rows of these tables, not an absolute. The judge-free columns (part-coverage, aggregation-recall, distractor-number) are string tests and are the more literal measurements.

Reproducing the numbers

Every figure here is recomputable from arabic-rag-chat-8k-eval: the test split, 15 models' raw replies, 13 sets of judge verdicts, and the rendered per-model reports. This model is chat-grpo-v3; its SFT base is chat-s1.

model.safetensors has SHA-256 5b5fea657c15b661a680b2976700f643f8f12bf32b4551ceffcbc6fd0f20b291 — the GRPO v3 policy.

Intended use

Research and prototyping of grounded, multi-turn Arabic question answering, and a baseline for how small a long-context conversational RAG model can get. No general instruction tuning, no safety alignment, and no knowledge of its own — outside the passages you give it, it has nothing to say.

Citation

bibtex

@misc{nawah50mrag8k,
  title  = {Nawah-50M-RAG-Chat-8K-GRPO: a 51.8M-parameter multi-turn long-context Arabic RAG model},
  author = {Wasfy, Ahmed},
  year   = {2026},
  url    = {https://huggingface.co/oddadmix/Nawah-50M-RAG-Chat-8K-GRPO}
}