A ~51.8M-parameter Gemma-3 architecture Arabic language model pretrained from scratch on 20 billion Arabic tokens with an 8192-token context window.

This is the long-context sibling of oddadmix/50M-2048-Emhotob: same corpus, same tokenizer, same parameter count, same token budget, same optimizer schedule. What differs is the context window (8192 vs 2048) and the attention layout that makes it affordable.

Base (pretrained) model: a next-token predictor with no instruction or chat tuning.

الملخص بالعربية: نموذج لغوي عربي صغير (~51.8 مليون معامل) بمعمارية Gemma-3، مُدرَّب من الصفر على 20 مليار رمز عربي بطول سياق 8192. نموذج أساس (Base) بدون ضبط تعليمات أو محادثة.

Model details


Parameters	~51.8M
Architecture	Gemma-3 (`Gemma3ForCausalLM`)
Hidden size	512 · Layers 12 · Heads 8 (GQA, 4 KV) · head_dim 64
Intermediate size	1408
Vocab size	32000 (custom Arabic byte-level BPE, shared with the 2048 model)
Context length	8192
Attention	interleaved: 10 sliding (window 1024) + 2 global
Positional encoding	RoPE, θ = 10,000 on sliding layers, 100,000 on global layers
Tied embeddings	yes
License	Apache-2.0

Why interleaved attention

All-global attention at 8192 tokens costs 453M attention FLOPs per token — roughly as much as the entire rest of the forward pass, and about 4× what the 2048 model spent. Restricting five of every six layers to a 1024-token window drops that to ~13M while leaving 2 layers that see the full window, so the long-range path survives and the run costs about what the original 2048 run cost. Sliding layers use the smaller RoPE base because they only ever have to resolve 1024 positions.

Training


Corpus	`kaust-generative-ai/fineweb-edu-ar` (config `ar`)
Budget	20,000,000,000 tokens, 1 epoch over the stream
Packing	documents separated by `</s>`; documents ≥ 4096 tokens duplicated once
Optimizer	AdamW (fused), β = (0.9, 0.95), wd 0.1, grad clip 1.0
Schedule	lr 6e-4, cosine, warmup ratio 0.02
Effective batch	32 sequences × 8192 = 262,144 tokens/step
Precision	bf16, `torch.compile`, a single consumer GPU

The corpus is short-document heavy — median 427 tokens, and only 0.47% of documents reach 8192 — so packing matters more than architecture for long context here. Documents are separated by </s> (the 2048 recipe used no separator), and documents of at least 4096 tokens are written twice to raise the share of tokens sitting inside a genuinely long single-document span.

Usage

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "oddadmix/50M-8192-Nawah-gemma"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16)

prompt = "اللغة العربية هي"
ids = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=128, do_sample=True,
                     temperature=0.8, top_p=0.9, repetition_penalty=1.2)
print(tok.decode(out[0], skip_special_tokens=True))

Limitations

At this size and budget the model has a narrow capability range and will produce factually unreliable and sometimes incoherent text. It is not instruction-tuned or aligned, and no safety filtering has been applied. Attention is not masked at document boundaries during training, so the two global layers were exposed to cross-document context within a packed window.