You can deploy this model and start earning money today!

AraFusion/
arafusion-mdlm-641m-vanillabpe-scratch-v2

Name: arafusion-mdlm-641m-vanillabpe-scratch-v2
Availability: InStock

Views

No views yet

AraFusion — Arabic Masked Diffusion Language Model

Trained with MDLM on FineWeb-2 Arabic subsets. Supports dialect-conditioned generation via Classifier-Free Guidance (CFG).

Model details


Architecture	DiT (BERT-base scale: 12L / 12H / 768d)
Vocab	96 000 (MorphBPE, see `AraFusion/arafusion-morphBPE`)
Sequence length	512 tokens
Diffusion	Linear noise schedule, 1 000 sampling steps
Dialects	MSA (`[MSA]`), Najdi (`[NAJDI]`), Egyptian (`[EGYPT]`)
CFG p_uncond	0.10
Training steps	200 000 (pre-train)
Precision	BF16

Data

Trained on three FineWeb-2 Arabic subsets:

Subset	Dialect	Split
`arb_Arab`	Modern Standard Arabic	train 10 %
`ars_Arab`	Najdi / Saudi Arabic	train (full)
`arz_Arab`	Egyptian Arabic	train (full)

See AraFusion/arafusion-arabic-raw for the raw text and AraFusion/arafusion-arabic-packed for the packed training sequences.

Checkpoints

Training job: vanillabpe-641m-51479

Path in repo
`checkpoints/0-34500-v1.ckpt`
`checkpoints/1-35000-v1.ckpt`
`checkpoints/1-35500.ckpt`
`checkpoints/1-36000.ckpt`
`checkpoints/1-36500.ckpt`
`checkpoints/1-37000.ckpt`
`checkpoints/1-37500.ckpt`
`checkpoints/1-38000.ckpt`
`checkpoints/1-38500.ckpt`
`checkpoints/1-39000.ckpt`
`checkpoints/1-39500.ckpt`
`checkpoints/1-40000.ckpt`
`checkpoints/1-40500.ckpt`
`checkpoints/1-41000.ckpt`
`checkpoints/1-41500.ckpt`
`checkpoints/1-42000.ckpt`
`checkpoints/1-42500.ckpt`
`checkpoints/1-43000.ckpt`
`checkpoints/1-43500.ckpt`
`checkpoints/1-44000.ckpt`
`checkpoints/1-44500.ckpt`
`checkpoints/1-45000.ckpt`
`checkpoints/1-45500.ckpt`
`checkpoints/1-46000.ckpt`
`checkpoints/1-46500.ckpt`
`checkpoints/1-47000.ckpt`
`checkpoints/1-47500.ckpt`
`checkpoints/1-48000.ckpt`
`checkpoints/1-48500.ckpt`
`checkpoints/1-49000.ckpt`
`checkpoints/1-49500.ckpt`
`checkpoints/1-50000.ckpt`
`checkpoints/1-50500.ckpt`
`checkpoints/1-51000.ckpt`
`checkpoints/1-51500.ckpt`
`checkpoints/1-52000.ckpt`
`checkpoints/1-52500.ckpt`
`checkpoints/1-53000.ckpt`
`checkpoints/1-53500.ckpt`
`checkpoints/1-54000.ckpt`
`checkpoints/1-54500.ckpt`
`checkpoints/1-55000.ckpt`
`checkpoints/1-55500.ckpt`
`checkpoints/1-56000.ckpt`
`checkpoints/1-56500.ckpt`
`checkpoints/1-57000.ckpt`
`checkpoints/1-57500.ckpt`
`checkpoints/1-58000.ckpt`
`checkpoints/1-58500.ckpt`
`checkpoints/1-59000.ckpt`
`checkpoints/1-59500.ckpt`
`checkpoints/1-60000.ckpt`
`checkpoints/1-60500.ckpt`
`checkpoints/1-61000.ckpt`
`checkpoints/1-61500.ckpt`
`checkpoints/1-62000.ckpt`
`checkpoints/1-62500.ckpt`
`checkpoints/1-63000.ckpt`
`checkpoints/1-63500.ckpt`
`checkpoints/1-64000.ckpt`
`checkpoints/1-64500.ckpt`
`checkpoints/1-65000.ckpt`
`checkpoints/1-65500.ckpt`
`checkpoints/1-66000.ckpt`
`checkpoints/1-66500.ckpt`
`checkpoints/1-67000.ckpt`
`checkpoints/1-67500.ckpt`
`checkpoints/1-68000.ckpt`
`checkpoints/1-68500.ckpt`
`checkpoints/1-69000.ckpt`
`checkpoints/2-100000.ckpt`
`checkpoints/2-100500.ckpt`
`checkpoints/2-101000.ckpt`
`checkpoints/2-101500.ckpt`
`checkpoints/2-102000.ckpt`
`checkpoints/2-102500.ckpt`
`checkpoints/2-103000.ckpt`
`checkpoints/2-103500.ckpt`
`checkpoints/2-69500.ckpt`
`checkpoints/2-70000.ckpt`
`checkpoints/2-70500.ckpt`
`checkpoints/2-71000.ckpt`
`checkpoints/2-71500.ckpt`
`checkpoints/2-72000.ckpt`
`checkpoints/2-72500.ckpt`
`checkpoints/2-73000.ckpt`
`checkpoints/2-73500.ckpt`
`checkpoints/2-74000.ckpt`
`checkpoints/2-74500.ckpt`
`checkpoints/2-75000.ckpt`
`checkpoints/2-75500.ckpt`
`checkpoints/2-76000.ckpt`
`checkpoints/2-76500.ckpt`
`checkpoints/2-77000.ckpt`
`checkpoints/2-77500.ckpt`
`checkpoints/2-78000.ckpt`
`checkpoints/2-78500.ckpt`
`checkpoints/2-79000.ckpt`
`checkpoints/2-79500.ckpt`
`checkpoints/2-80000.ckpt`
`checkpoints/2-80500.ckpt`
`checkpoints/2-81000.ckpt`
`checkpoints/2-81500.ckpt`
`checkpoints/2-82000.ckpt`
`checkpoints/2-82500.ckpt`
`checkpoints/2-83000.ckpt`
`checkpoints/2-83500.ckpt`
`checkpoints/2-84000.ckpt`
`checkpoints/2-84500.ckpt`
`checkpoints/2-85000.ckpt`
`checkpoints/2-85500.ckpt`
`checkpoints/2-86000.ckpt`
`checkpoints/2-86500.ckpt`
`checkpoints/2-87000.ckpt`
`checkpoints/2-87500.ckpt`
`checkpoints/2-88000.ckpt`
`checkpoints/2-88500.ckpt`
`checkpoints/2-89000.ckpt`
`checkpoints/2-89500.ckpt`
`checkpoints/2-90000.ckpt`
`checkpoints/2-90500.ckpt`
`checkpoints/2-91000.ckpt`
`checkpoints/2-91500.ckpt`
`checkpoints/2-92000.ckpt`
`checkpoints/2-92500.ckpt`
`checkpoints/2-93000.ckpt`
`checkpoints/2-93500.ckpt`
`checkpoints/2-94000.ckpt`
`checkpoints/2-94500.ckpt`
`checkpoints/2-95000.ckpt`
`checkpoints/2-95500.ckpt`
`checkpoints/2-96000.ckpt`
`checkpoints/2-96500.ckpt`
`checkpoints/2-97000.ckpt`
`checkpoints/2-97500.ckpt`
`checkpoints/2-98000.ckpt`
`checkpoints/2-98500.ckpt`
`checkpoints/2-99000.ckpt`
`checkpoints/2-99500.ckpt`
`checkpoints/3-104000.ckpt`
`checkpoints/3-104500.ckpt`
`checkpoints/3-105000.ckpt`
`checkpoints/3-105500.ckpt`
`checkpoints/3-106000.ckpt`
`checkpoints/3-106500.ckpt`
`checkpoints/3-107000.ckpt`
`checkpoints/3-107500.ckpt`
`checkpoints/3-108000.ckpt`
`checkpoints/3-108500.ckpt`
`checkpoints/3-109000.ckpt`
`checkpoints/3-109500.ckpt`
`checkpoints/3-110000.ckpt`
`checkpoints/3-110500.ckpt`
`checkpoints/3-111000.ckpt`
`checkpoints/3-111500.ckpt`
`checkpoints/3-112000.ckpt`
`checkpoints/3-112500.ckpt`
`checkpoints/3-113000.ckpt`
`checkpoints/3-113500.ckpt`
`checkpoints/3-114000.ckpt`
`checkpoints/3-114500.ckpt`
`checkpoints/3-115000.ckpt`
`checkpoints/3-115500.ckpt`
`checkpoints/3-116000.ckpt`
`checkpoints/3-116500.ckpt`
`checkpoints/3-117000.ckpt`
`checkpoints/3-117500.ckpt`
`checkpoints/3-118000.ckpt`
`checkpoints/3-118500.ckpt`
`checkpoints/3-119000.ckpt`
`checkpoints/3-119500.ckpt`
`checkpoints/3-120000.ckpt`
`checkpoints/3-120500.ckpt`
`checkpoints/3-121000.ckpt`
`checkpoints/3-121500.ckpt`
`checkpoints/3-122000.ckpt`
`checkpoints/3-122500.ckpt`
`checkpoints/3-123000.ckpt`
`checkpoints/3-123500.ckpt`
`checkpoints/3-124000.ckpt`
`checkpoints/3-124500.ckpt`
`checkpoints/3-125000.ckpt`
`checkpoints/3-125500.ckpt`
`checkpoints/3-126000.ckpt`
`checkpoints/3-126500.ckpt`
`checkpoints/3-127000.ckpt`
`checkpoints/3-127500.ckpt`
`checkpoints/3-128000.ckpt`
`checkpoints/3-128500.ckpt`
`checkpoints/3-129000.ckpt`
`checkpoints/3-129500.ckpt`
`checkpoints/3-130000.ckpt`
`checkpoints/3-130500.ckpt`
`checkpoints/3-131000.ckpt`
`checkpoints/3-131500.ckpt`
`checkpoints/3-132000.ckpt`
`checkpoints/3-132500.ckpt`
`checkpoints/3-133000.ckpt`
`checkpoints/3-133500.ckpt`
`checkpoints/3-134000.ckpt`
`checkpoints/3-134500.ckpt`
`checkpoints/3-135000.ckpt`
`checkpoints/3-135500.ckpt`
`checkpoints/3-136000.ckpt`
`checkpoints/3-136500.ckpt`
`checkpoints/3-137000.ckpt`
`checkpoints/3-137500.ckpt`
`checkpoints/3-138000.ckpt`
`checkpoints/3-138500.ckpt`
`checkpoints/4-139000.ckpt`
`checkpoints/4-139500.ckpt`
`checkpoints/4-140000.ckpt`
`checkpoints/4-140500.ckpt`
`checkpoints/4-141000.ckpt`
`checkpoints/4-141500.ckpt`
`checkpoints/4-142000.ckpt`
`checkpoints/4-142500.ckpt`
`checkpoints/4-143000.ckpt`
`checkpoints/4-143500.ckpt`
`checkpoints/4-144000.ckpt`
`checkpoints/4-144500.ckpt`
`checkpoints/4-145000.ckpt`
`checkpoints/4-145500.ckpt`
`checkpoints/4-146000.ckpt`
`checkpoints/4-146500.ckpt`
`checkpoints/4-147000.ckpt`
`checkpoints/4-147500.ckpt`
`checkpoints/4-148000.ckpt`
`checkpoints/4-148500.ckpt`
`checkpoints/4-149000.ckpt`
`checkpoints/4-149500.ckpt`
`checkpoints/4-150000.ckpt`
`checkpoints/4-150500.ckpt`
`checkpoints/4-151000.ckpt`
`checkpoints/4-151500.ckpt`
`checkpoints/4-152000.ckpt`
`checkpoints/4-152500.ckpt`
`checkpoints/4-153000.ckpt`
`checkpoints/4-153500.ckpt`
`checkpoints/4-154000.ckpt`
`checkpoints/4-154500.ckpt`
`checkpoints/4-155000.ckpt`
`checkpoints/4-155500.ckpt`
`checkpoints/4-156000.ckpt`
`checkpoints/4-156500.ckpt`
`checkpoints/4-157000.ckpt`
`checkpoints/4-157500.ckpt`
`checkpoints/4-158000.ckpt`
`checkpoints/4-158500.ckpt`
`checkpoints/4-159000.ckpt`
`checkpoints/4-159500.ckpt`
`checkpoints/4-160000.ckpt`
`checkpoints/4-160500.ckpt`
`checkpoints/4-161000.ckpt`
`checkpoints/4-161500.ckpt`
`checkpoints/4-162000.ckpt`
`checkpoints/4-162500.ckpt`
`checkpoints/4-163000.ckpt`
`checkpoints/4-163500.ckpt`
`checkpoints/4-164000.ckpt`
`checkpoints/4-164500.ckpt`
`checkpoints/4-165000.ckpt`
`checkpoints/4-165500.ckpt`
`checkpoints/4-166000.ckpt`
`checkpoints/4-166500.ckpt`
`checkpoints/4-167000.ckpt`
`checkpoints/4-167500.ckpt`
`checkpoints/4-168000.ckpt`
`checkpoints/4-168500.ckpt`
`checkpoints/4-169000.ckpt`
`checkpoints/4-169500.ckpt`
`checkpoints/4-170000.ckpt`
`checkpoints/4-170500.ckpt`
`checkpoints/4-171000.ckpt`
`checkpoints/4-171500.ckpt`
`checkpoints/4-172000.ckpt`
`checkpoints/4-172500.ckpt`
`checkpoints/4-173000.ckpt`
`checkpoints/5-173500.ckpt`
`checkpoints/5-174000.ckpt`
`checkpoints/5-174500.ckpt`
`checkpoints/5-175000.ckpt`
`checkpoints/5-175500.ckpt`
`checkpoints/5-176000.ckpt`
`checkpoints/5-176500.ckpt`
`checkpoints/5-177000.ckpt`
`checkpoints/5-177500.ckpt`
`checkpoints/5-178000.ckpt`
`checkpoints/5-178500.ckpt`
`checkpoints/5-179000.ckpt`
`checkpoints/5-179500.ckpt`
`checkpoints/5-180000.ckpt`
`checkpoints/5-180500.ckpt`
`checkpoints/5-181000.ckpt`
`checkpoints/5-181500.ckpt`
`checkpoints/5-182000.ckpt`
`checkpoints/5-182500.ckpt`
`checkpoints/5-183000.ckpt`
`checkpoints/5-183500.ckpt`
`checkpoints/5-184000.ckpt`
`checkpoints/5-184500.ckpt`
`checkpoints/5-185000.ckpt`
`checkpoints/5-185500.ckpt`
`checkpoints/5-186000.ckpt`
`checkpoints/5-186500.ckpt`
`checkpoints/5-187000.ckpt`
`checkpoints/5-187500.ckpt`
`checkpoints/5-188000.ckpt`
`checkpoints/5-188500.ckpt`
`checkpoints/5-189000.ckpt`
`checkpoints/5-189500.ckpt`
`checkpoints/5-190000.ckpt`
`checkpoints/5-190500.ckpt`
`checkpoints/5-191000.ckpt`
`checkpoints/5-191500.ckpt`
`checkpoints/5-192000.ckpt`
`checkpoints/5-192500.ckpt`
`checkpoints/5-193000.ckpt`
`checkpoints/5-193500.ckpt`
`checkpoints/5-194000.ckpt`
`checkpoints/5-194500.ckpt`
`checkpoints/5-195000.ckpt`
`checkpoints/5-195500.ckpt`
`checkpoints/5-196000.ckpt`
`checkpoints/5-196500.ckpt`
`checkpoints/5-197000.ckpt`
`checkpoints/5-197500.ckpt`
`checkpoints/5-198000.ckpt`
`checkpoints/5-198500.ckpt`
`checkpoints/5-199000.ckpt`
`checkpoints/5-199500.ckpt`
`checkpoints/best.ckpt`
`checkpoints/last.ckpt`

Usage

python

# Requires the MDLM library from https://github.com/kuleshov-group/mdlm
from transformers import PreTrainedTokenizerFast
tok = PreTrainedTokenizerFast.from_pretrained("AraFusion/arafusion-morphBPE")

# Load checkpoint and run a sampling step — see MDLM docs for full API.

Citation

bibtex

@misc{arafusion2026,
  title   = {AraFusion: Dialect-Conditioned Arabic Masked Diffusion Language Model},
  year    = {2026},
  url     = {https://huggingface.co/AraFusion}
}

AraFusion/arafusion-mdlm-641m-vanillabpe-scratch-v2