AraSSM-base
AraSSM is a bidirectional state-space (Mamba) encoder pretrained from scratch for Arabic via
masked language modeling. It is, to our knowledge, the first bidirectional Mamba/SSM encoder
pretrained specifically for Arabic, and was trained entirely on four consumer-grade NVIDIA RTX
2080Ti GPUs (11GB) rather than an accelerator cluster.
Model description
Each AraSSM layer runs a forward and a backward selective-scan (Mamba) mixer over the same
input and merges the two outputs, giving the model full bidirectional context while keeping
$O(L)$ complexity in sequence length $L$, instead of the $O(L^2)$ complexity of self-attention.
| |
|---|
| Layers | 12 |
| Hidden size | 512 |
| State dimension | 16 |
| Parameters | ~105M |
| Max sequence length | 512 |
| Vocabulary size | 64,000 |
Tokenizer
AraSSM uses the existing
AraBERTv02 tokenizer
(
aubmindlab/bert-base-arabertv02)
rather than a custom-trained one. This tokenizer was not trained as part of this work; it is
reused as-is, both to decouple corpus preprocessing from tokenizer choice and to keep results
comparable to AraBERT-family baselines that use the same vocabulary. All credit for the
tokenizer belongs to its original authors.
Training data
AraSSM is pretrained on a cleaned, deduplicated corpus of approximately 80GB of Arabic text
(79.6GB train / 0.4GB validation), combining Arabic Wikipedia and the Arabic portion of
CulturaX. Documents are stripped of diacritics and URLs, filtered for length and Arabic-script
ratio, chunked to at most 400 words, and deduplicated at the chunk level with a Bloom filter.
Training procedure
- Objective: standard BERT-style masked language modeling (15% masking, 80/10/10 split)
- Optimizer: AdamW, lr 3e-4, weight decay 0.01, 10,000 warmup steps, linear decay
- Precision: fp16 (Turing GPUs do not support accelerated bf16)
- Effective batch size: 256 (batch size 8 x grad accumulation 8 x 4 GPUs)
- Compute: 4x RTX 2080Ti, ~960 GPU-hours (~10 days)
How to use
AraSSM is a custom architecture (not a native transformers model class), so loading it
requires the model code from the project repository:
1from huggingface_hub import PyTorchModelHubMixin
2from models.mamba import MambaForMaskedLM # from the AraSSM repository
3
4class HubMambaForMaskedLM(MambaForMaskedLM, PyTorchModelHubMixin):
5 pass
6
7model = HubMambaForMaskedLM.from_pretrained("aliane29/arassm-base")
8
9from transformers import AutoTokenizer
10tokenizer = AutoTokenizer.from_pretrained("aliane29/arassm-base")
Intended use
This is a pretrained encoder intended to be fine-tuned on downstream Arabic NLU tasks
(classification, token classification, extractive question answering), similarly to how a
BERT-family encoder is used. It has not been fine-tuned for any specific task in this repository.
Limitations
- Pretrained on Modern Standard Arabic and web text (Wikipedia + CulturaX); performance on
dialectal Arabic is not evaluated.
- Maximum sequence length is 512 tokens.
- Trained on a fixed, publicly available compute budget (4 consumer GPUs); larger-scale
Transformer baselines were pretrained on substantially larger accelerator-cluster budgets.
