AraSSM-base

AraSSM is a bidirectional state-space (Mamba) encoder pretrained from scratch for Arabic via masked language modeling. It is, to our knowledge, the first bidirectional Mamba/SSM encoder pretrained specifically for Arabic, and was trained entirely on four consumer-grade NVIDIA RTX 2080Ti GPUs (11GB) rather than an accelerator cluster.

Model description

Each AraSSM layer runs a forward and a backward selective-scan (Mamba) mixer over the same input and merges the two outputs, giving the model full bidirectional context while keeping $O(L)$ complexity in sequence length $L$, instead of the $O(L^2)$ complexity of self-attention.

Layers	12
Hidden size	512
State dimension	16
Parameters	~105M
Max sequence length	512
Vocabulary size	64,000

Tokenizer

AraSSM uses the existing AraBERTv02 tokenizer (aubmindlab/bert-base-arabertv02) rather than a custom-trained one. This tokenizer was not trained as part of this work; it is reused as-is, both to decouple corpus preprocessing from tokenizer choice and to keep results comparable to AraBERT-family baselines that use the same vocabulary. All credit for the tokenizer belongs to its original authors.

Training data

AraSSM is pretrained on a cleaned, deduplicated corpus of approximately 80GB of Arabic text (79.6GB train / 0.4GB validation), combining Arabic Wikipedia and the Arabic portion of CulturaX. Documents are stripped of diacritics and URLs, filtered for length and Arabic-script ratio, chunked to at most 400 words, and deduplicated at the chunk level with a Bloom filter.

Training procedure

• Objective: standard BERT-style masked language modeling (15% masking, 80/10/10 split)
• Optimizer: AdamW, lr 3e-4, weight decay 0.01, 10,000 warmup steps, linear decay
• Precision: fp16 (Turing GPUs do not support accelerated bf16)
• Effective batch size: 256 (batch size 8 x grad accumulation 8 x 4 GPUs)
• Compute: 4x RTX 2080Ti, 960 GPU-hours (10 days)

How to use

AraSSM is a custom architecture (not a native transformers model class), so loading it requires the model code from the project repository:

from huggingface_hub import PyTorchModelHubMixin
from models.mamba import MambaForMaskedLM  # from the AraSSM repository

class HubMambaForMaskedLM(MambaForMaskedLM, PyTorchModelHubMixin):
    pass

model = HubMambaForMaskedLM.from_pretrained("aliane29/arassm-base")

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("aliane29/arassm-base")

Intended use

This is a pretrained encoder intended to be fine-tuned on downstream Arabic NLU tasks (classification, token classification, extractive question answering), similarly to how a BERT-family encoder is used. It has not been fine-tuned for any specific task in this repository.

Limitations

• Pretrained on Modern Standard Arabic and web text (Wikipedia + CulturaX); performance on dialectal Arabic is not evaluated.
• Maximum sequence length is 512 tokens.
• Trained on a fixed, publicly available compute budget (4 consumer GPUs); larger-scale Transformer baselines were pretrained on substantially larger accelerator-cluster budgets.