---
language:
- ar
license: apache-2.0
tags:
- mamba
- state-space-model
- arabic
- masked-language-modeling
- bidirectional
datasets:
- wikimedia/wikipedia
- uonlp/CulturaX
pipeline_tag: fill-mask
---

# AraSSM-base

AraSSM is a bidirectional state-space (Mamba) encoder pretrained from scratch for Arabic via
masked language modeling. It is, to our knowledge, the first bidirectional Mamba/SSM encoder
pretrained specifically for Arabic, and was trained entirely on four consumer-grade NVIDIA RTX
2080Ti GPUs (11GB) rather than an accelerator cluster.

## Model description

Each AraSSM layer runs a forward and a backward selective-scan (Mamba) mixer over the same
input and merges the two outputs, giving the model full bidirectional context while keeping
$O(L)$ complexity in sequence length $L$, instead of the $O(L^2)$ complexity of self-attention.

| | |
|---|---|
| Layers | 12 |
| Hidden size  | 512 |
| State dimension | 16 |
| Parameters | ~105M |
| Max sequence length | 512 |
| Vocabulary size | 64,000 |

## Tokenizer

AraSSM uses the existing **AraBERTv02 tokenizer**
([`aubmindlab/bert-base-arabertv02`](https://huggingface.co/aubmindlab/bert-base-arabertv02))
rather than a custom-trained one. This tokenizer was not trained as part of this work; it is
reused as-is, both to decouple corpus preprocessing from tokenizer choice and to keep results
comparable to AraBERT-family baselines that use the same vocabulary. All credit for the
tokenizer belongs to its original authors.

## Training data

AraSSM is pretrained on a cleaned, deduplicated corpus of approximately 80GB of Arabic text
(79.6GB train / 0.4GB validation), combining Arabic Wikipedia and the Arabic portion of
CulturaX. Documents are stripped of diacritics and URLs, filtered for length and Arabic-script
ratio, chunked to at most 400 words, and deduplicated at the chunk level with a Bloom filter.

## Training procedure

- Objective: standard BERT-style masked language modeling (15% masking, 80/10/10 split)
- Optimizer: AdamW, lr 3e-4, weight decay 0.01, 10,000 warmup steps, linear decay
- Precision: fp16 (Turing GPUs do not support accelerated bf16)
- Effective batch size: 256 (batch size 8 x grad accumulation 8 x 4 GPUs)
- Compute: 4x RTX 2080Ti, ~960 GPU-hours (~10 days)

## How to use

AraSSM is a custom architecture (not a native `transformers` model class), so loading it
requires the model code from the project repository:

```python
from huggingface_hub import PyTorchModelHubMixin
from models.mamba import MambaForMaskedLM  # from the AraSSM repository

class HubMambaForMaskedLM(MambaForMaskedLM, PyTorchModelHubMixin):
    pass

model = HubMambaForMaskedLM.from_pretrained("aliane29/arassm-base")

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("aliane29/arassm-base")
```

## Intended use

This is a pretrained encoder intended to be fine-tuned on downstream Arabic NLU tasks
(classification, token classification, extractive question answering), similarly to how a
BERT-family encoder is used. It has not been fine-tuned for any specific task in this repository.

## Limitations

- Pretrained on Modern Standard Arabic and web text (Wikipedia + CulturaX); performance on
  dialectal Arabic is not evaluated.
- Maximum sequence length is 512 tokens.
- Trained on a fixed, publicly available compute budget (4 consumer GPUs); larger-scale
  Transformer baselines were pretrained on substantially larger accelerator-cluster budgets.