---
tags:
- arabic
- tokenizer
- morphology
- nlp
- dialect
license: apache-2.0
language:
- ar
datasets:
- dataflare/arabic-dialect-corpus
- dataflare/egypt-legal-corpus
---

# DF-Arc

**DF-Arc** is a specialized Arabic tokenizer that minimizes the "Arabic Token Tax" by combining **Morphological Pre-tokenization** with **PMI-based Phrase Merging**.

It achieves near 1:1 fertility (1.26) and high semantic density.

## Key Highlights

- **Architecture**: Unigram SentencePiece (compatible with `LlamaTokenizer`).
- **Vocab Size**: 64,000 tokens.
- **Baked-in Logic**: Rules for morphology (prefixes) and identity (God/Prophet names) are built into the vocabulary. No custom code needed.
- **Dialect Native**: Trained on Egyptian dialogue, songs, and feedback corpora.

## Performance

| Model | Fertility | Total Tokens | Total Words |
|-------|-----------|--------------|-------------|
| DF-Arc | 1.260 | 144,734 | 114,882 |
| GPT-4 (cl100k) | 3.689 | 423,743 | 114,882 |
| AraBERT v2 | 1.555 | 178,609 | 114,882 |
| AraT5 | 1.193 | 137,107 | 114,882 |

## Usage

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("dataflare/df-arc")
text = "بسم الله الرحمن الرحيم، انا بحب الذكاء الاصطناعي جدا"

print(tokenizer.tokenize(text))
# Output: ['ب_سم', 'الله', 'ال_رحمن', 'ال_رحيم', '،', 'انا', 'ب_حب', 'ال_ذكاء_ال_اصطناع_ي', 'جدا']
```

## Citation

```bibtex
@misc{df_arc,
  title={DF-Arc: The Arabic Token Tax & Morphology-Aware Tokenization},
  author={Dataflare Lab},
  year={2026},
  publisher={Hugging Face}
}
```