README.md

---
language:
- ar
tags:
- sudanese
- arabic
- tokenizer
- bpe
- dialect
- sudaverse
license: mit
library_name: transformers
---
# 🇸🇩 Sudanizer: Sudanese Arabic BPE Tokenizer
**Creator: [O96a](https://huggingface.co/O96a)**  
**Initiative: [Sudaverse](https://www.sudaverse.com)**  
*For the Sudanese NLP Community*
## 🎯 Mission
This tokenizer is part of the **Sudaverse** initiative dedicated to preserving Sudanese linguistic heritage and advancing Sudanese NLP. Developed specifically for the Sudanese Arabic dialect to ensure our language is properly represented in AI systems.
## Model Description
**Sudanizer** is a production-ready Byte-Pair Encoding (BPE) tokenizer trained on ~49 million words of authentic Sudanese dialect text. It recognizes words and expressions unique to Sudanese culture and communication.
### Key Features
- **Vocabulary Size**: 72,000 tokens
- **Algorithm**: BPE with ByteLevel pre-tokenization
- **Dialect-Specific**: Optimized for Sudanese Arabic patterns
- **Arabic-Aware**: Handles clitics, prefixes, suffixes, emojis
- **Normalization**: Sudanese orthographic variants (كده→كدا, ده→دا, etc.)
### Performance Metrics
| Metric | Value |
|--------|-------|
| Average Characters per Token | 4.536 |
| Compression Ratio | 0.2205 |
| UNK Rate | 0.0000% |
| Avg Subwords per Word | 1.159 |
| Vocabulary Size | 72,000 |
## 🚀 Quick Start
```python
# Load tokenizer
tokenizer = PreTrainedTokenizerFast.from_pretrained("O96a/sudanizer-72k")
# Tokenize Sudanese text
text = "لو مغالطني ارح كاكا ليو"
tokens = tokenizer.encode(text)
print(f"Tokens: {tokens}")
# Decode back
decoded = tokenizer.decode(tokens)
print(f"Decoded: {decoded}")
# Get token strings
token_strs = tokenizer.tokenize(text)
print(f"Token strings: {token_strs}")
```
## Training Details
### Configuration
- **Tokenizer Version**: 1.0.0
- **Minimum Token Frequency**: 3
- **Pre-tokenization**: ByteLevel (Arabic-optimized)
- **Normalization**: NFKC + Sudanese dialect normalization
- **Corpus Domain**: social+stories+youtube+qa
### Special Tokens
| Token | Purpose | ID |
|-------|---------|-----|
| `<pad>` | Padding | 0 |
| `<unk>` | Unknown | 1 |
| `<s>` | Start of sequence (BOS) | 2 |
| `</s>` | End of sequence (EOS) | 3 |
| `<mask>` | Mask token (MLM) | 4 |
| `<user>` | User message marker | 5 |
| `<assistant>` | Assistant message marker | 6 |
| `<system>` | System message marker | 7 |

## 💚 For the Sudanese Community
✅ **Intended Use:**
- Sudanese Arabic NLP research & development
- Building Sudanese AI assistants and chatbots
- Fine-tuning models on Sudanese data
- Preserving Sudanese linguistic heritage
- Educational purposes for Sudanese language technology

⚠️ **Attribution Required:**
This work is developed by **Aamer Mihaysi** as part of the **Sudaverse** initiative for the Sudanese NLP community. When using this tokenizer, please cite appropriately and acknowledge the Sudaverse project.
## Comparison with Other Tokenizers
Sudanizer is designed to outperform general Arabic tokenizers on Sudanese text:

### Benchmark Tokenizers Tested
1. **Aranizer-SP-64k** (riotu-lab) — SentencePiece tokenizer tailored for Arabic (~64K vocab)
2. **Aranizer-PBE-64k** (riotu-lab) — PBE version optimized for Arabic morphology
3. **arabic-bpe-tokenizer** (IsmaelMousa) — Standard BPE for Arabic (~50K vocab)
4. **SARF-Tokenizer** (almaghrabima) — Morphologically-aware tokenizer with morpheme segmentation
5. **deeplatent-tokenizer** (almaghrabima) — SARF-based bilingual tokenizer
6. **AraBERTv2** — General Arabic BERT tokenizer
7. **CAMeLBERT-Mix** — Mixed Arabic dialect tokenizer

| Tokenizer | Sudanese Chars/Token | UNK Rate |
|-----------|---------------------|----------|
| **Sudanizer-72K-v2 (this)** | 4.54 | 0.00% |
| AraBERTv2 | ~3.5 | Higher |
| CAMeL-BERT | ~3.2 | Higher |
| Aranizers | ~3.0-3.5 | Moderate |

## Citation
If you use this tokenizer in your research, please cite:
```bibtex
@misc{sudanizer2026,
  title={Sudanizer: A Production-Ready BPE Tokenizer for Sudanese Arabic},
  author={Mihaysi, Aamer},
  year={2026},
  howpublished={\url{https://huggingface.co/O96a/sudanizer-72k}}
}
```
## License
MIT License — This work belongs to the Sudanese NLP community.
## Acknowledgments
- **Creator**: Aamer Mihaysi
- **Initiative**: [Sudaverse](https://www.sudaverse.com)
- HuggingFace for the tokenizers library
- Contributors to Sudanese Arabic corpora
## Contact
For questions or contributions, please contact via [Sudaverse](https://www.sudaverse.com).
---
*Made with 💙 for Sudan. This work belongs to the Sudanese NLP community.*