Update README.md
Browse files
README.md
CHANGED
|
@@ -1,14 +1,103 @@
|
|
| 1 |
---
|
|
|
|
|
|
|
| 2 |
license: mit
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
---
|
| 4 |
|
| 5 |
# DunbaaBERT
|
| 6 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 7 |
## Pre-training
|
| 8 |
- 52k vocab size
|
| 9 |
- 100k training steps
|
| 10 |
- computed on 4x H100 with 8k batch size
|
| 11 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 12 |
## Fairseq Checkpoint
|
| 13 |
Get the fairseq checkpoint [here](https://drive.proton.me/urls/3EWQWHQF0G#MHW5nydl7U5J).
|
| 14 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
language:
|
| 3 |
+
- ur
|
| 4 |
license: mit
|
| 5 |
+
library_name: transformers
|
| 6 |
+
pipeline_tag: fill-mask
|
| 7 |
+
tags:
|
| 8 |
+
- urdu
|
| 9 |
+
- roberta
|
| 10 |
+
- masked-language-modeling
|
| 11 |
+
- encoder
|
| 12 |
+
- dunbaabert
|
| 13 |
+
datasets:
|
| 14 |
+
- uonlp/CulturaX
|
| 15 |
---
|
| 16 |
|
| 17 |
# DunbaaBERT
|
| 18 |
|
| 19 |
+
DunbaaBERT is a family of Urdu RoBERTa-base encoder models trained from scratch on a deduplicated 17 GB Urdu corpus. The models use Byte-BPE vocabularies of 32k, 52k, and 96k tokens and are released under the MIT license.
|
| 20 |
+
|
| 21 |
+
## Model Details
|
| 22 |
+
|
| 23 |
+
- **Model type:** RoBERTa-style masked language model
|
| 24 |
+
- **Language:** Urdu
|
| 25 |
+
- **Architecture:** Encoder-only Transformer
|
| 26 |
+
- **Training objective:** Masked Language Modeling with Whole Word Masking (WWM)
|
| 27 |
+
- **Sequence length:** 512 tokens
|
| 28 |
+
- **Training corpus:** 17 GB deduplicated Urdu text
|
| 29 |
+
|
| 30 |
+
## Model Variants
|
| 31 |
+
|
| 32 |
+
| Model | Vocabulary Size | Parameters |
|
| 33 |
+
|---------|---------:|---------:|
|
| 34 |
+
| DunbaaBERT-32k | 32,009 | 110,625,024 |
|
| 35 |
+
| **DunbaaBERT-52k** | 52,009 | 125,985,024 |
|
| 36 |
+
| DunbaaBERT-96k | 96,009 | 159,777,024 |
|
| 37 |
+
|
| 38 |
+
## Training Data
|
| 39 |
+
|
| 40 |
+
The final corpus was constructed from multiple Urdu resources and deduplicated at line level.
|
| 41 |
+
|
| 42 |
+
| Corpus | Size |
|
| 43 |
+
|----------|---------:|
|
| 44 |
+
| mC4 | 17.0 GB |
|
| 45 |
+
| OSCAR-2019 | 869 MB |
|
| 46 |
+
| OSCAR-2109 | 604 MB |
|
| 47 |
+
| OSCAR-2201 | 344 MB |
|
| 48 |
+
| OSCAR-2301 | 982 MB |
|
| 49 |
+
| Urdu Wikipedia | 364 MB |
|
| 50 |
+
| Filtered NLLB Urdu | 2.1 GB |
|
| 51 |
+
| **Total before deduplication** | **22.3 GB** |
|
| 52 |
+
| **Final corpus** | **17.0 GB** |
|
| 53 |
+
|
| 54 |
## Pre-training
|
| 55 |
- 52k vocab size
|
| 56 |
- 100k training steps
|
| 57 |
- computed on 4x H100 with 8k batch size
|
| 58 |
|
| 59 |
+
## Evaluation Results
|
| 60 |
+
|
| 61 |
+
### Main Results
|
| 62 |
+
|
| 63 |
+
| Model | UrBLiMP | COUNT19 F1 | USADC F1 | PSL-Kabaddi F1 | IMDB Urdu F1 | Avg. Norm. Eff. |
|
| 64 |
+
|---------|---------:|---------:|---------:|---------:|---------:|---------:|
|
| 65 |
+
| DunbaaBERT-32k | 95.1 | 94.44 | **94.08** | 70.08 | 90.13 | **0.859** |
|
| 66 |
+
| DunbaaBERT-52k | **97.0** | 94.91 | 91.75 | 67.60 | 90.14 | 0.795 |
|
| 67 |
+
| DunbaaBERT-96k | 94.6 | 95.22 | 89.97 | 70.53 | 90.65 | 0.813 |
|
| 68 |
+
| Urdu-RoBERTa-small | 90.5 | 92.08 | 85.36 | 67.06 | 84.72 | 0.781 |
|
| 69 |
+
| HPLT-BERT-ur | **97.3** | **95.71** | 93.51 | **71.11** | 89.69 | 0.597 |
|
| 70 |
+
| mBERT | 75.5 | 90.88 | 83.03 | 65.78 | 85.47 | 0.744 |
|
| 71 |
+
| mmBERT-small | 89.5 | 92.36 | 73.09 | 70.36 | 85.44 | 0.494 |
|
| 72 |
+
| mmBERT-base | 92.4 | 93.97 | 77.77 | 67.75 | 87.31 | 0.495 |
|
| 73 |
+
| XLM-R-base | 89.6 | 93.72 | 85.22 | 60.56 | 88.69 | 0.754 |
|
| 74 |
+
| XLM-R-large | 94.1 | 94.38 | 83.55 | 69.62 | **91.15** | 0.492 |
|
| 75 |
+
|
| 76 |
+
### Efficiency
|
| 77 |
+
|
| 78 |
+
We report a normalized efficiency metric combining Macro-F1 and inference throughput.
|
| 79 |
+
Across benchmarks, the DunbaaBERT family consistently achieved stronger performance-efficiency trade-offs than most multilingual baselines.
|
| 80 |
+
|
| 81 |
+
DunbaaBERT-52k achieved the strongest linguistic probing performance on UrBLiMP, while DunbaaBERT-32k provided the strongest overall efficiency profile.
|
| 82 |
+
Interestingly, DunbaaBERT-96k ranked second in average efficiency despite having the largest vocabulary.
|
| 83 |
+
|
| 84 |
## Fairseq Checkpoint
|
| 85 |
Get the fairseq checkpoint [here](https://drive.proton.me/urls/3EWQWHQF0G#MHW5nydl7U5J).
|
| 86 |
|
| 87 |
+
## Citation
|
| 88 |
+
|
| 89 |
+
```bibtex
|
| 90 |
+
@misc{maab2026dunbaabertsacrificesemantics,
|
| 91 |
+
title={DunbaaBERT: From Sacrifice to Semantics},
|
| 92 |
+
author={Iffat Maab and Waleed Jamil and Raphael Schmitt},
|
| 93 |
+
year={2026},
|
| 94 |
+
eprint={2605.26935},
|
| 95 |
+
archivePrefix={arXiv},
|
| 96 |
+
primaryClass={cs.CL},
|
| 97 |
+
url={https://arxiv.org/abs/2605.26935},
|
| 98 |
+
}
|
| 99 |
+
```
|
| 100 |
+
|
| 101 |
+
## License
|
| 102 |
+
|
| 103 |
+
MIT License
|