README.md

---
language:
- ckb
- kur
license: apache-2.0
library_name: transformers
pipeline_tag: fill-mask
tags:
- kurdish
- sorani
- central-kurdish
- ckb
- roberta
- masked-lm
- fill-mask
---

# Kurdish Sorani RoBERTa

A RoBERTa language model for Central Kurdish (Sorani), trained from scratch on a
1-million-sentence Kurdish corpus with a custom 32k Kurdish BPE tokenizer. It's a
masked-language model meant as a **base for fine-tuning** on Kurdish tasks
(classification, NER, embeddings, and so on).

It's the base model behind
[haryads/kurdish-sentence-embeddings](https://huggingface.co/haryads/kurdish-sentence-embeddings).

## Usage

```python
from transformers import pipeline

fill = pipeline("fill-mask", model="haryads/kurdish-roberta")
fill("زمانی کوردی زۆر <mask> ە")
```

Or load it directly to fine-tune:

```python
from transformers import AutoModel, AutoTokenizer

tok = AutoTokenizer.from_pretrained("haryads/kurdish-roberta")
model = AutoModel.from_pretrained("haryads/kurdish-roberta")
```

## Architecture

| | |
|---|---|
| Type | RoBERTa (masked LM) |
| Layers | 6 |
| Hidden size | 512 |
| Attention heads | 8 |
| Max sequence length | 128 |
| Vocab size | 32,000 |
| Parameters | ~35M |

## Training

- Corpus: 1M Sorani sentences
- Tokenizer: custom 32k byte-level BPE for Kurdish
- Objective: masked language modeling (15% masking)

## Intended use and limits

A base model, not an instruct or chat model. Best used fine-tuned for a specific
Kurdish task. Sorani only, and the corpus is news-heavy, so very colloquial or
domain-specific text may need in-domain fine-tuning.

## License

Apache-2.0.