Add model card

README.md

@@ -0,0 +1,222 @@
+---
+language:
+- nso
+license: cc-by-nc-sa-4.0
+tags:
+- sepedi
+- northern-sotho
+- nso
+- african-languages
+- language-model
+- masked-language-modeling
+- low-resource
+- south-africa
+- limpopo
+datasets:
+- Sediba-AI/sepedi-training-v1
+base_model: xlm-roberta-base
+model-index:
+- name: Sediba-AI/sepedi-llama-v1
+  results: []
+---
+
+# Sepedi-Llama v1
+
+**First open-source continued pre-training of XLM-RoBERTa on Sepedi (Northern Sotho)**
+
+Developed by [Sediba AI](https://huggingface.co/Sediba-AI) / [TSEBO SOVEREIGN TECH (Pty) Ltd](https://github.com/Sediba-AI)  
+Mankweng, Limpopo, South Africa
+
+---
+
+## Model Description
+
+Sepedi-Llama v1 is a continued pre-training of `xlm-roberta-base` on a 376,803-record clean Sepedi corpus. It is the first publicly available language model specifically trained on Sepedi (ISO 639-3: `nso`) at this scale.
+
+Sepedi (Northern Sotho / Sesotho sa Leboa) is spoken by approximately **4.6 million people** in Limpopo, Gauteng, and Mpumalanga provinces of South Africa. Despite being one of South Africa's 11 official languages, it has near-zero representation in existing NLP infrastructure.
+
+This model is the foundation layer of the **Sediba AI** intelligence platform — a community-sovereign AI system serving Sepedi-speaking communities in Limpopo.
+
+---
+
+## Intended Uses
+
+### Primary uses
+- **Fine-tuning** for downstream Sepedi NLP tasks:
+  - Sentiment classification
+  - Named Entity Recognition (NER)
+  - Text classification
+  - Question answering
+- **EduIntel** — curriculum AI assistant for Grade 10-12 Sepedi teachers (Morutabana platform)
+- **Research** — low-resource African language NLP
+
+### Out-of-scope uses
+- Production text generation without further fine-tuning
+- Languages other than Sepedi/Northern Sotho
+- Commercial use without review of the CC BY-NC-SA 4.0 license terms
+
+---
+
+## Training Details
+
+### Training Data
+
+| Source | Records | License | Notes |
+|--------|---------|---------|-------|
+| Autshumato Monolingual Sepedi v2.1 | 91,347 | CC BY | SADiLaR |
+| Autshumato Bilingual Sepedi-English (NSO side) | 122,665 | CC BY | SADiLaR |
+| NCHLT RAW Sepedi corpus | 63,984 | CC BY 4.0 | CTexT/NWU |
+| FineWeb-2 NSO subset | 144,611 | CC BY 4.0 | HuggingFaceFW |
+| Sepedi Bible | 30,616 | Public domain | |
+| Wikipedia NSO | 18,347 | CC BY-SA | wikimedia |
+| DBE NSC exam papers 2017-2025 | 195 documents | Public domain | Grade 10-12 |
+| DBE Annual Teaching Plans 2023 | 9 documents | Public domain | Grade 10-12 |
+| **Total** | **376,803 records** | | |
+
+Full dataset: [Sediba-AI/sepedi-training-v1](https://huggingface.co/datasets/Sediba-AI/sepedi-training-v1)
+
+### Training Procedure
+
+| Parameter | Value |
+|-----------|-------|
+| Base model | `xlm-roberta-base` |
+| Training objective | Masked Language Modeling (MLM, 15% mask rate) |
+| Epochs | 1 |
+| Batch size (effective) | 32 (2 per device × 16 accumulation steps) |
+| Max sequence length | 64 tokens |
+| Learning rate | 5e-5 with linear warmup (200 steps) |
+| Weight decay | 0.01 |
+| Hardware | Tesla T4 (16GB VRAM) |
+| Training time | 10 hours 3 minutes |
+| Framework | HuggingFace Transformers 4.x |
+
+### Training Loss
+
+| Epoch | Step | Loss |
+|-------|------|------|
+| 0.00 | 0 | — |
+| 0.004 | 50 | 83.27 |
+| 0.085 | 1000 | ~35 |
+| 0.25 | 3000 | ~26 |
+| 0.50 | 5900 | ~22 |
+| 0.75 | 8800 | ~20 |
+| 1.00 | 11776 | **19.04** |
+
+**Final training loss: 25.45 (average) | 19.04 (final step)**  
+Loss reduction: 83.27 → 19.04 (**77% reduction**)
+
+---
+
+## Evaluation
+
+Formal evaluation benchmarks (FLORES-200, MasakhaNER) are pending. This is a v1 release intended to establish a baseline and receive community feedback.
+
+Downstream fine-tuning experiments:
+- Sepedi sentiment classification (in progress — Sediba-AI/sepedi-sentiment-classifier)
+- EduIntel curriculum QA (deployed to Morutabana dashboard)
+
+---
+
+## Limitations
+
+- **Short sequence length**: Trained with max_length=64 due to GPU memory constraints. May underperform on longer documents. Future versions will train at max_length=512 with larger GPU allocation.
+- **1 epoch only**: Single-epoch training is a starting point. Further training will improve performance significantly.
+- **Informal register**: Training data is primarily formal text (government, education, religious). Performance on informal/conversational Sepedi may be limited.
+- **Dialect coverage**: Sepedi has multiple dialect zones (Sekhukhune, Balobedu, Batlokwa etc.). Current corpus does not tag dialect zones.
+
+---
+
+## Ethical Considerations
+
+### Data sovereignty
+This model was developed under the **Sediba Sovereignty Framework**:
+- Training data sourced from publicly licensed corpora
+- Community data collected via Leotša la Sepedi with contributor consent (FPIC framework)
+- Commercial use governed by TSEBO SOVEREIGN TECH (Pty) Ltd — **25% of commercial revenue flows to Sediba AI NPC** (community non-profit)
+
+### Language sovereignty
+Sepedi speakers have historically been excluded from AI development. This model is built **by and for** the Sepedi-speaking community in Limpopo, with the explicit goal of returning AI capability to the community that speaks the language.
+
+### Bias and risks
+- Religious text (Sepedi Bible) may introduce theological framing in certain contexts
+- Government/educational text may reflect formal register bias
+- Model reflects biases present in web-crawled text (FineWeb-2)
+
+---
+
+## How to Use
+
+```python
+from transformers import AutoTokenizer, AutoModelForMaskedLM
+import torch
+
+tokenizer = AutoTokenizer.from_pretrained("Sediba-AI/sepedi-llama-v1")
+model = AutoModelForMaskedLM.from_pretrained("Sediba-AI/sepedi-llama-v1")
+
+# Example: fill-mask in Sepedi
+text = "Baithuti ba <mask> go bala dipuku tša Sepedi."
+inputs = tokenizer(text, return_tensors="pt")
+
+with torch.no_grad():
+    outputs = model(**inputs)
+
+# Get top predictions for masked token
+mask_idx = (inputs["input_ids"] == tokenizer.mask_token_id).nonzero()[0][1]
+logits = outputs.logits[0, mask_idx]
+top_tokens = tokenizer.convert_ids_to_tokens(logits.topk(5).indices)
+print("Top predictions:", top_tokens)
+```
+
+### Fine-tuning for classification
+
+```python
+from transformers import AutoModelForSequenceClassification
+
+model = AutoModelForSequenceClassification.from_pretrained(
+    "Sediba-AI/sepedi-llama-v1",
+    num_labels=3  # e.g. positive/neutral/negative
+)
+```
+
+---
+
+## Citation
+
+```bibtex
+@misc{sepedi-llama-v1-2026,
+  author       = {Lehlohonolo Lemekoana and Sediba AI},
+  title        = {Sepedi-Llama v1: Continued Pre-training of XLM-RoBERTa on Sepedi},
+  year         = {2026},
+  publisher    = {HuggingFace},
+  url          = {https://huggingface.co/Sediba-AI/sepedi-llama-v1},
+  note         = {TSEBO SOVEREIGN TECH (Pty) Ltd, Mankweng, Limpopo, South Africa}
+}
+```
+
+---
+
+## Related Resources
+
+| Resource | Link |
+|----------|------|
+| Training dataset | [Sediba-AI/sepedi-training-v1](https://huggingface.co/datasets/Sediba-AI/sepedi-training-v1) |
+| Sentiment classifier | [Sediba-AI/sepedi-sentiment-classifier](https://huggingface.co/Sediba-AI/sepedi-sentiment-classifier) |
+| Data collection platform | [Leotša la Sepedi](https://github.com/Sediba-AI) |
+| Organisation | [Sediba-AI on HuggingFace](https://huggingface.co/Sediba-AI) |
+| GitHub | [github.com/Sediba-AI](https://github.com/Sediba-AI) |
+
+---
+
+## Acknowledgements
+
+- **SADiLaR** (South African Centre for Digital Language Resources) — Autshumato and NCHLT corpora
+- **CTexT, NWU** — NCHLT speech and text resources  
+- **Masakhane** — African NLP community and benchmarks
+- **Department of Basic Education, South Africa** — NSC exam papers and Annual Teaching Plans (public domain)
+- **HuggingFaceFW** — FineWeb-2 NSO subset
+- **Anri Lombard** — MzansiLM reference architecture
+
+---
+
+*Built in Mankweng, Limpopo. Powering the invisible foundations.*  
+*Sediba AI — Making the unseen ours and seen.*