Upload .\README.md with huggingface_hub

.//README.md

@@ -0,0 +1,145 @@
+# BwengeAi - Rwandan AI that knows Kinyarwanda
+
+BwengeAi is a large-scale Kinyarwanda language model trained on 200 verified datasets from HuggingFace, Wikipedia, Rwandan news sources, and parallel corpora.
+
+## Current Stats
+
+| Metric | Value |
+|--------|-------|
+| Datasets | 200 |
+| Estimated Rows | 11.7B |
+| Estimated Tokens | 64.9B |
+
+## Project Structure
+
+```
+BwengeAi/
+├── configs/
+│   └── default.yaml              # All 200 dataset configs + model/training params
+├── src/
+│   ├── data_collection/          # 12 collection modules
+│   │   ├── huggingface_collector.py
+│   │   ├── parallel_corpus_collector.py
+│   │   ├── cc100_oscar_collector.py
+│   │   ├── masakhane_collector.py
+│   │   ├── wikipedia_collector.py
+│   │   ├── igihe_scraper.py
+│   │   ├── kigalitoday_scraper.py
+│   │   ├── rss_collector.py
+│   │   ├── wikimedia_collector.py
+│   │   ├── rbc_collector.py
+│   │   ├── web_utils.py          # robots.txt compliance
+│   │   └── data_processor.py     # Clean + deduplicate + language filter
+│   ├── models/
+│   │   └── bwenge_model.py
+│   ├── training/
+│   │   └── trainer.py
+│   └── evaluation/
+│       └── metrics.py
+├── scripts/
+│   ├── collect_all.py            # Main pipeline (12 steps)
+│   └── train.py
+├── data/
+│   ├── raw/                      # Collected JSONL files
+│   └── processed/                # Cleaned training data
+├── docs/
+│   ├── data-collection.md
+│   ├── training.md
+│   └── evaluation.md
+├── tests/
+│   └── test_basic.py
+└── pyproject.toml
+```
+
+## Data Sources (200 datasets)
+
+| Category | Count | Description |
+|----------|-------|-------------|
+| Monolingual Kinyarwanda | 3 | CircuitNotion, mbazaNLP |
+| Massive Web Crawl | 5 | fineweb-2, HPLT, CCMatrix, NLLB |
+| Parallel Corpus | 109 | michsethowusu sentence pairs |
+| Helsinki-NLP OPUS | 1 | 178K en-rw pairs |
+| AllenAI NLLB | 1 | NLLB-200 mined bitext |
+| CCAligned | 1 | Web-document pairs |
+| Instruction Tuning | 12 | Alpaca, African-Translated-Alpaca |
+| Chat/Conversation | 2 | Kinyarwanda chat, roleplay |
+| Wikipedia | 4 | rw wiki, wikidata, wikisource |
+| NER/POS | 3 | masakhaner, masakhapos |
+| QA Benchmarks | 13 | afriqa, uhura suite, tydiqa |
+| Sentiment | 3 | afrisenti, AfriSenti, safri |
+| NLI | 3 | afrixnli, afro-xnli, xnli |
+| Math/Logic | 2 | afrimgsm, uhura-gsm8k |
+| Dialogue | 1 | AfriADR |
+| Translation MT | 6 | flores, flores+, mafand, fleurs |
+| Code | 16 | Code-170k (multiple languages) |
+| Emotion | 7 | emotion-corpus (multiple languages) |
+| Speech-Text | 7 | speech-text (multiple languages) |
+| African Multilingual | 6 | NaolBM, afrolm, xlsum, LughaGen |
+| Health | 1 | DigitalUmuganda health |
+| News | 3 | KINNEWS, KIRNEWS, swahili_news |
+| Hate Speech | 1 | afri-hate-speech |
+| Common Voice | 2 | Kinyarwanda ASR transcripts |
+
+## Installation
+
+```bash
+git clone <repository-url>
+cd BwengeAi
+python -m venv venv
+source venv/bin/activate  # Windows: venv\Scripts\activate
+pip install -e ".[dev]"
+```
+
+## Usage
+
+### 1. Collect Data
+
+```bash
+python scripts/collect_all.py
+```
+
+Pipeline stages:
+1. HuggingFace datasets (streaming, with max_rows limits)
+2. Parallel corpora (michsethowusu, read from config)
+3. CC-100/OSCAR (with language validation)
+4. Masakhane benchmarks (filtered to Kinyarwanda)
+5. Wikipedia (API + XML dump with iterparse)
+6. Igihe news (with robots.txt)
+7. Kigali Today (with robots.txt)
+8. New Times RSS
+9. Wikisource
+10. Wiktionary
+11. RBC health content (with robots.txt)
+12. Process + deduplicate + language filter
+
+### 2. Train Model
+
+```bash
+python scripts/train.py
+```
+
+### 3. Evaluate
+
+Evaluation runs automatically after training.
+
+## Pipeline Fixes Applied
+
+| Issue | Fix |
+|-------|-----|
+| CC-100/OSCAR crash on `rw` | Language validation before loading, graceful skip |
+| HuggingFace OOM on large datasets | Streaming mode + per-dataset max_rows limits |
+| Parallel collector ignored 90+ datasets | Reads from config instead of hardcoded list |
+| Masakhane collected all languages | Added `kin_Latn` language filter |
+| No language detection | Added heuristic Kinyarwanda detection |
+| 20-char minimum dropped short texts | Reduced to 5 chars |
+| File copy name collisions | Prefixed with collector directory name |
+| No token counting | Added token counting at every stage |
+| No per-file processing stats | Added detailed per-file logging |
+| No streaming error recovery | Added .done marker files for resume |
+| Wikipedia XML parser fragile | Switched to ET.iterparse |
+| No robots.txt compliance | Added robots.txt checking to all scrapers |
+| Empty data/docs folders | Added .gitkeep, .gitignore, docs |
+
+## License
+
+MIT License