| # Data log |
|
|
| Every dataset touched by this project, how it is used, and every closed-API data-generation call (required |
| disclosure). Task data is never committed; `scripts/fetch_data.py` re-downloads it at the pinned revision. |
|
|
| ## Evaluation data |
|
|
| | Dataset | Revision | License | Use | Notes | |
| |---|---|---|---|---| |
| | SinhalaMMLU public Easy tier (`naist-nlp/SinhalaMMLU`, TEST/easy) | `7dd30681d9` | CC BY-NC-ND 4.0 | Dev evaluation | 1,851 questions, 14 subjects, grades 6-8, 4 options. `Geography/64` dropped (answer label `24`). Cite Pramodya et al., EMNLP 2025. | |
| | Global-MMLU `si` (`CohereLabs/Global-MMLU`) | `0e619dbeb3` | Apache-2.0 | Harder internal validation: fixed 2,000-question stratified sample of `test` (ids in `configs/splits/gmmlu_si_val_ids.txt`); `dev` split = few-shot pool | Validation ids must be excluded from any training mix. | |
| | Shared-task test set (`SinhalaMMLU-SharedTask/Sinhala-MMLU-SharedTask-Test-Dataset`) | `56fb2da50c` | (organisers) | **Prediction only.** Never used for training, tuning, prompt design or model selection; no answers included | 2,238 questions (easy 642, medium 1,111, hard 485), 4 options, same JSON layout as dev. Stored in `data/test/` (git-ignored). | |
| | Belebele `sin_Sinh` (`facebook/belebele`) | `7899cdfa4e` | CC BY-SA 4.0 | **Evaluation only**, never training | 900 reading-comprehension questions. | |
|
|
| ## Training data |
|
|
| | Dataset | Revision | License | Use | Processing | |
| |---|---|---|---|---| |
| | Global-MMLU `si` test (minus the 2,000 validation ids) + dev split | `0e619dbeb3` | Apache-2.0 | Fine-tuning MCQs: `data/sft/gmmlu_si_train.jsonl` via `scripts/build_sft_gmmlu.py` | 12,327 pool → 11,447 kept. Dropped 819 with char-5-gram TF-IDF cosine ≥ 0.6 to any validation item (MMLU repeats questions across subjects) and 61 within-pool copies (≥ 0.95). Max similarity of kept items to SinhalaMMLU dev: 0.42. Options shuffled to balanced gold positions except options that refer to other options. Stats: `reports/data_stats_sft_gmmlu.json` | |
| | FineWeb-2 `sin_Sinh` (`HuggingFaceFW/fineweb-2`) | `af9c13333e` | ODC-By 1.0 | Continued pretraining | `scripts/build_cpt_corpus.py` (filters below). Already MinHash-deduplicated upstream | |
| | Sinhala Wikipedia (`wikimedia/wikipedia`, `20231101.si`) | `b04c8d1ceb` | CC BY-SA 3.0 / GFDL | Continued pretraining | Same filters | |
| | NSINA news (`sinhala-nlp/NSINA`) | `a986af81ad` | CC BY-SA 4.0 | Continued pretraining | Same filters | |
|
|
| Corpus filters (`scripts/build_cpt_corpus.py`), first failing rule recorded per document; counts in |
| `reports/corpus_stats.json`: blocked exam / past-paper / curriculum domains; exam-like text (question-paper phrases or |
| ≥ 8 numbered option lines); < 200 characters; < 50% Sinhala letters; < 30% unique lines; exact duplicate across |
| sources (normalised text); any word 8-gram shared with a SinhalaMMLU dev or Global-MMLU validation question. |
| Not yet available: CulturaX `si` (gated, access not requested), MADLAD-400 `si` (path not found). |
|
|
| ### Contamination controls (apply to every training/pretraining source) |
|
|
| - SinhalaMMLU dev questions come from **pastpapers.wiki** (93%) and **e-thaksalawa.moe.gov.lk** (5%), i.e. Sri Lankan |
| term-test and exam papers. Drop all documents from these domains, and from any other past-paper / exam site, from |
| web corpora (FineWeb-2, CulturaX, MADLAD-400, CC-100, OSCAR) before training. |
| - N-gram overlap filter of every training corpus against SinhalaMMLU dev and the Global-MMLU validation split. |
| - No scraped Sri Lankan O/L or A/L past papers. Ask the organisers before using government textbooks. |
|
|
| ## Closed-API data generation (disclosure) |
|
|
| _None yet._ For each use: date, provider and model, purpose, prompt template location, number of items generated, |
| filtering applied, and where the output is stored. |
|
|
