Data log
Every dataset touched by this project, how it is used, and every closed-API data-generation call (required
disclosure). Task data is never committed; scripts/fetch_data.py re-downloads it at the pinned revision.
Evaluation data
| Dataset | Revision | License | Use | Notes |
|---|---|---|---|---|
SinhalaMMLU public Easy tier (naist-nlp/SinhalaMMLU, TEST/easy) |
7dd30681d9 |
CC BY-NC-ND 4.0 | Dev evaluation | 1,851 questions, 14 subjects, grades 6-8, 4 options. Geography/64 dropped (answer label 24). Cite Pramodya et al., EMNLP 2025. |
Global-MMLU si (CohereLabs/Global-MMLU) |
0e619dbeb3 |
Apache-2.0 | Harder internal validation: fixed 2,000-question stratified sample of test (ids in configs/splits/gmmlu_si_val_ids.txt); dev split = few-shot pool |
Validation ids must be excluded from any training mix. |
Shared-task test set (SinhalaMMLU-SharedTask/Sinhala-MMLU-SharedTask-Test-Dataset) |
56fb2da50c |
(organisers) | Prediction only. Never used for training, tuning, prompt design or model selection; no answers included | 2,238 questions (easy 642, medium 1,111, hard 485), 4 options, same JSON layout as dev. Stored in data/test/ (git-ignored). |
Belebele sin_Sinh (facebook/belebele) |
7899cdfa4e |
CC BY-SA 4.0 | Evaluation only, never training | 900 reading-comprehension questions. |
Training data
| Dataset | Revision | License | Use | Processing |
|---|---|---|---|---|
Global-MMLU si test (minus the 2,000 validation ids) + dev split |
0e619dbeb3 |
Apache-2.0 | Fine-tuning MCQs: data/sft/gmmlu_si_train.jsonl via scripts/build_sft_gmmlu.py |
12,327 pool → 11,447 kept. Dropped 819 with char-5-gram TF-IDF cosine ≥ 0.6 to any validation item (MMLU repeats questions across subjects) and 61 within-pool copies (≥ 0.95). Max similarity of kept items to SinhalaMMLU dev: 0.42. Options shuffled to balanced gold positions except options that refer to other options. Stats: reports/data_stats_sft_gmmlu.json |
FineWeb-2 sin_Sinh (HuggingFaceFW/fineweb-2) |
af9c13333e |
ODC-By 1.0 | Continued pretraining | scripts/build_cpt_corpus.py (filters below). Already MinHash-deduplicated upstream |
Sinhala Wikipedia (wikimedia/wikipedia, 20231101.si) |
b04c8d1ceb |
CC BY-SA 3.0 / GFDL | Continued pretraining | Same filters |
NSINA news (sinhala-nlp/NSINA) |
a986af81ad |
CC BY-SA 4.0 | Continued pretraining | Same filters |
Corpus filters (scripts/build_cpt_corpus.py), first failing rule recorded per document; counts in
reports/corpus_stats.json: blocked exam / past-paper / curriculum domains; exam-like text (question-paper phrases or
≥ 8 numbered option lines); < 200 characters; < 50% Sinhala letters; < 30% unique lines; exact duplicate across
sources (normalised text); any word 8-gram shared with a SinhalaMMLU dev or Global-MMLU validation question.
Not yet available: CulturaX si (gated, access not requested), MADLAD-400 si (path not found).
Contamination controls (apply to every training/pretraining source)
- SinhalaMMLU dev questions come from pastpapers.wiki (93%) and e-thaksalawa.moe.gov.lk (5%), i.e. Sri Lankan term-test and exam papers. Drop all documents from these domains, and from any other past-paper / exam site, from web corpora (FineWeb-2, CulturaX, MADLAD-400, CC-100, OSCAR) before training.
- N-gram overlap filter of every training corpus against SinhalaMMLU dev and the Global-MMLU validation split.
- No scraped Sri Lankan O/L or A/L past papers. Ask the organisers before using government textbooks.
Closed-API data generation (disclosure)
None yet. For each use: date, provider and model, purpose, prompt template location, number of items generated, filtering applied, and where the output is stored.