Upload README.md with huggingface_hub

README.md

@@ -33,12 +33,6 @@ tags:
 
 ## Datasets
 
-### Shared-task data (provided by the organisers)
-| Set | Questions | How we used it |
-|---|---|---|
-| Public training set (`SinhalaMMLU-SharedTask`, Easy level only, grades 6–8, 14 subjects, 4 options, with answers) | 1,851 | **Development and evaluation only**, never for training. 18 questions with a near-duplicate in the test set (character 5-gram Jaccard ≥ 0.5) were removed. The rest was split (seed 42) into a **screening set** of 280 (20 per subject) for comparing prompts and methods, a **few-shot pool** of 70 (5 per subject, the only source of in-context examples), and a **confirmation set** of 1,483 for confirming the screening winners. |
-| Test set (Easy / Medium / Hard, no answers) | 2,238 (642 / 1,111 / 485) | Only to produce the submission with `inference.py`. Never used for training. |
-
 ### Extra training data (for the LoRA adapter)
 | Dataset | Source and licence | Size | Preparation |
 |---|---|---|---|
@@ -89,7 +83,7 @@ Validation accuracy (2,745 held-out questions, digit prompt) during training:
 | Accuracy (%) | 44.4 | 47.4 | 49.9 | 51.7 | 51.7 | 52.8 | 53.2 | 54.5 | 54.8 |
 
 ### How we got here
-- **Baseline and prompts.** We first evaluated Gemma-3-4B-it with the prompts of the SinhalaMMLU paper and many variants on our development data: the public training questions, split into a 280-question screening set and a 1,483-question confirmation set. English step-by-step reasoning in the role of a Sri Lankan teacher was the clear winner: 45.6% vs 38.2% for the paper's prompt on the confirmation set.
+- **Baseline and prompts.** We first evaluated Gemma-3-4B-it with the prompts of the SinhalaMMLU paper and many variants on development data held out from the public training questions (a screening split and a separate confirmation split). English step-by-step reasoning in the role of a Sri Lankan teacher was the clear winner: 45.6% vs 38.2% for the paper's prompt on the confirmation set.
 - **Ideas that did not help enough.** Machine translation of the questions into English (NLLB, MADLAD, self-translation) often lost negations ("which is NOT …"). Few-shot examples, calibration, option-order averaging, self-consistency, a negation-check chain and prompt ensembles each changed accuracy by less than our +2-point threshold.
 - **Fine-tuning.** LoRA adapters on Global-MMLU Sinhala and on sinmulu_si improved Gemma-3-4B-it. In all our runs, validation accuracy peaked after one epoch.
 - **Final system.** We then moved to Qwen3.5-4B, a newer ≤ 8B open model, trained one LoRA epoch on the combined data, and switched on its thinking mode for the reasoning step.
@@ -98,14 +92,13 @@ Validation accuracy (2,745 held-out questions, digit prompt) during training:
 ## Results of our experiments
 
 All numbers are accuracy (%) on our own development data, never on the shared-task test set:
-- **Screening set:** 280 questions from the public training data.
-- **Confirmation set:** 1,483 different public training questions.
+- **Screening / confirmation:** two separate splits of the public training questions; methods were compared on the screening split and the best ones re-tested on the larger confirmation split.
 - **Validation sets:** held out from our fine-tuning data.
 
 A condition was adopted only if it beat the baseline by at least 1–2 points with an exact McNemar test (Holm-corrected) on the confirmation set.
 
 ### 1. Prompting Gemma-3-4B-it (no fine-tuning)
-| Prompt / condition | Screening (280) | Confirmation (1,483) |
+| Prompt / condition | Screening | Confirmation |
 |---|---|---|
 | Paper prompt, Sinhala, no subject (baseline A0) | 37.5 | 38.2 |
 | Paper prompt + subject (Sinhala) | 42.1 | – |
@@ -164,7 +157,7 @@ python inference.py
   - Our run also had `causal-conv1d==1.7.0` installed. It can't be installed from `requirements.txt` (it needs torch at build time). For identical kernels, run `pip install causal-conv1d==1.7.0 --no-build-isolation` after the requirements; without it, transformers uses its torch implementation.
   - Decoding is greedy and fully deterministic on a given GPU. Different GPUs, or different batch compositions, can change a small number of answers through bf16 rounding.
   - The final file was produced by our package (`training/sinmmlu`, `python -m sinmmlu.evaluate --model configs/models/qwen35_4b_lora_combined.yaml --data test --prompt cot_en_role_think --chat on --max-new-tokens 4096`). `inference.py` is a self-contained copy of that code path. We checked that it builds the same prompt for all 2,238 test questions and processes the reasoning texts the same way.
-  - On our side, the first 300 questions were generated in a separate run and reused, so batches were composed differently. A from-scratch run can therefore differ from `predictions.jsonl` in a few answers.
+  - On our side, part of the questions was generated in a separate run and reused, so batches were composed differently. A from-scratch run can therefore differ from `predictions.jsonl` in a few answers.
 
 ## Final run
 | | |