# Reproducing our SinhalaMMLU Shared Task submission

Team: QBitsLK · ICTer 2026 SinhalaMMLU Shared Task

Everything below runs with no human in the loop. Inference is deterministic: no sampling, no
generation, fixed seeds, and the prediction is an argmax over summed log-softmax scores.

## 1. Quick start — reproduce the submitted predictions

```bash
pip install unsloth
pip install --no-deps transformers==5.5.0 "tokenizers>=0.22.0,<=0.23.0"
pip install "huggingface_hub>=1.5.0,<2.0"
pip install --no-deps --upgrade timm
pip install hf_transfer

python check_env.py                 # verifies the whole stack; exits non-zero on a problem

bash run_inference.sh /path/to/test_root ./out
```

`test_root` is the directory holding `easy/`, `medium/` and `hard/`. The script writes
`out/predictions.jsonl` and `out/predictions.zip` (with `predictions.jsonl` at the archive root).

Hardware used for every reported run: one **NVIDIA A40, 48 GB**, driver 595.91, CUDA 13.2.
Inference over the 2,238-question test set takes about **18 minutes** and peaks around 27 GB of VRAM.

## 2. The submitted system in one line

| | |
|---|---|
| model | `QBitsLK/gemma4-e4b-sinhala-mcq-cpt-v3` (merged 16-bit) |
| parameters | **7,996,156,448** — the entire inference pipeline, one model, ≤ 8B |
| base | `QBitsLK/gemma4-e4b-sinhala-cpt` (Gemma 4 E4B + Sinhala continued pretraining) |
| adapter | LoRA r=32, α=32, merged into the base; merging adds no parameters |
| scoring | per-option log-likelihood, 5-prompt ensemble, argmax |
| prompt variants | `si_intro si_nointro en_intro si_rot1 si_rot2` |
| seed | 3407 |

The parameter count can be checked without downloading the weights:

```bash
python -c "
from huggingface_hub import get_safetensors_metadata
t = sum(get_safetensors_metadata('QBitsLK/gemma4-e4b-sinhala-mcq-cpt-v3').parameter_count.values())
print(f'{t:,}', 'OK, <= 8B' if t <= 8_000_000_000 else 'OVER 8B')"
```

**The adapter is merged on purpose.** Base + adapter loaded separately is 8,069,556,768 parameters,
which exceeds the cap; merging folds the LoRA into the existing weight matrices and returns the model
to 7,996,156,448.

## 3. Results reproduced by this package

Codabench, hidden test set (2,238 questions: easy 642, medium 1,111, hard 485):

| run | easy | medium | hard | **macro** |
|---|---|---|---|---|
| **submitted** — CPT + v3 data, 5-prompt ensemble | 58.26 | 62.02 | **52.58** | **57.62** |
| ablation — base Gemma 4 + v3 data, 5-prompt ensemble | 57.01 | 63.10 | 51.34 | 57.15 |
| ablation — base Gemma 4 + v3 data, single prompt | — | — | — | 55.66 |

To reproduce the first ablation, change `MODEL` in `run_inference.sh` to
`QBitsLK/gemma4-e4b-sinhala-mcq-base-v3b`. For the second, set `VARIANTS="si_intro"`.

## 4. Full reproduction from scratch

### 4.1 Rebuild the training data (CPU, a few minutes)

```bash
bash run_data_pipeline.sh
```

Requires the raw extraction files in `./data` (see section 5). Expected output: **7,872** questions
after deduplication, split into **7,151 train → 21,235 examples** and **721 validation**.

The public `TEST/easy` split is downloaded by this script and used **only** as a decontamination
blocklist. It removed **26 questions** that our past-paper extractions shared with the released test
data. It is never used for training, checkpoint selection, prompt design or hyperparameter choice.

### 4.2 Re-run the fine-tune (1 GPU, 88 minutes)

```bash
bash run_training.sh /workspace/outputs/mcq_sft_cpt_v3
```

This downloads the published SFT files from
`https://huggingface.co/datasets/QBitsLK/sinhala-mcq-sft-v3`, so section 4.1 is optional if you only
want to reproduce training.

Expected validation curve (721 held-out questions, accuracy):

| step | 0 | 150 | 300 | 450 | 600 | 664 | 750 | 900 | 1050 | 1200 | 1328 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| acc | .2316 | .5520 | .5395 | .5728 | .5506 | .5562 | .5867 | .5645 | .5742 | .5825 | **.5811** |

**The submitted checkpoint is `final_adapter` (step 1328), not `best_adapter`.** Rationale in
`SYSTEM_DESCRIPTION.md` §5.2: on a parallel run the validation-best checkpoint scored 0.7 points
*worse* on held-out test data than the final one, because our validation split shares the training
distribution and does not detect this failure mode.

### 4.3 Publish the merged model

```bash
export HF_TOKEN=...
python push_merged_to_hf.py \
  --adapter /workspace/outputs/mcq_sft_cpt_v3/final_adapter \
  --repo <your-namespace>/gemma4-e4b-sinhala-mcq-cpt-v3
```

Streams the merge directly to the Hub, so no 16 GB local copy is needed, and reads the parameter
count back for the ≤8B check.

## 5. Files

| file | purpose |
|---|---|
| `run_inference.sh` | reproduces the submitted predictions (the critical path) |
| `run_training.sh` | reproduces the fine-tune |
| `run_data_pipeline.sh` | rebuilds the SFT dataset from raw extractions |
| `make_predictions.py` | prompt builder, ensemble scorer, submission writer, format checks |
| `sft_gemma4_mcq.py` | LoRA SFT with answer-only loss; per-step candidate-scoring evaluation |
| `mcq_dataset.py` | extraction cleanup, validation, decontamination, splitting, prompt building |
| `add_new_data.py` | merges per-subject extractions into the dataset |
| `normalize_subjects.py` | aligns subject names with the official `metadata.subject_original` |
| `eval_sinhalammlu.py` | read-only analysis; also builds the decontamination blocklist |
| `push_merged_to_hf.py` | merges the adapter and publishes it |
| `check_env.py` | verifies the dependency stack before a run |
| `requirements.txt` | exact pins and install order |

**Not included:** the source examination PDFs. They are third-party past papers obtained from
publicly available sources (`pastpapers.wiki`, `e-thaksalawa.moe.gov.lk`), and we do not hold
redistribution rights. The derived question set — questions, options and answer keys — is published
at `QBitsLK/sinhala-mcq-sft-v3`, which is sufficient to reproduce training and inference. The raw
per-subject extraction files are included so `run_data_pipeline.sh` runs end-to-end.

## 6. Determinism

- No sampling anywhere; `model.generate` is never called on the submission path.
- `random.seed(3407)` and `torch.manual_seed(3407)`.
- The prediction is `argmax` over log-softmax sums; ties resolve to the lowest option index.
- Prompt variants are explicit in `run_inference.sh`; changing them changes the predictions.
- `make_predictions.py` refuses to write the zip unless the line count equals the question count,
  every line carries exactly `{q_no, subject, category, level, answer}`, `answer` is a `"1"`–`"5"`
  string, and every `level` is `easy`/`medium`/`hard`.

One deliberate deviation from the organisers' reference code: `evaluate.py` hard-codes the *hard*
instruction as "1, 2, 3, 4, 5" because the original SinhalaMMLU hard split had five options.
SinhalaMMLU 2.0 drops one incorrect option, so hard questions now have four. `fit_numbers()` in
`make_predictions.py` rewrites the number list to match the actual option count (verified: 0 of 2,238
prompts disagree). Without this, all 485 hard questions would be asked to choose from five options
when only four are shown.

## 7. Citation

This work evaluates on the SinhalaMMLU benchmark:

```bibtex
@inproceedings{pramodya-etal-2025-sinhalammlu,
    title = "SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhala",
    author = "Pramodya, Ashmari and Nelki, Nirasha and Shalinda, Heshan and Liyanage, Chamila and
              Sakai, Yusuke and Pushpananda, Randil and Weerasinghe, Ruvan and
              Kamigaito, Hidetaka and Watanabe, Taro",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    year = "2025",
    pages = "32943--32961",
    url = "https://aclanthology.org/2025.emnlp-main.1673/"
}
```

Verify this against the BibTeX on <https://sinhalammlu.ucsc.cmb.ac.lk/task.html> before submitting;
it is reproduced here from that page and the arXiv preprint (arXiv:2509.03162).