Reproducing our SinhalaMMLU Shared Task submission

Team: QBitsLK · ICTer 2026 SinhalaMMLU Shared Task

Everything below runs with no human in the loop. Inference is deterministic: no sampling, no generation, fixed seeds, and the prediction is an argmax over summed log-softmax scores.

1. Quick start — reproduce the submitted predictions

pip install unsloth
pip install --no-deps transformers==5.5.0 "tokenizers>=0.22.0,<=0.23.0"
pip install "huggingface_hub>=1.5.0,<2.0"
pip install --no-deps --upgrade timm
pip install hf_transfer

python check_env.py                 # verifies the whole stack; exits non-zero on a problem

bash run_inference.sh /path/to/test_root ./out

test_root is the directory holding easy/, medium/ and hard/. The script writes out/predictions.jsonl and out/predictions.zip (with predictions.jsonl at the archive root).

Hardware used for every reported run: one NVIDIA A40, 48 GB, driver 595.91, CUDA 13.2. Inference over the 2,238-question test set takes about 18 minutes and peaks around 27 GB of VRAM.

2. The submitted system in one line


model	`QBitsLK/gemma4-e4b-sinhala-mcq-cpt-v3` (merged 16-bit)
parameters	7,996,156,448 — the entire inference pipeline, one model, ≤ 8B
base	`QBitsLK/gemma4-e4b-sinhala-cpt` (Gemma 4 E4B + Sinhala continued pretraining)
adapter	LoRA r=32, α=32, merged into the base; merging adds no parameters
scoring	per-option log-likelihood, 5-prompt ensemble, argmax
prompt variants	`si_intro si_nointro en_intro si_rot1 si_rot2`
seed	3407

The parameter count can be checked without downloading the weights:

python -c "
from huggingface_hub import get_safetensors_metadata
t = sum(get_safetensors_metadata('QBitsLK/gemma4-e4b-sinhala-mcq-cpt-v3').parameter_count.values())
print(f'{t:,}', 'OK, <= 8B' if t <= 8_000_000_000 else 'OVER 8B')"

The adapter is merged on purpose. Base + adapter loaded separately is 8,069,556,768 parameters, which exceeds the cap; merging folds the LoRA into the existing weight matrices and returns the model to 7,996,156,448.

3. Results reproduced by this package

Codabench, hidden test set (2,238 questions: easy 642, medium 1,111, hard 485):

run	easy	medium	hard	macro
submitted — CPT + v3 data, 5-prompt ensemble	58.26	62.02	52.58	57.62
ablation — base Gemma 4 + v3 data, 5-prompt ensemble	57.01	63.10	51.34	57.15
ablation — base Gemma 4 + v3 data, single prompt	—	—	—	55.66

To reproduce the first ablation, change MODEL in run_inference.sh to QBitsLK/gemma4-e4b-sinhala-mcq-base-v3b. For the second, set VARIANTS="si_intro".

4. Full reproduction from scratch

4.1 Rebuild the training data (CPU, a few minutes)

bash run_data_pipeline.sh

Requires the raw extraction files in ./data (see section 5). Expected output: 7,872 questions after deduplication, split into 7,151 train → 21,235 examples and 721 validation.

The public TEST/easy split is downloaded by this script and used only as a decontamination blocklist. It removed 26 questions that our past-paper extractions shared with the released test data. It is never used for training, checkpoint selection, prompt design or hyperparameter choice.

4.2 Re-run the fine-tune (1 GPU, 88 minutes)

bash run_training.sh /workspace/outputs/mcq_sft_cpt_v3

This downloads the published SFT files from https://huggingface.co/datasets/QBitsLK/sinhala-mcq-sft-v3, so section 4.1 is optional if you only want to reproduce training.

Expected validation curve (721 held-out questions, accuracy):

step	0	150	300	450	600	664	750	900	1050	1200	1328
acc	.2316	.5520	.5395	.5728	.5506	.5562	.5867	.5645	.5742	.5825	.5811

The submitted checkpoint is final_adapter (step 1328), not best_adapter. Rationale in SYSTEM_DESCRIPTION.md §5.2: on a parallel run the validation-best checkpoint scored 0.7 points worse on held-out test data than the final one, because our validation split shares the training distribution and does not detect this failure mode.

4.3 Publish the merged model

export HF_TOKEN=...
python push_merged_to_hf.py \
  --adapter /workspace/outputs/mcq_sft_cpt_v3/final_adapter \
  --repo <your-namespace>/gemma4-e4b-sinhala-mcq-cpt-v3

Streams the merge directly to the Hub, so no 16 GB local copy is needed, and reads the parameter count back for the ≤8B check.

5. Files

file	purpose
`run_inference.sh`	reproduces the submitted predictions (the critical path)
`run_training.sh`	reproduces the fine-tune
`run_data_pipeline.sh`	rebuilds the SFT dataset from raw extractions
`make_predictions.py`	prompt builder, ensemble scorer, submission writer, format checks
`sft_gemma4_mcq.py`	LoRA SFT with answer-only loss; per-step candidate-scoring evaluation
`mcq_dataset.py`	extraction cleanup, validation, decontamination, splitting, prompt building
`add_new_data.py`	merges per-subject extractions into the dataset
`normalize_subjects.py`	aligns subject names with the official `metadata.subject_original`
`eval_sinhalammlu.py`	read-only analysis; also builds the decontamination blocklist
`push_merged_to_hf.py`	merges the adapter and publishes it
`check_env.py`	verifies the dependency stack before a run
`requirements.txt`	exact pins and install order

Not included: the source examination PDFs. They are third-party past papers obtained from publicly available sources (pastpapers.wiki, e-thaksalawa.moe.gov.lk), and we do not hold redistribution rights. The derived question set — questions, options and answer keys — is published at QBitsLK/sinhala-mcq-sft-v3, which is sufficient to reproduce training and inference. The raw per-subject extraction files are included so run_data_pipeline.sh runs end-to-end.

6. Determinism

No sampling anywhere; model.generate is never called on the submission path.
random.seed(3407) and torch.manual_seed(3407).
The prediction is argmax over log-softmax sums; ties resolve to the lowest option index.
Prompt variants are explicit in run_inference.sh; changing them changes the predictions.
make_predictions.py refuses to write the zip unless the line count equals the question count, every line carries exactly {q_no, subject, category, level, answer}, answer is a "1"–"5" string, and every level is easy/medium/hard.

One deliberate deviation from the organisers' reference code: evaluate.py hard-codes the hard instruction as "1, 2, 3, 4, 5" because the original SinhalaMMLU hard split had five options. SinhalaMMLU 2.0 drops one incorrect option, so hard questions now have four. fit_numbers() in make_predictions.py rewrites the number list to match the actual option count (verified: 0 of 2,238 prompts disagree). Without this, all 485 hard questions would be asked to choose from five options when only four are shown.

7. Citation

This work evaluates on the SinhalaMMLU benchmark:

@inproceedings{pramodya-etal-2025-sinhalammlu,
    title = "SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhala",
    author = "Pramodya, Ashmari and Nelki, Nirasha and Shalinda, Heshan and Liyanage, Chamila and
              Sakai, Yusuke and Pushpananda, Randil and Weerasinghe, Ruvan and
              Kamigaito, Hidetaka and Watanabe, Taro",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    year = "2025",
    pages = "32943--32961",
    url = "https://aclanthology.org/2025.emnlp-main.1673/"
}

Verify this against the BibTeX on https://sinhalammlu.ucsc.cmb.ac.lk/task.html before submitting; it is reproduced here from that page and the arXiv preprint (arXiv:2509.03162).