| # Reproducing our SinhalaMMLU Shared Task submission |
|
|
| Team: QBitsLK Β· ICTer 2026 SinhalaMMLU Shared Task |
|
|
| Everything below runs with no human in the loop. Inference is deterministic: no sampling, no |
| generation, fixed seeds, and the prediction is an argmax over summed log-softmax scores. |
|
|
| ## 1. Quick start β reproduce the submitted predictions |
|
|
| ```bash |
| pip install unsloth |
| pip install --no-deps transformers==5.5.0 "tokenizers>=0.22.0,<=0.23.0" |
| pip install "huggingface_hub>=1.5.0,<2.0" |
| pip install --no-deps --upgrade timm |
| pip install hf_transfer |
| |
| python check_env.py # verifies the whole stack; exits non-zero on a problem |
| |
| bash run_inference.sh /path/to/test_root ./out |
| ``` |
|
|
| `test_root` is the directory holding `easy/`, `medium/` and `hard/`. The script writes |
| `out/predictions.jsonl` and `out/predictions.zip` (with `predictions.jsonl` at the archive root). |
|
|
| Hardware used for every reported run: one **NVIDIA A40, 48 GB**, driver 595.91, CUDA 13.2. |
| Inference over the 2,238-question test set takes about **18 minutes** and peaks around 27 GB of VRAM. |
|
|
| ## 2. The submitted system in one line |
|
|
| | | | |
| |---|---| |
| | model | `QBitsLK/gemma4-e4b-sinhala-mcq-cpt-v3` (merged 16-bit) | |
| | parameters | **7,996,156,448** β the entire inference pipeline, one model, β€ 8B | |
| | base | `QBitsLK/gemma4-e4b-sinhala-cpt` (Gemma 4 E4B + Sinhala continued pretraining) | |
| | adapter | LoRA r=32, Ξ±=32, merged into the base; merging adds no parameters | |
| | scoring | per-option log-likelihood, 5-prompt ensemble, argmax | |
| | prompt variants | `si_intro si_nointro en_intro si_rot1 si_rot2` | |
| | seed | 3407 | |
|
|
| The parameter count can be checked without downloading the weights: |
|
|
| ```bash |
| python -c " |
| from huggingface_hub import get_safetensors_metadata |
| t = sum(get_safetensors_metadata('QBitsLK/gemma4-e4b-sinhala-mcq-cpt-v3').parameter_count.values()) |
| print(f'{t:,}', 'OK, <= 8B' if t <= 8_000_000_000 else 'OVER 8B')" |
| ``` |
|
|
| **The adapter is merged on purpose.** Base + adapter loaded separately is 8,069,556,768 parameters, |
| which exceeds the cap; merging folds the LoRA into the existing weight matrices and returns the model |
| to 7,996,156,448. |
|
|
| ## 3. Results reproduced by this package |
|
|
| Codabench, hidden test set (2,238 questions: easy 642, medium 1,111, hard 485): |
|
|
| | run | easy | medium | hard | **macro** | |
| |---|---|---|---|---| |
| | **submitted** β CPT + v3 data, 5-prompt ensemble | 58.26 | 62.02 | **52.58** | **57.62** | |
| | ablation β base Gemma 4 + v3 data, 5-prompt ensemble | 57.01 | 63.10 | 51.34 | 57.15 | |
| | ablation β base Gemma 4 + v3 data, single prompt | β | β | β | 55.66 | |
|
|
| To reproduce the first ablation, change `MODEL` in `run_inference.sh` to |
| `QBitsLK/gemma4-e4b-sinhala-mcq-base-v3b`. For the second, set `VARIANTS="si_intro"`. |
|
|
| ## 4. Full reproduction from scratch |
|
|
| ### 4.1 Rebuild the training data (CPU, a few minutes) |
|
|
| ```bash |
| bash run_data_pipeline.sh |
| ``` |
|
|
| Requires the raw extraction files in `./data` (see section 5). Expected output: **7,872** questions |
| after deduplication, split into **7,151 train β 21,235 examples** and **721 validation**. |
|
|
| The public `TEST/easy` split is downloaded by this script and used **only** as a decontamination |
| blocklist. It removed **26 questions** that our past-paper extractions shared with the released test |
| data. It is never used for training, checkpoint selection, prompt design or hyperparameter choice. |
|
|
| ### 4.2 Re-run the fine-tune (1 GPU, 88 minutes) |
|
|
| ```bash |
| bash run_training.sh /workspace/outputs/mcq_sft_cpt_v3 |
| ``` |
|
|
| This downloads the published SFT files from |
| `https://huggingface.co/datasets/QBitsLK/sinhala-mcq-sft-v3`, so section 4.1 is optional if you only |
| want to reproduce training. |
|
|
| Expected validation curve (721 held-out questions, accuracy): |
|
|
| | step | 0 | 150 | 300 | 450 | 600 | 664 | 750 | 900 | 1050 | 1200 | 1328 | |
| |---|---|---|---|---|---|---|---|---|---|---|---| |
| | acc | .2316 | .5520 | .5395 | .5728 | .5506 | .5562 | .5867 | .5645 | .5742 | .5825 | **.5811** | |
|
|
| **The submitted checkpoint is `final_adapter` (step 1328), not `best_adapter`.** Rationale in |
| `SYSTEM_DESCRIPTION.md` Β§5.2: on a parallel run the validation-best checkpoint scored 0.7 points |
| *worse* on held-out test data than the final one, because our validation split shares the training |
| distribution and does not detect this failure mode. |
|
|
| ### 4.3 Publish the merged model |
|
|
| ```bash |
| export HF_TOKEN=... |
| python push_merged_to_hf.py \ |
| --adapter /workspace/outputs/mcq_sft_cpt_v3/final_adapter \ |
| --repo <your-namespace>/gemma4-e4b-sinhala-mcq-cpt-v3 |
| ``` |
|
|
| Streams the merge directly to the Hub, so no 16 GB local copy is needed, and reads the parameter |
| count back for the β€8B check. |
|
|
| ## 5. Files |
|
|
| | file | purpose | |
| |---|---| |
| | `run_inference.sh` | reproduces the submitted predictions (the critical path) | |
| | `run_training.sh` | reproduces the fine-tune | |
| | `run_data_pipeline.sh` | rebuilds the SFT dataset from raw extractions | |
| | `make_predictions.py` | prompt builder, ensemble scorer, submission writer, format checks | |
| | `sft_gemma4_mcq.py` | LoRA SFT with answer-only loss; per-step candidate-scoring evaluation | |
| | `mcq_dataset.py` | extraction cleanup, validation, decontamination, splitting, prompt building | |
| | `add_new_data.py` | merges per-subject extractions into the dataset | |
| | `normalize_subjects.py` | aligns subject names with the official `metadata.subject_original` | |
| | `eval_sinhalammlu.py` | read-only analysis; also builds the decontamination blocklist | |
| | `push_merged_to_hf.py` | merges the adapter and publishes it | |
| | `check_env.py` | verifies the dependency stack before a run | |
| | `requirements.txt` | exact pins and install order | |
|
|
| **Not included:** the source examination PDFs. They are third-party past papers obtained from |
| publicly available sources (`pastpapers.wiki`, `e-thaksalawa.moe.gov.lk`), and we do not hold |
| redistribution rights. The derived question set β questions, options and answer keys β is published |
| at `QBitsLK/sinhala-mcq-sft-v3`, which is sufficient to reproduce training and inference. The raw |
| per-subject extraction files are included so `run_data_pipeline.sh` runs end-to-end. |
|
|
| ## 6. Determinism |
|
|
| - No sampling anywhere; `model.generate` is never called on the submission path. |
| - `random.seed(3407)` and `torch.manual_seed(3407)`. |
| - The prediction is `argmax` over log-softmax sums; ties resolve to the lowest option index. |
| - Prompt variants are explicit in `run_inference.sh`; changing them changes the predictions. |
| - `make_predictions.py` refuses to write the zip unless the line count equals the question count, |
| every line carries exactly `{q_no, subject, category, level, answer}`, `answer` is a `"1"`β`"5"` |
| string, and every `level` is `easy`/`medium`/`hard`. |
|
|
| One deliberate deviation from the organisers' reference code: `evaluate.py` hard-codes the *hard* |
| instruction as "1, 2, 3, 4, 5" because the original SinhalaMMLU hard split had five options. |
| SinhalaMMLU 2.0 drops one incorrect option, so hard questions now have four. `fit_numbers()` in |
| `make_predictions.py` rewrites the number list to match the actual option count (verified: 0 of 2,238 |
| prompts disagree). Without this, all 485 hard questions would be asked to choose from five options |
| when only four are shown. |
|
|
| ## 7. Citation |
|
|
| This work evaluates on the SinhalaMMLU benchmark: |
|
|
| ```bibtex |
| @inproceedings{pramodya-etal-2025-sinhalammlu, |
| title = "SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhala", |
| author = "Pramodya, Ashmari and Nelki, Nirasha and Shalinda, Heshan and Liyanage, Chamila and |
| Sakai, Yusuke and Pushpananda, Randil and Weerasinghe, Ruvan and |
| Kamigaito, Hidetaka and Watanabe, Taro", |
| booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing", |
| year = "2025", |
| pages = "32943--32961", |
| url = "https://aclanthology.org/2025.emnlp-main.1673/" |
| } |
| ``` |
|
|
| Verify this against the BibTeX on <https://sinhalammlu.ucsc.cmb.ac.lk/task.html> before submitting; |
| it is reproduced here from that page and the arXiv preprint (arXiv:2509.03162). |
|
|
