File size: 8,720 Bytes
b770ded e55c8a4 9411875 e55c8a4 9411875 e55c8a4 9411875 e55c8a4 9411875 e55c8a4 9411875 4b2ea75 9411875 e55c8a4 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 | ---
license: gemma
language:
- si
base_model: QBitsLK/gemma4-e4b-sinhala-cpt
tags:
- sinhala
- multiple-choice
- sinhalammlu
- shared-task
---
# Team QBitsLK – Sinhala MMLU Shared Task System
## System summary
- **Base model(s):** `google/gemma-4-E4B` (text-only), with Sinhala continued pretraining, then LoRA supervised fine-tuning on examination-paper MCQs.
- **What we did:** We built a 7,872-question Sinhala multiple-choice dataset by extracting from publicly available Sri Lankan O/L and Grade 6–11 past papers, cleaning OCR artefacts and decontaminating against the released SinhalaMMLU test split. We continued pretraining Gemma 4 E4B on 1M sentences from the public `Minuri/diverse_sinhala_dataset` corpus (LoRA r=128, alpha=32, rsLoRA, packed same-domain pseudo-documents, 1 epoch), then fine-tuned that checkpoint with LoRA (r=32, alpha=32, 2 epochs) using answer-only loss, and merged the adapter back into the weights. At inference each option is scored by per-option log-likelihood under a 5-prompt ensemble (Sinhala and English instructions, subject intro on/off, two option-order rotations) rather than by generation. Nothing is sampled.
- **Total inference parameters:** **7,996,157,418** (one merged model; ≤ 8B)
- **Extra data used for training:**
1. **`Minuri/diverse_sinhala_dataset`** (file `high_confidence_7_7M.csv`) - public Sinhala text corpus, used for the **continued-pretraining** stage. <https://huggingface.co/datasets/Minuri/diverse_sinhala_dataset>
2. **`SinhalaMMLU-SharedTask/team-qbits-mcq-data`** - our own past-paper MCQ dataset (7,872 questions), used for the **supervised fine-tuning** stage.
No retrieval, no external APIs, no closed models.
## Models and resources in this submission
| Repo | What it is |
|---|---|
| `SinhalaMMLU-SharedTask/team-qbits` | code, inference script, final predictions (this repo) |
| `SinhalaMMLU-SharedTask/team-qbits-cpt-v3` | **the submitted model** — merged 16-bit, 7,996,157,418 params |
| `QBitsLK/gemma4-e4b-sinhala-cpt` | the continued-pretrained **base** we fine-tuned (Gemma 4 E4B + Sinhala CPT), merged 16-bit |
| `SinhalaMMLU-SharedTask/team-qbits-cpt-v3-lora` | the LoRA adapter for the above (r=32), for inspection |
| `SinhalaMMLU-SharedTask/team-qbits-base-v3b` | ablation — same recipe on the plain Gemma 4 E4B base, merged |
| `SinhalaMMLU-SharedTask/team-qbits-base-v3b-lora` | the LoRA adapter for the ablation |
| `SinhalaMMLU-SharedTask/team-qbits-mcq-data` | the SFT dataset (prompt/completion files) |
`inference.py` uses **only** `team-qbits-cpt-v3`. The other repos are provided for transparency and
are not part of the inference pipeline.
## How to run
```bash
pip install -r requirements.txt
python inference.py
```
- **GPU used:** 1x NVIDIA A40 48 GB (any GPU with ≥ 20 GB should work; the model is 16 GB in bf16)
- **Approximate run time on the test set:** ~18 minutes for 2,238 questions (5 prompt variants × 4 options = ~44,800 scored candidates); peak VRAM ~27 GB
- **Anything else the organizers need to know:**
- The Gemma license must be accepted on the Hugging Face account used to run this. Our merged model is self-contained, so no separate base-model download is required.
- `inference.py` scores options by log-likelihood and never calls `generate`, so there is no sampling. With the fixed seed, repeated runs produce byte-identical `predictions.jsonl`.
- `VARIANTS` at the top of `inference.py` defines the prompt ensemble. **Changing that list changes the predictions**; it is set exactly as used for our final Codabench submission.
- The reference `util_prompt.py` hard-codes the *hard* instruction as "1, 2, 3, 4, 5", which was correct for the original five-option hard split. SinhalaMMLU 2.0 reduced hard questions to four options, so `fit_numbers()` derives the number list from the actual option count. Verified: 0 of 2,238 prompts disagree with their option count.
## Results (Codabench)
| system | easy | medium | hard | **macro** |
|---|---|---|---|---|
| **submitted** — `team-qbits-cpt-v3`, 5-prompt ensemble | 58.10 | 62.56 | 52.79 | **57.81** |
| the same model with the adapter applied at load time instead of merged | 58.26 | 62.02 | 52.58 | 57.62 |
| ablation — `team-qbits-base-v3b` (plain Gemma 4 base), 5-prompt ensemble | 57.01 | 63.10 | 51.34 | 57.15 |
| ablation — plain base, single prompt, no ensemble | — | — | — | 55.66 |
`inference.py` reproduces the submitted row exactly: it loads the merged model, which is the system
we ship. Row 2 is the same fine-tune evaluated with the LoRA applied at load time (base + adapter).
That form is 8,069,557,738 parameters and therefore **exceeds the 8B cap**, so it is not the
submitted system; it is listed only because merging is mathematically equivalent yet not
bit-identical in bf16 (`W + BA` rounded once, versus `Wx + B(Ax)`), which resolves about 1% of
near-tied questions differently.
## Files
- `inference.py` – the exact script that produced our final Codabench submission
- `predictions.jsonl` – the exact file we submitted to the final leaderboard (macro 57.81),
byte-for-byte reproducible by running `inference.py`
- `predictions_ablation_base_v3b.jsonl` – an earlier submission from the ablation model
(`team-qbits-base-v3b`, macro 57.15), included for transparency. To regenerate it, set
`MODEL_ID` to `SinhalaMMLU-SharedTask/team-qbits-base-v3b` and change nothing else.
- `requirements.txt` – package versions we used
- `training/` – training, fine-tuning and data-preparation code
- `cpt_gemma4_sinhala.py` – stage 1: continued pretraining on `Minuri/diverse_sinhala_dataset`
- `eval_cpt_success.py` – stage 1 diagnostics (held-out bits-per-character, English regression)
- `sft_gemma4_mcq.py` – stage 2: LoRA SFT with answer-only loss and candidate-scoring evaluation
- `mcq_dataset.py` – extraction cleanup, validation, decontamination, splitting, prompt building
- `add_new_data.py`, `normalize_subjects.py` – dataset assembly and subject-name alignment
- `merge_adapter.py`, `push_merged_to_hf.py` – adapter merging and publishing
- `eval_sinhalammlu.py` – read-only analysis; also builds the decontamination blocklist
- `check_env.py` – dependency verification
- `run_data_pipeline.sh`, `run_training.sh`, `run_inference.sh` – end-to-end drivers
- `REPRODUCE.md` – exact commands, expected counts and the training curve
## Training pipeline (two stages)
**Stage 1 - continued pretraining.** `training/cpt_gemma4_sinhala.py` on `Minuri/diverse_sinhala_dataset` (`high_confidence_7_7M.csv`)[Minuri/diverse_sinhala_dataset], 1M sentences, domain-balanced sampling, 16 same-domain sentences joined into each pseudo-document, packing enabled, LoRA r=128 alpha=32 with rsLoRA, 1 epoch on one A40. Result: `QBitsLK/gemma4-e4b-sinhala-cpt` (merged). Diagnostics in `training/eval_cpt_success.py` - held-out bits-per-character improved 19.6% while English perplexity regressed 21.7%.
```bash
python training/cpt_gemma4_sinhala.py --max_sentences 1000000 --epochs 1
```
**Stage 2 - supervised fine-tuning.** `training/sft_gemma4_mcq.py` on our MCQ dataset, LoRA r=32 alpha=32, lr 1e-4, 2 epochs, batch 8 x accumulation 4, answer-only loss; checkpoint `final_adapter` (step 1328), merged into the stage-1 weights to give the submitted model.
```bash
bash training/run_training.sh
```
Note the CPT stage used an r=128 adapter and the SFT stage r=32; both were merged, and merging adds no parameters, so the submitted model is 7,996,157,418 - the same count as the Gemma 4 E4B lineage.
## Data and decontamination
Source: publicly available Sri Lankan examination papers (`pastpapers.wiki`,
`e-thaksalawa.moe.gov.lk`), O/L and Grades 6–11. We do not redistribute the source PDFs; the derived
question set is published at `team-qbits-mcq-data`.
The released `TEST/easy` split was used **only** as a decontamination blocklist. It matched and
removed **26** questions from our training data. It was never used for training, checkpoint
selection, prompt design or hyperparameter choice. Model selection used a held-out split of our own
data, grouped so that no source paper appears in both training and validation.
## Citation
This work evaluates on the SinhalaMMLU benchmark:
> Ashmari Pramodya, Nirasha Nelki, Heshan Shalinda, Chamila Liyanage, Yusuke Sakai, Randil
> Pushpananda, Ruvan Weerasinghe, Hidetaka Kamigaito, and Taro Watanabe. 2025. **SinhalaMMLU: A
> Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhala.** In
> *Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, pages
> 32943–32961. https://aclanthology.org/2025.emnlp-main.1673/
|