| --- |
| license: mit |
| --- |
| # CTRL — Sinhala MMLU Shared Task |
|
|
| Sinhala multiple-choice answering with a 4-bit Llama 3.2 3B base and choice-supervised LoRA adaptation. |
|
|
| **Members:** Jayendra Matarage · Prashan Samarathunge · Isuru Udayanga · Mithun Nimesh |
|
|
| [Team code and predictions](https://huggingface.co/SinhalaMMLU-SharedTask/team-ctrl) · [Final adapter](https://huggingface.co/SinhalaMMLU-SharedTask/team-ctrl-llama-lora) |
|
|
| ## Architecture |
|
|
| A frozen [Llama 3.2 3B Instruct base](https://huggingface.co/unsloth/Llama-3.2-3B-Instruct-bnb-4bit) with one PEFT LoRA adapter, loaded through Unsloth in NF4 4-bit with double quantization and bfloat16 compute. |
|
|
| | Component | Configuration | |
| |---|---| |
| | Llama decoder | 28 layers, hidden size 3,072, MLP 8,192; 24 attention / 8 KV heads | |
| | LoRA | Rank 16, alpha 32, dropout 0; attention q/k/v/o and MLP gate/up/down projections | |
| | Parameters | 3,212,749,824 base + 24,313,856 adapter = **3,237,063,680 total** | |
| | Base revision | `bb1d317a108579fb40e646af8924a5e7ec5604b1` | |
|
|
| Counts use the saved architecture and adapter tensor shapes; quantization does not reduce parameter count. |
|
|
| Inference: **question + choices → chat prompt → base + LoRA → logits for `1`–`4` → argmax**. The exported tokenizer/chat template is retained; scoring uses no generation, sampling, retrieval or hosted API. |
|
|
| ## Training approach |
|
|
| We fine-tuned pretrained Llama 3.2 3B Instruct for Sinhala multiple-choice answering using **QLoRA with a single supervised alignment stage**. Each subject, question and four choices becomes a chat prompt; the correct answer digit is the supervision target. The final run is `llama32-all-questions-gold-v1-gold-alignment`. |
|
|
| | Setting | Recorded value | |
| |---|---| |
| | Examples / epochs | 3,968 valid labeled rows / 3 epochs | |
| | Objective | Cross-entropy restricted to the four answer-token logits | |
| | Optimization | AdamW 8-bit; learning rate `1e-4`; cosine schedule; 3% warmup | |
| | Batch / context | Batch 1 × accumulation 8 = effective batch 8; 8,192-token limit | |
| | Stability | bfloat16, gradient checkpointing, gradient clipping 1.0, weight decay 0 | |
| | Selection | Seed 42; evaluate/save every 350 steps; best `logit_accuracy` | |
|
|
| The four-choice loss updates only the 24.3M LoRA parameters while the quantized base stays frozen. We select the checkpoint with the best fit accuracy and export its adapter and tokenizer for direct next-token choice scoring. |
|
|
| Selection reuses the training rows: the recorded 22.48% → 47.91% improvement is a fit diagnostic, not held-out or leaderboard accuracy. Export: `checkpoint-1488` (`best_gold_fit`). |
|
|
| CPT, synthetic training and the optional two-epoch continuation were not used by this final adapter. |
|
|
| ## Data provenance and limitation |
|
|
| Our broader data workflow covered **Sinhala school textbook and lesson content, past examination and term-test papers, public SinhalaMMLU questions, and textbook-derived synthetic QA**. The submitted adapter used labeled question-answer records. Textbook content was prepared separately as an optional CPT corpus (12,235 train / 759 validation rows), alongside 486 synthetic examples; these corpora were not used in the final run. |
|
|
| Training used 2,264 labels with closed-LLM provenance and 1,704 supplied labels. **Inputs include competition test questions labeled/validated using closed LLMs, conflicting with the prohibition on test-question training.** Excluding their files does not undo training; organizer judgment is required. Original metadata preserves this lineage. |
|
|
| Task questions/fit dumps are excluded; original easy-eval remains local. Sources: [rules](https://sinhalammlu.ucsc.cmb.ac.lk/rules.html), [task dataset](https://huggingface.co/datasets/SinhalaMMLU-SharedTask/Sinhala-MMLU-SharedTask-Test-Dataset), [public SinhalaMMLU](https://huggingface.co/datasets/naist-nlp/SinhalaMMLU), [dataset paper](https://aclanthology.org/2025.emnlp-main.1673/). Optional textbook materials retain source/generation provenance, without a new redistribution license. |
|
|
| ## Run inference |
|
|
| Use Linux CUDA 12.8 / Python 3.12 with one NVIDIA GPU: |
|
|
| ```bash |
| pip install -r requirements.txt |
| mkdir -p reproduction |
| cp inference.ipynb reproduction/ |
| cd reproduction |
| jupyter nbconvert --execute --to notebook --ExecutePreprocessor.timeout=-1 --output executed-inference.ipynb inference.ipynb |
| ``` |
|
|
| Or run the copied notebook top to bottom in Jupyter. Change only `DATA_REPO` for another organizer dataset. Models/tokenizer download before scoring. Pinned adapter: [`75fbe4639bec76973c1fbaad5bbf4c7638075785`](https://huggingface.co/SinhalaMMLU-SharedTask/team-ctrl-llama-lora/tree/75fbe4639bec76973c1fbaad5bbf4c7638075785). |
|
|
| Inference uses left padding, explicit positions, batch 16, a 16,384-token batch budget and OOM backoff. Organizer dataset-loading/writing sections are preserved; Hub loading, seeding and output ordering are packaging adaptations. |
|
|
| The separate execution directory protects the exact **2,238-row submitted `predictions.jsonl`**. SHA-256: |
|
|
| `e02a2c364adfb924ae9dbdabe3f8e7b36a2cf544520231009f4260a1b016793c` |
|
|
| ## Environment and verification |
|
|
| Recorded training: **H100 80GB HBM3**, Python 3.12.3, Torch 2.8.0+cu128, CUDA 12.8; **4,146.525 seconds (69.11 minutes)**. Test-inference runtime is unavailable. |
|
|
| Checked: notebook/data contracts, all 2,238 prompt lengths, tokenizer answer boundaries, Linux dependency resolution and remote hashes. Full installation, CUDA loading and reproduction comparison remain unverified. Both repositories require authorized Hugging Face access. |
|
|
| ## Files and records |
|
|
| - `inference.ipynb`, `requirements.txt`, `predictions.jsonl`: organizer inference and preserved final predictions. |
| - `training/train.ipynb`, `training/sinhala_mmlu.ipynb`: actual notebooks, with credentials removed and outputs cleared. |
| - [`training/metadata/`](https://huggingface.co/SinhalaMMLU-SharedTask/team-ctrl/tree/main/training/metadata): original configuration, dataset identity, fit metrics and package/runtime records. |
| - `training/data/cpt/`, `training/data/synthetic/`: optional textbook materials and provenance. |
|
|
| Excluded questions prevent independent retraining from this filtered package; the original code and paths remain documented. |
|
|
