| --- |
| base_model: unsloth/Llama-3.2-3B-Instruct-bnb-4bit |
| library_name: peft |
| license: llama3.2 |
| language: |
| - si |
| tags: |
| - lora |
| - sinhala-mmlu |
| --- |
| # CTRL — Llama 3.2 3B LoRA adapter |
|
|
| An answer-choice adapter for Sinhala multiple-choice questions. This repository contains adapter weights and the exported tokenizer; the base weights are loaded separately. |
|
|
| **Members:** Jayendra Matarage · Prashan Samarathunge · Isuru Udayanga · Mithun Nimesh |
|
|
| [Team code, inference and predictions](https://huggingface.co/SinhalaMMLU-SharedTask/team-ctrl) · [Adapter repository](https://huggingface.co/SinhalaMMLU-SharedTask/team-ctrl-llama-lora) |
|
|
| ## Architecture |
|
|
| A frozen [Llama 3.2 3B Instruct base](https://huggingface.co/unsloth/Llama-3.2-3B-Instruct-bnb-4bit) with one PEFT LoRA adapter, loaded through Unsloth in NF4 4-bit with double quantization and bfloat16 compute. |
|
|
| | Component | Configuration | |
| |---|---| |
| | Llama decoder | 28 layers, hidden size 3,072, MLP 8,192; 24 attention / 8 KV heads | |
| | LoRA | Rank 16, alpha 32, dropout 0; attention q/k/v/o and MLP gate/up/down projections | |
| | Parameters | 3,212,749,824 base + 24,313,856 adapter = **3,237,063,680 total** | |
| | Base revision | `bb1d317a108579fb40e646af8924a5e7ec5604b1` | |
|
|
| Counts use the saved architecture and adapter tensor shapes; quantization does not reduce parameter count. |
|
|
| Inference: **question + choices → chat prompt → base + LoRA → logits for `1`–`4` → argmax**. The exported tokenizer/chat template is retained; scoring uses no generation, sampling, retrieval or hosted API. |
|
|
| ## Training |
|
|
| Three epochs on 3,968 labeled rows using four-choice cross-entropy, AdamW 8-bit, LR `1e-4`, cosine scheduling and 3% warmup. Batch 1 × accumulation 8, context 8,192, bfloat16 and gradient checkpointing; seed 42. Only LoRA weights were updated. |
|
|
| Selection used training-fit `logit_accuracy` (47.91%) on those same rows, not held-out validation; export: `checkpoint-1488`. CPT, synthetic training and the optional continuation were not used. See the [team training description](https://huggingface.co/SinhalaMMLU-SharedTask/team-ctrl#training-approach) for settings, fit diagnostics and optional textbook materials. |
|
|
| ## Data provenance and limitation |
|
|
| Training used 2,264 labels with closed-LLM provenance and 1,704 supplied labels. **Inputs include competition test questions labeled/validated using closed LLMs, conflicting with the prohibition on test-question training.** Excluding their files does not undo training; organizer judgment is required. Original metadata preserves this lineage. |
|
|
| Task questions/fit dumps are excluded; original easy-eval remains local. Sources: [rules](https://sinhalammlu.ucsc.cmb.ac.lk/rules.html), [task dataset](https://huggingface.co/datasets/SinhalaMMLU-SharedTask/Sinhala-MMLU-SharedTask-Test-Dataset), [public SinhalaMMLU](https://huggingface.co/datasets/naist-nlp/SinhalaMMLU), [dataset paper](https://aclanthology.org/2025.emnlp-main.1673/). Optional textbook materials retain source/generation provenance, without a new redistribution license. |
|
|
| ## Loading and artifacts |
|
|
| Run [`inference.ipynb`](https://huggingface.co/SinhalaMMLU-SharedTask/team-ctrl/blob/main/inference.ipynb) using the [team instructions](https://huggingface.co/SinhalaMMLU-SharedTask/team-ctrl#run-inference). It downloads the pinned base, uses the exported configuration/tokenizer, and attaches this adapter once with `PeftModel.from_pretrained(..., is_trainable=False)`. Loaded adapter tensors are checked; weights are not merged or requantized. |
|
|
| Weights SHA-256: `1ebee225446871aeb74215ee38554d39b2259f5182acd974234bc70e6f3ff812`. |
|
|
| Weights/tokenizer/configuration are unchanged. [`adapter_integrity.json`](adapter_integrity.json) records hashes; [`run_config.json`](run_config.json), [`dataset_identity.json`](dataset_identity.json), [`training_provenance.json`](training_provenance.json) and [`gold_fit_summary.json`](gold_fit_summary.json) preserve the run record. Versions: [`installed_versions.json`](installed_versions.json), [`pip-freeze.txt`](pip-freeze.txt). Per-question fit predictions and pickled training arguments are omitted. Llama 3.2 license/use conditions apply. |
|
|
| ## Environment and verification |
|
|
| Recorded training: **H100 80GB HBM3**, Python 3.12.3, Torch 2.8.0+cu128, CUDA 12.8; **4,146.525 seconds (69.11 minutes)**. Test-inference runtime is unavailable. |
|
|
| Checked: notebook/data contracts, all 2,238 prompt lengths, tokenizer answer boundaries, Linux dependency resolution and remote hashes. Full installation, CUDA loading and reproduction comparison remain unverified. Both repositories require authorized Hugging Face access. |
|
|
