Delete folder .ipynb_checkpoints with huggingface_hub

.ipynb_checkpoints/README-checkpoint.md

@@ -1,98 +0,0 @@
----
-license: gemma
-language:
-- si
-base_model: QBitsLK/gemma4-e4b-sinhala-cpt
-tags:
-- sinhala
-- multiple-choice
-- sinhalammlu
-- shared-task
----
-
-# Team QBitsLK – Sinhala MMLU Shared Task System
-
-## System summary
-- **Base model(s):** `google/gemma-4-E4B` (text-only), with Sinhala continued pretraining, then LoRA supervised fine-tuning on examination-paper MCQs.
-- **What we did:** We built a 7,872-question Sinhala multiple-choice dataset by extracting from publicly available Sri Lankan O/L and Grade 6–11 past papers, cleaning OCR artefacts and decontaminating against the released SinhalaMMLU test split. We continued pretraining Gemma 4 E4B on Sinhala text, then fine-tuned with LoRA (r=32) using answer-only loss, and merged the adapter back into the base weights. At inference each option is scored by per-option log-likelihood under a 5-prompt ensemble (Sinhala and English instructions, subject intro on/off, two option-order rotations) rather than by generation. Nothing is sampled.
-- **Total inference parameters:** **7,996,157,418** (one merged model; ≤ 8B)
-- **Extra data used for training or retrieval:** our own past-paper MCQ dataset, published at `SinhalaMMLU-SharedTask/team-qbits-mcq-data`. No retrieval, no external APIs, no closed models.
-
-## Models and resources in this submission
-| Repo | What it is |
-|---|---|
-| `SinhalaMMLU-SharedTask/team-qbits` | code, inference script, final predictions (this repo) |
-| `SinhalaMMLU-SharedTask/team-qbits-cpt-v3` | **the submitted model** — merged 16-bit, 7,996,157,418 params |
-| `SinhalaMMLU-SharedTask/team-qbits-cpt-v3-lora` | the LoRA adapter for the above (r=32), for inspection |
-| `SinhalaMMLU-SharedTask/team-qbits-base-v3b` | ablation — same recipe on the plain Gemma 4 E4B base, merged |
-| `SinhalaMMLU-SharedTask/team-qbits-base-v3b-lora` | the LoRA adapter for the ablation |
-| `SinhalaMMLU-SharedTask/team-qbits-mcq-data` | the SFT dataset (prompt/completion files) |
-
-`inference.py` uses **only** `team-qbits-cpt-v3`. The other repos are provided for transparency and
-are not part of the inference pipeline.
-
-## How to run
-```bash
-pip install -r requirements.txt
-python inference.py
-```
-- **GPU used:** 1x NVIDIA A40 48 GB (any GPU with ≥ 20 GB should work; the model is 16 GB in bf16)
-- **Approximate run time on the test set:** ~18 minutes for 2,238 questions (5 prompt variants × 4 options = ~44,800 scored candidates); peak VRAM ~27 GB
-- **Anything else the organizers need to know:**
-  - The Gemma license must be accepted on the Hugging Face account used to run this. Our merged model is self-contained, so no separate base-model download is required.
-  - `inference.py` scores options by log-likelihood and never calls `generate`, so there is no sampling. With the fixed seed, repeated runs produce byte-identical `predictions.jsonl`.
-  - `VARIANTS` at the top of `inference.py` defines the prompt ensemble. **Changing that list changes the predictions**; it is set exactly as used for our final Codabench submission.
-  - The reference `util_prompt.py` hard-codes the *hard* instruction as "1, 2, 3, 4, 5", which was correct for the original five-option hard split. SinhalaMMLU 2.0 reduced hard questions to four options, so `fit_numbers()` derives the number list from the actual option count. Verified: 0 of 2,238 prompts disagree with their option count.
-
-## Results (Codabench)
-
-| system | easy | medium | hard | **macro** |
-|---|---|---|---|---|
-| **submitted** — `team-qbits-cpt-v3`, 5-prompt ensemble | 58.10 | 62.56 | 52.79 | **57.81** |
-| the same model with the adapter applied at load time instead of merged | 58.26 | 62.02 | 52.58 | 57.62 |
-| ablation — `team-qbits-base-v3b` (plain Gemma 4 base), 5-prompt ensemble | 57.01 | 63.10 | 51.34 | 57.15 |
-| ablation — plain base, single prompt, no ensemble | — | — | — | 55.66 |
-
-`inference.py` reproduces the submitted row exactly: it loads the merged model, which is the system
-we ship. Row 2 is the same fine-tune evaluated with the LoRA applied at load time (base + adapter).
-That form is 8,069,557,738 parameters and therefore **exceeds the 8B cap**, so it is not the
-submitted system; it is listed only because merging is mathematically equivalent yet not
-bit-identical in bf16 (`W + BA` rounded once, versus `Wx + B(Ax)`), which resolves about 1% of
-near-tied questions differently.
-
-## Files
-- `inference.py` – the exact script that produced our final Codabench submission
-- `predictions.jsonl` – the exact file we submitted to the final leaderboard (macro 57.81),
-  byte-for-byte reproducible by running `inference.py`
-- `predictions_ablation_base_v3b.jsonl` – an earlier submission from the ablation model
-  (`team-qbits-base-v3b`, macro 57.15), included for transparency. To regenerate it, set
-  `MODEL_ID` to `SinhalaMMLU-SharedTask/team-qbits-base-v3b` and change nothing else.
-- `requirements.txt` – package versions we used
-- `training/` – training, fine-tuning and data-preparation code
-  - `sft_gemma4_mcq.py` – LoRA SFT with answer-only loss and candidate-scoring evaluation
-  - `mcq_dataset.py` – extraction cleanup, validation, decontamination, splitting, prompt building
-  - `add_new_data.py`, `normalize_subjects.py` – dataset assembly and subject-name alignment
-  - `merge_adapter.py`, `push_merged_to_hf.py` – adapter merging and publishing
-  - `eval_sinhalammlu.py` – read-only analysis; also builds the decontamination blocklist
-  - `check_env.py` – dependency verification
-  - `run_data_pipeline.sh`, `run_training.sh`, `run_inference.sh` – end-to-end drivers
-  - `REPRODUCE.md` – exact commands, expected counts and the training curve
-
-## Data and decontamination
-Source: publicly available Sri Lankan examination papers (`pastpapers.wiki`,
-`e-thaksalawa.moe.gov.lk`), O/L and Grades 6–11. We do not redistribute the source PDFs; the derived
-question set is published at `team-qbits-mcq-data`.
-
-The released `TEST/easy` split was used **only** as a decontamination blocklist. It matched and
-removed **26** questions from our training data. It was never used for training, checkpoint
-selection, prompt design or hyperparameter choice. Model selection used a held-out split of our own
-data, grouped so that no source paper appears in both training and validation.
-
-## Citation
-This work evaluates on the SinhalaMMLU benchmark:
-
-> Ashmari Pramodya, Nirasha Nelki, Heshan Shalinda, Chamila Liyanage, Yusuke Sakai, Randil
-> Pushpananda, Ruvan Weerasinghe, Hidetaka Kamigaito, and Taro Watanabe. 2025. **SinhalaMMLU: A
-> Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhala.** In
-> *Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, pages
-> 32943–32961. https://aclanthology.org/2025.emnlp-main.1673/