SinhalaMMLU-SharedTask/team-shilpa-shastra

Team Shilpa Shastra — SinhalaMMLU Shared Task (ICTer 2026)

Inference code and submitted predictions for our final Codabench entry: 49.6269 macro on the hidden Shared Task test.

Team details

Field	Value
Team	Shilpa Shastra
Members	Malindra Senalankadhikara (1 of 1)
Corresponding member	Malindra Senalankadhikara — sole holder of the team's Codabench account
Contact	mail@malindra.lk
Codabench username	malindra
Competition	https://sinhalammlu.duckdns.org/competitions/1/ (the organisers' own Codabench server) · https://www.codabench.org/competitions/18187/
Task and rules site	https://sinhalammlu.ucsc.cmb.ac.lk/ — rules at https://sinhalammlu.ucsc.cmb.ac.lk/rules.html
Scored submission	board 2026-10-28 09:48 — macro 49.6269, easy 48.1308, medium 45.9046, hard 54.8454

What you asked for

Requested	In this repo
Inference script that produced the final Codabench submission	sinhalammlu-easy-gate.ipynb
The predictions.jsonl submitted	predictions.jsonl — 2 238 lines, md5 005c9d229c11738786094f8985cec6eb, byte-identical to the file inside artifacts/predictions_easy-gate-v40.zip
requirements.txt	requirements.txt
README with team details	this file
Training and data-preparation code	none — nothing was trained. No fine-tuning, no adapter tuning, no generated data; both models are used exactly as released upstream. There is therefore no team-shilpa-shastra-<model> weights repository.

Also here: MODEL-MANIFEST.md (rule 06), kernel-metadata-easy-gate.json, and artifacts/ with the submitted archive and the scored run's provenance and counters.

Method

Training-free, two stages, two models — 3B + 4.0B = 7.0B total, within the 8B limit. Items are answered in English: a model that is strong in English reads the text after it has been translated out of the Sinhala script.

1. MADLAD-400 3B MT translates each item — question and four choices as one block — Sinhala→English. float32, greedy (num_beams=1), 256-token input cap, 512-token output budget, <2en> target prefix. If the block output does not parse as a stem plus four distinct numbered choices, an "apart" pass translates the parts separately and whichever parse yields more choices wins.
2. Qwen3-4B-Instruct-2507 answers the multiple choice on the English surface. float16, four option-order rotations, mean option-token logprob margin, TAU = 0.75. At or above the margin the logprob choice is taken directly; below it a short English chain-of-thought is generated (cap 256 tokens); hard items always get CoT. Rows that produce no usable English are answered from the Sinhala surface instead.

No classifier, router, reranker, embedding model, retrieval or ensemble beyond those two models; nothing quantised, no adapter applied. Parameter, licence and load-setting detail — including the one disclosed load-time fix to MADLAD's weight tying under transformers 5 — is in MODEL-MANIFEST.md.

Re-running it

• Kaggle kernel malindra/sinhalammlu-pilot version 40 (COMPLETE), RUN_MODE = "full", machine shape NvidiaTeslaT4 (2× T4), internet off (enable_internet: false, local_files_only=True). 17 948.9 s for 2 238 items.
• The script requires two GPUs: gpu_plan() raises pipeline needs 2 GPUs below two, placing MADLAD on cuda:0 and Qwen on cuda:1. It was never run on a single accelerator.
• transformers must be 5.x — the tie fix keys on the dict form of T5ForConditionalGeneration._tied_weights_keys.
• Inputs are the Kaggle dataset mirrors malindra/sinhalammlu-data, malindra/sinhalammlu-sharedtask-test, malindra/qwen3-4b-instruct-2507, seshurajup/madlad400-3b-mt. Substituting the public sources — naist-nlp/SinhalaMMLU, SinhalaMMLU-SharedTask/Sinhala-MMLU-SharedTask-Test-Dataset, Qwen/Qwen3-4B-Instruct-2507, google/madlad400-3b-mt — gives the same models and the same method.
• artifacts/run_provenance.json names branch release/easy-margin-gate-v40 and commits d3247e6/1095ac4 in our private repository, not on Hugging Face. The check that the script here is the scored one is its git blob hash, d544395e0145c9a071debf46cd05ec06dcf88d8d.

Prompt wording and TAU were chosen on the organisers' released development set only, never on the hidden test: all 2 238 scored items carry gold: null, and artifacts/run_meta.json records graded: 0, accuracy: null.

License

Code is MIT (LICENSE) — free for any purpose, attribution appreciated. That covers neither the model weights (both Apache-2.0) nor the benchmark data, and this repository redistributes neither.

Reference

Ashmari Pramodya, Nirasha Nelki, Heshan Shalinda, Chamila Liyanage, Yusuke Sakai, Randil Pushpananda, Ruvan Weerasinghe, Hidetaka Kamigaito and Taro Watanabe. SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhala. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 32943–32961, 2025. https://aclanthology.org/2025.emnlp-main.1673/