Team SinBotics LoRA adapter (adapter/)
This is Team SinBotics' LoRA adapter for the Sinhala MMLU Shared Task @ ICTer 2026, on top of Qwen/Qwen3.5-4B (revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a). Code, inference script and predictions are in the main folder of this repo, SinhalaMMLU-SharedTask/team-sinbotics.
| Training data | Global-MMLU Sinhala (CohereLabs/Global-MMLU, si) + sinmulu_si (our Sri Lankan question collection): 25,799 training / 2,745 validation questions |
| Objective | the answer digit after a plain multiple-choice prompt with the reply pre-filled "Answer: " (loss on that token only) |
| LoRA | rank 16, alpha 32, dropout 0.05; attention, MLP and linear-attention projections of the 32 text layers; 30,474,240 parameters |
| Optimiser | AdamW, lr 1e-4, cosine, 3% warm-up, 32 questions per step, 1 epoch (807 steps), seed 42 |
| Hardware | 1× RTX 3090 24 GB, 8.8 h |
| Validation accuracy | 44.4% (base) → 54.8% |
| Parameters at inference | 4,205,751,296 (Qwen3.5-4B text model) + 30,474,240 (adapter), merged |
Files: adapter_config.json and adapter_model.safetensors (adapter), the tokenizer files of the base model, train_config.json and train_log.jsonl (training run).
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = "Qwen/Qwen3.5-4B"; rev = "851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a"
tok = AutoTokenizer.from_pretrained(base, revision=rev)
model = AutoModelForCausalLM.from_pretrained(base, revision=rev, dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, "SinhalaMMLU-SharedTask/team-sinbotics", subfolder="adapter").merge_and_unload()
The final system adds Qwen's thinking mode and a teacher-role English reasoning prompt; see inference.py in the main repo.