RECIPE.md

# Top 1 Private Solution - Magic Team
**Multilingual Health QA in Low-Resource African Languages Challenge**

Submitter: **Magic**
Best submission: **`go.csv`** (Zindi ID `vtbP7bCH`, submitted 21 Jun 17:12)
Public LB: **0.738783** · Private LB: **Top 1 🏆**

---

## 1. Solution overview


**19-candidate ensemble** built with **V2 medoid_ngram** consensus:
- Base: 12 fewshot LoRA adapters across Qwen3.5/Qwen3.6/Qwen3-32B × K=3,4,5,7
- v8 prompt swap: 1 Qwen3.5-27B K=5 LoRA trained with anchored-extraction prompt
- 3 cross-arch "less-overfit" checkpoints
- 3 no-fewshot mediators

Per Test row, the medoid (highest sum of pairwise ROUGE-1.F + ROUGE-2.F) is chosen as the final answer.

### Why it wins
Six independent decorrelation axes simultaneously:
1. **Architecture**: Qwen3.5-27B + Qwen3.6-27B + Qwen3-32B (3 base models)
2. **K-count**: K=3, K=4, K=5, K=7 fewshot demos
3. **Prompt recipe**: v1 baseline vs v8 anchored-extraction
4. **Training schedule**: 3ep vs 5ep
5. **Mediator**: 3 no-fewshot baselines

---

## 2. Workspace layout

**All paths in this package are hard-coded to `/mnt/msrh/Magic_submission/`.** Extract this archive to that exact location (or mount/symlink it there) so every script — data builders, training YAMLs, inference launcher, ensemble — works without editing.

```bash
# One-time setup — choose ONE of these:

# (a) Extract the archive to /mnt/msrh/
sudo mkdir -p /mnt/msrh && sudo chown $USER /mnt/msrh
tar -xf Magic_submission.tar -C /mnt/msrh/        # extracts to /mnt/msrh/Magic_submission/

# (b) OR symlink an existing copy
ln -s /path/to/existing/Magic_submission /mnt/msrh/Magic_submission

# Verify
ls /mnt/msrh/Magic_submission/{go.csv,checkpoints,configs,scripts,data_builders}
```

After extraction, you must also place under `/mnt/msrh/Magic_submission/`:
- `data/Train.csv`, `data/Val.csv`, `data/Test.csv`, `data/SampleSubmission.csv` (Zindi competition data — download separately). Or you can manually download it from the Zindi data source.
- `hub/Qwen3.5-27B/`, `hub/Qwen3.6-27B/`, `hub/Qwen3-32B/` (HF snapshot of each base model)
- `hub/AfriE5-Large-instruct/` (HF snapshot of `McGill-NLP/AfriE5-Large-instruct`)

```
/mnt/msrh/Magic_submission/
├── go.csv                            # Final submission CSV (== Zindi sub vtbP7bCH)
├── README.md                         # This file — full reproduction guide
├── environment.md                    # All env setup (conda, packages, models)
├── checkpoints/                      # ⭐ 19 LoRA adapters (35 GB) — INPUT to inference
│   # Each folder = 1 LoRA adapter with:
│   #   adapter_model.safetensors  (1.8 GB for 27B, 2.1 GB for 32B)
│   #   adapter_config.json
│   #   chat_template.jinja
│   #   tokenizer_config.json + tokenizer.json + processor_config.json
│   ├── Qwen3.5-27B-3fewshots-bs64-3eps-ckpt-1200/
│   ├── Qwen3.5-27B-3fewshots-bs64-3eps-ckpt-1100/
│   ├── Qwen3.5-27B-3fewshots-bs64-5eps-ckpt-1200/
│   ├── Qwen3.5-27B-4fewshots-bs64-3eps-ckpt-1600/
│   ├── Qwen3.5-27B-5fewshots-bs64-3eps-v8prompt-ckpt-1500/    
│   ├── Qwen3.5-27B-7fewshots-bs64-3eps-ckpt-1600/
│   ├── Qwen3.5-27B-7fewshots-bs64-3eps-ckpt-1200/             
│   ├── Qwen3.6-27B-3fewshots-bs64-3eps-ckpt-1600/
│   ├── Qwen3.6-27B-4fewshots-bs64-3eps-ckpt-1400/
│   ├── Qwen3.6-27B-5fewshots-bs64-3eps-ckpt-1200/
│   ├── Qwen3.6-27B-5fewshots-bs64-3eps-ckpt-1000/             
│   ├── Qwen3.6-27B-7fewshots-bs64-3eps-ckpt-1600/
│   ├── Qwen3-32B-3fewshots-bs64-3eps-ckpt-1400/
│   ├── Qwen3-32B-5fewshots-bs64-3eps-ckpt-1700/
│   ├── Qwen3-32B-7fewshots-bs64-3eps-ckpt-1600/
│   ├── Qwen3-32B-7fewshots-bs64-3eps-ckpt-1200/              
│   ├── Qwen3.5-27B-NoFewshots-bs64-5eps-ckpt-2800/            
│   ├── Qwen3.6-27B-NoFewshots-bs64-5eps-ckpt-2600/            
│   └── Qwen3-32B-NoFewshots-bs64-4eps-ckpt-6500/              
├── candidate_csvs/                   # OUTPUT folder for 19 per-model prediction CSVs
│   # NOTE: any CSVs here are reference/audit-trail copies.
│   # For verification, REGENERATE all 19 from checkpoints/ via launch_all_predicts.sh.
│   # The ensemble script (build_ensemble.py) reads CSVs from this folder.
├── LF/
│   └── data/
│       └── dataset_info.json         
├── requirements/                     # per-phase pip pinned deps
│   ├── train.txt                     #   Phase 1 (data builders) + Phase 2 (SFT)
│   ├── infer.txt                     #   Phase 3 (vLLM inference)
│   └── ensemble.txt                  #   Phase 4 (CPU medoid ensemble)
├── data_builders/                    
│   ├── build_afrie5_k5.py            
│   ├── build_fewshot_train.py        
│   ├── build_fewshot_train_k4.py
│   ├── build_fewshot_train_k5.py
│   ├── build_fewshot_train_k7.py
│   ├── build_fewshot_test.py         
│   ├── build_fewshot_test_k4.py
│   ├── build_fewshot_test_k5.py
│   └── build_fewshot_test_k7.py
├── configs/                          
│   # ── Fewshot SFT runs (13) ──
│   ├── qwen35_27b_fewshot_3ep_8gpu.yaml             
│   ├── qwen35_27b_fewshot_k4_3ep_8gpu.yaml          
│   ├── qwen35_27b_fewshot_k5_v8_3ep_8gpu.yaml       
│   ├── qwen35_27b_fewshot_k7_3ep_8gpu.yaml         
│   ├── qwen35_27b_fewshot_2ep_8gpu.yaml             
│   ├── qwen36_27b_fewshot_k4_3ep_8gpu.yaml          
│   ├── qwen36_27b_fewshot_k5_3ep_8gpu.yaml          
│   ├── qwen36_27b_fewshot_k7_v8_3ep_8gpu.yaml       # Q3.6 K=7 v8 prompt 3ep  → ck-1600
│   ├── qwen3_32b_fewshot_3ep_bs64_r128_all_8gpu.yaml      
│   ├── qwen3_32b_fewshot_3ep_r128_all_8gpu.yaml           
│   ├── qwen3_32b_fewshot_k5_3ep_bs64_r128_all_8gpu.yaml   
│   ├── qwen3_32b_fewshot_k7_3ep_bs64_r128_all_8gpu.yaml   
│   # ── NoFewshots (mediator) SFT runs (3) ──
│   ├── qwen35_27b_nofewshots_5ep_bs64_8gpu.yaml     
│   ├── qwen36_27b_nofewshots_5ep_bs64_8gpu.yaml    
│   ├── qwen3_32b_nofewshots_4ep_bs64_8gpu.yaml      
│   ├── fewshot_e3_Q36.sh                                  
│   ├── ds_z3_config.json                                  
├── scripts/                          # Inference + ensemble scripts
│   ├── build_v8_k5_fewshot.py        # v8 prompt swap (V1 → V8 prefix)
│   ├── vllm_predict_extra.py         # vLLM batch inference w/ LoRA
│   ├── jsonl_rowidx_to_csv.py        # jsonl → CSV converter
│   ├── rag_to_zindi.py               # Alternate CSV builder (Plan D path)
│   ├── launch_all_predicts.sh        # ⭐ Orchestrator: runs all 19 inferences → candidate_csvs/<descriptive>.csv
│   ├── build_ensemble.py  # ⭐ FINAL ensemble medoid (uses candidate_csvs/)
│   └── ds_z3_config.json
└── docs/
    └── (additional documentation if needed)
```

---

## 3. Verification flow (RECOMMENDED for reviewers — uses provided checkpoints)

The 19 LoRA adapters in `checkpoints/` are the WINNING trained weights — the official artifacts that produced `go.csv`. **Reviewers do NOT need to retrain.** Use them directly to regenerate the 19 prediction CSVs, then run the ensemble.

All commands below run from `/mnt/msrh/Magic_submission/` (paths are hard-coded — no editing required).

```bash
cd /mnt/msrh/Magic_submission

# Step 1: Setup envs (see environment.md for full conda setup)
conda activate llama-qa35 && pip install -r requirements/train.txt    # Phase 1 + 2
conda activate vllm       && pip install -r requirements/infer.txt    # Phase 3
conda activate rouge      && pip install -r requirements/ensemble.txt # Phase 4

# Step 2: Build TEST data (AfriE5 retrieval + fewshot demos) — train data not needed for verification
conda activate llama-qa35
python data_builders/build_afrie5_k5.py --k 3       # K=3 base retrieval (NoFewshots test JSON)
python data_builders/build_fewshot_test.py          # K=3 fewshot test demos
python data_builders/build_fewshot_test_k4.py       # K=4 fewshot test demos
python data_builders/build_fewshot_test_k5.py       # K=5 fewshot test demos
python data_builders/build_fewshot_test_k7.py       # K=7 fewshot test demos
# v8 variants — REQUIRE K=5 / K=7 fewshot files to already exist:
python scripts/build_v8_k5_fewshot.py               # rebuilds *_k5_v8.json from K=5 fewshot
python scripts/build_v8_k7_fewshot.py               # rebuilds *_k7_v8.json from K=7 fewshot

# Step 3: Generate 19 prediction CSVs from provided checkpoints (Reads checkpoints/, writes candidate_csvs/)
conda activate vllm
bash scripts/launch_all_predicts.sh

# Step 4: Run ensemble (V2 medoid_ngram) → final CSV
conda activate rouge
python scripts/build_ensemble.py
# Writes the regenerated submission CSV alongside the shipped go.csv.
# Reference md5 of shipped go.csv: a2ecca4a8e1aa01acf9a8b9a1d56ebf2
```

### Note on byte-identity vs functional reproducibility

We E2E-verified this pipeline on a fresh 8×H100 box (vLLM 0.19.1 + torch 2.10+cu128).
**Expect the regenerated CSV to NOT be md5-equal to the shipped `go.csv`** — but the LB
will reproduce within ~0.005 public / ~0.001 private of the winning result.

| Run | Public LB | Private LB | Row match vs `go.csv` |
|---|---:|---:|---|
| Original `vtbP7bCH` (shipped `go.csv`) | 0.738783 | 0.730865 | byte-identical (md5 `a2ecca4a...`) |
| Re-run #1 — `max_num_seqs=32` hard-coded | 0.734035 | 0.729336 | 65.0% |
| Re-run #2 — per-cand `max_num_seqs` matched | 0.734104 | 0.729553 | 67.8% |

**Root cause of the ~0.005 gap**: vLLM 0.19.1 paged-attention has **inherent non-determinism**
between runs even at `--temperature 0.0` with identical flags (CUDA atomic ops, bfloat16
rounding-order in attention, KV-cache block scheduling races). This affects ~30-40% of
generated tokens at single-cand level; the V2 medoid_ngram ensemble suppresses most of the
drift, leaving ~32% of per-row picks different but only a ~0.001 private-LB hit.

**Acceptance criterion**: if the regenerated CSV yields **Private LB ≥ 0.728** (within −0.003
of the shipped 0.730865), reproduction is considered successful and the pipeline is verified.
md5 byte-equality is NOT achievable on a different machine/run.

---

## 3b. Full from-scratch reproduction (optional — trains 13 LoRAs from base models)

Only needed if you want to retrain the LoRA adapters yourself instead of using the provided ones.

```bash
cd /mnt/msrh/Magic_submission
conda activate llama-qa35   # training env

# Build train data (same as Step 2 above but for train set)
python data_builders/build_afrie5_k5.py --k 3
python data_builders/build_afrie5_k5.py --k 5
python data_builders/build_afrie5_k5.py --k 7
python data_builders/build_fewshot_train.py
python data_builders/build_fewshot_train_k4.py
python data_builders/build_fewshot_train_k5.py
python data_builders/build_fewshot_train_k7.py
# v8 prompt swap — runs AFTER build_fewshot_train_k5.py and build_fewshot_test_k5.py
python scripts/build_v8_k5_fewshot.py               # rebuilds *_k5_v8.json (train + test)

# Train each of the 16 LoRA adapters (configs/*.yaml). Outputs go to
# /mnt/msrh/Magic_submission/checkpoints_trained/<config-name>/ as set inside each YAML.
# Example for the key v8 model:
FORCE_TORCHRUN=1 NPROC_PER_NODE=8 \
  llamafactory-cli train /mnt/msrh/Magic_submission/configs/qwen35_27b_fewshot_k5_v8_3ep_8gpu.yaml
# ... repeat for all 16 configs (~14h each on 8× H100)

# Replace adapters in checkpoints/<descriptive_name>/ with your fresh ckpts, then
# follow Step 3 + Step 4 from Section 3 above.
```

---

## 4. Data preparation (full pipeline)

### Inputs (required)
- Zindi competition data: `Train.csv`, `Val.csv`, `Test.csv`, `SampleSubmission.csv` (from competition page)
- Path expected: `/mnt/msrh/Magic_submission/data/{Train,Val,Test,SampleSubmission}.csv` (no editing needed — every builder and script reads from this fixed location)

### Pipeline

1. **AfriE5 base retrieval** (`build_afrie5_k5.py`)
   - For each train query, retrieve top-K from Train+Val pool using AfriE5 cosine similarity
   - For each test query, retrieve top-K from Train+Val pool
   - Output: `msrh_rag_train_afrie5_TV_k{K}.json` + `msrh_rag_test_k3_AfriE5_TV.json`
   - Includes language-specific instruction tag (Akan/Amharic/Luganda/Swahili/English)

2. **Fewshot demo prep** (`build_fewshot_train_k{K}.py` + `build_fewshot_test_k{K}.py`)
   - Per row, retrieve K AfriE5-similar same-subset (Q,A) pairs from Train+Val
   - Prepend as "Example N:" demos in user message
   - Output: `msrh_rag_train_afrie5_TV_k{K}_fewshot.json`

3. **v8 prompt swap** (`build_v8_k5_fewshot.py`)
   - Replace v1 prompt:
     ```
     Use the retrieved contexts as your primary sources — copy exact phrasing where
     the contexts already address the question. Be concise and factually accurate.
     ```
   - With v8 prompt (33 words, 220 chars):
     ```
     The retrieved contexts are your source of truth — copy or paraphrase their
     exact phrasing to answer. Reply in the same language and script as the question.
     Plain prose, no disclaimers or meta-commentary.
     ```
   - Output: `msrh_rag_train_afrie5_TV_k5_fewshot_v8.json`

---

## 5. Training (13 LoRA adapters)

All training uses **LlamaFactory + DeepSpeed ZeRO-3** on 8× H100 (80GB).

### Common hyperparameters (all 13 runs)
```yaml
finetuning_type: lora
lora_rank: 128
lora_alpha: 256
lora_dropout: 0.05
lora_target: all          # all linear layers
learning_rate: 2.0e-4
lr_scheduler_type: cosine
warmup_ratio: 0.05
bf16: true
gradient_checkpointing: true
deepspeed: scripts/ds_z3_config.json
```

### Per-model details

### Per-model details (16 SFT runs total — all reproducible via configs/*.yaml)

The 19 ensemble candidates come from **16 distinct SFT training runs** (3 of the runs export 2 checkpoints each as "early-tap" anti-overfit cands).

#### Fewshot runs (13 SFT, each prepends top-K AfriE5-similar (Q,A) demos)

| # | Model | K | Prompt | Epochs | cutoff_len | eff_bs | YAML | Best ckpt | Cand(s) |
|---:|---|:---:|:---:|:---:|:---:|:---:|---|---|---|
| 1 | Qwen3.5-27B | 3 | v1 | 3 | 4096 | 64 | qwen35_27b_fewshot_3ep_8gpu.yaml | ck-1200 | ck-1200 + ck-1100 (early-tap) |
| 2 | Qwen3.5-27B | 3 | v1 | 5 | 4096 | 64 | qwen35_27b_fewshot_5ep_8gpu.yaml (modify epochs=5 in run 1's YAML) | ck-1200 | 1 cand |
| 3 | Qwen3.5-27B | 4 | v1 | 3 | 5120 | 64 | qwen35_27b_fewshot_k4_3ep_8gpu.yaml | ck-1600 | 1 cand |
| 4 | **Qwen3.5-27B** | **5** | **v8** | **3** | **6144** | **64** | **qwen35_27b_fewshot_k5_v8_3ep_8gpu.yaml** | **ck-1500** ⭐ | 1 cand ⭐ |
| 5 | Qwen3.5-27B | 7 | v1 | 3 | 8192 | 64 | qwen35_27b_fewshot_k7_3ep_8gpu.yaml | ck-1600 | ck-1600 + ck-1200 (early-tap) |
| 6 | Qwen3.6-27B | 3 | v1 (RecA) | 3 | 4096 | 64 | qwen36_27b_fewshot_3eps_bs64.yaml (Q3.5 K=3 YAML w/ model+template swap) | ck-1600 | 1 cand |
| 7 | Qwen3.6-27B | 4 | v1 | 3 | 5120 | 64 | qwen36_27b_fewshot_k4_3ep_8gpu.yaml | ck-1400 | 1 cand |
| 8 | Qwen3.6-27B | 5 | v1 | 3 | 6144 | 64 | qwen36_27b_fewshot_k5_3ep_8gpu.yaml | ck-1200 | ck-1200 + ck-1000 (early-tap) |
| 9 | Qwen3.6-27B | 7 | **v8** | 3 | 8192 | 64 | qwen36_27b_fewshot_k7_v8_3ep_8gpu.yaml | ck-1600 | 1 cand |
| 10 | Qwen3-32B | 3 | v1 | 3 | 4096 | 64 | qwen3_32b_fewshot_3ep_bs64_r128_all_8gpu.yaml | ck-1400 | 1 cand |
| 11 | Qwen3-32B | 5 | v1 | 3 | 6144 | 64 | qwen3_32b_fewshot_k5_3ep_bs64_r128_all_8gpu.yaml | ck-1700 | 1 cand |
| 12 | Qwen3-32B | 7 | v1 | 3 | 8192 | 64 | qwen3_32b_fewshot_k7_3ep_bs64_r128_all_8gpu.yaml | ck-1600 | ck-1600 + ck-1200 (early-tap) |

#### NoFewshots runs (3 SFT, mediators — same K=3 AfriE5 retrieval but WITHOUT demos prepended)

These 3 are **trained the same way** as the fewshot runs but use the **non-fewshot K=3 dataset** (`msrh_rag_train_afrie5_TV_k3.json` — output of `build_afrie5_k5.py --k 3` only, no `build_fewshot_train*.py` step). They serve as decorrelated "no-context-demos" mediators in the ensemble.

| # | Model | K | Prompt | Epochs | cutoff_len | eff_bs | YAML | Best ckpt | Cand |
|---:|---|:---:|:---:|:---:|:---:|:---:|---|---|---|
| 13 | Qwen3.5-27B | 3 | v1 | 5 | 4096 | 64 | **qwen35_27b_nofewshots_5ep_bs64_8gpu.yaml** | ck-2800 | 1 cand |
| 14 | Qwen3.6-27B | 3 | v1 | 5 | 4096 | 64 | **qwen36_27b_nofewshots_5ep_bs64_8gpu.yaml** | ck-2600 | 1 cand |
| 15 | Qwen3-32B | 3 | v1 | 4 | 4096 | 64 | **qwen3_32b_nofewshots_4ep_bs64_8gpu.yaml** | ck-6500 | 1 cand |

**Differences vs fewshot runs**: only `dataset: msrh_rag_train_afrie5_TV_k3` (vs `*_k3_fewshot`) and longer epoch budget (4-5 vs 3) since no-demo trains more slowly.

#### Total: 15 SFT runs → 19 ensemble cands (3 runs contribute 2 ckpts each)

### Launch example (any config)
```bash
HF_DATASETS_OFFLINE=1 TRANSFORMERS_OFFLINE=1 \
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
FORCE_TORCHRUN=1 NNODES=1 NPROC_PER_NODE=8 \
llamafactory-cli train /mnt/msrh/Magic_submission/configs/qwen35_27b_fewshot_k5_v8_3ep_8gpu.yaml \
  > /mnt/msrh/Magic_submission/training_logs/q35_k5_v8.log 2>&1
```

Save_steps=100-500 (depends on config), save_total_limit=25 (keep multiple ckpts for early-tap predictions).

### Wall time per run (8× H100 80GB)
- Q3.5-27B 3ep: ~7-8h
- Q3.6-27B 3ep: ~8-10h
- Q3-32B 3ep: ~13-14h
- Q3-32B 4ep (NoFewshots): ~18h
- Q3.5/3.6-27B 5ep (NoFewshots): ~12-15h
- Total for 15 SFT runs (sequential): ~7-9 days
- With 2-3 nodes parallel: ~3-4 days

---

## 6. Inference (each LoRA adapter)

Single-GPU inference per adapter via vLLM (env: `vllm`). The wrapper `launch_all_predicts.sh` orchestrates all 19 runs and writes outputs into `candidate_csvs/<descriptive_name>.csv` ready for ensemble.

### One-shot run all 19 predictions
```bash
bash /mnt/msrh/Magic_submission/scripts/launch_all_predicts.sh
# Outputs: /mnt/msrh/Magic_submission/candidate_csvs/{Qwen3.5-27B-..., Qwen3.6-27B-..., Qwen3-32B-...}.csv (19 files)
```

### Manual single-cand example (the key v8 ckpt)
```bash
ROOT=/mnt/msrh/Magic_submission
CUDA_VISIBLE_DEVICES=0 python $ROOT/scripts/vllm_predict_extra.py \
  --base   $ROOT/hub/Qwen3.5-27B \
  --adapter $ROOT/checkpoints/Qwen3.5-27B-5fewshots-bs64-3eps-v8prompt-ckpt-1500 \
  --rag_test $ROOT/LF/data/msrh_rag_test_k3_AfriE5_TV_fewshot_k5_v8.json \
  --out_jsonl $ROOT/_predict_workdir/Qwen3.5-27B-5fewshots-bs64-3eps-v8prompt-ckpt-1500/predictions.jsonl \
  --max_lora_rank 128 --max_new 512 \
  --max_model_len 8192 --mem_util 0.88 --max_num_seqs 32 \
  --temperature 0.0 --top_p 1.0 --best_of 1 --no_think

python $ROOT/scripts/jsonl_rowidx_to_csv.py \
  --jsonl $ROOT/_predict_workdir/Qwen3.5-27B-5fewshots-bs64-3eps-v8prompt-ckpt-1500/predictions.jsonl \
  --out_csv $ROOT/candidate_csvs/Qwen3.5-27B-5fewshots-bs64-3eps-v8prompt-ckpt-1500.csv
```

### Critical inference flags
- `--no_think` — passes `enable_thinking=False` to chat template (Qwen3.5/6 trained with `qwen3_5_nothink` template). Default ON wastes generation budget on reasoning; LB drops 0.10-0.25.
- `--temperature 0.0 --best_of 1` — greedy deterministic decoding
- `--max_model_len`: 6144 for K=3/4; 8192 for K=5; 10240 for K=7 (long prompts)

**Provide `--adapter checkpoints/<descriptive_name>`** pointing into the provided `checkpoints/` folder. Run inference for all 19 candidates (each on a different GPU in parallel via the launcher). The output 19 CSVs go to `candidate_csvs/`, ready for ensemble.

> **Note**: This package does NOT ship pre-computed `candidate_csvs/*.csv`.
> They must be regenerated from `checkpoints/` via `scripts/launch_all_predicts.sh`
> so the reviewer can independently verify the inference step.

---

## 7. Final ensemble (V2 medoid_ngram)

```bash
# (no GPU needed — pure CPU + ROUGE scoring)
pip install rouge-score==0.1.2

python scripts/build_ensemble.py
# Reads 19 CSVs from candidate_csvs/<descriptive_name>.csv
# Computes per-row medoid, writes the regenerated submission CSV
# Compares md5 against the shipped go.csv: a2ecca4a8e1aa01acf9a8b9a1d56ebf2
```

### V2 medoid_ngram algorithm
For each Test row:
1. Compute pairwise ROUGE-1.F + ROUGE-2.F between all 19 candidate answers
2. For each candidate, sum its similarities to all OTHERS
3. Pick the candidate with HIGHEST sum (the "medoid" of the cluster)
4. Use its answer text as final prediction

Implementation (excerpt from `build_ensemble.py`):
```python
from rouge_score import rouge_scorer
scorer = rouge_scorer.RougeScorer(["rouge1", "rouge2"], use_stemmer=False)

def medoid(texts):
    best, best_s = 0, -1.0
    for i in range(len(texts)):
        s = sum(
            scorer.score(texts[j], texts[i])["rouge1"].fmeasure
          + scorer.score(texts[j], texts[i])["rouge2"].fmeasure
            for j in range(len(texts)) if i != j
        )
        if s > best_s: best_s, best = s, i
    return best
```

---

## 8. Reproducibility

### Hash of final submission
```
$ md5sum go.csv
a2ecca4a8e1aa01acf9a8b9a1d56ebf2  go.csv
```

### Pinned versions (env)
See `environment.md` — exact `torch` / `transformers` / `vllm` / `llamafactory` versions.

### Random seed
- LlamaFactory default seed (42) used for all trainings — reproducible LoRA weights given same env/data
- vLLM inference is deterministic with `--temperature 0.0 --best_of 1`
- Ensemble medoid is purely deterministic (no random selection)

### Sanity checks
1. Verify CSV row order matches `data/Test.csv` (2618 rows, 4 target columns: TargetRLF1, TargetR1F1, TargetLLMJudge, all = same answer)
2. Confirm md5sum of `go.csv` matches above

---

## 9. Acknowledgments

- **AfriE5** (`McGill-NLP/AfriE5-Large-instruct`) — multilingual retrieval encoder
- **LlamaFactory** — LoRA training framework
- **vLLM** — inference engine (LoRA + batched generation)
- **DeepSpeed** — ZeRO-3 distributed training
- **Qwen team** (Alibaba) — Qwen3.5/3.6/Qwen3 base models

---

## 10. Contact

Submitter: Magic
Submission deadline: 23 Jun 2026 17:00 GMT (handled in time ✓)