# Environment Requirements

## Hardware
- 8× NVIDIA H100 80GB (or A100 80GB) per training run
- ~500 GB shared disk for model weights + checkpoints + train data
- ~200 GB system RAM recommended (DeepSpeed ZeRO-3 large weights gathering)

## OS
- Ubuntu 22.04 LTS (kernel 5.15+)
- CUDA 12.8 drivers (12.4+ should also work)

## Three conda environments

We use **THREE separate environments** to avoid dependency conflicts between
training (LlamaFactory + DeepSpeed) and inference (vLLM), plus a tiny CPU-only
env for the ensemble. Each phase has a pinned `requirements/<phase>.txt`.

### Env 1: `llama-qa35` (TRAINING + Phase 1 data builders)
Used by data_builders/*.py and `llamafactory-cli train configs/*.yaml`.
```bash
conda create -n llama-qa35 python=3.12 -y
conda activate llama-qa35
pip install -r requirements/train.txt
# LlamaFactory installed in dev mode (git clone)
git clone https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory && pip install -e ".[deepspeed]"
```

### Env 2: `vllm` (INFERENCE — Phase 3)
Used by `scripts/launch_all_predicts.sh` → 19 LoRA cand CSVs.
```bash
conda create -n vllm python=3.12 -y
conda activate vllm
pip install -r requirements/infer.txt
```

### Env 3: `rouge` (ENSEMBLE — Phase 4, CPU only, any python 3.10+)
Used by `scripts/build_ensemble.py` to write the regenerated submission CSV.
```bash
conda create -n rouge python=3.12 -y          # or use any base python
conda activate rouge
pip install -r requirements/ensemble.txt
```

## Workspace layout

Every script, YAML, and builder in this package is hard-coded to the workspace root **`/mnt/msrh/Magic_submission/`**. Extract the archive (or symlink) so the tree lives at that exact location. Inside it you also need:

```
/mnt/msrh/Magic_submission/
├── data/                                  # Zindi competition CSVs (Train/Val/Test/SampleSubmission)
├── hub/                                   # base + retrieval HF snapshots (see below)
│   ├── Qwen3.5-27B/
│   ├── Qwen3.6-27B/
│   ├── Qwen3-32B/
│   └── AfriE5-Large-instruct/
├── LF/data/                               # generated train/test JSONLs (created by data_builders/)
├── hf_cache/                              # HF cache for retrieval encoder
├── checkpoints/                           # 19 LoRA adapters shipped with this package
├── checkpoints_trained/                   # output dir for any from-scratch retrains
└── candidate_csvs/                        # generated per-cand prediction CSVs
```

## Base models (HuggingFace)

Download into `/mnt/msrh/Magic_submission/hub/`:
```bash
HUB=/mnt/msrh/Magic_submission/hub

# Qwen3.5-27B-Base — main model
huggingface-cli download Qwen/Qwen3.5-27B --local-dir $HUB/Qwen3.5-27B

# Qwen3.6-27B-Base
huggingface-cli download Qwen/Qwen3.6-27B --local-dir $HUB/Qwen3.6-27B

# Qwen3-32B-Base
huggingface-cli download Qwen/Qwen3-32B --local-dir $HUB/Qwen3-32B
```

## Retrieval model

```bash
# Multilingual E5 instruct (AfriE5) — used by all data_builders/*.py
huggingface-cli download McGill-NLP/AfriE5-Large-instruct \
  --local-dir /mnt/msrh/Magic_submission/hub/AfriE5-Large-instruct
```

## Key packages cheatsheet

| Package | Train env | Infer env | Notes |
|---|---|---|---|
| torch | 2.11.0+cu128 | 2.10.0+cu128 | DIFFERENT versions — keep envs separate |
| transformers | 5.2.0 | 4.57.6 | DIFFERENT — train needs newer for Qwen3.5/6 |
| vllm | — | 0.19.1 | Inference only |
| llamafactory | 0.9.5.dev0 | — | Train only (editable install) |
| peft | 0.18.1 | 0.19.1 | LoRA support |
| deepspeed | 0.18.8 | — | ZeRO-3 |
| rouge-score | — | — (CPU env) | 0.1.2 for ensemble medoid_ngram |