File size: 2,535 Bytes
0b65f8f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 | # External benchmark evaluation
Isambard-AI is blocked by a CPU-minutes quota, so we evaluate the trained
checkpoints on a GPU elsewhere. The models are a custom LLaMA-style Transformer
(`src/xscript/model.py`) + SentencePiece tokenizer — pure PyTorch, using
`F.scaled_dot_product_attention`, no flash-attn / triton / custom kernels — so
they run on any stock GPU (or CPU, slowly). Each model is ~1B params (fits any
16GB GPU).
The benchmark harness (`src/xscript/eval/bench.py`) wraps our model into
lm-evaluation-harness and scores Global-MMLU, Belebele, and XNLI on each run's
training languages. It is the *same* harness we would have run on-cluster, so
numbers are directly comparable.
## 1. Export from Isambard (already done by `upload_to_hf.py`)
The private HF repo mirrors the on-cluster layout:
```
src/xscript/** # bundled model + harness code
tokenizers/unigram_{starved,destarved}/{sp.model,meta.json}
runs/<name>/checkpoints/final.pt # 15 checkpoints, fp32, ~4GB each
models.json # friendly name -> tokenizer + langs + orig run
run_benchmarks.py requirements.txt README.md
```
Models use friendly names `<mixture>-<starved|fair>` (e.g. `en-fair`,
`en-ar-starved`). `models.json` maps each to its real tokenizer.
## 2. Run on your GPU
```bash
# clone just the runner (or download run_benchmarks.py + requirements.txt from the repo)
pip install torch --index-url https://download.pytorch.org/whl/cu121 # match your CUDA
pip install -r requirements.txt
export HF_TOKEN=hf_... # while the repo is private
# quick validation pass over all 15 runs (~200 examples/task) -- do this FIRST
python run_benchmarks.py --repo jvonrad/xscript-eval --limit 200
# full suite once the quick pass looks sane
python run_benchmarks.py --repo jvonrad/xscript-eval
```
The runner downloads one checkpoint at a time and deletes it after eval
(`--keep-checkpoints` to retain), so peak disk is ~5GB. Results:
```
xscript_bench/results/bench/<run>_final.json # per-run task accuracies
xscript_bench/results/summary.json # everything combined
```
Send those JSONs back for analysis.
## Notes
- `--runs en-starved en-fair` limits to a subset (friendly names).
- `--tasks xnli_en xnli_de` overrides the task list (default = the run's langs).
- Mono runs get 3 tasks (their one language), bilingual runs get 6 (both langs).
- Scores are ordinary accuracy (`acc,none`); raw harness output is preserved in
each per-run JSON for length-normalized variants.
|