code/local_eval/README.md

# Local Albedo eval harness

Offline loop for SN97 after the Aug 21–22 reset. Uses the same generation settings, submit protocol, loop detector, and pre-eval gates as the live pipeline. It does **not** call GLM-5.2, so `proxy_score` is a gate/behaviour stand-in — survive this first or the live duel never runs.

## Layout

| path | what |
|---|---|
| `/workspace/models/albedo-qwen3.6-35b-king-genesis` | pinned genesis weights |
| `/workspace/data/albedo-datasets` | local parquet shards |
| `/workspace/data/eval-runs/<id>` | verdict + trajectories |

Pinned king: `dendriteholdings/albedo-qwen3.6-35b-king-genesis@abe89194`  
Weight hash: `sha256:efd5b8d0…89c165`  
Sampling: temperature 1.0, top_p 0.95, top_k 20, thinking on, stop at `<|im_end|>`.

## Setup

From the albedo repo root, with the workspace venv (or any env that has `huggingface_hub`, `pyarrow`, `vllm`):

```bash
source /workspace/.venv/bin/activate
export PYTHONPATH=src:.
export HF_HUB_ENABLE_HF_TRANSFER=1

# genesis (~104 GB, 26 shards) — already started if you used this harness
python -m local_eval download-genesis

# two mini-coder shards so `run` can sample prefixes without the full 4-corpus tree
python -m local_eval download-lite-data

python -m local_eval verify --path /workspace/models/albedo-qwen3.6-35b-king-genesis
```

Full official datasets (slow, large):

```bash
python scripts/prepare_datasets.py --dataset-root /workspace/data/albedo-datasets
```

## Run a local duel

8× H200 default: king on GPUs 0–3, challenger on 4–7, tensor-parallel 4.

```bash
# smoke-test genesis (you do not have a trained challenger yet)
python -m local_eval run --self-check --samples 1 --turns 1 --seed smoke

# later: duel your finetune against genesis
python -m local_eval run \
  --challenger /workspace/models/your-finetune \
  --king /workspace/models/albedo-qwen3.6-35b-king-genesis \
  --dataset-root /workspace/data/albedo-datasets \
  --samples 8 --turns 4 --seed local-eval
```

`--challenger` must be a real local directory with `config.json`. The placeholder `/path/to/your/qwen3.6-35b` will now exit immediately.

## Official GLM judge (OpenRouter)

`proxy_score` is not the live checklist. To score a finished duel the same way the subnet does — SOTA reference (`z-ai/glm-5.2`, 8 turns) → checklist → yes/no judge on both stored trajectories:

```bash
export ALBEDO_JUDGE_OPENROUTER_API_KEY=...
# optional: ALBEDO_JUDGE_ENGY_API_KEY, ALBEDO_JUDGE_REPO_CONTEXT_URL

# 2-sample smoke on the v11 8×12 duel
python -m local_eval judge --run /workspace/data/eval-runs/20260824T182728Z-cdd1388b --limit 2

# rebuild JudgeSamples only (no API calls)
python -m local_eval judge --dry-run --limit 2

# full 8
python -m local_eval judge --run /workspace/data/eval-runs/20260824T182728Z-cdd1388b --limit 0
```

Writes `official-judge.json` and `official-judge-records.jsonl` next to the duel artifacts. Without a repo-context URL, SOTA observations use the official LLM simulator (DeepSeek flash, then GLM fallback) — not leftover gold parquet.

Useful flags:

- `--self-check` — run genesis alone (implies `--skip-king`)
- `--skip-king` — gates on the challenger only
- `--gpu-memory-utilization 0.80` — default; raise only if the GPUs are empty
- `--turns 12` — closer to live horizons (12/16)
- `--samples 100 --seed <block_hash>` — same draw size as production (needs the full manifest)

## What is scored locally

Instant fail (same as live pre-eval / judge short-circuit):

- truncated / unclosed `<think>` / too short / empty
- looped commands (dup ratio ≥ 0.5 or same cmd 4×)
- reserved token leak
- no ` ```bash ` command

Warnings that tank the live checklist:

- no edit (`sed -i`, heredoc, patch, …)
- missing the **exact** rewritten submit command
- amputated thinking (`</think>`-first turns → official score × 0.5)

Environment between turns: leftover gold user observations from the parquet when present; otherwise a format-correct stub (`RETURNCODE` / `SWE_AGENT` / `OPENHANDS`). This is **not** the live repo-context executor.

## Install eval deps

Workspace venv: `/workspace/.venv` (torch 2.11.0+cu130, vllm 0.23.0).

`run` points `CUDA_HOME` at the venv CUDA 13 toolkit (`nvidia/cu13`), links `/usr/local/cuda` if missing, disables the flashinfer sampler JIT (`VLLM_USE_FLASHINFER_SAMPLER=0`), and uses the Triton GDN backend. Each vLLM worker gets its own Triton cache under `/workspace/data/triton-cache/pid-*` so four TP ranks do not race on overlayfs (and `/dev/shm` is noexec). First vLLM start may compile Triton MoE kernels for several minutes.

```bash
source /workspace/.venv/bin/activate
export PYTHONPATH=/workspace/gpt/albedo/src:/workspace/gpt/albedo
python -m local_eval run --self-check --skip-king --samples 1 --turns 1 --seed smoke
```