Blame for artifacts/HANDOFF.md

# INGOT Exploit Verification — Handoff for Troy

Goal: produce a q2 GGUF that beats the current king (Ingot-IX-27B-Q2, KL 0.10294 vs teacher,
dist ~0.32) by the win margins (**≥1.5% KL** and **≥0.84% dist**, both relative to the king,
measured in the same eval), verify it offline, and submit it. This proves the SN97 exploit:
duels are fully reproducible from public data, so only offline-verified winners get submitted.

## Access

- Gateway (flaky): `ssh nx7uk9r5ieqbsr-64410ce7@ssh.runpod.io -i ~/.ssh/ingot_a100_ed25519` (needs `-tt` + a tty: use `script -q /dev/null ssh -tt ...`)
- Direct (reliable, SCP works): `ssh -p 18956 root@216.81.245.29 -i <your key>` — your key is already in `~/.ssh/authorized_keys`
- Pod: H100 NVL 96GB, 144 cores, 200GB disk, **dies ~02:40 UTC Oct 26 (4h rental)**. If it died, rerun `scripts/setup_h100.sh` (works, but the vllm patch step needs `patch --batch`), then re-scp the scripts below and redo: config patch → king_hf → gen_teacher → qat.
- On a fresh pod the fork must build with `-DCMAKE_CUDA_ARCHITECTURES=90` (H100) or `80` (A100).

## Layout on the pod

- Repo: `/root/ingot` (commit 7bb8e7d, submodule llama-prism at e3de715, built for sm_90: `build/llama_states/llama-states` + `llama-prism/build/bin`)
- venv: `/root/ingot/.venv` (vLLM 0.31.0 + torch + ingot `.[eval]`, PR #56822 applied, EVAL BOX OK verified)
- Models: `/root/models/teacher` (Qwen3.8-27B BF16), `/root/models/king-ix` (Ingot-IX-27B-Q2.gguf)
- `/dev/shm/king_hf.pt`: the king dequantized to HF layout (851 tensors) — the surrogate checkpoint
- Scripts in /root: `eval_seq.py` (official-path measurement), `gen_teacher.py`, `qat_norms.py`, `qat.py` (unused draft), `scale_refit_gpu.py`, `make_traindata.py`, `run_measure.sh`/`run_m3.sh`
- Dev patches in `src/eval_service/config.py`: `LLAMA_PER_GPU=1`, `MEMORY_SHARE` teacher 0.72, `MAX_MODEL_LEN=4096` (single-GPU reality; scores unaffected)

## Proven so far

1. **Oracle fully reproduced.** eval 156's draw = 1079/1080 exact (1 sample = tokenizer-version
   boundary flip on a 30k-token zh prompt). Deployed eval uses `SAMPLES_PER_DOMAIN=60` for ALL
   18 domains (public repo says 40 + swe/terminal 150 — repo is out of sync with the live
   validator!). Seed = miner's r2ready block hash; corpus pinned by dataset_hash =
   sha256("dendriteholdings/ingot_corpus@<commit>:chungus").
2. **Offline harness works.** `eval_seq.py` runs the official scorer (vLLM teacher + fork
   llama-states reads + metrics) sequentially. Sanity: king==challenger gives identical scores.
   Measured king KL ≈ official values on short samples.
3. **Surrogate validated.** transformers' qwen3_5 forward with the king's HF-layout tensors
   matches the fork's read: L08 cos 0.998, L16-L56 0.96-0.997, L64 0.94 (rel 2.6% at L64 —
   this systematic error matters, see below).
4. Registration burn ≈ 0.05 τ (~$12). Queue empty. Kings crown every few hours (treadmill live:
   IX crowned Oct 25 11:21, margins 1.51-1.52% — challengers are calibrated perturbations).

## Critical technical knowledge (cost hours to discover)

- The fork's converter (`llama-prism/conversion/qwen.py`) reorders by value heads:
  `A_log → -exp(A_log)`, `dt_bias → ssm_dt.bias`, conv1d squeezed, and **A_log/dt_bias/conv1d-V
  channels are row-tiled [rep,n_k] ↔ [n_k,rep]**. `GgufCheckpoint` handles weights; the ssm
  params must be untiled by hand (see build_king_hf.py — correct version is on the pod).
- Norm weights are stored as `hf + 1` (except `linear_attn.norm`/ssm_norm, stored as-is).
  Output norm = hf+1 too.
- Hadamard rotation: block_size=1024, per-tensor sign vectors (`prism.hadamard.sign_widths`
  [5120, 6144, 17408]), rotation axis = gguf shape[0]. `GgufCheckpoint.get()` = the correct
  unrotation (`(W'@H)*signs`), proven by the surrogate fidelity check.
- PQ2_0: 34 B per 128 weights (2B fp16 scale + 32B codes), value = (code−1)·d, code 3 (+2)
  rejected by validation. **Blocks tile the gguf COLUMN axis** (i.e., numpy rows of the
  reversed layout) — NOT the row axis. Any scale refit must fit along that axis.
- vLLM writes hidden-state files ASYNCHRONOUSLY — always poll-wait for the file after
  `generate()`/`complete()` before reading (both gen_teacher and eval_seq do now).
- The eval's LAYERS = 8..64 step 8 (0-based capture 7..63). Sample: only answer tokens scored;
  KL from last capture layer + model's own head; dist over all 8 capture layers.

## What failed and the open problem

1. **scale_refit (chal-v1): KL ~12, dist ~1.0 — garbage.** The LS scale refit toward the
   teacher's rotated weights still corrupts the model even after the block-axis fix. Remaining
   suspect: the rotation transform `hf_to_stored` ((W·signs)@H per 1024-block on the HF tensor's
   last dim) — run this decisive 5-min test: for one tensor (e.g. blk.0.ffn_gate), correlate
   `hf_to_stored(teacher)` against the king's raw dequant `W'` (stored space). If cos is not
   ~0.9+, the rotation side/order is wrong; then the identity test (refit vs the king's own
   stored tensor must return d* = d_orig) will find the axis bug.
2. **qat_norms: trains well on its own samples (loss −33%) but overfits / doesn't transfer to
   the fork measurement.** chal-norms improved dist +3.4% but KL −247%. The surrogate's L64
   systematic error (~2.6% rel) is LARGER than the 0.84% dist margin, so surrogate-guided tuning
   is below the noise floor at the margins. Ideas: (a) train with the fork in the loop (slow),
   (b) fix the rotation issue and use weight-level refits, (c) bigger, more diverse training set
   (the eval draws 40-60 per domain incl. long agent trajectories; my 16 rows were short math),
   (d) KL-weighted loss (alpha 0.15 tried; hurt both).

## Commands

```bash
cd /root && export INGOT_TEACHER_GPUS=0 INGOT_KING_GPUS=0 INGOT_CHALLENGER_GPUS=0
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True

# train data (teacher trajectories for 16 short samples -> /root/train_samples.json)
/root/ingot/.venv/bin/python /root/gen_teacher.py --samples 16 --max-prompt 1200 --workers 24

# norm QAT (surrogate)
/root/ingot/.venv/bin/python /root/qat_norms.py --steps 25 --alpha 0.15 --max-tokens 1500 --out /root/models/chal-X

# GPU scale refit
/root/ingot/.venv/bin/python /root/scale_refit_gpu.py --out /root/models/chal-v1

# validate
cd /root/ingot && .venv/bin/python scripts/validate_model.py --path /root/models/<dir>/<file>.gguf

# official-path measurement (8-16 samples, ~10-35 min): THE ground truth
nohup bash /root/run_m3.sh &   # edit challenger/dataset lines; log /root/measure.log

# submit (needs funded coldkey ~0.5 tau; .env: INGOT_COLDKEY/INGOT_HOTKEY/INGOT_NETUID=97/INGOT_NETWORK=finney)
uv run ingot register && uv run ingot check-model --path <file> && uv run ingot submit-private --path <file>
# eval on the validator ~27 min; kings crowned immediately on win
```

## Decision state

- chal-norms / chal-n2 / chal-v1: all measured LOSING. Nothing to submit yet.
- The rotation/block-axis correspondence in the stored GGUF is the one open mystery blocking the
  weight-refit path. The surrogate path works for exploration but its noise floor exceeds the
  dist margin at deep layers.
- The exploit itself is already proven without a submission: full draw reproduction +
  margins-hugging-the-bar + r=0.9985 king/challenger correlation in the official artifacts.