| # INGOT Exploit Verification β Handoff for Troy |
|
|
| Goal: produce a q2 GGUF that beats the current king (Ingot-IX-27B-Q2, KL 0.10294 vs teacher, |
| dist ~0.32) by the win margins (**β₯1.5% KL** and **β₯0.84% dist**, both relative to the king, |
| measured in the same eval), verify it offline, and submit it. This proves the SN97 exploit: |
| duels are fully reproducible from public data, so only offline-verified winners get submitted. |
|
|
| ## Access |
|
|
| - Gateway (flaky): `ssh nx7uk9r5ieqbsr-64410ce7@ssh.runpod.io -i ~/.ssh/ingot_a100_ed25519` (needs `-tt` + a tty: use `script -q /dev/null ssh -tt ...`) |
| - Direct (reliable, SCP works): `ssh -p 18956 root@216.81.245.29 -i <your key>` β your key is already in `~/.ssh/authorized_keys` |
| - Pod: H100 NVL 96GB, 144 cores, 200GB disk, **dies ~02:40 UTC Oct 26 (4h rental)**. If it died, rerun `scripts/setup_h100.sh` (works, but the vllm patch step needs `patch --batch`), then re-scp the scripts below and redo: config patch β king_hf β gen_teacher β qat. |
| - On a fresh pod the fork must build with `-DCMAKE_CUDA_ARCHITECTURES=90` (H100) or `80` (A100). |
|
|
| ## Layout on the pod |
|
|
| - Repo: `/root/ingot` (commit 7bb8e7d, submodule llama-prism at e3de715, built for sm_90: `build/llama_states/llama-states` + `llama-prism/build/bin`) |
| - venv: `/root/ingot/.venv` (vLLM 0.31.0 + torch + ingot `.[eval]`, PR #56822 applied, EVAL BOX OK verified) |
| - Models: `/root/models/teacher` (Qwen3.8-27B BF16), `/root/models/king-ix` (Ingot-IX-27B-Q2.gguf) |
| - `/dev/shm/king_hf.pt`: the king dequantized to HF layout (851 tensors) β the surrogate checkpoint |
| - Scripts in /root: `eval_seq.py` (official-path measurement), `gen_teacher.py`, `qat_norms.py`, `qat.py` (unused draft), `scale_refit_gpu.py`, `make_traindata.py`, `run_measure.sh`/`run_m3.sh` |
| - Dev patches in `src/eval_service/config.py`: `LLAMA_PER_GPU=1`, `MEMORY_SHARE` teacher 0.72, `MAX_MODEL_LEN=4096` (single-GPU reality; scores unaffected) |
|
|
| ## Proven so far |
|
|
| 1. **Oracle fully reproduced.** eval 156's draw = 1079/1080 exact (1 sample = tokenizer-version |
| boundary flip on a 30k-token zh prompt). Deployed eval uses `SAMPLES_PER_DOMAIN=60` for ALL |
| 18 domains (public repo says 40 + swe/terminal 150 β repo is out of sync with the live |
| validator!). Seed = miner's r2ready block hash; corpus pinned by dataset_hash = |
| sha256("dendriteholdings/ingot_corpus@<commit>:chungus"). |
| 2. **Offline harness works.** `eval_seq.py` runs the official scorer (vLLM teacher + fork |
| llama-states reads + metrics) sequentially. Sanity: king==challenger gives identical scores. |
| Measured king KL β official values on short samples. |
| 3. **Surrogate validated.** transformers' qwen3_5 forward with the king's HF-layout tensors |
| matches the fork's read: L08 cos 0.998, L16-L56 0.96-0.997, L64 0.94 (rel 2.6% at L64 β |
| this systematic error matters, see below). |
| 4. Registration burn β 0.05 Ο (~$12). Queue empty. Kings crown every few hours (treadmill live: |
| IX crowned Oct 25 11:21, margins 1.51-1.52% β challengers are calibrated perturbations). |
| |
| ## Critical technical knowledge (cost hours to discover) |
| |
| - The fork's converter (`llama-prism/conversion/qwen.py`) reorders by value heads: |
| `A_log β -exp(A_log)`, `dt_bias β ssm_dt.bias`, conv1d squeezed, and **A_log/dt_bias/conv1d-V |
| channels are row-tiled [rep,n_k] β [n_k,rep]**. `GgufCheckpoint` handles weights; the ssm |
| params must be untiled by hand (see build_king_hf.py β correct version is on the pod). |
| - Norm weights are stored as `hf + 1` (except `linear_attn.norm`/ssm_norm, stored as-is). |
| Output norm = hf+1 too. |
| - Hadamard rotation: block_size=1024, per-tensor sign vectors (`prism.hadamard.sign_widths` |
| [5120, 6144, 17408]), rotation axis = gguf shape[0]. `GgufCheckpoint.get()` = the correct |
| unrotation (`(W'@H)*signs`), proven by the surrogate fidelity check. |
| - PQ2_0: 34 B per 128 weights (2B fp16 scale + 32B codes), value = (codeβ1)Β·d, code 3 (+2) |
| rejected by validation. **Blocks tile the gguf COLUMN axis** (i.e., numpy rows of the |
| reversed layout) β NOT the row axis. Any scale refit must fit along that axis. |
| - vLLM writes hidden-state files ASYNCHRONOUSLY β always poll-wait for the file after |
| `generate()`/`complete()` before reading (both gen_teacher and eval_seq do now). |
| - The eval's LAYERS = 8..64 step 8 (0-based capture 7..63). Sample: only answer tokens scored; |
| KL from last capture layer + model's own head; dist over all 8 capture layers. |
| |
| ## What failed and the open problem |
| |
| 1. **scale_refit (chal-v1): KL ~12, dist ~1.0 β garbage.** The LS scale refit toward the |
| teacher's rotated weights still corrupts the model even after the block-axis fix. Remaining |
| suspect: the rotation transform `hf_to_stored` ((WΒ·signs)@H per 1024-block on the HF tensor's |
| last dim) β run this decisive 5-min test: for one tensor (e.g. blk.0.ffn_gate), correlate |
| `hf_to_stored(teacher)` against the king's raw dequant `W'` (stored space). If cos is not |
| ~0.9+, the rotation side/order is wrong; then the identity test (refit vs the king's own |
| stored tensor must return d* = d_orig) will find the axis bug. |
| 2. **qat_norms: trains well on its own samples (loss β33%) but overfits / doesn't transfer to |
| the fork measurement.** chal-norms improved dist +3.4% but KL β247%. The surrogate's L64 |
| systematic error (~2.6% rel) is LARGER than the 0.84% dist margin, so surrogate-guided tuning |
| is below the noise floor at the margins. Ideas: (a) train with the fork in the loop (slow), |
| (b) fix the rotation issue and use weight-level refits, (c) bigger, more diverse training set |
| (the eval draws 40-60 per domain incl. long agent trajectories; my 16 rows were short math), |
| (d) KL-weighted loss (alpha 0.15 tried; hurt both). |
| |
| ## Commands |
| |
| ```bash |
| cd /root && export INGOT_TEACHER_GPUS=0 INGOT_KING_GPUS=0 INGOT_CHALLENGER_GPUS=0 |
| PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True |
| |
| # train data (teacher trajectories for 16 short samples -> /root/train_samples.json) |
| /root/ingot/.venv/bin/python /root/gen_teacher.py --samples 16 --max-prompt 1200 --workers 24 |
| |
| # norm QAT (surrogate) |
| /root/ingot/.venv/bin/python /root/qat_norms.py --steps 25 --alpha 0.15 --max-tokens 1500 --out /root/models/chal-X |
|
|
| # GPU scale refit |
| /root/ingot/.venv/bin/python /root/scale_refit_gpu.py --out /root/models/chal-v1 |
|
|
| # validate |
| cd /root/ingot && .venv/bin/python scripts/validate_model.py --path /root/models/<dir>/<file>.gguf |
| |
| # official-path measurement (8-16 samples, ~10-35 min): THE ground truth |
| nohup bash /root/run_m3.sh & # edit challenger/dataset lines; log /root/measure.log |
|
|
| # submit (needs funded coldkey ~0.5 tau; .env: INGOT_COLDKEY/INGOT_HOTKEY/INGOT_NETUID=97/INGOT_NETWORK=finney) |
| uv run ingot register && uv run ingot check-model --path <file> && uv run ingot submit-private --path <file> |
| # eval on the validator ~27 min; kings crowned immediately on win |
| ``` |
| |
| ## Decision state |
| |
| - chal-norms / chal-n2 / chal-v1: all measured LOSING. Nothing to submit yet. |
| - The rotation/block-axis correspondence in the stored GGUF is the one open mystery blocking the |
| weight-refit path. The surrogate path works for exploration but its noise floor exceeds the |
| dist margin at deep layers. |
| - The exploit itself is already proven without a submission: full draw reproduction + |
| margins-hugging-the-bar + r=0.9985 king/challenger correlation in the official artifacts. |
| |
