onf/docs/RUNBOOK.md

# Runbook — every command, in order

Typed top to bottom, this file takes a fresh checkout to a scored benchmark for one suite. Every
command below was run on this box; the CLI flags are copied from `--help`, not guessed.

The README's Quickstart is the same sequence with the commands only. This file adds what each step
writes, how long it takes, and how it fails.

It assumes setup is already done — third-party trees under `external/`, the three runtimes installed,
checkpoints and demonstration HDF5 on disk. [`SETUP.md`](SETUP.md) is that, step by step, including
which directory each download has to land in. On this box everything is already in place.

Two rules that apply to **every** command here:

- `OMP_NUM_THREADS=4` — without it the training loop oversubscribes threads and runs an order of
  magnitude slower.
- `PYTHONPATH=src` — the `onf` package is not installed into the eval envs.

Shorthand used below:

```bash
cd /path/to/your/checkout                       # NOT hardcoded: sibling worktrees of this repo
export ONF=$(pwd)                                # exist on this box, and they are different branches
export PY=/home/quang/miniconda3/envs/stablevla/bin/python
export OMP_NUM_THREADS=4 PYTHONPATH=src
```

### Just want to run a LIBERO benchmark? Skip to §5.

§1–§4 build artifacts, and on this box they are **already built for all four suites**
(`g_nodes`/`g_edges`/`g_head`/`g_track` under `outputs/<suite>/latest/artifacts/`). Check with:

```bash
$PY -c "from onf.config import default_paths as d; import os; \
        print({s: sorted(os.listdir(d().graph(s))) for s in ('object','spatial','goal','long')})"
```

If that prints four artifact sets, go straight to **§5 Benchmark**. Read §1–§4 only when you need to
rebuild a graph or retrain a head. The three commands worth running first, in this order:

```bash
$PY scripts/run_sr.py --check-env                        # GPUs, load, LIBERO-Plus episode counts
$PY scripts/run_sr.py --rung R1 --gpus 2,3 --dry-run     # what would run, and for how long
$PY scripts/run_sr.py --rung R1 --gpus 2,3               # 35 episodes, ~15 min
```

---

Pick one suite and keep it fixed through §1–§4. Valid values: `object`, `spatial`, `goal`, `long`
(`long` is LIBERO-10; the mapping to the benchmark's own suite names lives in `configs/suites.yaml`).

```bash
export SUITE=goal
```

---

## 0. Inputs

Clones, environments, weights and demonstrations: [`SETUP.md`](SETUP.md). Point the repo at its trees
with:

```bash
export ONF_DATA=/path/to/data          # demo hdf5 + per-suite fwm/ artifacts
export ONF_OUTPUTS=/path/to/outputs    # graph artifacts land here
export ONF_RESULTS=/path/to/results    # rollout mp4s land here
```

`onf.config.Paths` resolves everything from these three; nothing else is hardcoded.

| Input | Where | Note |
|---|---|---|
| demo HDF5 | `data/` (`Paths.hdf5(_hdf5_name(SUITE))`) | [`SETUP.md`](SETUP.md) step 4 |
| policy checkpoint | `stablevla/hf_weights/<suite>/`, `external/Isaac-GR00T/ckpts_n17_libero/<...>` | paths in `configs/suites.yaml` |
| LIBERO-Plus | `external/LIBERO-plus` | [`SETUP.md`](SETUP.md) step 1 |

On the author's box `data/`, `outputs/`, `results/` and `external/` are **symlinks into trees shared
with sibling git worktrees** — a fresh clone has real directories and none of this applies. Where they
are shared, another branch rebuilding an artifact will change it underneath you. `tests/golden/parity.json`
records a sha256 of each artifact for exactly this reason — if the parity test reports `ARTIFACT DRIFT`,
someone else rebuilt a file, it is not a regression in your code.

---

## 1. Field artifacts — `q_flow.npz` → `onf_head.npz`

The retrieval head pools a query window by *cleanliness*, and cleanliness is read off a trained
distance-to-manifold field. That field is needed both to train the graph head and at deploy time, so
it comes first.

The two directories the builders need are resolved by `onf.config.Paths`, but the HDF5 one goes
through a suite -> dataset-name mapping (`goal` -> `libero_goal`, `long` -> `libero_10`, ...), so
resolve both once and reuse them:

```bash
eval "$($PY -c "
from onf.config import default_paths
from onf.graph.build.from_demos import _hdf5_name
p = default_paths()
print(f'HDF5={p.hdf5(_hdf5_name(\"$SUITE\"))}')
print(f'FWM={p.fwm(\"$SUITE\")}')")"
echo "$HDF5 -> $FWM"

# demo velocity cloud -> $FWM/q_flow.npz, q_home.npz
$PY -m onf.field.build qflow "$HDF5" "$FWM"

# q_flow.npz -> $FWM/onf_head.npz   (the field; ~10 s on CPU for `long`)
$PY -m onf.field.train "$FWM"
```

Or both in one step, which also runs the field training:

```bash
$PY -m onf.field.build suite --suite "$SUITE"
```

Skip this section if `<fwm_dir>/onf_head.npz` already exists. On this box all four suites already
have one (`data/fwm/onf_head.npz` for `object`, `data/fwm/<suite>/onf_head.npz` for the rest).

If it is missing, nothing breaks loudly — `cleanliness()` warns once and falls back to uniform window
weights. That is a different (worse) query encoder, so train and deploy must agree: never train with
a field and deploy without one.

---

## 2. Build the demonstration graph

```bash
$PY -m onf.graph build --suite "$SUITE"
```

Writes `g_nodes.npz` + `g_edges.npz` into `outputs/$SUITE/<run>/artifacts/` and **repoints
`outputs/$SUITE/latest` at that run**.

> **Read this before building `long`.** `outputs/` is a symlink into a tree shared with sibling
> worktrees, and `tests/golden/parity.json` pins the sha256 of `outputs/long/latest/artifacts/*`. So
> repointing `latest` does not just affect you: every checkout's `tests/test_parity.py` starts
> reporting `ARTIFACT DRIFT`, and any eval reading `latest` silently switches to your new graph.
> Build somewhere else unless you specifically mean to replace the shipped artifacts:
>
> ```bash
> export ONF_OUTPUTS=/tmp/my_build      # `Paths` resolves everything from this; nothing else changes
> ```
>
> Steps 2–4 then read and write only under `$ONF_OUTPUTS`. Step 1 is separate: it writes into
> `Paths.fwm($SUITE)`, i.e. the shared `data/` tree, so pass an explicit output directory there
> (the recipe in §1 already takes one as an argument).

Useful flags: `--limit-demos 8 --limit-tasks 2` for a fast smoke build, `--dry-run` to print the
resolved config and exit without touching `outputs/`, `--coarsen N` to change raw frames per node.

> **On this box, `outputs/object/latest` is a real directory, not a symlink** — a stale build from an
> earlier run. `Paths.graph("object")` therefore resolves to it and a new `build` cannot repoint
> `latest`. It contains a valid `g_nodes`/`g_edges` pair, so training into it works; just be aware
> that "latest" is frozen for that one suite.

## 3. Train the retrieval head

```bash
$PY -m onf.graph train --suite "$SUITE" --device cuda:3
```

Writes `g_head.npz` next to the graph it trained on. ~1 hour per suite on one GPU. `--epochs 1` for a
smoke run; `--dry-run` prints the resolved config and the planned stages without writing anything.

This is the only learned component on the retrieval path. The blend weight below is the second and
last one in the repo, and it is trained against this head frozen.

## 4. Fit the sentinel artifacts

```bash
ART=$($PY -c "from onf.config import default_paths; print(default_paths().graph('$SUITE'))")
$PY scripts/build_sentinel_artifacts.py "$ART" --suite "$SUITE" --device cuda:3
```

Fits the transition kernel (`pi`, `beta`, `leak`) and the basin geometry into `g_track.npz`, plus a
`sentinel_artifacts.json` provenance sidecar. ~30 min per suite.

This is what the sentinel belief filter needs. A missing `g_track.npz` is a hard error at launch, not
a silent fallback.

### The blend's own artifacts

`base` needs nothing below. Every `blend*` mode needs the first of them; only `blend_learned` needs
all three. Design: [`technical/03-action-chunk-blend.md`](technical/03-action-chunk-blend.md).

```bash
# metres/radians per action unit, fitted off the demos -> <graph dir>/action_scale.json
$PY scripts/fit_action_scale.py "$SUITE"
```

Seconds, CPU, no policy needed. It **refuses to write** a fit whose position or rotation block scores
R² < 0.8 — that gate catches a rotation delta subtracted instead of composed, which is the mistake
that silently corrupts the orientation channel. A blend mode with no `action_scale.json` raises at
launch and names this command; there is no default calibration.

```bash
# the frozen policy's own action chunks over the whole corpus -> <graph dir>/a_pi_raw.npz
$PY scripts/cache_policy_actions.py "$SUITE"

# train the blend weight against a FROZEN retriever -> <graph dir>/g_alpha.npz
$PY -m onf.graph train --suite "$SUITE" --objective chunk --device "cuda:$GPU"
```

`cache_policy_actions.py` talks to a running policy server (`--host` / `--port`, default port 10093)
and is resumable per demo; `scripts/run_cache_policy_actions.sh` brings the server up around it.
`--objective chunk` optimises only the weight head — it loads `g_head.npz`, freezes it, and writes
`g_alpha.npz` beside it without rewriting the head it froze.

### Build everything from scratch, in one block

Verified end to end on 2026-08-07 with a smoke-sized graph (`--limit-demos 8 --limit-tasks 2`,
CPU, `long`). Timings are from that run; a full-suite build is the per-step estimates above.

```bash
export SUITE=long
export SCRATCH=/tmp/onf_scratch                 # nothing here touches the shared outputs/ tree
export ONF_OUTPUTS=$SCRATCH/outputs
mkdir -p "$SCRATCH/fwm"

HDF5=$($PY -c "from onf.config import default_paths as d
from onf.graph.build.from_demos import _hdf5_name
print(d().hdf5(_hdf5_name('$SUITE')))")

$PY -m onf.field.build qflow "$HDF5" "$SCRATCH/fwm"        # 1s   -> q_flow.npz, q_home.npz
$PY -m onf.field.train "$SCRATCH/fwm"                       # 8s   -> onf_head.npz
$PY -m onf.graph build --suite "$SUITE" --limit-demos 8 --limit-tasks 2   # 12s -> g_nodes, g_edges
$PY -m onf.graph train --suite "$SUITE" --epochs 1 --device cpu           # 86s -> g_head.npz
ART=$($PY -c "from onf.config import default_paths; print(default_paths().graph('$SUITE'))")
$PY scripts/build_sentinel_artifacts.py "$ART" --suite "$SUITE" --device cpu   # 126s -> g_track.npz
```

Drop the two `--limit-*` flags and `--epochs 1`, and use `--device cuda:N`, for a real build.

**Verify what you built** — loads the artifacts and runs one step of each regime:

```bash
$PY - <<'EOF'
import numpy as np
from onf.config import GraphConfig, default_paths
from onf.graph.run.track import GraphTracker
tr = GraphTracker.load(default_paths().graph("long"), cfg=GraphConfig(hist=8, device="cpu"), device="cpu")
n = tr.retriever.nodes
print(f"V={len(n)} demos={n.n_demos} tasks={n.n_tasks} beta={tr.kernel.beta:.3f} leak={tr.leak:.3f}")
q = n.q_raw[:8].astype(np.float64); tr.reset()
print("step:", tr.step(q).phase_hat, "| where:", tr.where_target(q[-1]).depth)
EOF
```

A build that produces all four `.npz` files but cannot be loaded here is the failure worth catching
early; `g_head.npz` and `g_track.npz` both carry a `graph_hash` stamp and are refused against a graph
they were not fit on.

### All three steps for the three suites that only have a graph

```bash
GPU=3                                        # sequential, ~1.5 h per suite
for SUITE in goal spatial object; do
    $PY -m onf.graph build --suite "$SUITE"
    $PY -m onf.graph train --suite "$SUITE" --device "cuda:$GPU"
    ART=$($PY -c "from onf.config import default_paths; print(default_paths().graph('$SUITE'))")
    $PY scripts/build_sentinel_artifacts.py "$ART" --suite "$SUITE" --device "cuda:$GPU"
done
```

---

## 5. Benchmark

Modes are defined in one place (`evals/common/modes.sh`):

| mode | what it does |
|---|---|
| `base` | frozen policy, no intervention |
| `blend` | the action-chunk blend at alpha 0.0 — bit-identical to `base`, the plumbing check |
| `blend_a05` | the blend at a fixed alpha 0.5, ADVANCE=0 head, no bound. The arm [`results.md`](results.md) §5 measures — **superseded**, kept because it is still the best arm on Camera_Viewpoints |
| `blend_a015` | the same at 0.15 — intervention magnitude only |
| `blend_bounded` | fixed alpha 0.3 with the row-0 saturation bound at 1.0 |
| `blend_a015_bounded` | the same at 0.15 — the best fixed weight measured |
| `blend_scaled015` | the trained per-row, per-block weight from `g_alpha.npz`, rescaled by `SN_BLEND_SCALE` |
| `blend_full` | **the current recipe.** Learned weight at scale 0.27 off the base-frame head, plus the behavioural task lane. [`results.md`](results.md) §6 |

`evals/common/modes.sh` holds a dozen further probe arms; the table above is the ladder that leads to
the current recipe.

Every `blend*` mode needs `action_scale.json` and every learned-weight mode needs `g_alpha.npz`; both
are §4's "blend's own artifacts". Both are missing-file **errors** at launch, never silent fallbacks.

`configs/sr_ladder.yaml`'s `axis_default_mode` picks a mode per axis; both drivers read that table, so
it cannot drift between them. It is `base` everywhere today — the blend arms are run by passing
`--mode` explicitly, because at a fixed alpha the blend is net negative across the seven axes and
nothing has earned the default.

Result and log names share one grammar — the result tag and the log directory name are the same
string, so a number can always be traced back to its logs:

```
results/plus_{bench}/{axis}/{policy}_{mode}_{suite}_{axis}/     # rollout mp4s + run.json
logs/{utc_timestamp}_{policy}_{mode}_{suite}_{axis}/            # server.log + client_shard*.log
```

### StableVLA

```bash
# always look at the schedule first
$PY scripts/run_sr.py --rung R4 --gpus 2,3 --dry-run

# smoke: three small instance-filtered cells
$PY scripts/run_sr.py --rung R1 --gpus 2,3

# the full 4 suites x 7 axes grid
$PY scripts/run_sr.py --rung R4 --gpus 2,3

# a subset of a rung, keeping each cell's own episode filter / expected count / gate
$PY scripts/run_sr.py --rung R1 --cells long:Robot_Initial_States,long:Sensor_Noise --gpus 2,3
```

Other flags: `--resume` (skip cells already complete in the ledger, purge and rerun partial ones),
`--mode <mode>` (override the per-axis default), `--score` / `--ledger` (report on existing results
without launching anything), `--check-env` (GPU/load/LIBERO-Plus preflight only).

### GR00T-N1.7

```bash
$PY evals/gr00t/run_gr00t.py --suites "$SUITE" --gpu 2 --num-clients 6 --dry-run
$PY evals/gr00t/run_gr00t.py --suites "$SUITE" --gpu 2 --num-clients 6
```

One suite per GPU (`--gpu` takes a single index). Add `--axes "Sensor Noise"` to restrict axes, or
`--mode base` for a raw-GR00T baseline. It refuses to write into a result directory that already
holds rollouts unless `--resume`/`--force`: re-running a cell in place would leave both the success
and the failure mp4 of a flipped episode and inflate `n`.

### Score

```bash
# success-rate grid; --methods takes name:tag_prefix pairs
$PY evals/libero_plus/score.py --suites "$SUITE" --methods base:stablevla_base

# paired McNemar between two arms of one suite x axis
$PY evals/libero_plus/mcnemar.py --suite 10 --axis Sensor_Noise \
    --tags stablevla_base_long_Sensor_Noise stablevla_other_long_Sensor_Noise
```

McNemar is only meaningful for a **deterministic** policy. StableVLA is deterministic (0 episode
churn), so its arms compare exactly, episode by episode. GR00T-N1.7 is flow-matching and stochastic
(~11% episode churn, roughly ±2.4 pp): compare its arms only **within one session**, against a
baseline measured alongside them, and read the sign rather than the digits.

---

## 6. Reproducibility

```bash
$PY -m pytest tests/ -q
$PY -m pytest tests/test_parity.py -q          # must be "2 passed"
```

`tests/test_parity.py` is the golden bit-parity net: it replays 64 recorded t=0 retrievals and 64 t>0
tracker steps against the real trained artifacts and asserts every float is **exactly** equal (via
`float.hex()`, not `approx`) to `tests/golden/parity.json`. It skips cleanly when the artifacts are
absent, so a fresh clone never fails it.

To re-record it — only after you have confirmed the artifacts on disk are the ones you mean to ship:

```bash
$PY scripts/capture_golden.py
```

Every result directory also carries a `run.json` recording policy, mode, suite, axis, expected episode
count, graph dir, `graph_hash`, head mtime, git commit and the full mode env.

---

## 7. GPU etiquette on this box

GPUs 0 and 1 are in use by other work. Use `--gpus 2,3` (StableVLA) or `--gpu 2` / `--gpu 3` (GR00T).
`run_sr.py` also refuses to launch a large grid when the 1-minute load average is above its preflight
threshold; `--check-env` prints that report on its own.