onf/docs/RUNBOOK.md

Runbook — every command, in order

Typed top to bottom, this file takes a fresh checkout to a scored benchmark for one suite. Every command below was run on this box; the CLI flags are copied from --help, not guessed.

The README's Quickstart is the same sequence with the commands only. This file adds what each step writes, how long it takes, and how it fails.

It assumes setup is already done — third-party trees under external/, the three runtimes installed, checkpoints and demonstration HDF5 on disk. SETUP.md is that, step by step, including which directory each download has to land in. On this box everything is already in place.

Two rules that apply to every command here:

• OMP_NUM_THREADS=4 — without it the training loop oversubscribes threads and runs an order of magnitude slower.
• PYTHONPATH=src — the onf package is not installed into the eval envs.

Shorthand used below:

cd /path/to/your/checkout                       # NOT hardcoded: sibling worktrees of this repo
export ONF=$(pwd)                                # exist on this box, and they are different branches
export PY=/home/quang/miniconda3/envs/stablevla/bin/python
export OMP_NUM_THREADS=4 PYTHONPATH=src

Just want to run a LIBERO benchmark? Skip to §5.

§1–§4 build artifacts, and on this box they are already built for all four suites (g_nodes/g_edges/g_head/g_track under outputs/<suite>/latest/artifacts/). Check with:

$PY -c "from onf.config import default_paths as d; import os; \
        print({s: sorted(os.listdir(d().graph(s))) for s in ('object','spatial','goal','long')})"

If that prints four artifact sets, go straight to §5 Benchmark. Read §1–§4 only when you need to rebuild a graph or retrain a head. The three commands worth running first, in this order:

$PY scripts/run_sr.py --check-env                        # GPUs, load, LIBERO-Plus episode counts
$PY scripts/run_sr.py --rung R1 --gpus 2,3 --dry-run     # what would run, and for how long
$PY scripts/run_sr.py --rung R1 --gpus 2,3               # 35 episodes, ~15 min

---

Pick one suite and keep it fixed through §1–§4. Valid values: object, spatial, goal, long (long is LIBERO-10; the mapping to the benchmark's own suite names lives in configs/suites.yaml).

export SUITE=goal

---

0. Inputs

Clones, environments, weights and demonstrations: SETUP.md. Point the repo at its trees with:

export ONF_DATA=/path/to/data          # demo hdf5 + per-suite fwm/ artifacts
export ONF_OUTPUTS=/path/to/outputs    # graph artifacts land here
export ONF_RESULTS=/path/to/results    # rollout mp4s land here

onf.config.Paths resolves everything from these three; nothing else is hardcoded.

Input	Where	Note
demo HDF5	data/ (Paths.hdf5(_hdf5_name(SUITE)))	SETUP.md step 4
policy checkpoint	stablevla/hf_weights/<suite>/, external/Isaac-GR00T/ckpts_n17_libero/<...>	paths in configs/suites.yaml
LIBERO-Plus	external/LIBERO-plus	SETUP.md step 1

On the author's box data/, outputs/, results/ and external/ are symlinks into trees shared with sibling git worktrees — a fresh clone has real directories and none of this applies. Where they are shared, another branch rebuilding an artifact will change it underneath you. tests/golden/parity.json records a sha256 of each artifact for exactly this reason — if the parity test reports ARTIFACT DRIFT, someone else rebuilt a file, it is not a regression in your code.

---

1. Field artifacts — q_flow.npz → onf_head.npz

The retrieval head pools a query window by cleanliness, and cleanliness is read off a trained distance-to-manifold field. That field is needed both to train the graph head and at deploy time, so it comes first.

The two directories the builders need are resolved by onf.config.Paths, but the HDF5 one goes through a suite -> dataset-name mapping (goal -> libero_goal, long -> libero_10, ...), so resolve both once and reuse them:

eval "$($PY -c "
from onf.config import default_paths
from onf.graph.build.from_demos import _hdf5_name
p = default_paths()
print(f'HDF5={p.hdf5(_hdf5_name(\"$SUITE\"))}')
print(f'FWM={p.fwm(\"$SUITE\")}')")"
echo "$HDF5 -> $FWM"

# demo velocity cloud -> $FWM/q_flow.npz, q_home.npz
$PY -m onf.field.build qflow "$HDF5" "$FWM"

# q_flow.npz -> $FWM/onf_head.npz   (the field; ~10 s on CPU for `long`)
$PY -m onf.field.train "$FWM"

Or both in one step, which also runs the field training:

$PY -m onf.field.build suite --suite "$SUITE"

Skip this section if <fwm_dir>/onf_head.npz already exists. On this box all four suites already have one (data/fwm/onf_head.npz for object, data/fwm/<suite>/onf_head.npz for the rest).

If it is missing, nothing breaks loudly — cleanliness() warns once and falls back to uniform window weights. That is a different (worse) query encoder, so train and deploy must agree: never train with a field and deploy without one.

---

2. Build the demonstration graph

$PY -m onf.graph build --suite "$SUITE"

Writes g_nodes.npz + g_edges.npz into outputs/$SUITE/<run>/artifacts/ and repoints outputs/$SUITE/latest at that run.

Read this before building long. outputs/ is a symlink into a tree shared with sibling worktrees, and tests/golden/parity.json pins the sha256 of outputs/long/latest/artifacts/*. So repointing latest does not just affect you: every checkout's tests/test_parity.py starts reporting ARTIFACT DRIFT, and any eval reading latest silently switches to your new graph. Build somewhere else unless you specifically mean to replace the shipped artifacts:

export ONF_OUTPUTS=/tmp/my_build      # `Paths` resolves everything from this; nothing else changes

Steps 2–4 then read and write only under $ONF_OUTPUTS. Step 1 is separate: it writes into Paths.fwm($SUITE), i.e. the shared data/ tree, so pass an explicit output directory there (the recipe in §1 already takes one as an argument).

Useful flags: --limit-demos 8 --limit-tasks 2 for a fast smoke build, --dry-run to print the resolved config and exit without touching outputs/, --coarsen N to change raw frames per node.

On this box, outputs/object/latest is a real directory, not a symlink — a stale build from an earlier run. Paths.graph("object") therefore resolves to it and a new build cannot repoint latest. It contains a valid g_nodes/g_edges pair, so training into it works; just be aware that "latest" is frozen for that one suite.

3. Train the retrieval head

$PY -m onf.graph train --suite "$SUITE" --device cuda:3

Writes g_head.npz next to the graph it trained on. ~1 hour per suite on one GPU. --epochs 1 for a smoke run; --dry-run prints the resolved config and the planned stages without writing anything.

This is the only learned component on the retrieval path. The blend weight below is the second and last one in the repo, and it is trained against this head frozen.

4. Fit the sentinel artifacts

ART=$($PY -c "from onf.config import default_paths; print(default_paths().graph('$SUITE'))")
$PY scripts/build_sentinel_artifacts.py "$ART" --suite "$SUITE" --device cuda:3

Fits the transition kernel (pi, beta, leak) and the basin geometry into g_track.npz, plus a sentinel_artifacts.json provenance sidecar. ~30 min per suite.

This is what the sentinel belief filter needs. A missing g_track.npz is a hard error at launch, not a silent fallback.

The blend's own artifacts

base needs nothing below. Every blend* mode needs the first of them; only blend_learned needs all three. Design: technical/03-action-chunk-blend.md.

# metres/radians per action unit, fitted off the demos -> <graph dir>/action_scale.json
$PY scripts/fit_action_scale.py "$SUITE"

Seconds, CPU, no policy needed. It refuses to write a fit whose position or rotation block scores R² < 0.8 — that gate catches a rotation delta subtracted instead of composed, which is the mistake that silently corrupts the orientation channel. A blend mode with no action_scale.json raises at launch and names this command; there is no default calibration.

# the frozen policy's own action chunks over the whole corpus -> <graph dir>/a_pi_raw.npz
$PY scripts/cache_policy_actions.py "$SUITE"

# train the blend weight against a FROZEN retriever -> <graph dir>/g_alpha.npz
$PY -m onf.graph train --suite "$SUITE" --objective chunk --device "cuda:$GPU"

cache_policy_actions.py talks to a running policy server (--host / --port, default port 10093) and is resumable per demo; scripts/run_cache_policy_actions.sh brings the server up around it. --objective chunk optimises only the weight head — it loads g_head.npz, freezes it, and writes g_alpha.npz beside it without rewriting the head it froze.

Build everything from scratch, in one block

Verified end to end on 2026-08-07 with a smoke-sized graph (--limit-demos 8 --limit-tasks 2, CPU, long). Timings are from that run; a full-suite build is the per-step estimates above.

export SUITE=long
export SCRATCH=/tmp/onf_scratch                 # nothing here touches the shared outputs/ tree
export ONF_OUTPUTS=$SCRATCH/outputs
mkdir -p "$SCRATCH/fwm"

HDF5=$($PY -c "from onf.config import default_paths as d
from onf.graph.build.from_demos import _hdf5_name
print(d().hdf5(_hdf5_name('$SUITE')))")

$PY -m onf.field.build qflow "$HDF5" "$SCRATCH/fwm"        # 1s   -> q_flow.npz, q_home.npz
$PY -m onf.field.train "$SCRATCH/fwm"                       # 8s   -> onf_head.npz
$PY -m onf.graph build --suite "$SUITE" --limit-demos 8 --limit-tasks 2   # 12s -> g_nodes, g_edges
$PY -m onf.graph train --suite "$SUITE" --epochs 1 --device cpu           # 86s -> g_head.npz
ART=$($PY -c "from onf.config import default_paths; print(default_paths().graph('$SUITE'))")
$PY scripts/build_sentinel_artifacts.py "$ART" --suite "$SUITE" --device cpu   # 126s -> g_track.npz

Drop the two --limit-* flags and --epochs 1, and use --device cuda:N, for a real build.

Verify what you built — loads the artifacts and runs one step of each regime:

$PY - <<'EOF'
import numpy as np
from onf.config import GraphConfig, default_paths
from onf.graph.run.track import GraphTracker
tr = GraphTracker.load(default_paths().graph("long"), cfg=GraphConfig(hist=8, device="cpu"), device="cpu")
n = tr.retriever.nodes
print(f"V={len(n)} demos={n.n_demos} tasks={n.n_tasks} beta={tr.kernel.beta:.3f} leak={tr.leak:.3f}")
q = n.q_raw[:8].astype(np.float64); tr.reset()
print("step:", tr.step(q).phase_hat, "| where:", tr.where_target(q[-1]).depth)
EOF

A build that produces all four .npz files but cannot be loaded here is the failure worth catching early; g_head.npz and g_track.npz both carry a graph_hash stamp and are refused against a graph they were not fit on.

All three steps for the three suites that only have a graph

GPU=3                                        # sequential, ~1.5 h per suite
for SUITE in goal spatial object; do
    $PY -m onf.graph build --suite "$SUITE"
    $PY -m onf.graph train --suite "$SUITE" --device "cuda:$GPU"
    ART=$($PY -c "from onf.config import default_paths; print(default_paths().graph('$SUITE'))")
    $PY scripts/build_sentinel_artifacts.py "$ART" --suite "$SUITE" --device "cuda:$GPU"
done

---

5. Benchmark

Modes are defined in one place (evals/common/modes.sh):

mode	what it does
base	frozen policy, no intervention
blend	the action-chunk blend at alpha 0.0 — bit-identical to base, the plumbing check
blend_a05	the blend at a fixed alpha 0.5, ADVANCE=0 head, no bound. The arm results.md §5 measures — superseded, kept because it is still the best arm on Camera_Viewpoints
blend_a015	the same at 0.15 — intervention magnitude only
blend_bounded	fixed alpha 0.3 with the row-0 saturation bound at 1.0
blend_a015_bounded	the same at 0.15 — the best fixed weight measured
blend_scaled015	the trained per-row, per-block weight from g_alpha.npz, rescaled by SN_BLEND_SCALE
blend_full	the current recipe. Learned weight at scale 0.27 off the base-frame head, plus the behavioural task lane. results.md §6

evals/common/modes.sh holds a dozen further probe arms; the table above is the ladder that leads to the current recipe.

Every blend* mode needs action_scale.json and every learned-weight mode needs g_alpha.npz; both are §4's "blend's own artifacts". Both are missing-file errors at launch, never silent fallbacks.

configs/sr_ladder.yaml's axis_default_mode picks a mode per axis; both drivers read that table, so it cannot drift between them. It is base everywhere today — the blend arms are run by passing --mode explicitly, because at a fixed alpha the blend is net negative across the seven axes and nothing has earned the default.

Result and log names share one grammar — the result tag and the log directory name are the same string, so a number can always be traced back to its logs:

results/plus_{bench}/{axis}/{policy}_{mode}_{suite}_{axis}/     # rollout mp4s + run.json
logs/{utc_timestamp}_{policy}_{mode}_{suite}_{axis}/            # server.log + client_shard*.log

StableVLA

# always look at the schedule first
$PY scripts/run_sr.py --rung R4 --gpus 2,3 --dry-run

# smoke: three small instance-filtered cells
$PY scripts/run_sr.py --rung R1 --gpus 2,3

# the full 4 suites x 7 axes grid
$PY scripts/run_sr.py --rung R4 --gpus 2,3

# a subset of a rung, keeping each cell's own episode filter / expected count / gate
$PY scripts/run_sr.py --rung R1 --cells long:Robot_Initial_States,long:Sensor_Noise --gpus 2,3

Other flags: --resume (skip cells already complete in the ledger, purge and rerun partial ones), --mode <mode> (override the per-axis default), --score / --ledger (report on existing results without launching anything), --check-env (GPU/load/LIBERO-Plus preflight only).

GR00T-N1.7

$PY evals/gr00t/run_gr00t.py --suites "$SUITE" --gpu 2 --num-clients 6 --dry-run
$PY evals/gr00t/run_gr00t.py --suites "$SUITE" --gpu 2 --num-clients 6

One suite per GPU (--gpu takes a single index). Add --axes "Sensor Noise" to restrict axes, or --mode base for a raw-GR00T baseline. It refuses to write into a result directory that already holds rollouts unless --resume/--force: re-running a cell in place would leave both the success and the failure mp4 of a flipped episode and inflate n.

Score

# success-rate grid; --methods takes name:tag_prefix pairs
$PY evals/libero_plus/score.py --suites "$SUITE" --methods base:stablevla_base

# paired McNemar between two arms of one suite x axis
$PY evals/libero_plus/mcnemar.py --suite 10 --axis Sensor_Noise \
    --tags stablevla_base_long_Sensor_Noise stablevla_other_long_Sensor_Noise

McNemar is only meaningful for a deterministic policy. StableVLA is deterministic (0 episode churn), so its arms compare exactly, episode by episode. GR00T-N1.7 is flow-matching and stochastic (~11% episode churn, roughly ±2.4 pp): compare its arms only within one session, against a baseline measured alongside them, and read the sign rather than the digits.

---

6. Reproducibility

$PY -m pytest tests/ -q
$PY -m pytest tests/test_parity.py -q          # must be "2 passed"

tests/test_parity.py is the golden bit-parity net: it replays 64 recorded t=0 retrievals and 64 t>0 tracker steps against the real trained artifacts and asserts every float is exactly equal (via float.hex(), not approx) to tests/golden/parity.json. It skips cleanly when the artifacts are absent, so a fresh clone never fails it.

To re-record it — only after you have confirmed the artifacts on disk are the ones you mean to ship:

$PY scripts/capture_golden.py

Every result directory also carries a run.json recording policy, mode, suite, axis, expected episode count, graph dir, graph_hash, head mtime, git commit and the full mode env.

---

7. GPU etiquette on this box

GPUs 0 and 1 are in use by other work. Use --gpus 2,3 (StableVLA) or --gpu 2 / --gpu 3 (GR00T). run_sr.py also refuses to launch a large grid when the 1-minute load average is above its preflight threshold; --check-env prints that report on its own.