onf-v3/docs/usage.md

# Usage

Copy-pasteable path from a fresh checkout to a scored benchmark. Every command assumes the repo root
as cwd. **Always set `OMP_NUM_THREADS=4`** — without it the training loop oversubscribes threads and
slows down by an order of magnitude.

---

## 0. Environments

Four separate runtimes; they cannot be merged (conflicting pins). See `envs/README.md`.

| env | file | used for |
|---|---|---|
| `onf-core` | `envs/onf-core.yml` | the `onf` package: graph build/train, artifacts, tests |
| `stablevla` | `envs/stablevla.yml` | the StableVLA policy server |
| libero-plus | via `scripts/setup_external.sh` | the simulator client |
| gr00t | separate conda env, path given by `GR00T_PY` | the GR00T-N1.7 policy server |

```bash
conda env create -f envs/onf-core.yml
pip install -e .                       # inside onf-core
bash scripts/setup_external.sh         # links LIBERO-plus and the vendored deps into external/
```

Data and checkpoints are **not** downloaded by the setup script. Point the repo at them with:

```bash
export ONF_DATA=/path/to/data          # demo hdf5 + per-suite fwm/ artifacts
export ONF_OUTPUTS=/path/to/outputs    # graph artifacts land here
export ONF_RESULTS=/path/to/results    # rollout mp4s land here
```

`onf.config.Paths` resolves everything from these three; nothing else is hardcoded.

---

## 1. Build the demonstration graph

```bash
OMP_NUM_THREADS=4 python -m onf.graph build --suite long
```

Writes `g_nodes.npz` + `g_edges.npz` into `$ONF_OUTPUTS/long/<run>/artifacts/`, and points
`$ONF_OUTPUTS/long/latest` at it. Use `--suite all` for all four suites, or
`--limit-demos 8 --limit-tasks 2` for a fast smoke build.

## 2. Train the retrieval head

```bash
OMP_NUM_THREADS=4 python -m onf.graph train --suite long
```

Writes `g_head.npz` next to the graph it trained on. Roughly one hour per suite on one GPU. Add
`--epochs 1` for a smoke run.

## 3. Fit the sentinel artifacts

```bash
OMP_NUM_THREADS=4 python scripts/build_sentinel_artifacts.py \
    "$ONF_OUTPUTS/long/latest/artifacts" --suite long
```

Fits the transition kernel (`π`, `β`, `leak`), the three null banks and the basin geometry into
`g_track.npz`, plus a `sentinel_artifacts.json` provenance sidecar.

> **t=0 needs steps 1–2; t>0 needs 1–3.** A missing `g_track.npz` is a hard error at launch, not a
> silent fallback. Today only `long` has a trained head and a fitted tracker; `object`, `spatial` and
> `goal` need steps 2–3 run once each before they can be benchmarked.

---

## 4. Benchmark

Three modes exist, and only three: `base` (frozen policy, no recovery), `entry` (t=0), `sentinel`
(t>0). They are defined in one place, `evals/common/modes.sh`.

Result and log names share one grammar:

```
results/plus_{bench}/{axis}/{policy}_{mode}_{suite}_{axis}/     # rollout mp4s + run.json
logs/{utc_timestamp}_{policy}_{mode}_{suite}_{axis}/            # server.log + client_shard*.log
```

### StableVLA

```bash
# smoke first (12 episodes per regime)
OMP_NUM_THREADS=4 python scripts/run_sr.py --rung R1 --gpus 2,3

# the full 4 suites x 7 axes grid
OMP_NUM_THREADS=4 python scripts/run_sr.py --rung R4 --gpus 2,3
```

Useful flags: `--dry-run` (print the schedule and exit), `--check-env` (validate paths/ports only),
`--resume` (skip cells already complete in the ledger), `--mode <mode>` (override the per-axis default
from `configs/sr_ladder.yaml`), and `--cells suite:axis[,...]` — on its own it builds ad hoc full-axis
cells; combined with `--rung` it runs a **subset** of that rung, keeping each cell's own episode
filter, expected count, comparators and gate:

```bash
# just the two `long` smoke cells (12 episodes each), one per regime
python scripts/run_sr.py --rung R1 --cells long:Robot_Initial_States,long:Sensor_Noise --gpus 2,3
```

### GR00T-N1.7

```bash
GR00T_PY=/path/to/gr00t/env/bin/python \
OMP_NUM_THREADS=4 python evals/gr00t/run_gr00t.py \
    --suites object,spatial,goal,long --mode sentinel --gpus 2,3
```

Refuses to start if `g_track.npz` is missing, and refuses to write into a result directory that
already holds rollouts unless `--resume`/`--force` — re-running a cell in place would otherwise leave
both the success and the failure mp4 of a flipped episode and inflate `n`.

### Score

```bash
# success-rate grid; --methods takes name:tag_prefix pairs
python evals/libero_plus/score.py --suites long \
    --methods base:stablevla_base entry:stablevla_entry sentinel:stablevla_sentinel

# paired McNemar between two arms of one suite x axis
python evals/libero_plus/mcnemar.py --suite 10 --axis Sensor_Noise \
    --tags stablevla_base_long_Sensor_Noise stablevla_sentinel_long_Sensor_Noise
```

McNemar is only meaningful for a **deterministic** policy (StableVLA). GR00T-N1.7 is stochastic
(~11.2% episode churn); compare its arms only within one session, and read the sign, not the digits.

---

## 5. Reproducibility

```bash
OMP_NUM_THREADS=4 PYTHONPATH=src pytest tests/ -q
OMP_NUM_THREADS=4 PYTHONPATH=src pytest tests/test_parity.py -q     # must be 2 passed
```

`tests/test_parity.py` is the golden bit-parity net: it replays 64 recorded t=0 retrievals and 64 t>0
tracker steps against the real trained artifacts and asserts every float is **exactly** equal (via
`float.hex()`, not `approx`) to `tests/golden/parity.json`. It skips cleanly when the artifacts are
absent, so a fresh clone never fails it.

Every result directory also carries a `run.json` recording policy, mode, suite, axis, expected
episode count, graph dir, `graph_hash`, head mtime, git commit and the full mode env — so any number
can be traced back to the exact code and artifacts that produced it.

## GPU etiquette on this box

GPUs 0 and 1 belong to another user. Use `--gpus 2,3` only.