onf-c2/docs/technical/05-artifacts.md

# 05. Artifacts

**Reads:** every `.npz` in this repo
**Writes:** nothing — reference chapter
**Code:** `src/onf/graph/core/schema.py` (filenames), `src/onf/graph/core/geometry.py::graph_hash`,
`src/onf/graph/run/params.py`, `src/onf/config.py::Paths`
**Stage:** BUILD and RUNTIME
**Read after:** [`03-action-chunk-blend.md`](03-action-chunk-blend.md)

Every artifact the repo produces or consumes, who writes it, who reads it, and the one check that
stops the worst failure in the subsystem.

---

## 5.1 The table

BUILD = an offline build/train script touches it. RUNTIME = an eval or deploy process reads it while
an episode is running. The shipped runtime regime is **sentinel** (`onf.sentinel`).

| artifact | produced by | read by | build / runtime | regime |
|---|---|---|---|---|
| `q_flow.npz` | `onf.field.build` (`build_qflow`, `build_suite`) | `onf.field.train.DemoCloud.load` | BUILD only | indirectly via `onf_head.npz` |
| `q_home.npz` | `onf.field.build` (`_save_home`, from both entry points) | *nothing* | dead — written, never loaded | neither |
| `onf_head.npz` | `onf.field.train.train_onf` (from `q_flow.npz`) | `onf.field.field.ONFField.load`, reached via `onf.graph.core.geometry.load_cleanliness_field` | RUNTIME | `field_dir` is threaded through `sentinel/sentinel.py` into `GraphRetriever.load` |
| `q_manifold.npz` | `onf.field.build.SuiteBuilder.build_suite` (pre-grasp slice) | *nothing* | dead — written, never loaded | neither |
| `q_manifold_full.npz` | `onf.field.build.SuiteBuilder.build_suite` (full trajectories) | *nothing* | dead — written, never loaded | neither |
| `g_nodes.npz` | `onf.graph.build.from_demos` (`NodeTable.save`, name from `schema.NODES_NPZ`) | `GraphRetriever.load` (`NodeTable.load`), `onf.graph.train.loop`, `scripts/build_sentinel_artifacts.py` | RUNTIME | every retriever/tracker load reads it |
| `g_edges.npz` | `onf.graph.build.from_demos` (`EdgeSet.save`, `schema.EDGES_NPZ`) | same as `g_nodes.npz` | RUNTIME | as above |
| `g_head.npz` | `onf.graph.train.loop.save_checkpoint` (`schema.HEAD_NPZ`), `graph_hash`-stamped | `GraphRetriever.load`, `scripts/build_sentinel_artifacts.py`, `scripts/capture_golden.py`; `onf.graph.cli` checks only that it exists | RUNTIME | the trained GNN retrieval head |
| `g_track.npz` | `scripts/build_sentinel_artifacts.py` via `onf.graph.run.params.TrackParams.save` (`schema.TRACK_NPZ`), `graph_hash`-stamped | `onf.graph.run.track.GraphTracker.load` → `TrackParams.load` | RUNTIME | sentinel |
| `sentinel_artifacts.json` | `scripts/build_sentinel_artifacts.py` | nothing in code | BUILD-time sidecar | sentinel (documents how `g_track.npz` was fit) |
| `action_scale.json` | `scripts/fit_action_scale.py` | `onf.blend.ee_track.ActionScale.from_json`, reached from `Sentinel._action_scale` | RUNTIME | blend — metres/radians per action unit. **Mandatory** for any blend mode; absence raises |
| `a_pi_raw.npz` | `scripts/cache_policy_actions.py` | `onf.blend.target.PolicyActionCache` | BUILD only | the frozen policy's own chunks, needed to train the blend weight |
| `g_alpha.npz` | `onf.graph.train.loop` under `--objective chunk`, via `onf.blend.alpha.AlphaNet.save` | `AlphaNet.load`, reached from `Sentinel._alpha_net` | RUNTIME | blend — only under `SN_BLEND_LEARNED`; absence raises rather than falling back to a fixed weight |

`g_track.npz` holds the fitted belief-filter transition kernel (`pi` / `beta` / `leak`) and the basin
geometry (`basin_r` / `basin_h`), all `f8`. It is fit by `onf.graph.build.tracker_fit`
(`build_likelihood_sequences`, `fit_transition_kernel`) from the graph alone — no simulator, no
success-rate outcome. Files written before the e-process null banks were retired still carry those
keys; the loader ignores them.

**`g_track.npz` is the one optional artifact.** `GraphTracker.load` checks whether the file exists; if
it does not, the tracker still loads on documented unfit defaults (uniform advance kernel) rather
than crashing. Every other artifact in the table is mandatory for its regime. A
`g_track.npz` that *is* present but stale is refused — §5.2.

### Where they live

| group | directory | resolver | override |
|---|---|---|---|
| field: `q_flow`, `q_home`, `onf_head`, `q_manifold*` | `data/fwm/<suite>/` | `Paths.fwm(suite)` | `QNDF_DIR` |
| graph: `g_nodes`, `g_edges`, `g_head`, `g_track`, `sentinel_artifacts.json` | `outputs/<suite>/latest/artifacts/` | `Paths.graph(suite)` | `GR_GRAPH_DIR` |
| blend: `action_scale.json`, `a_pi_raw.npz`, `g_alpha.npz` | beside the graph artifacts | `Paths.graph(suite)` | `GR_GRAPH_DIR`; `SN_ACTION_SCALE` for the calibration alone |

The `object` suite's field lives at the `fwm` **root**, not at `fwm/object` — the mapping is read
from `configs/suites.yaml`, not assumed. `latest` is a symlink repointed by
`onf.graph.report.StageLogger` at every new build.

---

## 5.2 `graph_hash` — the most important check in the subsystem

`src/onf/graph/core/geometry.py::graph_hash(nodes, edges)` is a SHA-256 over
`V`, `E`, `dim` and then, in order:

| hashed | fields |
|---|---|
| node | `q`, `qdot`, `owner`, `task_id`, `stage`, `t_idx`, `phase` |
| edge | `src`, `dst`, `rel`, `indptr` |

`w` and `log_idf` are excluded: they are pure functions of the hashed columns, so hashing them adds
nothing.

### Why this check, and not any other

**Node ids are positional.** A node id means "row 14,203 of *this* `g_nodes.npz`" and nothing more.
Two builder runs over the same demos with one parameter changed — a different `COARSEN`, one extra
demo, a re-sorted HDF5 directory — renumber every node.

Now load a head trained on graph A against graph B:

- the head's parameter tensors have **no dimension equal to `|V|`** (the inductivity guarantee,
  [chapter 01 §1.6](01-data-and-graph.md#16-why-task_id-is-not-a-feature)), so there is **no shape
  mismatch**;
- the forward pass runs to completion over B's nodes and returns a well-formed `[V]` logit vector;
- the softmax is peaked, the abstain logit is calibrated-looking, the readout returns a joint config
  inside the workspace;
- no NaN, no exception, no warning.

The output is **confident, correctly shaped, and points at the wrong demonstration strands**. Its
end-effector segment then becomes a tracking chunk that is mixed straight into the actions a real
arm executes ([chapter 03](03-action-chunk-blend.md)) — and row 0 of that chunk is a pose error, so
the further wrong the target is, the harder the blend drives towards it. Nothing downstream is
capable of noticing: every check in the pipeline is a shape check, a finiteness check or a
reachability mask, and a foreign head passes all three.

The same argument applies with more force to `g_track.npz`. The advance operators `S_a` are built by
raw-frame arithmetic over node ids (`NodeTable.node_at_raw`), so a foreign kernel advances the belief
along the wrong strands *and* projects onto basin geometry fitted on a different graph. The target
and the basin no-op decision are then both wrong, consistently and silently.

So both files carry a `graph_hash` key, and both loaders refuse on mismatch:

| loader | file | on mismatch |
|---|---|---|
| `GraphRetriever.load` | `g_head.npz` | `ValueError`, before any network weight is touched |
| `TrackParams.load` | `g_track.npz` | `ValueError`, before the kernel is built |

A file predating the stamp reports `"<none: head predates graph_hash stamping>"` and is refused on
the same path — absence is treated as mismatch, not as permission.

`schema.NpzSchema.hash_stamped` is the declaration: `HEAD` and `TRACK` are `True`, `NODES` and
`EDGES` are `False` (they *are* the hash input, so there is nothing to check them against).

**The three blend artifacts carry no stamp.** `action_scale.json` is a property of the robot and the
controller rather than of the graph, and `g_alpha.npz` / `a_pi_raw.npz` predate any decision about
stamping them. `AlphaNet.load` therefore accepts a head trained against a different graph, and its
failure mode is the mild version of the one above — the weight is wrong, not the target. The
calibration is the one worth watching: a wrong `action_scale.json` scales every blended action, and
the only guard is that a missing file raises rather than defaulting.

---

## 5.3 `sentinel_artifacts.json` — what it certifies, and what it does not

Written next to `g_track.npz` by `scripts/build_sentinel_artifacts.py`. Nothing in the code reads it
back; it is a provenance record so a later run can be checked against exactly what an earlier
`g_track.npz` was built from.

**It records:**

| key | content |
|---|---|
| `graph_hash`, `head_npz`, `graph_dir` | which graph and which head the fit ran against |
| `n_nodes`, `n_edges`, `n_demos`, `n_tasks`, `device` | graph shape |
| `split` | the FIT / HELD-OUT owner partition rule, `held_out_stride`, `fit_clean_stride`, both set sizes, `disjoint_fit_vs_held_out` |
| `fit_sequences` | counts per kind (clean / drift / cross_strand) and `seed` |
| `fit_transition_kernel` | `pi`, `beta`, `leak`, `beta_is_zero`, `log_evidence`, the uniform-`pi` baseline and the delta, `n_checks`, `n_sweeps`, elapsed |
| basin summary | `basin_r` / `basin_h` ranges, backfilled-cell count, spacing |

**The split is deterministic.** `owner % HELD_OUT_STRIDE == 0` is held out; everything else is a FIT
owner. A stride partition, no RNG, exactly reproducible. With owners assigned in contiguous per-task
blocks — as this repo's builder does — the stride spreads held-out demos evenly across tasks instead
of clumping them into one.

**What it does not certify.** `split.head_training_disjointness` is written verbatim as
`"not verified (no recorded head train/held-out split in this repo's graph-build pipeline)"`. The
FIT / HELD-OUT disjointness is a guarantee about the kernel fit *relative to the probe set*. It says nothing about either relative to the demos that trained `g_head.npz`, because the
graph-build pipeline records no such split for the GNN. The field is spelled out in the JSON
specifically so nobody reads a stronger claim into it.

`beta_is_zero` is written unconditionally and warned about on the console. A fitted `beta == 0.0`
would contradict `schema.py`'s "sibling mixing is mandatory"; the script reports it verbatim rather
than papering over it.

---

## 5.4 `q_manifold.npz` / `q_manifold_full.npz` are dead

Both are written by `SuiteBuilder.build_suite` (`src/onf/field/build.py`, the `manifold_full.save` /
`manifold_pre.save` pair). Neither has a reader.

**Re-verified against today's tree.** Every `np.load` / `np.savez*` call site in `src/`, `tests/`,
`scripts/` and `evals/` was enumerated, and every textual occurrence of `manifold` was checked. The
only surviving references to either filename are docstrings:

| file | mention |
|---|---|
| `src/onf/graph/core/nodes.py` (×2) | contrasts the `q_manifold_full.npz` phase convention (`/T`) against `NodeTable`'s (`/(T−1)`) — the `phase_conv` stamp check |
| `src/onf/graph/train/data.py` | cites a procedure the deleted `build_qmanifold_full` used |
| `src/onf/field/build.py` | the builder's own docstrings |

None of them loads a file. The standalone `build_qmanifold_full` CLI function no longer exists;
`build_suite` still writes both files inline. They are not deleted from disk here because `data/` is
a shared symlink (§5.6), and `build_suite` is not changed to stop writing them because that is a
behaviour change outside this documentation pass.

---

## 5.5 How the pieces connect

```
raw demos ──field.build──▶ q_flow.npz ──field.train──▶ onf_head.npz ──ONFField.f_value──┐
                        └▶ q_home.npz ─────────────────────────────────────────┐        │
                                                                                │        │
raw demos ──graph.build.from_demos──▶ g_nodes.npz, g_edges.npz                 │        │
                                            │                                   │        │
                                            └──graph.train.loop──▶ g_head.npz   │        │
                                                                   │            │        │
                                                                   ▼            │        ▼
                                              onf.graph.run.retrieve.GraphRetriever
                                              (cleanliness() pools the query window,
                                               weighted by f_value)
                                                                   │            │
                                                                                ▼
                                                                   SENTINEL (onf.sentinel)
                                                        Sentinel._build_tracker passes field_dir to
                                                        GraphTracker.load, which additionally reads
                                                        g_track.npz — built offline by
                                                        scripts/build_sentinel_artifacts.py from
                                                        g_nodes/g_edges/g_head, no simulator, no SR
                                                                   │
                                                                   ▼
                                                        BLEND (onf.blend), reading
                                                        action_scale.json and, under
                                                        SN_BLEND_LEARNED, g_alpha.npz —
                                                        trained from a_pi_raw.npz + the graph
```

`q_manifold.npz` and `q_manifold_full.npz` are off this chain entirely.

---

## 5.6 `outputs/` and `data/` are shared symlinks

At this checkout's root:

```
data     -> ../vla-bottleneck/data
outputs  -> /srv/data/VR-SmallVLA/onf/outputs
results  -> ../vla-bottleneck/results
```

All three point into trees shared with sibling worktrees and checkouts. A rebuild started from
another branch can overwrite `q_flow.npz`, `onf_head.npz`, `g_nodes.npz`, `g_edges.npz`,
`g_head.npz` or `g_track.npz` underneath an eval that is reading them from *this* checkout.

The hash stamp covers only half of that risk:

| overwrite | caught? |
|---|---|
| rebuilt `(g_nodes, g_edges)` with a new head | **yes** — `graph_hash` mismatch, hard refusal |
| same graph, refit `g_head.npz` / `g_track.npz` with different values | **no** — loads silently, results move, no local commit explains it |

`scripts/capture_golden.py` records a SHA-256 of the four artifact files it read into
`tests/golden/parity.json` for exactly this reason. Do not delete or rebuild artifacts from this
repo without first checking whether another checkout points at the same `data` / `outputs` target.

---

## 5.7 Other files written

Not part of the load path, listed so they are not mistaken for one:

| file | written by | read by |
|---|---|---|
| `<run>/config.json`, `manifest.json`, `metrics.json`, `log.txt` | `onf.graph.report.StageLogger` | humans; `config.json` is the run-reproduction record |
| `<root>/<suite>/latest` symlink (or `latest.txt` fallback) | `StageLogger` | `Paths.graph` resolves the **symlink** only — nothing reads `latest.txt` |
| `trace_w<wid>_ep<n>.npz` | `onf.sentinel.sentinel.Sentinel._flush`, only when `SN_GRAPH_TRACE_DIR` is set | debugging |
| `tests/golden/parity.json` | `scripts/capture_golden.py` | `tests/test_parity.py` |

---

## 5.8 Easy to get wrong

1. **Assuming a foreign head or kernel will fail loudly.** It will not. Shapes match, values are
   finite, the answer is confidently wrong. `graph_hash` is the only thing between that and a moving
   arm — §5.2.
2. **Treating a missing `graph_hash` key as "old file, probably fine".** Both loaders map absence to
   mismatch and refuse. That is deliberate.
3. **Rebuilding artifacts without checking the symlink target.** A same-shaped refit passes the hash
   check and silently changes results — §5.6.
4. **Reading `sentinel_artifacts.json` as a held-out guarantee against the head.** It is not, and it
   says so in its own `head_training_disjointness` field — §5.3.
5. **Deleting `q_manifold*.npz` from `data/`.** Dead in *this* repo's code; `data/` is shared with
   others that may still want them.
6. **Expecting a missing `g_track.npz` to raise.** `GraphTracker.load` falls back to unfit defaults
   and runs — a sentinel whose kernel is uniform, which is not what
   the results table was measured with.

---

## 5.9 Constants

| constant | value | CHOSEN / DERIVED | what breaks |
|---|---|---|---|
| `NODES_NPZ` | `g_nodes.npz` | CHOSEN | filename contract; every loader resolves through it |
| `EDGES_NPZ` | `g_edges.npz` | CHOSEN | as above |
| `HEAD_NPZ` | `g_head.npz` | CHOSEN | default `head_npz=` argument of both `load` classmethods |
| `TRACK_NPZ` | `g_track.npz` | CHOSEN | `_resolve_track_npz`'s directory-to-file expansion |
| `ArtifactSchemas.HEAD.hash_stamped` | `True` | CHOSEN | disabling it re-opens the foreign-head failure |
| `ArtifactSchemas.TRACK.hash_stamped` | `True` | CHOSEN | disabling it re-opens the foreign-kernel failure |
| `_HASH_NODE_FIELDS` | `q, qdot, owner, task_id, stage, t_idx, phase` | CHOSEN | dropping a field lets two distinguishable graphs collide |
| `_HASH_EDGE_FIELDS` | `src, dst, rel, indptr` | CHOSEN | as above |
| `HELD_OUT_STRIDE` | 5 | CHOSEN — 100/500 on `long` | shrinks the kernel fit set or the probe set |
| `FIT_CLEAN_STRIDE` | 3 | CHOSEN — wall-clock bound only | fit-set size, not composition |
| `N_DRIFT_SEQ` / `N_CROSS_STRAND_SEQ` | 80 / 80 | CHOSEN | the regimes that make `beta` identifiable at all |
| `DRIFT_SEQ_SEED` | 0 | CHOSEN | reproducibility of the sampled sequences |

---

**Next:** [`06-config-reference.md`](06-config-reference.md) — every constant and environment
variable, with its default and its provenance.