onf-c2/docs/technical/01-data-and-graph.md

# 01. Demonstrations to graph

**Reads:** LIBERO demo HDF5
**Writes:** `g_nodes.npz`, `g_edges.npz`
**Code:** `src/onf/graph/build/from_demos.py`, `src/onf/graph/core/{nodes,edges,schema}.py`
**Stage:** BUILD (offline, CPU)
**Read after:** [`../concepts.md`](../concepts.md)

```bash
$PY -m onf.graph build --suite long        # -> g_nodes.npz + g_edges.npz
```

---

## 1.1 Input: LIBERO HDF5

`src/onf/graph/build/from_demos.py`

```python
SUITE_HDF5_DIRS = {
    "object": "libero_object", "spatial": "libero_spatial",
    "goal":   "libero_goal",   "long":    "libero_10",   # NOT libero_long
}
```

Suite `long`: 10 task files × 50 demos = 500 demos. Each demo is a trajectory of `T` raw frames.

Extracted per demo:

| field | shape | dtype | source |
|---|---|---|---|
| `q` | `[T, 7]` | f32 | joint positions (Panda, 7-DoF) |
| `qdot` | `[T, 7]` | f32 | finite difference of `q` |
| `grip` | `[T]` | u1 | `mean(\|obs/gripper_states\|) < 0.035` → `1` |
| `stage` | `[T]` | i2 | cumulative count of gripper releases |
| `task_id` | scalar | i2 | sort order of the HDF5 file |

**Gripper polarity: `grip = 1` means CLOSED.** This is the opposite of the intuitive reading and the
opposite of the raw `gripper_states` magnitude it is derived from. `measured_grip()` prefers the
measured channel (`obs/gripper_states`) and falls back to the command channel
(`actions[:, 6] > 0.5`) only when the measured one is absent. Any code deriving a gripper flag for a
query must go through `grip_flag()` (`GRIP_OPEN_THR = 0.035`) so the polarity agrees with
`NodeTable.grip`.

---

## 1.2 Coarsening: raw frames to nodes

`src/onf/graph/core/nodes.py::coarsen_segments`

Merge `COARSEN = 5` consecutive raw frames into one node, but cut a node short at every gripper state
change, so that each node holds exactly one gripper state.

### Worked example

Demo with `T = 263`. `grip` flips at frames 97 and 210.

```
frame:  0..4  5..9  ...  90..94  95,96 | 97..101  102..106  ... | 210..214  ...
node:     0     1          18      19  |    20       21          |     n     ...
                                  ^^^^^^ only 2 frames — cut short by the flip at 97
```

Node 19 has `n_members = 2`. Nodes 0–18 have `n_members = 5`.

### Mechanism

Vectorised; no Python loop over frames.

```python
changed[0] = True
changed[i] = grip[i] != grip[i - 1]
last_reset = np.maximum.accumulate(np.where(changed, idx, -1))
pos_in_run = idx - last_reset
is_new_node = (pos_in_run % coarsen) == 0
node_id = np.cumsum(is_new_node) - 1
```

### Result on `long`

28,476 nodes / 500 demos / 10 tasks. ~57 nodes per demo, ~4.8 raw frames per node — below `COARSEN`
because of the gripper-induced cuts.

---

## 1.3 `NodeTable` on disk (`g_nodes.npz`)

For `long`: `V = 28,476`, `D = 7`, `n_demos = 500`, `Nraw = 138,090`.

### Per-node arrays, length `V`

| name | shape | dtype | value | note |
|---|---|---|---|---|
| `q` | `[V, 7]` | f32 | `q_raw[start]` | the **first** frame of the node, not the mean |
| `qdot` | `[V, 7]` | f32 | `q_raw[end] − q_raw[start]` | **net displacement**, not a per-step delta |
| `grip` | `[V]` | u1 | `1 = closed` | see §1.1 |
| `stage` | `[V]` | i2 | cumulative releases | diagnostic only; nothing reads it |
| `t_idx` | `[V]` | i4 | node index within its demo | resets to 0 at each demo |
| `t_frac` | `[V]` | f4 | `t_idx / (n_nodes_in_demo − 1)` | **node**-space coordinate, ∈ [0,1] |
| `t_raw` | `[V]` | i4 | global raw offset of the node's first frame | |
| `n_members` | `[V]` | i2 | raw frames merged, 1..5 | |
| `owner` | `[V]` | i4 | demo id | the **strand** |
| `task_id` | `[V]` | i2 | task id | the **lane** |
| `phase` | `[V]` | f4 | `arange(T)/(T−1)` at the first frame | **raw**-space coordinate, ∈ [0,1] |

### CSR pointers, length `n_demos + 1 = 501`

| name | shape | dtype | meaning |
|---|---|---|---|
| `demo_ptr` | `[501]` | i4 | `demo_ptr[i] : demo_ptr[i+1]` is the node range of demo `i` |
| `raw_ptr` | `[501]` | i4 | the same, in raw-frame indices |

### Full raw trajectories, retained

| name | shape | dtype | why kept |
|---|---|---|---|
| `q_raw` | `[~138k, 7]` | f32 | so `SEG_K` (segment length) stays a runtime choice |
| `qdot_raw` | `[~138k, 7]` | f32 | same |
| `ee_raw` | `[~138k, 6]` | f32 | `concat(obs/ee_pos, obs/ee_ori)` as recorded, i.e. **world frame** — the orientation half is robosuite's **unnormalized** axis-angle (max norm 4.87 rad on `long`, no shortest-arc wrap). Not what the blend reads: the derived `NodeTable.ee_base` FKs `q_raw` into the **robot base** frame instead, because the world position of the base changes per scene (see [`03-action-chunk-blend.md`](03-action-chunk-blend.md)). This column stays as the ground truth that FK is checked against |
| `a_raw` | `[~138k, 7]` | f32 | the demo's own action commands, verbatim |

### Easy to get wrong

Three mistakes this table invites. The first and third have a check; its name is given so you can
find out what you broke. The second has none — it is silent by construction.

**1. `qdot` is a net displacement, not a mean per-step delta.**
`check_coarsening()` asserts `qdot == q_raw[last] − q_raw[first]`. Writing the mean instead shrinks
every node's apparent motion by ≈5×. Nothing raises — the retrieval head just gets quietly worse, and
the seeding velocity term (§2.3 in [chapter 02](02-retrieval-head.md)) silently loses most of its
discriminative power.

**2. `t_frac` is not `phase`.**

| | space | used for |
|---|---|---|
| `t_frac` | node index / node count | network input, via `psi(t_frac)` |
| `phase` | raw frame / raw frame count | binning, masking, comparing demos of unequal length |

They differ because coarsening is not uniform: a node cut short by a gripper flip covers 2 raw frames
while its neighbours cover 5. Substituting one for the other makes the entry-stratum mask
(`phase ≤ 0.05`) select a different node set, with no error raised.

**3. `phase_conv` is stamped into the file and re-checked on load.**
The stamp is `"arange(T)/(T-1) full"`. Other artifacts in this repo use different conventions —
`q_flow.npz` truncates at grasp, so its maximum `phase` is 0.436 on `long`, and the legacy
`q_manifold_full.npz` divided by `T` instead of `T−1`. Loading a file whose stamp disagrees raises
immediately rather than silently mixing two coordinate systems; `check_phase_convention()` catches
the same thing from the values, by requiring the table-wide `phase` to reach both endpoints.

---

## 1.4 `EdgeSet` on disk (`g_edges.npz`)

`R = 12` relations. `rel` is an `i1` index into this table; **the order is load-bearing** — it is the
index the relation embedding table is looked up with.

| rel index | name | meaning | fan-out |
|---|---|---|---|
| 0–4 | `next1, next2, next4, next8, next16` | forward along one demo, dilated | 1 each |
| 5–9 | `prev1 … prev16` | backward along one demo, dilated | 1 each |
| 10 | `sibling` | kNN: different demo, same task, same gripper state | `K_SIBLING = 8` |
| 11 | `align` | kNN: different task, same phase bin, same gripper state | `K_ALIGN = 4` |

`sibling` and `align` are symmetrised and deduplicated after the kNN, so `k` is the fan-out before
symmetrisation; on `long` the realised mean degree is 11.0 and 7.0.

There is deliberately no `stage` relation — see §1.5.

### Why dilation

Message-passing depth `L = 3` with `next1` only reaches 3 nodes ≈ 15 raw frames. A pre-grasp approach
spans ~80 raw frames. With dilations up to 16:

```
reach = layers × max(dilations) × coarsen = 3 × 16 × 5 = 240 raw frames
```

That sits just under the p50 demo length of `long` (263 frames). Going deeper does not help: from a
256-node seed set, 6 layers touch 100% of the graph, which carries no information.

### Storage

CSR sorted by `dst`, with `indptr`, so `index_add_` can coalesce collisions during message passing.
Also stored: `log_idf [V] f4`, an additive prior on the output logit computed from sibling
reachability — `log(1 + reached / reachable)` strands in the node's (task, grip) pool
(GFM-RAG eq. 15–16).

---

## 1.5 Why there is no `stage` relation

An earlier version derived a per-frame stage label from gripper-release events, used it as a hard
constraint on `align`, and added a dedicated cross-demo handoff relation. All of it was removed.

- **The label is unfixable.** One misread teleop sample shifts every later frame's label, and 58 of
  500 `object` demos contain a genuine mid-carry re-grip.
- **The constraint changed nothing.** 97.5% of `align` candidates already agreed with it.
- **The handoff relation was redundant** with what `next^d` and `sibling` already reach.
- **The architectural reason, which is the real one.** The network's relation gate is
  query-dependent, so it can learn *when* to trust an `align` edge. A hand-derived stage mask does by
  heuristic exactly what that gate exists to learn. See
  [chapter 02 §2.2](02-retrieval-head.md#22-the-relation-gate-is-recomputed-every-layer).

`stage` remains on the node table as an inspection-only diagnostic. Nothing in edge construction,
training or retrieval reads it.

---

## 1.6 Why `task_id` is not a feature

The network feature vector is `[q(7), qdot(7), grip(1), psi(t_frac)(8)]` and never contains `task_id`
or `stage`.

Adding `task_id` would give the input layer a width that depends on the suite, destroying inductivity
— one head trained on one suite could no longer run on a graph built from another. Task membership
enters through the relations (`sibling` is within-task, `align` is cross-task) instead.

`tests/test_gnn.py::test_inductive_state_dict_has_no_v_sized_dim` holds this: no saved parameter
tensor has a dimension equal to `|V|`. Its two siblings,
`test_inductive_param_count_independent_of_graph_size` and
`test_inductive_same_state_dict_runs_on_both_graph_sizes`, hold the rest: the parameter count does
not move with graph size, and one state dict runs on two differently-sized graphs.

---

## 1.7 Constants

| constant | value | CHOSEN / DERIVED | why |
|---|---|---|---|
| `COARSEN` | 5 raw frames | CHOSEN | splits forced at gripper changes |
| `DILATIONS` | (1,2,4,8,16) | DERIVED | `3 × 16 × 5 = 240` raw frames ≈ p50 demo length |
| `K_SIBLING` | 8 | CHOSEN | |
| `K_ALIGN` | 4 | CHOSEN | |
| `NBINS_ALIGN` | 20 | CHOSEN | one bin = 0.05 phase = `GR_ENTRY_BAND` |
| `KERNEL_BW` | 0.15 rad | CHOSEN | joint-space kernel bandwidth for edge weights and seeding |
| `GRIP_OPEN_THR` | 0.035 | CHOSEN | |

---

**Next:** [`02-retrieval-head.md`](02-retrieval-head.md) — how a query window becomes a distribution
over these nodes.