onf-c2/docs/technical/01-data-and-graph.md

01. Demonstrations to graph

Reads: LIBERO demo HDF5 Writes: g_nodes.npz, g_edges.npz Code: src/onf/graph/build/from_demos.py, src/onf/graph/core/{nodes,edges,schema}.py Stage: BUILD (offline, CPU) Read after: ../concepts.md

$PY -m onf.graph build --suite long        # -> g_nodes.npz + g_edges.npz

---

1.1 Input: LIBERO HDF5

src/onf/graph/build/from_demos.py

SUITE_HDF5_DIRS = {
    "object": "libero_object", "spatial": "libero_spatial",
    "goal":   "libero_goal",   "long":    "libero_10",   # NOT libero_long
}

Suite long: 10 task files × 50 demos = 500 demos. Each demo is a trajectory of T raw frames.

Extracted per demo:

field	shape	dtype	source
q	[T, 7]	f32	joint positions (Panda, 7-DoF)
qdot	[T, 7]	f32	finite difference of q
grip	[T]	u1	mean(|obs/gripper_states|) < 0.035 → 1
stage	[T]	i2	cumulative count of gripper releases
task_id	scalar	i2	sort order of the HDF5 file

Gripper polarity: grip = 1 means CLOSED. This is the opposite of the intuitive reading and the opposite of the raw gripper_states magnitude it is derived from. measured_grip() prefers the measured channel (obs/gripper_states) and falls back to the command channel (actions[:, 6] > 0.5) only when the measured one is absent. Any code deriving a gripper flag for a query must go through grip_flag() (GRIP_OPEN_THR = 0.035) so the polarity agrees with NodeTable.grip.

---

1.2 Coarsening: raw frames to nodes

src/onf/graph/core/nodes.py::coarsen_segments

Merge COARSEN = 5 consecutive raw frames into one node, but cut a node short at every gripper state change, so that each node holds exactly one gripper state.

Worked example

Demo with T = 263. grip flips at frames 97 and 210.

frame:  0..4  5..9  ...  90..94  95,96 | 97..101  102..106  ... | 210..214  ...
node:     0     1          18      19  |    20       21          |     n     ...
                                  ^^^^^^ only 2 frames — cut short by the flip at 97

Node 19 has n_members = 2. Nodes 0–18 have n_members = 5.

Mechanism

Vectorised; no Python loop over frames.

changed[0] = True
changed[i] = grip[i] != grip[i - 1]
last_reset = np.maximum.accumulate(np.where(changed, idx, -1))
pos_in_run = idx - last_reset
is_new_node = (pos_in_run % coarsen) == 0
node_id = np.cumsum(is_new_node) - 1

Result on long

28,476 nodes / 500 demos / 10 tasks. ~57 nodes per demo, ~4.8 raw frames per node — below COARSEN because of the gripper-induced cuts.

---

1.3 NodeTable on disk (g_nodes.npz)

For long: V = 28,476, D = 7, n_demos = 500, Nraw = 138,090.

Per-node arrays, length V

name	shape	dtype	value	note
q	[V, 7]	f32	q_raw[start]	the first frame of the node, not the mean
qdot	[V, 7]	f32	q_raw[end] − q_raw[start]	net displacement, not a per-step delta
grip	[V]	u1	1 = closed	see §1.1
stage	[V]	i2	cumulative releases	diagnostic only; nothing reads it
t_idx	[V]	i4	node index within its demo	resets to 0 at each demo
t_frac	[V]	f4	t_idx / (n_nodes_in_demo − 1)	node-space coordinate, ∈ [0,1]
t_raw	[V]	i4	global raw offset of the node's first frame
n_members	[V]	i2	raw frames merged, 1..5
owner	[V]	i4	demo id	the strand
task_id	[V]	i2	task id	the lane
phase	[V]	f4	arange(T)/(T−1) at the first frame	raw-space coordinate, ∈ [0,1]

CSR pointers, length n_demos + 1 = 501

name	shape	dtype	meaning
demo_ptr	[501]	i4	demo_ptr[i] : demo_ptr[i+1] is the node range of demo i
raw_ptr	[501]	i4	the same, in raw-frame indices

Full raw trajectories, retained

name	shape	dtype	why kept
q_raw	[~138k, 7]	f32	so SEG_K (segment length) stays a runtime choice
qdot_raw	[~138k, 7]	f32	same
ee_raw	[~138k, 6]	f32	concat(obs/ee_pos, obs/ee_ori) as recorded, i.e. world frame — the orientation half is robosuite's unnormalized axis-angle (max norm 4.87 rad on long, no shortest-arc wrap). Not what the blend reads: the derived NodeTable.ee_base FKs q_raw into the robot base frame instead, because the world position of the base changes per scene (see 03-action-chunk-blend.md). This column stays as the ground truth that FK is checked against
a_raw	[~138k, 7]	f32	the demo's own action commands, verbatim

Easy to get wrong

Three mistakes this table invites. The first and third have a check; its name is given so you can find out what you broke. The second has none — it is silent by construction.

1. qdot is a net displacement, not a mean per-step delta. check_coarsening() asserts qdot == q_raw[last] − q_raw[first]. Writing the mean instead shrinks every node's apparent motion by ≈5×. Nothing raises — the retrieval head just gets quietly worse, and the seeding velocity term (§2.3 in chapter 02) silently loses most of its discriminative power.

2. t_frac is not phase.

	space	used for
t_frac	node index / node count	network input, via psi(t_frac)
phase	raw frame / raw frame count	binning, masking, comparing demos of unequal length

They differ because coarsening is not uniform: a node cut short by a gripper flip covers 2 raw frames while its neighbours cover 5. Substituting one for the other makes the entry-stratum mask (phase ≤ 0.05) select a different node set, with no error raised.

3. phase_conv is stamped into the file and re-checked on load. The stamp is "arange(T)/(T-1) full". Other artifacts in this repo use different conventions — q_flow.npz truncates at grasp, so its maximum phase is 0.436 on long, and the legacy q_manifold_full.npz divided by T instead of T−1. Loading a file whose stamp disagrees raises immediately rather than silently mixing two coordinate systems; check_phase_convention() catches the same thing from the values, by requiring the table-wide phase to reach both endpoints.

---

1.4 EdgeSet on disk (g_edges.npz)

R = 12 relations. rel is an i1 index into this table; the order is load-bearing — it is the index the relation embedding table is looked up with.

rel index	name	meaning	fan-out
0–4	next1, next2, next4, next8, next16	forward along one demo, dilated	1 each
5–9	prev1 … prev16	backward along one demo, dilated	1 each
10	sibling	kNN: different demo, same task, same gripper state	K_SIBLING = 8
11	align	kNN: different task, same phase bin, same gripper state	K_ALIGN = 4

sibling and align are symmetrised and deduplicated after the kNN, so k is the fan-out before symmetrisation; on long the realised mean degree is 11.0 and 7.0.

There is deliberately no stage relation — see §1.5.

Why dilation

Message-passing depth L = 3 with next1 only reaches 3 nodes ≈ 15 raw frames. A pre-grasp approach spans ~80 raw frames. With dilations up to 16:

reach = layers × max(dilations) × coarsen = 3 × 16 × 5 = 240 raw frames

That sits just under the p50 demo length of long (263 frames). Going deeper does not help: from a 256-node seed set, 6 layers touch 100% of the graph, which carries no information.

Storage

CSR sorted by dst, with indptr, so index_add_ can coalesce collisions during message passing. Also stored: log_idf [V] f4, an additive prior on the output logit computed from sibling reachability — log(1 + reached / reachable) strands in the node's (task, grip) pool (GFM-RAG eq. 15–16).

---

1.5 Why there is no stage relation

An earlier version derived a per-frame stage label from gripper-release events, used it as a hard constraint on align, and added a dedicated cross-demo handoff relation. All of it was removed.

• The label is unfixable. One misread teleop sample shifts every later frame's label, and 58 of 500 object demos contain a genuine mid-carry re-grip.
• The constraint changed nothing. 97.5% of align candidates already agreed with it.
• The handoff relation was redundant with what next^d and sibling already reach.
• The architectural reason, which is the real one. The network's relation gate is query-dependent, so it can learn when to trust an align edge. A hand-derived stage mask does by heuristic exactly what that gate exists to learn. See chapter 02 §2.2.

stage remains on the node table as an inspection-only diagnostic. Nothing in edge construction, training or retrieval reads it.

---

1.6 Why task_id is not a feature

The network feature vector is [q(7), qdot(7), grip(1), psi(t_frac)(8)] and never contains task_id or stage.

Adding task_id would give the input layer a width that depends on the suite, destroying inductivity — one head trained on one suite could no longer run on a graph built from another. Task membership enters through the relations (sibling is within-task, align is cross-task) instead.

tests/test_gnn.py::test_inductive_state_dict_has_no_v_sized_dim holds this: no saved parameter tensor has a dimension equal to |V|. Its two siblings, test_inductive_param_count_independent_of_graph_size and test_inductive_same_state_dict_runs_on_both_graph_sizes, hold the rest: the parameter count does not move with graph size, and one state dict runs on two differently-sized graphs.

---

1.7 Constants

constant	value	CHOSEN / DERIVED	why
COARSEN	5 raw frames	CHOSEN	splits forced at gripper changes
DILATIONS	(1,2,4,8,16)	DERIVED	3 × 16 × 5 = 240 raw frames ≈ p50 demo length
K_SIBLING	8	CHOSEN
K_ALIGN	4	CHOSEN
NBINS_ALIGN	20	CHOSEN	one bin = 0.05 phase = GR_ENTRY_BAND
KERNEL_BW	0.15 rad	CHOSEN	joint-space kernel bandwidth for edge weights and seeding
GRIP_OPEN_THR	0.035	CHOSEN

---

Next: 02-retrieval-head.md — how a query window becomes a distribution over these nodes.