docs/ARCHITECTURE_latent16.md

System architecture: latent-subgoal planner and controller on RoboMME

This document describes the current system as trained and evaluated on the 16 RoboMME tasks. It reaches 510/800 = 63.8% success on the official test split with the planner in the loop and no simulator subgoal text at test time (50 episodes per task). Every number comes from the configs, checkpoints, and logs listed in the appendix. Code lives in triquang/graphvla/HAMLET-Isaac-GR00T, branch latent-subgoal.

1. Overview

Two learned parts run in a closed loop on top of a frozen vision-language backbone.

1. A planner reads the demo video, the robot's own execution so far, and the instruction. At every graph node it emits a latent subgoal Z (4 tokens of width 256) and a grounded target point in the front image.
2. A controller, the GR00T N1.6 flow-matching DiT action head, reads the current observation, Z, and the point, and decodes a 50-step joint-position chunk.

Z is the only channel through which the past reaches the controller. During training the planner learns Z with three kinds of supervision: classification of the step kind and ordinal, an entity pointer for the target, and a small sentence decoder that must reproduce the subgoal text and its coordinates from Z alone. The sentence decoder exists only at training time. At test time nothing is decoded to text and no text is re-encoded.

          demo video + execution frames (front 256x256, wrist 256x256), instruction
                                     |
                 +-------------------v--------------------+
                 | Eagle-2B VLM, frozen, LM cut at layer 16 |
                 | + 4 TCL moment tokens appended          |
                 +------+--------------------------+-------+
                        | 9x9 front patches,       | 2 x 81 image tokens, text,
                        | mean wrist, instruction  | 4 moment tokens (2048-d)
   CoTracker (frozen)   v                          |
   192 tracked  +-------------------------+        |
   points ----->| PLANNER (9.0M at test)  |        |
                |  frame stream: 4 graph  |        |
                |   blocks over 82 tokens |        |
                |   per node, event count |        |
                |  entity stream: 2 layers|        |
                |   over 192 entities     |        |
                |  latent readout -> Z    |        |
                |  heads: kind, ordinal,  |        |
                |   pointer + shift       |        |
                |  [train only] sentence  |        |
                |   decoder Z -> text+xy  |        |
                +-----+-----------+-------+        |
                      | Z (4x256) | point          |
                      v           v                v
            +--------------------------+   +----------------------------------+
            | memory tokens (6 x 256)  |   | GR00T N1.6 DiT, 32 blocks        |
            |  point Fourier token     |-->|  cross-attn: image / non-image   |
            |  4 projected Z tokens    |   |  self-attn: state + 50 actions   |
            |  fovea goal token        |   |  FFN input FiLM from memory      |
            | mark on 81 front patches |-->|                                  |
            | fovea 64x64 crop ->      |-->|  image tokens + 81 fovea tokens  |
            |  81 extra image tokens   |   +----------------+-----------------+
            +--------------------------+                    |
                                                            v
                                 joint-position chunk, execute <= 32 steps

Training runs in this order: TCL moment tokens, feature caches, fovea reader pre-training, planner, Z cache, controller. Eagle and CoTracker are never updated.

2. Task, observations, actions

Item	Value
Environment	RoboMME, Franka Panda, 10 fps, 16 tasks
Cameras	front_view and wrist_view, RGB, 256 px shortest edge, center crop fraction 0.95
Language	the task instruction string
State	8 dims: 7 joint positions and the gripper position
Action	8 dims: 7 absolute joint positions and the gripper command
Action normalisation	min-max to [-1, 1], statistics fit on the training demos (runs/tcl_k4/processor_official/statistics.json), no relative actions
Action chunk	50 action tokens in the DiT, loss on the first 40 steps, at most 32 executed
Padding	state and action padded to 128 dims inside the action head
Dataset	data/robomme16: 1600 episodes (100 per task), 768,897 frames

An episode is a demo video segment (is_demo = 1, empty for tasks without a video) followed by the scripted execution, which starts at exec_start_idx. Only the official RoboMME demonstrations are used. No data is generated.

3. Frozen backbone

The backbone is GR00T N1.6-3B with the Eagle-Block2A-2B-v2 VLM, frozen in every stage.

• The language model is truncated to select_layer = 16. The feature is the last hidden state of the truncated model, 2048-d.
• Each frame gives two views of 9 x 9 patches (162 image tokens), plus the text tokens and 4 moment tokens, 209 to 228 tokens in total.
• The 4 moment tokens are learned vectors appended after the text (TCL, Section 3.1). Eagle is causal, so they do not change the image or text features. Their outputs stay in the sequence as non-image tokens.
• The action head applies a LayerNorm (vlln, 2048) to the backbone features.

3.1 TCL moment tokens

Run runs/tcl_k4. Four learned 2048-d tokens and a projection head (Linear 2048 to 2048, SiLU, Linear 2048 to 2048; 8.4M parameters) are trained with InfoNCE (temperature 0.07): the anchor frame, an augmented view of the same frame (positive) and another frame (negative) pass through the frozen backbone, and the mean-pooled, projected, L2-normalised moment outputs are contrasted. 5000 steps, global batch 32, lr 1e-4, AdamW (weight decay 1e-5), cosine schedule with 5% warmup, 4 GPUs, bf16. Training data: the 240 training episodes of PickXtimes, VideoPlaceOrder and VideoUnmaskSwap (data/robomme3_train). The checkpoint is used unchanged for all 16 tasks, at cache time and at test time.

4. Subgoal labels

Labels are parsed per frame from the dataset column grounded_subgoal_online, the same string the simulator reports during a rollout (qgraph/memory/oracle.py, file cache/oracle_labels_16k7wc.pt). They are used only as training targets. At test time the simulator string is never read.

Field	Form
kind	first matching regex among ^pick up, ^(drop .* onto target|place |put down), onto table, ^press, ^move to, ^remain static, ^insert; otherwise none. 8 classes
ordinal	for the (\w+) time with first to fifth, otherwise 0. 6 classes
point	the first <row, col> in the 256 px image, mapped to crop coordinates in [0, 1] as (pixel - 6.4) / 243.2; -1 when the text has no point
words	lowercase [a-z]+ words of the text with the point removed, at most 20; vocabulary of 72 training words plus pad and unknown

5. Planner

Run runs/online/planner16z_s0, config configs/qgraph/planner16z.yaml, code qgraph/memory/online.py, qgraph/memory/entities.py, qgraph/memory/latent.py, qgraph/memory/attend.py.

5.1 Graph

The planner keeps a graph that grows by one node every 4 frames, on the grid f ≡ exec_start (mod 4), so demo and execution share the grid.

Frame tokens. Node n holds 82 tokens: the 81 front-camera patches of Eagle on a fixed 9 x 9 grid (the front camera does not move, so slot p is always the same place on the table) and the mean of the 81 wrist patches. The node also carries the gripper opening g_n and a phase bit e_n (0 for demo, 1 for execution).

Entity tokens. Online CoTracker follows a pool of 384 points over every front frame. The points are seeded on the first frame by farthest-point sampling over foreground pixels (summed absolute RGB difference above 60 from an empty-table image). The planner uses 192 of them (Section 5.6). At node n, entity k has its tracked position x_{n,k} in crop coordinates, a visibility bit, and two 12 x 12 RGB patches around it, the current one and the one at the first node.

Instruction. The instruction is encoded once by the frozen Eagle LM (one 2048-d vector per token, 19 to 24 tokens) and projected to 256-d.

Edges. Each layer passes messages along four edge types (multi-head attention, pre-norm, residual).

Edge	From, to	Rule
query	instruction tokens to every frame and entity token	full cross-attention
temporal	same slot (frame stream) or same entity (entity stream) at all earlier nodes	causal attention with rotary position encoding over the node index
spatial, frame	the 82 tokens of one node	full attention
spatial, entity	the 192 entities of one node	attention with bias -‖x_k - x_k'‖² / 2σ_h², σ_h learned per head, initialised from 0.03 to 0.5 of the image

The graph is causal: nothing at node n reads a later frame, so the output at node n can be computed while the robot acts. Because the instruction enters every layer before the temporal and spatial edges, the same video gives different node states under different instructions.

5.2 Frame stream

Four GraphBlocks, width 256, 8 heads, FFN 1024, dropout 0.1.

1. Input: h_{n,p} = LN(W_in Eagle_{n,p}) + P_p + MLP_g(25 g_n) + E(e_n), with P a learned embedding of the 82 slots.
2. Block: query edge, temporal edge, spatial edge, FFN.
3. Event counter after the first block. For each node and each of the 7 kinds, s_{n,k} = σ(Linear(LN(mean_p h_{n,p}))). The running count C_{n,k} = Σ_{m ≤ n} s_{m,k} is an exact prefix sum. It is encoded as a triangular one-hot over 0..8 and its embedding is added to all 82 tokens of node n.
4. Final LayerNorm.

5.3 Latent readout

LatentReadout, 4 learned query tokens of width 256.

1. Read. The 4 queries attend the 82 final frame tokens of node n (8 heads), followed by an FFN (1024). The mean of the 4 tokens is the node context used by the entity stream.
2. Entity stream (5.4) runs with that context.
3. Refine. The 4 tokens attend the 192 final entity tokens of the same node (128-d, projected to 256), then an FFN and a LayerNorm. The result is Z_n ∈ R^{4 x 256}.

The kind and ordinal heads are linear layers on mean(Z_n). Every output of the planner, and every gradient from its losses, therefore passes through Z.

5.4 Entity stream and pointer

EntityStream, 2 layers, width 128, 4 heads, FFN 512, dropout 0.1.

1. Input: a_{n,k} = MLP_pix([patch_{n,k}, patch_{1,k}]) + MLP_mot([10 (x_{n,k} - x_{n-1,k}), 10 (x_{n,k} - x_{1,k}), vis_{n,k}]) + c_n, where c_n = Linear(LN([context_n, C_n])) carries the latent read of 5.3 and the 7 x 9 count code.
2. Layer: query edge, FiLM by c_n (zero-initialised), temporal edge, spatial edge, FFN.
3. Heads: per entity a pointer logit and a shift 0.06 tanh(.) in crop units (at most 14.6 px); per node a null logit.

Point output: x_{n,k*} + shift_{n,k*}, with k* the argmax over the 192 pointer logits and the null logit. If the null logit wins, the point is absent.

5.5 Sentence decoder (training only)

SentenceDecoder: a causal Transformer decoder, 2 layers, width 256, 8 heads, FFN 1024, GELU, pre-norm, dropout 0.1, learned positions. Its cross-attention reads only the 4 tokens of Z_n, so the sentence can be produced only from what Z holds.

Target sequence: the subgoal words (Section 4), then two coordinate tokens for the target point when one exists, then EOS. Coordinates are quantised into 128 bins per axis, with separate row and column token ranges. The vocabulary has 332 entries: 74 word entries, 2 x 128 coordinate bins, BOS and EOS. Input is BOS followed by the target shifted by one (teacher forcing). The loss is the token cross-entropy averaged over the sequence.

The words "left" and "right" in a subgoal would be wrong after a symmetry turn of the image (5.6), so the sentence loss is applied only to rows without a turn and without re-query.

5.6 Training

Data: the 1600 official episodes split 1280 train / 320 val (20 val per task, qgraph_split_planner16.json). Inputs are cached once (Eagle node tokens in cache/online16_s4, CoTracker tracks and patches in cache/entities16_p384_s4). Both demo and execution nodes are supervised with the label of their frame.

Losses, per node, with execution nodes weighted 2.0 and demo nodes 1.0:

Term	Weight	Definition
kind	1.0	cross-entropy over 8 classes
ordinal	1.0	cross-entropy over 6 classes
events	1.0	each labelled step (a run of nodes with one kind) holds event mass 1 for its own kind and 0 for the others: Σ_k (Σ_{n ∈ step} s_{n,k} - [k = kind(step)])², averaged over steps
pointer	1.0	-log Σ_{k ∈ P} softmax(logits)_k, with P the entities within 8 px of the label point, or the null entry when there is no point
locate	0.1	smooth-L1 (β = 1, px) between x_{n,k} + shift_{n,k} and the label point, over P
sentence	1.0	Section 5.5, on rows without a turn or re-query

The counter has no count labels. Its target comes from the step boundaries in the kind labels.

Augmentation:

Item	Value
Time warp	execution nodes dropped with p 0.1, or repeated 2 to 4 times with p 0.15; the newest node is kept
Symmetry	with p 0.5 no turn; otherwise one of the 8 rotations and flips of the square, applied to the patch grid, entity coordinates, entity patches and label point
Entity subsets	8 fixed subsets of 192 of the 384 tracked points per episode (farthest-point orders from 8 starts), one drawn per sample
Re-query	VideoPlaceOrder, p 0.5: the instruction is replaced by that of another training episode that differs only in the ordinal word. The pointer target of the execution place step becomes the entities within 8 px of the matching demo drop location

Optimisation:

Item	Value
Hardware	4 A100 80GB, DDP (NCCL); each rank keeps the token store of 4 tasks (sorted tasks [r::4]) in GPU memory
Batch	each rank draws one of its tasks per step (round robin) and 8 episodes of it: 32 episodes and 4 tasks per step; every task has the same weight
Steps	6500
Optimiser	AdamW, lr 3e-4, weight decay 0.01, grad clip 1.0, bf16 autocast
Schedule	200 warmup steps, cosine to 0
Kernels	rotary temporal attention through causal flash SDPA; graph blocks and the entity layer compiled with torch.compile(dynamic=True)
Wall time	43 min

5.7 Parameters

Module	Params (M)
node projection (2048 to 256, LN)	0.525
instruction projection (2048 to 256, LN)	0.525
slot position, gripper MLP, phase	0.088
4 graph blocks	5.273
event counter	0.019
entity stream and heads	0.965
latent readout	1.615
kind and ordinal heads	0.004
final norm	0.001
inference path	9.01
sentence decoder (training only)	2.284
unused in this configuration (node pool attention, heat head)	0.265
total in planner.pt	11.562

5.8 Validation (320 val episodes, step 6500)

Node set	Nodes	Kind acc.	Ordinal acc.	Point within 4 px	Point within 8 px
all execution nodes	23,350	0.989	0.990	0.872	0.885
decision nodes (execution nodes within 16 frames after the subgoal changes)	3,975	0.976	0.991	0.856	0.867

Point accuracy is over nodes whose label has a point. Ordinal accuracy is over the tasks that use ordinals (PickXtimes, VideoRepick, SwingXtimes).

6. Z cache for controller training

qgraph/latents.py runs the trained planner (eval mode, first entity subset, no augmentation) once over all 1600 episodes. Each frame f takes the outputs of the latest node at or before f: Z (4 x 256, stored in bf16), point, kind and ordinal. The controller reads these at its anchor frames, so it trains on the planner's own predictions rather than on labels. The run is sharded by task over 4 GPUs and takes 5 min. Output: cache/oracle_labels_16z.pt (1.6 GB).

7. Controller

GR00T N1.6 action head (Gr00tN1d6ActionHead with AlternateVLDiT), plus an adapter that builds memory tokens, a memory conditioning module, and a fovea. Config configs/qgraph/z16.yaml, code qgraph/model.py, qgraph/memory/oracle.py, qgraph/memory/modulate.py, qgraph/memory/fovea.py.

7.1 DiT flow-matching head

Item	Value
Blocks	32: 16 cross-attention (even index), 16 self-attention (odd index)
Width	32 heads x 48 = 1536
Norm, activation, dropout	AdaLayerNorm on the flow time, GELU (tanh), 0.2
Cross-attention routing	blocks with index ≡ 0 (mod 4) attend the non-image tokens (text and moment tokens); blocks with index ≡ 2 (mod 4) attend the image tokens, including the fovea tokens
Self-attention	over the state token and 50 action tokens
Flow time	t = 0.999 (1 - u), u ~ Beta(1.5, 1); 1000 buckets
Interpolant	x_t = (1 - t) ε + t a, target velocity a - ε
Sampling	4 Euler steps at t = 0, 0.25, 0.5, 0.75

7.2 Memory tokens

The memory is 6 tokens of width 256:

1. Point token: Fourier features of the point (6 bands, frequencies 2^k π, sin and cos, 24 inputs), Linear(24, 256), GELU, Linear(256, 256). A learned absent vector replaces it when the point is -1.
2. Four Z tokens: Z_n passed through LayerNorm(256) and Linear(256, 256).
3. Fovea goal token (7.4).

Tokens 1 and 2 share a final LayerNorm.

7.3 Memory conditioning

Each of the 32 DiT blocks has a MemoryModulator on the input of its feed-forward layer, which is the post-attention normalised hidden state x.

• Memory LayerNorm(256); query Linear(1536 to 256) of x; key and value Linear(256 to 256) of the memory; 4-head attention.
• FiLM: Linear(256, 256), SiLU, Linear(256, 3072), zero-initialised, giving scale and shift; x' = (1 + scale) x + shift.
• Keys and values are computed once per chunk and reused across the 4 denoising steps.

7.4 Mark and fovea

Mark. A learned 2048-d vector (initial scale 0.5) is added to the 81 front-image tokens with weight exp(-d² / (2 · 0.06²)), d the distance between the patch centre and the point. No mark when the point is absent.

Fovea. A 64 x 64 crop of the front image centred on the point.

• PixelReader: CNN with channels 32, 32, pool, 64, 64, pool, 128, 128; adaptive average pooling to 9 x 9 = 81 tokens; LayerNorm, Linear(128, 2048) and a Fourier encoding of each token's image position. The 81 tokens are appended to the image tokens.
• Goal reader: an MLP over the pooled CNN feature, the Fourier code of the crop centre, the mean memory token, and a cross-attention readout of the scene features (hidden 256, 4 heads, with a null key). It predicts a 10-d end-pose label and produces the goal token (zero-initialised output layer).
• Goal labels (cache/subgoal_end5r.pt), from demo forward kinematics: sin and cos of the full-turn yaw of the end pose, 10 x the end TCP position, sin and cos of the remaining yaw, 10 x the displacement from gripper to end, and a valid flag.
• The reader is pre-trained on fovea crops of the training split (cache/subgoal_reader5.safetensors, 24,000 steps, batch 128, lr 1e-3) and loaded before controller training.
• Fovea crops and goal labels exist for 5 tasks (VideoUnmaskSwap, PickXtimes, VideoPlaceOrder, StopCube, InsertPeg; cache/fovea5d). In training, anchors of the other 11 tasks have no fovea tokens.

7.5 Training-time regularisation

• Token drop: each front or wrist image token is dropped with p 0.6 (fovea tokens are kept).
• Prefix: a length d ~ U{0..16}; the first d action tokens enter clean at time bucket 999 and are excluded from the loss.
• Proprioception: the state token is zeroed for the whole sample with p 0.5.
• Point drop: the point is set to -1 with p 0.15, which trains the unconditional branch for guidance.
• Fovea jitter: Gaussian shift of the crop, σ 4 px, crops stored at 96 x 96.

7.6 Losses

L = L_flow + 1.0 L_kin + 0.1 L_goal

• L_flow: mean squared error between predicted and target velocity over valid action steps.
• L_kin: the clean chunk estimate â = x_t + (1 - t) v̂ is un-normalised and passed through differentiable Panda forward kinematics. The TCP position (flange plus 0.1034 m along z) is compared with that of the label chunk, squared error divided by 0.2, averaged over valid steps.
• L_goal: squared error of the goal reader output against the 10-d label, masked by the valid flag.

7.7 Guidance

v = v_uncond + 2.0 (v_cond - v_uncond). The unconditional branch replaces the point by the absent token and removes the mark. Z and the fovea crop stay. Both branches run as one doubled batch.

7.8 Parameters (checkpoint-12000)

Module	Params (M)
DiT	1091.722
action encoder	232.931
state encoder	54.608
action decoder	37.786
action position embedding	1.573
vlln	0.004
subtotal, N1.6 action head	1418.62
memory conditioning (32 modulators)	44.188
adapter (point MLP 0.072, Z projection 0.066, mark 0.002, norm, absent)	0.141
fovea (reader 0.287, projection 0.264, scene 0.529, attention 0.263, scene query 0.105, goal 0.236, goal token 0.066, position 0.051, goal output 0.003, other 0.001)	1.805
total	1464.76

8. Controller training

Item	Value
Initialisation	action head from GR00T N1.6-3B; fovea reader from its pre-training; adapter and conditioning random, FiLM output zero
Feature cache	cache/tcl_k4_16d: Eagle and moment-token features at the anchors, 2048-d
Anchors	every 8 frames of execution (phases 0 and 8 of a stride-16 grid); label chunk of 40 steps
Split	qgraph_split5.json: 1500 train, 100 val episodes; 1489 train episodes and 31,543 train anchors after 11 PatternLock episodes without anchors
Conditioning inputs	Z and point from the Z cache (Section 6) at the anchor frame
Sampling	uniform over training anchors
Hardware	4 A100 80GB, DDP; feature store resident in GPU memory
Batch	global 256 (64 per GPU)
Steps	12,000 (3.07M samples)
Optimiser	AdamW (fused), weight decay 1e-5, grad clip 1.0, bf16 autocast, TF32
Learning rates	1e-4 for the action head; 3e-4 for adapter, conditioning and fovea
Schedule	linear warmup over 600 steps, cosine to 0
Final	train loss 0.0113; val loss 0.0688 (100 val episodes); 0.415 s per step; 83 min

9. Inference

1. Before the robot moves, the demo frames are replayed through CoTracker and the planner; every 4th frame on the grid becomes a node with phase 0.
2. During execution CoTracker sees every frame. A node with phase 1 is added every 4 environment steps and at every controller query. After each new node the planner runs over the whole graph and the newest node gives (Z, point, kind, ordinal).
3. The controller decodes a chunk from noise with 4 Euler steps and guidance 2.0, and the robot executes at most 32 steps of it.
4. Watch rule: when the planner's (kind, ordinal) at a new node differs from the pair the running chunk was generated with, the chunk is cut and the controller is queried at once. A change of Z or point alone does not cut the chunk.
5. The mark and the fovea crop are built from the planner's point. Every oracle_* input that the policy could build from the simulator's subgoal text is removed before the planner's outputs are merged, so the controller sees only planner outputs.
6. Episode limit 1300 steps. Inference seed 0. One entity subset (the first 192 seeds). No voting and no task-specific logic.

Measured over the 800 test episodes, with 3 clients sharing one GPU and one policy server: 405 environment steps and 14.5 controller queries per episode (27.9 steps per query), 62.5 s per episode, 0.154 s per environment step. These times include CoTracker, planner re-runs and server contention.

10. Evaluation protocol

• Official RoboMME test split (env_split = test), all 16 tasks.
• 50 episodes per task, 800 in total, stratified by difficulty from env_metadata (subset seed 0). One run, one seed.
• 4 GPUs, one policy server per GPU (ZeroMQ), 3 environment clients per server, 2 episodes per job.
• Metric: success rate per task and its mean over tasks.

11. Results

Task	Success / 50	%
BinFill	17	34
ButtonUnmask	46	92
ButtonUnmaskSwap	30	60
InsertPeg	6	12
MoveCube	24	48
PatternLock	4	8
PickHighlight	39	78
PickXtimes	42	84
RouteStick	12	24
StopCube	21	42
SwingXtimes	43	86
VideoPlaceButton	49	98
VideoPlaceOrder	40	80
VideoRepick	45	90
VideoUnmask	49	98
VideoUnmaskSwap	43	86
Mean	510 / 800	63.8

Outcomes: 510 success, 208 failure, 82 timeout.

12. Compute

Training stages only, on NVIDIA A100-SXM4-80GB. Feature extraction passes (Eagle, CoTracker, fovea crops, Z) are not counted. Wall times come from train_log.jsonl, log.jsonl and file timestamps.

Stage	GPUs	Wall time	GPU-hours	Note
TCL moment tokens	4	about 1 h 24 min	5.6	estimated from the step rate in the log
Fovea reader pre-training	1	not timed	under 0.5	1.1 MB model, 24,000 steps; can run beside TCL
Planner	4	43 min	2.9	measured
Controller	4	84 min	5.6	measured
Total		about 3 h 31 min	about 14.1 (at most 14.6 with the fovea reader)

TCL, planner and controller run one after another on 4 GPUs, so the wall time is their sum. Without TCL, the planner and controller take 2 h 07 min and 8.5 GPU-hours. The 800-episode evaluation takes 83 min on 4 GPUs (5.6 GPU-hours).

13. Facts to state with the results

• Z on training anchors. The controller trains on Z that the planner computed for the controller's training episodes. 1280 of the 1500 controller training episodes are also planner training episodes, so Z there comes from episodes the planner has fitted. The 100 controller val episodes are all planner val episodes. Test episodes are new to both.
• Fovea coverage. Fovea crops in training are centred on the label point and exist for 5 tasks. At test time the crop is centred on the planner's point and is used whenever a point exists, for all 16 tasks. The mark uses the planner's point in training and at test.
• Sampler. Training is uniform over anchors, not over tasks. PatternLock has 1.4% of the anchors and PickXtimes 16.3%.
• Prefix. The head is trained with clean action prefixes, but inference samples every chunk from noise without a prefix.
• TCL scope. The moment tokens were trained on 3 tasks and are applied to all 16.
• Test-time inputs. The controller sees camera images, proprioception, the instruction (through Eagle), and the planner's Z and point. The simulator's subgoal text is not used at test time.

Appendix: paths and commands

Working directory triquang/graphvla/HAMLET-Isaac-GR00T, interpreter .venv/bin/python, PYTHONPATH=$PWD.

Item	Path
Planner config	configs/qgraph/planner16z.yaml
Controller config	configs/qgraph/z16.yaml
Planner run	../runs/online/planner16z_s0 (planner.pt, log.jsonl)
Z cache	../cache/oracle_labels_16z.pt
Controller run	../runs/stage_b/z16_k32_alr0.0001_mlr0.0003_s0/checkpoint-12000
Evaluation	../runs/eval_t50_z16/z16_k32_alr0.0001_mlr0.0003_s0/checkpoint-12000/summary.json
TCL run	../runs/tcl_k4/checkpoint-5000
Split manifests	../data/robomme16/meta/qgraph_split_planner16.json, qgraph_split5.json
Scripts	../logs/planner/train_planner16z.sh, latents16z.sh, train_z16.sh, eval_z16.sh

Planner:

torchrun --nnodes 1 --nproc_per_node 4 --master_addr 127.0.0.1 --master_port 29517 \
  -m qgraph.online --config configs/qgraph/planner16z.yaml --variant planner16z --seed 0

Z cache (one process per GPU, then merge):

python -m qgraph.latents --config configs/qgraph/planner16z.yaml --planner ../runs/online/planner16z_s0 \
  --out ../cache/oracle_labels_16z.pt --part $i --parts 4 --gpu $i
python -m qgraph.latents --config configs/qgraph/planner16z.yaml --out ../cache/oracle_labels_16z.pt --parts 4 --merge

Controller:

torchrun --nnodes 1 --nproc_per_node 4 --master_addr 127.0.0.1 --master_port 29518 \
  -m qgraph.train --config configs/qgraph/z16.yaml --variant z16 --seed 0

Evaluation:

python -m qgraph.evaluate --config configs/qgraph/z16.yaml \
  --checkpoint ../runs/stage_b/z16_k32_alr0.0001_mlr0.0003_s0/checkpoint-12000 \
  eval.replan=false eval.watch=true eval.guide=prefix eval.cfg=2.0 eval.state_sigma=0.0 eval.fill=true \
  "eval.gpus=[0,1,2,3]" eval.port_base=25000 eval.online=../runs/online/planner16z_s0 \
  "eval.tasks=[BinFill,ButtonUnmask,ButtonUnmaskSwap,InsertPeg,MoveCube,PatternLock,PickHighlight,PickXtimes,RouteStick,StopCube,SwingXtimes,VideoPlaceButton,VideoPlaceOrder,VideoRepick,VideoUnmask,VideoUnmaskSwap]" \
  eval.subset.per_task=50 eval.runs=../runs/eval_t50_z16