srlwam — stacking policies

LiLa-WAM policies for the two-cube Franka stacking task, trained in Isaac Lab simulation and packaged for deployment on the real robot. Training data lives in the companion dataset repo aabyaneh/srlwam.

Checkpoint	Weights	Trained on	Status
policy_stack_v0	epoch 40 of sft_2026-09-01_19-41-16	data_stack_v0 — 900 of the 1800 train demos (data_fraction: 0.5)	scored on real data; deploy this
policy_stack_v0_fulldata	epoch 105 of sft_2026-09-06_11-54-23	data_stack_v0 — all 1800 train demos	better in sim, not yet scored on real data
runs/stack_real_v0_video	epoch 43 selected / epoch 50 final	real camera-only video adaptation + labeled sim replay, initialized from full-data epoch 105	video-loss candidate; robot control not yet evaluated
archive/policy_stack_v0_ep6.pt	epoch 6, best validation loss	same as policy_stack_v0	superseded; kept for comparison

Epoch 40 was chosen on evidence rather than on validation loss — see Which checkpoint.

policy_stack_v0

Flow-matching action model over a frozen DINOv3 ViT-L/16 encoder. Two camera views plus a 9-D proprioceptive state in, 32-step chunks of 7-D actions out. Weights are epoch 40 of sft_2026-09-01_19-41-16; sim validation at that epoch was action L1 0.0172, position 0.0126 m, gripper accuracy 0.975.

policy_stack_v0/
├── policy_stack_v0.pt        the action model, epoch 40 (2.4 GB)
├── model/
│   ├── config.yaml           camera order, state keys, chunk size, image size
│   ├── dinov3_pretrain/      frozen vision encoder (1.2 GB), loaded at construction
│   ├── models/, utils/       inference code and the normalisation statistics
│   └── robotwin_infer.py
├── lilawam_policy.py         observation -> action chunk
├── lilawam_deploy.py         the sim-to-real adapter; every convention lives here
├── check_deployment.py       five-check self-test, run it before enabling the arm
├── fixture.npz               16 real observations + reference predictions (19 MB)
├── fixture_meta.json
└── ZEROSHOT_PLAN.md          packaging, transfer, calibration, control loop, eval protocol

Quick start

hf download aabyaneh/srlwam --repo-type model --include "policy_stack_v0/*" \
    --local-dir ./srlwam --max-workers 8
cd ./srlwam/policy_stack_v0
python check_deployment.py --bundle .          # nothing moves a robot

Needs Python 3.10+, torch with CUDA (bf16, so Ampere or newer), torchvision, transformers, omegaconf, numpy, opencv-python, scipy, h5py. About 1.1 GB of GPU memory; inference is 36 ms median at batch 1 on an RTX 4080 SUPER, which fits a 10 Hz control loop with room to re-plan every step.

Interface

Observations — the policy speaks the simulator's dialect, and each row below is a conversion the robot side must apply. All were measured against 20 real captures, not assumed; lilawam_deploy.policy_observation implements them and check_deployment.py proves it.

Policy input	Shape	From the robot	Conversion
table_cam	(B, 256, 256, 3) uint8 RGB	agentview 640×480	crop raw[:, 85:565], resize 256² INTER_AREA
wrist_cam	(B, 256, 256, 3) uint8 RGB	wrist 640×480	crop raw[:, 125:605], resize 256² INTER_AREA
eef_pos	(B, 3) float32 [m]	O_T_EE translation (panda_hand origin)	+0.1034 m along the hand's own z
eef_quat	(B, 4) float32	O_T_EE rotation (panda_link8 flange)	right-multiply by Rz(−45°), keep (x, y, z, w)
gripper_pos	(B, 2) float32 [m]	gripper width	(w/2, −w/2)

Three of these fail silently if you get them wrong — the policy still emits actions, they are just meaningless:

• The two camera crops are different windows. The wrist is 40 px right of centre. A naive centre crop disagrees with the reference observations by 23.6/255 against a correct 1.5/255.
• Do not reorder the quaternion. Both sides use (x, y, z, w) despite Isaac Lab's nominal (w, x, y, z). Reordering puts 100% of frames outside the training normalisation range on three of four components.
• The flange→hand 45° roll is required, same failure mode.

Actions — 7 numbers per step, in the simulator's convention:

Index	Channel	Meaning	On the robot
0:3	dx, dy, dz	IK-relative command, realised at 0.192× by the sim controller	tool displacement 0.192 × a[0:3] per 10 Hz period
3:6	d roll, d pitch, d yaw	a Mimic data-generation artefact; the task needs none	drop, hold orientation
6	gripper	+1 = OPEN, −1 = CLOSE	> 0 → open, ≤ 0 → close

The gripper sign is inverted relative to the real recordings, where +1 means close. Consume the policy's convention on the robot; apply the flip only when scoring against recorded real labels. Getting this backwards makes the arm open on every grasp, and nothing errors.

Control rate is 10 Hz. The tool point is the ee_frame, 0.1034 m ahead of panda_hand.

Which checkpoint

Validation loss and validation L1 disagree across this training run: loss is best at epoch 6 (0.356, rising to 0.669 by epoch 40) while action L1 keeps improving (0.0251 → 0.0172). Rather than pick on either, both checkpoints were scored open-loop against the 20 real episodes — the metric that actually bears on deployment. Epoch 40 wins on nearly everything, at the executed step and across the chunk:

on real data	epoch 6	epoch 40
direction cosine, executed step	+0.380	+0.490
cosine > 0, executed step	0.661	0.716
translation L1, executed step [m]	0.01049	0.01020
gripper accuracy, executed step	0.907	0.880
direction cosine, full chunk	0.263	0.308
gripper accuracy, full chunk	0.721	0.759
action L1, full chunk	0.0879	0.0752
spurious rotation L1 [rad]	0.00675	0.00539

Epoch 40 is better at where to move; epoch 6 holds a 2.7-point edge on the immediate gripper sign. Since reaching dominates and the adapter re-plans every step, epoch 40 is the deployment choice — but the grasp-timing caveat below applies to both, so comparing them on the robot is worthwhile.

Honest performance expectations

Scored open-loop against 20 real teleoperated episodes (predict at every frame, compare to what the operator did next), this checkpoint is not expected to succeed zero-shot:

	held-out sim	real
action L1	0.0181	0.0752
direction cosine, executed step	+0.626	+0.490
gripper accuracy, executed step	0.989	0.880
gripper accuracy by chunk step 31	0.972	0.579

It responds to real images — direction and current gripper state carry real signal — but open-loop error grows with horizon on real data while staying flat in sim, and gripper accuracy collapses below chance partway through the episode: it mistimes the grasp. The likeliest cause is object placement. In those recordings every cube sat 3–7 cm beyond the training spawn box, putting eef x outside the training normalisation range for 26.9% of frames — the only state dimension that leaves it. Place cubes inside x ∈ [0.40, 0.50] m, |y| ≤ 0.10 m before drawing conclusions.

ZEROSHOT_PLAN.md has the calibration procedure, the safety envelope, and the evaluation protocol.

Contents note

fixture.npz contains 16 frames captured in the lab (both camera streams, raw 640×480) with the matching robot state, used to verify a deployment reproduces the reference pipeline.

Training data

data_stack_v0: 2000 Mimic-generated trajectories from 19 teleoperated source demonstrations, domain-randomised over lighting, table material, cube appearance and camera mount jitter.

The rotation in this policy's action space is an artefact of data generation, not of the task: the teleoperated demonstrations contain exactly zero rotation, and Mimic manufactures it from isotropic action noise plus an orientation feedback term. A successor corpus regenerated without it (data_stack_v1_norot, 2004 demos, rotation identically zero, slightly better generation yield) exists for retraining.

policy_stack_v0_fulldata

Same corpus, same architecture, more of the data. policy_stack_v0 was trained on 900 of the 1800 training demos (data_fraction: 0.5); this one uses all 1800 for 120 epochs. The name says v0 because the data is still data_stack_v0 — only the fraction consumed changed.

policy_stack_v0_fulldata/
├── policy_stack_v0_fulldata.pt        epoch 105, the default (2.5 GB)
├── policy_stack_v0_fulldata_ep120.pt  epoch 120, for comparison (2.5 GB)
├── model/                             config, encoder, inference code, norm stats
│   └── dataloader/                    (also present here; absent from the v0 bundle)
├── lilawam_policy.py, lilawam_deploy.py, check_deployment.py
├── fixture.npz, fixture_meta.json     references regenerated for epoch 105
└── ZEROSHOT_PLAN.md

It is a drop-in replacement. model/config.yaml is identical to v0's apart from three training knobs (data_fraction 0.5→1.0, epochs 40→120, save_interval_epoch 1→5). Architecture, image_size [192, 192], action_dim, state_dim, chunk size, encoder layers and action_execution_horizon all match, so every convention in Interface — the camera crops, the +0.1034 m z-offset, the Rz(−45°) flange roll, the inverted gripper sign, the 0.192× action scaling, 10 Hz — applies unchanged.

In held-out sim, at epoch 105 against v0's epoch 40:

held-out sim	policy_stack_v0 (ep 40)	policy_stack_v0_fulldata (ep 105)
action L1	0.0172	0.0143
position error	0.0126 m	0.0116 m
rotation error	0.00971 rad	0.00934 rad
gripper accuracy	0.975	0.983

Epochs 90 through 120 are flat within noise (action L1 0.0142–0.0146), so epoch 105 is not a sharp optimum — it is where act_l1_h16, the L1 over the 16 steps actually executed before re-planning, was lowest. That metric replaced validation loss for checkpoint selection during this run, for the reason Which checkpoint documents: loss again bottomed early (epoch 5) and rose throughout while action error kept falling.

Not yet scored on the robot

policy_stack_v0 remains the deployment choice until this one is scored on real data. The sim gain above is real but sim gain is not what the v0 decision was made on — that comparison used 20 real teleoperated episodes, and v0's real action L1 (0.0752) was four times its sim figure.

Two observations from the 16 real fixture frames, which is all the real data in this bundle: predictions sit close to v0's, median |difference| 0.0078 against the self-test's 0.02 tolerance, so this is not a wildly different policy. But the gripper sign differs on 14% of steps, and mistimed grasping is precisely v0's known real-world failure. Whether that 14% is an improvement or a regression cannot be read off sim metrics — it needs the open-loop scoring protocol in ZEROSHOT_PLAN.md.

fixture.npz's reference_actions were regenerated with epoch 105, so check_deployment.py verifies this bundle rather than reporting drift against v0's weights. The 16 observations, the recorded images and the states are byte-identical to v0's — only the reference predictions changed. The cube-placement caveat in Honest performance expectations applies here too: the training spawn box did not change.

Real-video adaptation

The completed four-GPU video-adaptation run is available in runs/stack_real_v0_video, alongside the real-world scratch and warmstart runs. It initializes from the full-data sim policy at epoch 105 and updates the visual adapter using real camera videos plus labeled simulation replay, without real action or state labels. The selected checkpoint is epoch 43; the final checkpoint is epoch 50 after 4,100 updates. Selection minimizes held-out real-video feature loss subject to a 10% sim-action regression limit. This is an evaluation candidate, not evidence of improved closed-loop robot control. See the run README for metrics, limitations, normalization, source snapshots, and download instructions.