JingwuLuo/ur3_sim_pick_and_place_cube

π0.5 policies for UR3 cube pick-and-place: simulation pre-training, fine-tuning on real recordings, and one-stage mixtures

openpi π0.5 policies of a dual-arm UR3 workcell in which one arm picks one of three cubes and puts it on the table, the plate, the white box or another cube. Eight groups of checkpoints:

1. Stage 1, simulation only, 500 episodes (stage1_sim500/): the action expert trained on 500 scripted MuJoCo episodes (JingwuLuo/ur3_sim_pick_and_place_cube, revision 7fa3c586).

2. Stage 1, simulation only, 1000 episodes (stage1_sim1000/): the action expert trained on 1000 scripted episodes of the same simulator (revision 5cd89567 of the same dataset) with relays between the two arms and perturbed approaches.

3. Stage 2, real recordings (stage2_lora_real/): LoRA adapters on the action expert, trained on 223 teleoperated episodes of the real workcell, starting from the stage1_sim500/ stage-1 checkpoint of the same label. See the real-robot test below: these did not perform the task on the robot.

4. Stage 2 on a mixture (stage2_mix_sim1000_real/): the full action expert, starting from the stage1_sim1000/ stage-1 checkpoint of the same label, trained on the 223 real episodes with simulated rows mixed into every batch, the simulated share falling during the run. One policy also takes the robot's state as input, switched on gradually.

5. Real recordings of the moved workcell (real_quick_new/): the action expert trained from pi05_base on 62 teleoperated episodes recorded after the workcell was moved, with the robot's state as input.

6. One-stage mixture (joint_sim300_real_quick_new/): the same real episodes mixed with 300 simulated episodes of the moved workcell in a single run from pi05_base, with the robot's state as input.

7. Real recordings after the top camera was adjusted (real_quick_new2/): the action expert trained from pi05_base on 72 teleoperated episodes re-recorded after the top camera of the workcell was adjusted, with the robot's state as input; the curvature policy only.

8. One-stage mixture after the top camera was adjusted (joint_sim300_real_quick_new2/): the 72 real episodes of group 7 mixed with 300 simulated episodes in a single run from pi05_base, with the robot's state as input; the curvature policy only.

Groups 1 to 4 were trained for the earlier placement of the workcell.

Within each of groups 1 to 6 the policies differ only in how the 10-point action chunk is labelled; everything else (data, model, recipe) is identical. Groups 7 and 8 hold the curvature policy only.

policy	label	points per chunk	reach of one chunk
per-frame	the next 10 control frames	10	10 frames (0.33 s)
two-phase	evenly spaced in time over 48 frames, one point per frame around grasp, lift and release	10	10 to 48 frames
curvature	curvature-resampled over 48 frames: points placed by path curvature, denser per unit of joint path in turns (for example the reversal at a grasp or release), with no extra density in time or at gripper events	10	48 frames (1.6 s)

folder	stage	steps published
stage1_sim500/<policy>/	1, 500 simulated episodes	5000, 9999 (the last)
stage1_sim1000/<policy>/	1, 1000 simulated episodes	14999 (the last)
stage2_lora_real/<policy>/	2, LoRA adapters on the real recordings	3999 (the last)
stage2_mix_sim1000_real/<policy>/	2, mixture of real and simulated rows	7999 (the last)
real_quick_new/<policy>/	real recordings of the moved workcell only	5499 (the last)
joint_sim300_real_quick_new/<policy>/	one-stage mixture of real and simulated rows, moved workcell	8999 (the last)
real_quick_new2/curvature/	real recordings after the camera adjustment only	6449 (the last)
joint_sim300_real_quick_new2/curvature/	one-stage mixture of real and simulated rows, after the camera adjustment	10099 (the last)

<policy> is per_frame, two_phase or curvature in every stage folder, and additionally curvature_state in stage2_mix_sim1000_real/ (curvature with the gated state input). Each step folder is one checkpoint, for example stage2_lora_real/curvature/3999/. The --name used when serving is the configuration name listed in the serving table below, not the folder name.

Real-robot test

Closed-loop trials on the real workcell with the policies served as described below (qualitative; success rates were not recorded):

• the stage-1 curvature policy of stage1_sim500/ (simulation only) performs the task;
• a policy trained on the real episodes only (see Results) performs the task;
• the stage-2 policies of stage2_lora_real/ (LoRA adapters) do not perform the task at full adapter scale; with the adapter update scaled by 0.5 the task sometimes succeeds.

The open-loop table below did not predict this: stage 2 has lower real held-out error than stage 1. The stage1_sim1000/ policies have not been run on the robot. No closed-loop result in the simulator is reported.

Results

The table is open loop. It covers stage1_sim500/ and stage2_lora_real/; the stage1_sim1000/ policies have not been evaluated. The evaluation feeds recorded observations to the served policy and compares its output with the recording, so it does not measure task success. Setup: 5 real held-out episodes (raw recording indices 54, 102, 122, 180, 185) and 10 simulated held-out episodes (raw indices 2, 100, 231, 234, 243, 287, 323, 325, 349, 463); neither set is in the training data of either stage. The server runs with --uniform --cap 1 so that its targets line up with the recorded frames (the deployment timing is faster than the recording and cannot be compared frame by frame); a new inference every 8 frames. Each cell is the median, in rad, of the largest error over the 12 joints:

• points: the model's 10 points against the label's 10 points, per inference;
• head: the first 8 targets of each response against the recorded actions, per inference;
• stitched: the targets of consecutive responses joined, 8 per response, against the recorded actions, per frame (padding rows excluded).

label	model	simulated held-out: points / head / stitched	real held-out: points / head / stitched
per-frame	stage 1	0.138 / 0.112 / 0.045	0.157 / 0.130 / 0.066
per-frame	stage 2	0.174 / 0.140 / 0.066	0.132 / 0.108 / 0.055
two-phase	stage 1	0.236 / 0.106 / 0.046	0.296 / 0.119 / 0.063
two-phase	stage 2	0.276 / 0.122 / 0.054	0.229 / 0.107 / 0.055
curvature	stage 1	0.472 / 0.110 / 0.053	0.418 / 0.116 / 0.061
curvature	stage 2	0.499 / 0.135 / 0.067	0.382 / 0.107 / 0.058
curvature	trained on the real episodes only, see below	0.522 / 0.154 / 0.078	0.288 / 0.087 / 0.052

• Stage 2 lowers the real held-out error of every label against stage 1 and raises the simulated held-out error.
• The curvature policy trained on the real episodes only (full action expert from pi05_base, 20,000 steps, folder 223TrialsTrainData/pi05_ur3_cube_quick_v4h5_curvature_s48_pp/19999 of JingwuLuo/pick_and_place_cube_quick_ur3) has lower real held-out error than every stage-2 policy and higher simulated held-out error.
• Open-loop error did not predict the closed-loop outcome on the real robot (see above).
• "points" compares against labels of different reach, so compare it within a label, not across labels.
• Gripper commands: every recorded open/close edge appears in the stitched trajectory of every stage-1 and stage-2 policy, on both held-out sets.

Stage 1: the simulated data

Raw episodes: JingwuLuo/ur3_sim_pick_and_place_cube, revision 7fa3c586627f34b55d179fbc4e762e5e9106ba44 (500 episodes, 81,087 steps at 30 Hz, 3.2–10.9 s per episode, median 4.9 s). The dataset card describes how the episodes were generated. The recordings used here, as the dataset card describes them:

• Scripted from the simulator's privileged state (object poses); not teleoperated and not produced by a learned policy; only successful episodes are kept. The scripted motion is faster and smoother than the teleoperated recordings of the real workcell.
• In every episode one arm moves and the other stays at its start pose (240 episodes left arm, 260 right arm). 83 episodes have a late close and regrasp: the arm reaches the grasp point with open fingers, pauses and starts lifting, the gripper closes on air too late, reopens after 0.1–0.3 s, and at least 0.7 s later the arm descends straight back and grasps the cube (two close and two open commands in these episodes). The transport distance from pick to place is short in 172 episodes, medium in 240 and long in 88.
• Cubes: pink 173, blue 180, orange 147 episodes. Where the cube starts: table 270, plate 98, white box 88, on another cube 44. Where it is put: table 160, plate 105, white box 92, on another cube 143.
• Camera set: the top camera at 240×320 and two wrist cameras at 240×424, 30 Hz, the 14-dimensional state and action of the real workcell (left joints 0–5, left gripper 6, right joints 7–12, right gripper 13; grippers 0 open, 1 closed). Both wrist cameras sit on the table side of their grippers, as on the real rig.

Preparation before training (the converted datasets are not uploaded):

1. Conversion of the HDF5 episodes to a LeRobot dataset.
2. Instructions. The episodes carry no language, so five phrasings per episode were generated from each episode's sim/task group (the arm is not named; 211 distinct sentences over the 2,500): Put the pink cube on the white box., Pick up the pink cube from the table and place it on the white box., Move the pink block from the table onto the white box., Could you pick the pink cube up from the table and set it on the white box?, pink cube from table to white box. One of the five is drawn at random per training sample.
3. Still frames. Stretches of at least 5 consecutive frames in which the action changes by less than 0.002 per frame (rad for the joints) were dropped (6,475 rows), and 60 frames of the final pose were appended to each episode as padding rows (30,000 rows), so that the policy learns to hold still after the task is done. 104,612 rows in total.
4. Keyframes (grasp, rest, lift, release) were detected per arm from the gripper signal and the arm's motion; rest is the first frame after the grasp at which the arm has stopped (joint speed below 0.003 rad per frame for 3 frames), because the simulator issues the close command while the arm is still decelerating. In all 500 episodes the number of detected grasps and releases equals the number of gripper commands the simulator recorded. The two-phase label uses the keyframes.
5. Labels. The per-frame, two-phase and curvature action-chunk labels were built from the timed episodes.
6. Held-out episodes. 10 episodes (raw indices 2, 100, 231, 234, 243, 287, 323, 325, 349, 463; 7 right arm, 3 left arm, 2 late-close; 5 short, 4 medium and 1 long transport) are excluded from training; the other 490 episodes (102,500 rows) are the training set.

Stage 1 on 1000 simulated episodes (stage1_sim1000/)

Raw episodes: JingwuLuo/ur3_sim_pick_and_place_cube, revision 5cd89567862dbe659abc900edc81b5467cb81ce1 (1000 episodes, 268,700 steps at 30 Hz, 3.4–17.2 s per episode, median 9.0 s). The dataset card describes how the episodes were generated. As the card describes them:

• Scripted from the simulator's privileged state; not teleoperated and not produced by a learned policy; only successful episodes are kept.
• 388 episodes: one arm does the task while the other stays still (right 203, left 185). 612 episodes are relays: the picking arm cannot reach the place, so it puts the cube on a handover spot in the middle of the table and returns; the other arm starts when the first one begins its return, takes the cube and puts it at the place.
• 252 episodes are perturbed: in one leg the arm is pushed 1–5 cm off its path during the approach (105) or the transport (147), stops, and corrects from there; the pushed steps are tagged.
• 236 episodes have a late close and regrasp. Pick-to-place distances are spread evenly from 0.08 to about 0.80 m.
• Cubes: pink 351, orange 323, blue 326. Camera set, state and action as in the 500-episode set.

Preparation before training (the converted datasets are not uploaded):

1. Instructions. Five phrasings per episode (231 distinct sentences over the 5,000), generated from each episode's sim/task group; the arm and the relay are not mentioned. One of the five is drawn at random per training sample.
2. Still frames. Stretches of at least 5 consecutive frames in which the action changes by less than 0.002 per frame were dropped (13,776 rows: 425 leading, 12,325 trailing, 1,026 pauses outside keyframe windows); 41,750 still frames inside keyframe windows were kept. 60 frames of the final pose were appended to each episode as padding rows (60,000 rows). 314,924 rows in total.
3. Keyframes (grasp and release of each arm). In all 1000 episodes the detected grasps and releases equal the gripper commands the simulator recorded. The two-phase label uses them.
4. Labels. The per-frame, two-phase and curvature action-chunk labels (same configurations as stage1_sim500/). Chunk points that lie on a pushed step get label weight 0 and do not enter the loss: 0.82 % (per-frame), 1.12 % (curvature) and 0.56 % (two-phase) of the points. The median reach of one chunk is 10.0 frames (per-frame), 20.3 (two-phase) and 48.0 (curvature).
5. Held-out episodes. 40 episodes, stratified (relay 27 and single arm 13; left arm 15 and right arm 25; perturbed 11; late close 8), are excluded from training; the other 960 episodes (302,218 rows) are the training set.

Training: the same as stage 1 of stage1_sim500/ except the data and the length.

item	value
start	pi05_base, action horizon 10
trained	action expert (Gemma 300M) and the action and time projections; SigLIP and Gemma 2B frozen
actions, state input	joints as deltas from the state, grippers absolute, quantile normalisation per chunk point; no state input
data	302,218 simulated rows
optimisation	batch 32, 15,000 steps (about 1.6 passes), AdamW (β2 0.95, weight decay 1e-4, gradient clipping 1.0), cosine learning rate 5e-5 to 5e-6 after 500 warm-up steps
weight averaging	EMA, decay 0.999; the saved params are the averaged weights
parallelism	one GPU per policy, no model sharding; each run trained from step 0 to step 14,999 without interruption

Batch loss in normalised space (the labels predict different quantities, so the values are not comparable); the logged loss is that of the training weights, not of the averaged weights that are saved. Every array of every published checkpoint is finite.

label	step 0	mean of steps 14,500–14,950
per-frame	0.171	0.0173
two-phase	0.170	0.0167
curvature	0.184	0.0187

The statistics in assets/ are constructed as for stage1_sim500/ (next section), from the 960 training episodes: the wrist-2 range at the last chunk point is 0.002 rad in the simulation and 0.05 (left) and 0.214–0.391 (right) after the union with the real recordings.

Stage 2 on a mixture of real and simulated rows (stage2_mix_sim1000_real/)

Starting weights: the stage1_sim1000/ stage-1 checkpoint (step 14999) of the same label. Trained: the action expert (Gemma 300M) and the action and time projections, as in stage 1; SigLIP and Gemma 2B frozen. Data: every row of a batch comes from the simulated training set of stage1_sim1000/ (960 episodes, 302,218 rows) with a probability that falls linearly from 0.5 at the first step to 0.1 at the last, and from the 223 real training episodes (62,410 rows) otherwise; each set is traversed in shuffled passes. Rows of the simulated set at pushed steps keep the zero label weight of stage 1. Prompts, labels, normalisation statistics and the 5 real held-out episodes are those of the sections above.

item	value
optimisation	batch 32, 8,000 steps, AdamW (β2 0.95, weight decay 1e-4, gradient clipping 1.0), cosine learning rate 2.5e-5 to 2.5e-6 after 300 warm-up steps
weight averaging	EMA, decay 0.999; the saved params are the averaged weights
parallelism	one GPU per policy, no model sharding

Batch loss in normalised space, over the mixed batches (the labels predict different quantities, so the values are not comparable; the logged loss is that of the training weights, not of the averaged weights that are saved). Every array of every published checkpoint is finite.

policy	step 0	mean of steps 7,500–7,950
per-frame	0.043	0.0183
two-phase	0.055	0.0156
curvature	0.074	0.0180
curvature, gated state input	0.075	0.0176

Gated state input (curvature_state/, configuration pi05_ur3_sim2_mix_curvature_s48_state_pp). pi0.5 can take the robot's state as tokens in the prompt: the 14-dimensional state, normalised with the statistics in assets/, in 256 bins (State: 12 87 ...). Stage 1 was trained without them, so during this run each training sample carries the state tokens with a probability that rises linearly from 0 at the first step to 1 at step 6,000 and is 1 for the last 2,000 steps; a sample without them has exactly the prompt of stage 1. The model is therefore always used with the state tokens after training. The state statistics of this policy differ from the others' in one entry: the range of the left arm's wrist-2 (state dimension 4, 0.0012 rad in the data) is widened to 0.05 rad around its centre, as the same joint's range is for the actions, because its encoder noise would otherwise fill the whole token range. The 14-dimensional observation/state the client sends is the one described under "Request and response"; the left arm stays parked in the real recordings, so its state tokens barely change there.

The policies of this group have no open-loop or closed-loop evaluation reported here.

Stage 2: the real recordings

Teleoperated demonstrations of the real dual-arm UR3 workcell (30 Hz, the same three cameras and 14-dimensional state and action): only the right arm moves and the left arm stays parked; an orange, blue or pink cube is put on a white box, a plate or another cube. 223 training episodes (62,410 rows) and 5 held-out episodes. The recordings of the real workcell and their preparation are described in JingwuLuo/pick_and_place_cube_quick_ur3; the 223-episode set is not uploaded. The prompts are that repository's: five phrasings per episode (480 distinct sentences), one drawn at random per sample, for example Put the orange cube on the white box.

Real recordings of the moved workcell only (real_quick_new/)

Policies trained from pi05_base on the real recordings JingwuLuo/pick_and_place_quick_new (revision 83f57589, taken after the workcell was moved), with the robot's state as input. They are not related to the checkpoints of the other folders, which were trained for the earlier placement of the workcell.

Data: 71 teleoperated episodes of 30 Hz, the same three cameras and 14-dimensional state and action; a cube is put on the white box, the plate or another cube. Four episodes that are scene resets (they end with a cube put back on the table) are excluded, 67 are kept; the right arm moves in all of them except two, in which the left arm picks. After dropping still frames outside the windows around each gripper switch and adding a 60-row end hold: 62 training episodes (17,605 rows) and 5 held-out episodes (dataset episodes 10, 11, 19, 31, 55), chosen by clustering the start poses, the motion and the first top-camera frame. Six episodes begin with a failed grasp followed by a second grasp and are kept. Prompts: five phrasings per episode, one drawn at random per sample. The policies differ only in how the 10-point action chunk is labelled (table above).

item	value
start	pi05_base, action horizon 10
trained	action expert (Gemma 300M) and the action and time projections; SigLIP and Gemma 2B frozen
actions, state input	joints as deltas from the state, grippers absolute, quantile normalisation per chunk point; the 14-dimensional state is written into the prompt as 256-bin tokens (--state-input)
statistics	per-point statistics of the 62 training episodes, no range widened
optimisation	batch 32, 5,500 steps (10.0 passes), AdamW (β2 0.95, weight decay 1e-4, gradient clipping 1.0), cosine learning rate 2.5e-5 to 2.5e-6 after 150 warm-up steps
weight averaging	EMA, decay 0.999; the saved params are the averaged weights
parallelism	one GPU per policy, no model sharding, 3.2 s per step

Batch loss in normalised space (the labels predict different quantities, so the values are not comparable; the logged loss is that of the training weights, not of the averaged weights that are saved). Every array of every published checkpoint is finite.

policy	step 0	mean of steps 5,000–5,450
per-frame	0.223	0.0156
two-phase	0.267	0.0158
curvature	0.292	0.0145

The policies of this group have no open-loop or closed-loop evaluation reported here.

One-stage mixture of real and simulated rows (joint_sim300_real_quick_new/)

The same real recordings as above mixed with 300 scripted simulated episodes of the moved workcell (JingwuLuo/ur3_sim_pick_and_place_cube revision 98ee80d7: right arm only, camera, backdrop and light colours fitted to the real recordings of the moved workcell), in one training run from pi05_base, with the robot's state as input from the first step; there is no simulation-only stage and no second stage.

Data: every row of a batch comes from the simulated training set (288 episodes, 55,359 rows after the same trimming and a 60-row end hold; 12 held-out episodes by clustering) with a probability that falls linearly from 0.6 at the first step to 0.15 at the last (mean 0.375), and from the 62 real training episodes (17,605 rows) otherwise; each set is traversed in shuffled passes. The 9,000 steps make 10.2 passes over the real rows and 1.95 over the simulated rows, the numbers of passes of the two single-source recipes (10 and 2). Rows of the simulated set at pushed steps (the arm is pushed off its path in 72 episodes) carry the label weight 0. The simulated episodes are shorter than the real ones (median 134 rows against 295), so a chunk of the same span covers about twice the motion there. The normalisation statistics are the union of the real and the simulated per-point statistics (the lower of the two q01 and the higher of the two q99 per entry): the right arm's shoulder pan and wrist 3, and slightly wrist 1, have a wider range than in the real data alone (curvature, last chunk point: 1.58 to 2.45 rad and 2.27 to 3.24 rad), the other joints keep the real range. The simulated wrist 2 hardly moves, which is why the simulated statistics alone cannot be used.

item	value
start, trained, actions, state input	as in the real-only group above
optimisation	batch 32, 9,000 steps, AdamW (β2 0.95, weight decay 1e-4, gradient clipping 1.0), cosine learning rate 2.5e-5 to 2.5e-6 after 300 warm-up steps
weight averaging	EMA, decay 0.999; the saved params are the averaged weights
parallelism	one GPU per policy, no model sharding, 3.2 s per step

Batch loss in normalised space over the mixed batches (same remarks as above). Every array of every published checkpoint is finite.

policy	step 0	mean of steps 8,500–8,950
per-frame	0.334	0.0133
two-phase	0.525	0.0136
curvature	0.376	0.0133

arc_resample.json of these checkpoints is the real dataset's: its robot block carries the real gripper's hold times (0.50 s after a close, 0.52 s after an open), which the server applies. The policies of this group have no open-loop or closed-loop evaluation reported here.

Real recordings after the camera adjustment (real_quick_new2/)

One curvature policy trained from pi05_base on the re-recorded real episodes JingwuLuo/pick_and_place_quick_new (revision 88bfa623, taken after the top camera was adjusted; top camera 640x360), with the robot's state as input. It is not related to the checkpoints of the other folders.

Data: 80 teleoperated episodes of 30 Hz. Eight are excluded: episode 18 (68 steps, the gripper never closes, the cube ends on the plate without a grasp), 25 (no cube moved, 33 s mostly idle), 26, 27 and 28 (the gripper never closes and the arm does not move; the scene is rearranged by hand) and 29, 30 and 31 (no arm motion). The 72 kept episodes (21,090 rows) move a cube (pink, orange or blue) to the white box, the plate or another cube; the left arm never moves. After dropping still frames outside the windows around each gripper switch and adding a 60-row end hold the training set has 72 episodes and 20,610 rows, 31.0 % of them end padding; every kept episode is in the training set, there is no held-out episode. Ten episodes contain a failed grasp that is repeated, episode 0 starts with a closed gripper and episode 22 carries the cube for 188 frames; all are kept. Prompts: five phrasings per episode, one drawn at random per sample. The label is the curvature label of the table above.

item	value
start	pi05_base, action horizon 10
trained	action expert (Gemma 300M) and the action and time projections; SigLIP and Gemma 2B frozen
actions, state input	as in real_quick_new/ (joints as deltas from the state, grippers absolute, quantile normalisation per chunk point; the 14-dimensional state is written into the prompt as 256-bin tokens, --state-input)
statistics	per-point statistics of the 72 training episodes; the parked left arm (state and action dimensions 0 to 5) has a quantile range of 1e-4 rad, given a minimum range of 0.05 rad
optimisation	batch 32, 6,450 steps (10.0 passes), AdamW (β2 0.95, weight decay 1e-4, gradient clipping 1.0), cosine learning rate 2.5e-5 to 2.5e-6 after 150 warm-up steps
weight averaging	EMA, decay 0.999; the saved params are the averaged weights
parallelism	one GPU, no model sharding, 3.2 s per step

Batch loss in normalised space (the logged loss is that of the training weights, not of the averaged weights that are saved). Every array of the published checkpoint is finite.

policy	step 0	mean of steps 6,000–6,400
curvature	0.139	0.0088

arc_resample.json of this checkpoint carries the real gripper's hold times (0.50 s after a close, 0.52 s after an open), which the server applies. The policy has no open-loop or closed-loop evaluation reported here.

One-stage mixture after the camera adjustment (joint_sim300_real_quick_new2/)

One curvature policy trained from pi05_base on the real episodes of the group above mixed with 300 scripted simulated episodes (JingwuLuo/ur3_sim_pick_and_place_cube revision 493421dd: right arm only, top camera 640x360 at the pose fitted to the real recording after the adjustment), in one training run, with the robot's state as input from the first step; there is no simulation-only stage and no second stage. It is not related to the checkpoints of the other folders.

Data: every row of a batch comes from the simulated set (all 300 episodes, 58,408 rows after still-frame trimming and a 60-row end hold; no held-out episodes) with a probability that falls linearly from 0.6 at the first step to 0.15 at the last (mean 0.375), and from the 72 real episodes (20,610 rows) otherwise; each set is traversed in shuffled passes. The 10,100 steps make 9.8 passes over the real rows and 2.08 over the simulated rows, close to the numbers of passes of the single-source recipes (10 and 2). Rows of the simulated set at pushed steps (the arm is pushed off its path) carry the label weight 0, 1.29 % of the points. The normalisation statistics are the union of the real per-point statistics (the parked left arm given a minimum range of 0.05 rad, as above) and the simulated ones (the lower of the two q01 and the higher of the two q99 per entry).

item	value
start, trained, actions, state input	as in the real-only group above
optimisation	batch 32, 10,100 steps, AdamW (β2 0.95, weight decay 1e-4, gradient clipping 1.0), cosine learning rate 2.5e-5 to 2.5e-6 after 300 warm-up steps
weight averaging	EMA, decay 0.999; the saved params are the averaged weights
parallelism	one GPU, no model sharding, 3.1 to 3.2 s per step

The run was stopped once after the checkpoint at step 5000 and resumed from it on another GPU, with the mixture schedule continuing from step 5000; the steps after 5000 of the first process were discarded.

Batch loss in normalised space over the mixed batches (same remarks as above). Every array of the published checkpoint is finite.

policy	step 0	mean of steps 9,500–10,050
curvature	0.168	0.0088

arc_resample.json of this checkpoint is the real dataset's: its robot block carries the real gripper's hold times (0.50 s after a close, 0.52 s after an open), which the server applies. The policy has no open-loop or closed-loop evaluation reported here.

Normalisation statistics

assets/<repo_id>/norm_stats.json inside each checkpoint holds the quantile statistics (q01, q99) the model was trained with: state (14) and actions per chunk point (10, 14), joints as deltas from the observed state, grippers absolute. The file of a stage-2 checkpoint has the same content as the file of the stage-1 checkpoint of the same label.

They are not the statistics of the simulated data alone. In the simulated episodes the wrist-2 joints (state and action dimensions 4 and 11) barely move: the q01–q99 range of their per-frame delta is 0.002 rad, against 0.21 rad for the right wrist-2 of the real recordings. Normalising real fine-tuning data with simulation-only statistics would scale those targets by about 140. For each label, the q01 of every statistic is therefore the minimum and the q99 the maximum of the simulated and the real recordings' statistics, in both stages. For the simulation-only training this means the model is trained with targets normalised by the wider range. Left-arm wrist-2 has a range floor of 0.05 rad. Do not replace this file when serving or fine-tuning.

label	left joints 0–5, range over the 10 points (rad)	right joints 7–12, range over the 10 points (rad)
per-frame	0.050–0.863	0.027–0.884
two-phase	0.050–1.933	0.113–1.848
curvature	0.050–2.550	0.102–2.506

(smallest and largest q99 − q01 over the 10 chunk points and the joints of the arm; the gripper dimensions have range 1.)

Training

item	stage 1	stage 2
start	pi05_base, action horizon 10	the stage-1 checkpoint at step 9999 of the same label
trained	action expert (Gemma 300M) and the action and time projections, about 430 M parameters; SigLIP and Gemma 2B frozen	LoRA adapters (rank 32) on the attention and feed-forward layers of the action expert only; SigLIP, Gemma 2B, the action expert's own weights and the projections frozen; the adapters start as a no-op
actions	joints as deltas from the state, grippers absolute; quantile normalisation per chunk point	same
state input	none	none
data	102,500 simulated rows	62,410 real rows
optimisation	batch 32, 10,000 steps (about 3 passes), AdamW (β2 0.95, weight decay 1e-4, gradient clipping 1.0), cosine learning rate 5e-5 to 5e-6 after 500 warm-up steps	batch 32, 4,000 steps (about 2 passes), same optimiser, cosine learning rate 1e-4 to 1e-5 after 200 warm-up steps
weight averaging	EMA, decay 0.999; the saved params are the averaged weights	EMA, decay 0.998; the LoRA arrays in the saved params are the averaged weights; see below for the frozen arrays
prompt	one of the episode's five phrasings, drawn at random per sample	same
parallelism	one GPU per policy, no model sharding	same, 1.3 s per step

Batch loss in normalised space (the three labels predict different quantities, so the values are not comparable). The logged loss is that of the training weights, not of the averaged weights that are saved.

label	stage 1: step 0 / mean of steps 9,500–9,950	stage 2: mean within 100 steps of step 100 / of step 3,900
per-frame	0.16 / 0.0138	0.064 / 0.0226
two-phase	0.17 / 0.0126	0.0665 / 0.0205
curvature	0.18 / 0.0125	0.0899 / 0.0262

Every array of every published checkpoint is finite. Each run trained from step 0 to the last step without interruption.

The frozen arrays of the stage-2 params are not the EMA of the training run. The averaging step decay * old + (1 - decay) * new runs in the dtype of the array; for the frozen bfloat16 arrays the decay 0.998 rounds to 0.99609375, so averaging an unchanged array is not the identity: all 51 frozen arrays of the averaged weights differed from their stage-1 values by about 0.3 in relative norm (the 10 LoRA arrays are float32 and unaffected; at decay 0.999 the averaging is the identity for bfloat16). The training-objective loss of the curvature policy's averaged params on real training batches was 0.2385, against 0.0265 logged. The published stage-2 params are therefore rebuilt: the 51 frozen arrays are taken from the stage-1 step-9999 checkpoint (stage 2 does not change them) and the 10 LoRA arrays from the stage-2 EMA at step 3999. The training-objective loss of the rebuilt curvature checkpoint is 0.0293 on real training batches; the other two were rebuilt the same way, and the open-loop table above was measured with the rebuilt checkpoints. The optimiser state is not published.

Contents of a checkpoint

<stage1_sim500 | stage1_sim1000 | stage2_lora_real | stage2_mix_sim1000_real>/<per_frame | two_phase | curvature | curvature_state>/<step>/
  params/                                   model weights (orbax)
  assets/<repo_id>/norm_stats.json          quantile statistics: state (14), actions per chunk point (10, 14)
  assets/<repo_id>/arc_resample.json        label configuration and robot block (30 Hz, joint velocity limits, gripper holds)
  _CHECKPOINT_METADATA

params of a stage-1 checkpoint is the EMA of the training weights of the action expert. params of a stage-2 checkpoint holds every array of the model including the LoRA adapters, so a stage-2 folder is complete and does not need the stage-1 folder to be served. The optimiser state is not included, so training cannot be resumed from these files; they can be used for serving and as the initial weights of a new run (params).

<repo_id>:

policy	stage 1 stage1_sim500/	stage 1 stage1_sim1000/	stage 2 stage2_lora_real/ and stage2_mix_sim1000_real/
per-frame	ur3-sim-cube-time-s10-train	ur3-sim2-time-s10-train	pick-and-place-cube-quick-v4h5-time-s10-train
two-phase	ur3-sim-cube-two-phase-s48-train	ur3-sim2-two-phase-s48-train	pick-and-place-cube-quick-v4h5-two-phase-s48-train
curvature	ur3-sim-cube-curvature-s48-d0-train	ur3-sim2-curvature-s48-d0-train	pick-and-place-cube-quick-v4h5-curvature-s48-d0-train

The keyframes field of arc_resample.json names the file the keyframes were read from; the path was shortened to the file name for this upload and is not used when serving. In a stage-2 file the robot block carries the real gripper's hold times (0.50 s after a close, 0.52 s after an open), which the server applies; in a stage1_sim1000/ file it carries the simulated gripper's (0.30 s after a close, 0.50 s after an open).

Serving

• NVIDIA GPU with CUDA 12. JAX preallocates 75% of GPU memory by default; set XLA_PYTHON_CLIENT_MEM_FRACTION=0.5 to share the GPU.
• openpi with the patches in luo-jingw/VLA_AGENT integrations/openpi/patches/ (all of them) applied, and VLA_AGENT (commit c3d22ea or later for the stage2_mix_sim1000_real/, real_quick_new/ and joint_sim300_real_quick_new/ policies, which need patches 0004 and 0005; commit 0e68c29 or later for the real_quick_new2/ and joint_sim300_real_quick_new2/ policies, whose recipes it adds; commit 0baeb4d or later for the --lora serving flag of stage2_lora_real/) installed in the same environment (pip install -e <VLA_AGENT>).
• openpi fetches the PaliGemma tokenizer (gs://big_vision/paligemma_tokenizer.model) on first use; without internet access, copy ~/.cache/openpi/big_vision/ from a machine that has it (or set OPENPI_DATA_HOME).

Download one policy, stage 2 curvature as the example:

python -c "from huggingface_hub import snapshot_download; snapshot_download('JingwuLuo/ur3_sim_pick_and_place_cube', \
  allow_patterns=['stage2_lora_real/curvature/3999/**'], local_dir='ckpt')"

Serve it, from the openpi root:

export PYTHONPATH=scripts:<VLA_AGENT>/src:<VLA_AGENT>/examples/ur3_dual_arm
python <VLA_AGENT>/examples/ur3_dual_arm/run.py serve \
  --name pi05_ur3_sim_real_curvature_s48_pp \
  --repo-id pick-and-place-cube-quick-v4h5-curvature-s48-d0-train \
  --data-root /unused \
  --checkpoint ckpt/stage2_lora_real/curvature/3999 \
  --lora --init-checkpoint /unused \
  --port 8000 --cap 4 --velocity-fraction 0.5

--name is the configuration name and --repo-id the assets/ folder name of the checkpoint; --data-root is not read when serving. A stage-2 policy needs --lora, which builds the model with the LoRA adapters the weights contain; --init-checkpoint is required together with --lora but its path is not read when serving. The other policies:

policy	stage	--name	--repo-id	extra flags
per-frame	1 (stage1_sim500/)	pi05_ur3_sim_time_s10_pp	ur3-sim-cube-time-s10-train	none
two-phase	1 (stage1_sim500/)	pi05_ur3_sim_two_phase_s48_pp	ur3-sim-cube-two-phase-s48-train	none
curvature	1 (stage1_sim500/)	pi05_ur3_sim_curvature_s48_pp	ur3-sim-cube-curvature-s48-d0-train	none
per-frame	1 (stage1_sim1000/)	pi05_ur3_sim2_time_s10_pp	ur3-sim2-time-s10-train	none
two-phase	1 (stage1_sim1000/)	pi05_ur3_sim2_two_phase_s48_pp	ur3-sim2-two-phase-s48-train	none
curvature	1 (stage1_sim1000/)	pi05_ur3_sim2_curvature_s48_pp	ur3-sim2-curvature-s48-d0-train	none
per-frame	2 (stage2_mix_sim1000_real/)	pi05_ur3_sim2_mix_time_s10_pp	pick-and-place-cube-quick-v4h5-time-s10-train	none
two-phase	2 (stage2_mix_sim1000_real/)	pi05_ur3_sim2_mix_two_phase_s48_pp	pick-and-place-cube-quick-v4h5-two-phase-s48-train	none
curvature	2 (stage2_mix_sim1000_real/)	pi05_ur3_sim2_mix_curvature_s48_pp	pick-and-place-cube-quick-v4h5-curvature-s48-d0-train	none
curvature, gated state input	2 (stage2_mix_sim1000_real/)	pi05_ur3_sim2_mix_curvature_s48_state_pp	pick-and-place-cube-quick-v4h5-curvature-s48-d0-train	--state-input
per-frame	2 (stage2_lora_real/)	pi05_ur3_sim_real_time_s10_pp	pick-and-place-cube-quick-v4h5-time-s10-train	--lora --init-checkpoint /unused
two-phase	2 (stage2_lora_real/)	pi05_ur3_sim_real_two_phase_s48_pp	pick-and-place-cube-quick-v4h5-two-phase-s48-train	--lora --init-checkpoint /unused
curvature	2 (stage2_lora_real/)	pi05_ur3_sim_real_curvature_s48_pp	pick-and-place-cube-quick-v4h5-curvature-s48-d0-train	--lora --init-checkpoint /unused
per-frame	real only (real_quick_new/)	pi05_ur3_quick_new_time_s10_state_pp	pick-and-place-quick-new-time-s10-train	--state-input
two-phase	real only (real_quick_new/)	pi05_ur3_quick_new_two_phase_s48_state_pp	pick-and-place-quick-new-two-phase-s48-train	--state-input
curvature	real only (real_quick_new/)	pi05_ur3_quick_new_curvature_s48_state_pp	pick-and-place-quick-new-curvature-s48-d0-train	--state-input
per-frame	one-stage mixture (joint_sim300_real_quick_new/)	pi05_ur3_joint_time_s10_state_pp	pick-and-place-quick-new-time-s10-train	--state-input
two-phase	one-stage mixture (joint_sim300_real_quick_new/)	pi05_ur3_joint_two_phase_s48_state_pp	pick-and-place-quick-new-two-phase-s48-train	--state-input
curvature	one-stage mixture (joint_sim300_real_quick_new/)	pi05_ur3_joint_curvature_s48_state_pp	pick-and-place-quick-new-curvature-s48-d0-train	--state-input
curvature	real only, after the camera adjustment (real_quick_new2/)	pi05_ur3_quick_new2_curvature_s48_state_pp	pick-and-place-quick-new2-curvature-s48-d0-train	--state-input
curvature	one-stage mixture, after the camera adjustment (joint_sim300_real_quick_new2/)	pi05_ur3_joint2_curvature_s48_state_pp	pick-and-place-quick-new2-curvature-s48-d0-train	--state-input

No --absolute-joints flag: all of them predict deltas. The gated-state policy of stage2_mix_sim1000_real/ and all policies of real_quick_new/, real_quick_new2/, joint_sim300_real_quick_new/ and joint_sim300_real_quick_new2/ take the state as input (--state-input); the others take none. Always pass --lora for a stage2_lora_real/ checkpoint, never for stage1_sim500/, stage1_sim1000/ or stage2_mix_sim1000_real/. Without it the server starts without an error, builds a model without adapters, and answers differently to the same input (the adapters are not applied); nothing in the log says so. The state policy of stage2_mix_sim1000_real/ must be served with --state-input (without it the prompt has no state tokens, the format of the other policies, and the answers differ); the state has to be sent as observation/state in both cases.

The two-phase and curvature policies do not output the next 10 control frames; their 10 points lie along the next 10–48 frames. The server fits a cubic spline from the observed joints through the points and re-times it with velocity timing (no joint faster than velocity_fraction × the UR3 limits, 3.14 rad/s for the base, shoulder and elbow and 6.28 rad/s for the wrists; never faster than cap × the recordings' pace; the arm holds still for 0.50 s after a gripper close and 0.52 s after an open), returning one absolute target per 30 Hz control frame. Execute the server's actions, never the raw points. Do not pass --uniform for these two policies; --uniform --cap 1 (points at fixed frame spacing) is meaningful for the per-frame policy only.

On a real arm:

• The teleoperation recordings never exceeded 2.5 rad/s; start with --velocity-fraction 0.5 and raise it after the first trials.
• The number of targets per response varies with the path; read len(actions) and execute at most that many, the first 4–8 before querying again with a fresh observation.
• Around a gripper switch the response repeats the same target for the hold time; keep sending those repeated targets, they are what makes the arm wait for the gripper. Close the right gripper when actions[:, 13] > 0.5.
• Send the measured joints as observation/state; the spline starts there, so a stale state makes the first targets jump. With asynchronous execution, drop the first d targets of a new response, d = latency × 30 frames.
• The left arm stayed parked in all stage-2 recordings. Keep it where it is (dimensions 0–6 only reproduce its parked pose).

Request and response

Client: openpi_client.websocket_client_policy.WebsocketClientPolicy(host, port).infer(obs).

key	value
observation/image	top camera, uint8 H×W×3 RGB (240×320 in the data)
observation/left_wrist_image	left wrist camera, uint8 H×W×3 (240×424)
observation/right_wrist_image	right wrist camera, uint8 H×W×3 (240×424)
observation/state	float32 (14): left joints 0–5 (rad), left gripper 6, right joints 7–12 (rad), right gripper 13; a gripper entry is the last gripper command sent (0 open, 1 closed), not the measured opening
prompt	English instruction, e.g. Put the pink cube on the white box.

Response: actions float32 (N, 14), one absolute target per 30 Hz control frame (actions[j] is the target for frame t + j + 1 after the observation); arc_points (10, 14), the model's points (absolute); arc_point_frames, the frames the points are placed at. The state is added to the predicted deltas by the server and is also the start of the spline.

The instructions do not name the arm; the model has to infer from the scene which arm moves. Name the cube's colour (pink, orange, blue), where it is (table, plate, white box, or "on the cube") and where it goes. Stage 2 saw only right-arm episodes; its behaviour on a scene where the left arm should move has not been evaluated. The stage1_sim1000/ policies saw both arms, including relays in which one arm hands the cube to the other.

Known limits

• The stage2_mix_sim1000_real/ policies are not evaluated here, neither open loop nor closed loop; the state policy has seen the state tokens only in the later part of its run.
• Stage 1 saw simulation only: no real image, joint trajectory or language annotation.
• Stage 2 saw 223 real episodes of one workcell, in which only the right arm moves.
• The idle arm's wrist-camera images in the simulation are bit-identical from frame to frame (492 runs of identical frames in the episodes, all on the idle arm), unlike a real camera.
• The mount side of the wrist cameras matches the real rig; the rest of the mount pose and the field of view come from the gripper model. The real wrist views frame the fingers differently: the fingertips appear only along the top edge, while in the simulation the gripper body fills the upper part of the image.
• Wrist-2 of both arms is almost constant in the simulation, so these dimensions carry little signal there (see above).
• The scripted motion is smoother and faster than teleoperation and contains no failures other than the deliberate late close followed by a re-grasp.
• Stage 2 raises the open-loop error on the simulated held-out episodes (table above), and the stage-2 policies did not perform the task on the real robot (see above).
• The policies of real_quick_new/ and joint_sim300_real_quick_new/ saw 62 real episodes of the moved workcell (one placement of the objects' start and goal region per episode, five held-out episodes not seen) and are not evaluated here; the simulated part of the mixture is faster than the real recordings (median episode 4.4 s against 9.1 s).
• The policy of real_quick_new2/ saw 72 real episodes of one placement of the workcell after the top camera was adjusted, all of them in training (no held-out episode), and is not evaluated here.
• The policy of joint_sim300_real_quick_new2/ saw the same 72 real episodes as real_quick_new2/ plus 300 simulated episodes, all of them in training (no held-out episode), and is not evaluated here.
• The stage1_sim1000/ policies have no open-loop or closed-loop evaluation yet; the only measurement reported for them is the training loss.