| # Input/output and frame contract |
|
|
| Robot IO reads both wrists and two proprio frames `[previous,current]`, one native model frame apart in executed progress. Wire poses are `[x,y,z,qx,qy,qz,qw]`; Flexiv native poses are wxyz and the IO handles that boundary. The dataset is a camera optical-frame pose, not automatically a robot flange pose. |
|
|
| For each hand, E = inverse(T_flange_camera): |
|
|
| - Model camera input: `T_world_camera = T_world_flange @ inverse(E)`. |
| - Relative training state: `inverse(T_current_camera) @ T_previous_camera`, current gripper. |
| - Relative target: `inverse(T_current_camera) @ T_future_camera`. |
| - Robot target: `T_current_camera @ T_predicted_relative @ E`. |
|
|
| 20 dimensions total: each hand uses translation3 + first-two-rotation-columns6 + gripper1. Raw pose buffers are16 dimensions, wxyz, metres, gripper degrees; they must be encoded. Anchor each chunk using the same current pose captured when its observation was made. No cross-hand common SLAM world frame is assumed by the training input. |
|
|
| FLUX uses32 steps /30Hz and both wrist images. π0.5 uses50 model steps with a32-D model head padded from20-D UMI targets; its adapter returns20-D actions and restores absolute camera poses, then flange poses. Optional upsampling changes sample count/dt; robot-side playback scaling changes wall time without redefining the model's previous/current state contract. Local servo cadence is separate from request cadence. |
|
|
| This sample's Observation/Action msgpack protocol is defined in `flexiv-umi-infer/umi_infer/protocol.py`. Generic OpenPI clients use a different wire envelope. Use the provided robot_agent and policy_server together. |
|
|
| ## Deployment processes |
|
|
| The GPU host runs `scripts/run_flux_server.sh` or `scripts/run_pi05_server.sh`. Both use the UMI `policy_server` entrypoint, selecting a FLUX policy-export loader or the OpenPI checkpoint loader. The wrappers under `flexiv-umi-infer/umi_infer/policies/` encode observations and restore robot-frame actions. The Flexiv host runs `scripts/run_robot.sh`; `robot_agent` only calls WebSocket inference and drives local RobotIO/servo execution. It does not load checkpoints. |
|
|
| For separate hosts, bind the service with `SERVER_HOST=0.0.0.0` and configure the robot's `SERVER_URI=ws://<GPU-host>:<port>`. Simultaneous backends need distinct ports. The generic OpenPI server's initial metadata handshake and msgpack-numpy payload differ from this UMI protocol; the bundled UMI wrapper is required for the supplied robot client. |
|
|
