Input/output and frame contract
Robot IO reads both wrists and two proprio frames [previous,current], one native model frame apart in executed progress. Wire poses are [x,y,z,qx,qy,qz,qw]; Flexiv native poses are wxyz and the IO handles that boundary. The dataset is a camera optical-frame pose, not automatically a robot flange pose.
For each hand, E = inverse(T_flange_camera):
- Model camera input:
T_world_camera = T_world_flange @ inverse(E). - Relative training state:
inverse(T_current_camera) @ T_previous_camera, current gripper. - Relative target:
inverse(T_current_camera) @ T_future_camera. - Robot target:
T_current_camera @ T_predicted_relative @ E.
20 dimensions total: each hand uses translation3 + first-two-rotation-columns6 + gripper1. Raw pose buffers are16 dimensions, wxyz, metres, gripper degrees; they must be encoded. Anchor each chunk using the same current pose captured when its observation was made. No cross-hand common SLAM world frame is assumed by the training input.
FLUX uses32 steps /30Hz and both wrist images. π0.5 uses50 model steps with a32-D model head padded from20-D UMI targets; its adapter returns20-D actions and restores absolute camera poses, then flange poses. Optional upsampling changes sample count/dt; robot-side playback scaling changes wall time without redefining the model's previous/current state contract. Local servo cadence is separate from request cadence.
This sample's Observation/Action msgpack protocol is defined in flexiv-umi-infer/umi_infer/protocol.py. Generic OpenPI clients use a different wire envelope. Use the provided robot_agent and policy_server together.
Deployment processes
The GPU host runs scripts/run_flux_server.sh or scripts/run_pi05_server.sh. Both use the UMI policy_server entrypoint, selecting a FLUX policy-export loader or the OpenPI checkpoint loader. The wrappers under flexiv-umi-infer/umi_infer/policies/ encode observations and restore robot-frame actions. The Flexiv host runs scripts/run_robot.sh; robot_agent only calls WebSocket inference and drives local RobotIO/servo execution. It does not load checkpoints.
For separate hosts, bind the service with SERVER_HOST=0.0.0.0 and configure the robot's SERVER_URI=ws://<GPU-host>:<port>. Simultaneous backends need distinct ports. The generic OpenPI server's initial metadata handshake and msgpack-numpy payload differ from this UMI protocol; the bundled UMI wrapper is required for the supplied robot client.