中文 / English
iUMI Inference · 双臂UMI推理与部署
Crobotia iUMI的模型推理与机器人部署代码，支持OpenPI（π0.5）和FLUX后端，包含配置模板、启动脚本及双臂名义URDF。
模型演示与 sample rollouts
本仓库提供 OpenPI π0.5 和 FLUX 的叠衣推理权重、部署源码、两段演示视频，以及每个后端各 1 条机器人执行样例。任务为 fold_the_clothes，硬件为 Flexiv Rizon 4 双臂与左右 ego 相机。
| 后端 | 演示视频（静音剪辑） | 完整 rollout | 帧数 / 30 Hz | 完整记录时长 |
|---|---|---|---|---|
| OpenPI π0.5 | 4分15.4秒 | openpi_pi05 | 6,540 | 218 秒 |
| FLUX | 4分29.7秒 | flux | 30,619 | 1,020.633 秒 |
OpenPI π0.5
FLUX
视频用于快速浏览，剪辑时间轴不与 rollout 逐帧对应。两条 rollout 合计 37,159 帧、约 20分38.6秒，保留左右两路 960×960 / 30 Hz AV1 视频、16 维状态/动作、关节数据、双侧 IMU、标定和机器人 H5；不包含音频或 rear 相机。
只下载视频和 rollout，无需下载约 21.37 GB 的模型文件：
pip install -U huggingface_hub
hf download Crobotia/iUMI_Inference --local-dir iUMI_Inference --include "demos/*" "rollout_data/*"
媒体与 rollout 合计约 2.35 GB。分别以 rollout_data/openpi_pi05/ 和 rollout_data/flux/ 为独立 LeRobot v3 数据集根目录；两者都从 episode 0 开始。见 rollout 读取说明 和 视频说明。这些文件存放在本模型仓库中，下载时使用默认 repo_type="model"。
模型用途与数据约定
两种后端用于双臂 UMI 衣物折叠的研究、离线接口集成和匹配设备上的部署验证。每次推理使用左右腕部图像、前一/当前位姿和任务文本，输出动作块，再由配套适配器完成坐标还原。
| 项目 | 约定 |
|---|---|
| 记录的状态 / 动作 | 16 维，左手在前；每手 [x, y, z, qw, qx, qy, qz, gripper]；米 / wxyz / 夹爪角度 |
| rollout 动作语义 | action_mode=next_state，action[t] = observation.state[t+1]（末帧除外），不是原始策略输出 |
| 模型内部 UMI 表示 | 两手合计 20 维；每手相对平移 3 + 旋转前两列 6 + 夹爪 1 |
| FLUX 动作块 | 32 步 / 30 Hz，使用双腕图像 |
| π0.5 动作块 | 50 个模型步；32 维模型头由 20 维 UMI 目标补齐，适配器还原为双臂目标位姿 |
记录位姿来源为 robot_fk_cad。相机光学坐标、机器人法兰坐标和模型相对表示需要显式转换，见 输入输出与坐标约定。
训练来源、验证与局限
发布的 FLUX 为 step 14418 的 BF16 推理导出；π0.5 为 pi05_umi_v3 的 epoch 2 checkpoint。完整训练语料清单、划分和精确运行配置未随本次新增材料提供；这两条 rollout 是机器人执行样例，不能据此反推训练集。
本次新增材料通过文件哈希、视频流/包计数、Parquet 索引与下一帧动作关系、IMU 和 H5 结构检查，详见 rollout 验证。每个模型只有 1 条样例，未建立评测协议或任务成功率，也未运行完整运动质检。演示记录的确切 checkpoint 哈希、采样配置和剪辑到数据的逐帧映射未随材料提供，因此不能把当前启动脚本默认值视为视频的精确复现配方。
现有权重与源码的验证记录仍见 验证范围；此次上传没有重新运行 GPU 推理或机器人。实机部署需要匹配的编码器、厂商 SDK、设备标定、工作空间限制与现场验证。
获取代码
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/Crobotia/iUMI_Inference
cd iUMI_Inference
| 目录 | 内容 |
|---|---|
| flexiv-umi-infer/ | 策略服务、机器人/相机/夹爪接口及数据记录 |
| flux-umi/ | FLUX模型与UMI数据表示 |
| openpi-umi/ | OpenPI模型、数据变换与客户端 |
| scripts/ | 模型服务和机器人端启动脚本 |
| configs/ | 推理配置模板 |
| checkpoints/ | FLUX BF16 和 π0.5 叠衣推理权重 |
| demos/ | 两段静音叠衣演示视频 |
| rollout_data/ | OpenPI π0.5 / FLUX 各 1 条完整执行记录 |
OpenPI使用Linux（glibc ≥2.31）；按openpi-umi/.python-version配置Python3.12，在该目录运行uv sync --frozen --python 3.12。
FLUX使用scripts/run_flux_server.sh，OpenPI使用scripts/run_pi05_server.sh，机器人端使用scripts/run_robot.sh。两类模型使用独立环境，按各子项目说明安装依赖。启动前配置模型资源路径及设备标定；机器人脚本默认为dry run。
运行前准备所需编码器、厂商SDK及设备标定。输入输出约定见docs/INFERENCE_CONTRACT.md，安装与设备配置见各子项目README。
服务部署关系
flowchart LR
Robot["Flexiv 主机：robot_agent / 相机 / 本地伺服"] <-->|"WebSocket Observation / Action"| Server["GPU 主机：UMI policy_server"]
Server --> Flux["FLUX 后端：flux-umi + FLUX checkpoint"]
Server --> Pi["π0.5 后端：openpi-umi + OpenPI checkpoint"]
每个服务进程选择一个后端。run_flux_server.sh 和 run_pi05_server.sh 都启动 umi_infer.policy_server，分别调用对应的模型加载器；UMI 服务包装层位于 flexiv-umi-infer/。机器人端只运行 robot_agent，通过接口请求推理，不加载权重。模型输入编码、归一化和动作坐标还原在服务包装层执行；伺服执行在 Flexiv 主机上进行。
跨机器部署时，GPU 主机设置 SERVER_HOST=0.0.0.0、SERVER_PORT=8765；Flexiv 主机设置 SERVER_URI=ws://<GPU主机地址>:8765，再运行 bash scripts/run_robot.sh。同时部署两种模型时使用不同端口。设备标定路径 UMI_INFER_DATA 应在对应主机可访问；机器人脚本默认为 dry run。
请使用这套匹配的服务／客户端。OpenPI 原生 WebSocket 服务有不同的握手和数据封装，不能直接连接此 robot_agent。
下载权重与启动
两份权重已放在本仓库的 checkpoints/ 下，共约 21.37 GB。在仓库根目录下载（需要 Hugging Face CLI）：
pip install -U huggingface_hub
hf download Crobotia/iUMI_Inference --local-dir . --include "checkpoints/*"
| 后端 | 启动脚本默认权重目录 | 模型文件大小 |
|---|---|---|
| FLUX | checkpoints/flux_fold_clothes_bf16/ |
13,894,466,592 B |
| π0.5 | checkpoints/pi05_fold_clothes/epoch_2/ |
7,473,091,464 B |
启动脚本自动使用上述目录；也可用 FLUX_EXPORT_DIR 或 PI05_CHECKPOINT_DIR 指定其他位置。按后端安装依赖后，复制配置并填写资源、Python 环境和设备标定路径：
cp configs/inference.env.example .env
# 编辑 .env：FLUX_RESOURCES、TOKENIZER_DIR、UMI_INFER_DATA 及 Python 路径
set -a
source .env
set +a
bash scripts/run_flux_server.sh
# 或在 π0.5 环境中运行：bash scripts/run_pi05_server.sh
FLUX 仍需单独准备 FLUX_RESOURCES/video_vae.safetensors 和 FLUX_RESOURCES/text_encoder/；π0.5 仍需 TOKENIZER_DIR/paligemma-3b-pt-224/tokenizer.model。这些外部资源未包含在此次权重发布中。归一化参数随权重提供，启动时自动加载对应文件。
FLUX 导出未保存采样参数。当前脚本使用 Euler、4 步、shift 42、video/action guidance 1.0；这是可覆盖的脚本默认值，尚未验证叠衣效果。π0.5 默认 10 步去噪，VLA_QUAT_PERM=0 对应该权重的 xyz_quat_wxyz 配置。
权重已完成下载校验、文件结构及脱敏检查；CPU 检查不代表 GPU 推理或实机叠衣验证。详情见 FLUX 权重说明、π0.5 权重说明 和 验证范围。
iUMI Inference · Bimanual UMI Inference and Deployment
Model inference and robot deployment code for Crobotia iUMI, with OpenPI (π0.5) and FLUX backends, configuration templates, launch scripts and a nominal dual-arm URDF.
Demos and sample rollouts
This repository includes OpenPI π0.5 and FLUX cloth-folding inference checkpoints, deployment code, two demo videos and one robot rollout per backend. The task is fold_the_clothes on a dual-arm Flexiv Rizon 4 setup with left/right ego cameras.
| Backend | Muted presentation edit | Complete rollout | Frames at 30 Hz | Recording duration |
|---|---|---|---|---|
| OpenPI π0.5 | 4 min 15.4 sec | openpi_pi05 | 6,540 | 218 sec |
| FLUX | 4 min 29.7 sec | flux | 30,619 | 1,020.633 sec |
The embedded previews are above. Presentation edits have their own timelines; they are not frame-aligned to the datasets. The two rollouts contain 37,159 frames / 20 min 38.6 sec in total. Each retains left/right 960×960 / 30 Hz AV1 video, 16-D states/actions, joint data, bilateral IMU, calibration and robot H5 records. Audio and rear-camera views are excluded.
Download only the demos and rollouts, without the approximately 21.37 GB of model files:
pip install -U huggingface_hub
hf download Crobotia/iUMI_Inference --local-dir iUMI_Inference --include "demos/*" "rollout_data/*"
The media and rollout files total approximately 2.35 GB. Read rollout_data/openpi_pi05/ and rollout_data/flux/ as separate LeRobot v3 roots; both start at episode 0. See the rollout guide and demo guide. These assets live in this model repository; downloads use the default repo_type="model".
Intended use and model interface
The backends support bimanual UMI cloth-folding research, offline integration and deployment validation on a matching setup. Inference consumes both wrist images, previous/current poses and task text. The matching adapter restores predicted action chunks to robot target poses.
| Item | Convention |
|---|---|
| Recorded state / action | 16-D, left then right; each hand [x, y, z, qw, qx, qy, qz, gripper]; metres / wxyz / gripper degrees |
| Rollout action meaning | action_mode=next_state: action[t] = observation.state[t+1], excluding the last frame; not raw policy predictions |
| Internal UMI representation | 20-D total: translation 3 + first two rotation columns 6 + gripper 1 per hand |
| FLUX action chunk | 32 steps at 30 Hz, using both wrist images |
| π0.5 action chunk | 50 model steps; a 32-D model head padded from 20-D UMI targets, restored to dual-arm target poses by the adapter |
Recorded pose provenance is robot_fk_cad. Camera optical poses, robot flange poses and model-relative representations require explicit conversion. See the frame and input/output contract.
Training provenance, evaluation and limitations
The released FLUX checkpoint is the step-14418 BF16 inference export; π0.5 is the epoch-2 pi05_umi_v3 checkpoint. A complete training corpus inventory, split definition and exact run configuration are not supplied with these added assets. The two rollouts are execution examples and do not identify the training set.
The new assets pass file-hash checks, video stream/packet counts, Parquet index and next-state checks, IMU checks and H5 structure inspection; see rollout validation. There is one example per model, with no benchmark protocol, task-success measurement or complete motion QC. Exact checkpoint hashes and sampling settings used for the recordings, and frame mappings between presentation edits and datasets, were not supplied. Current launcher defaults therefore do not constitute an exact reproduction recipe for the videos.
Existing checkpoint/source validation remains documented in validation scope. This upload does not rerun GPU inference or the robot. Physical deployment requires matching encoder assets, vendor SDKs, device calibration, workspace constraints and on-site validation.
Get the code
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/Crobotia/iUMI_Inference
cd iUMI_Inference
| Directory | Purpose |
|---|---|
| flexiv-umi-infer/ | Policy service, robot/camera/gripper interfaces and recording |
| flux-umi/ | FLUX models and UMI representation |
| openpi-umi/ | OpenPI models, data transforms and client |
| scripts/ | Policy-server and robot-agent launchers |
| configs/ | Inference configuration templates |
| checkpoints/ | FLUX BF16 and π0.5 cloth-folding inference weights |
| demos/ | Two muted cloth-folding demo videos |
| rollout_data/ | One complete execution recording per backend |
For OpenPI, use Linux (glibc ≥2.31) and Python 3.12 as pinned in openpi-umi/.python-version and run uv sync --frozen --python 3.12 from that directory.
Launch FLUX with scripts/run_flux_server.sh, OpenPI with scripts/run_pi05_server.sh, and the robot agent with scripts/run_robot.sh. Use separate environments for the model backends and follow each project's dependency instructions. Configure model resource paths and cell calibration before launch. The robot script defaults to dry run.
Before running, prepare required encoders, vendor SDKs and cell calibration. See docs/INFERENCE_CONTRACT.md for input/output conventions and the subproject READMEs for installation and device configuration.
Service deployment
Each service process selects one backend. Both launchers run umi_infer.policy_server: FLUX loads through flux-umi, while π0.5 loads through openpi-umi. The UMI service wrappers live under flexiv-umi-infer/. On the Flexiv host, robot_agent sends observations and receives actions over WebSocket without loading model weights. Input encoding, normalization and action-frame restoration run in the server wrapper; servo execution runs locally on the robot host.
For separate hosts, set SERVER_HOST=0.0.0.0 and SERVER_PORT=8765 on the GPU host, then set SERVER_URI=ws://<GPU-host>:8765 on the Flexiv host and run bash scripts/run_robot.sh. Use different ports for simultaneous FLUX and π0.5 servers. Make UMI_INFER_DATA calibration files available on the relevant hosts. The robot launcher defaults to dry run.
Use the matching UMI server/client pair. The native OpenPI WebSocket service uses a different handshake and message envelope and cannot directly serve this robot_agent.
Download weights and launch
Both inference checkpoints are included under checkpoints/ (about 21.37 GB total). From the repository root:
pip install -U huggingface_hub
hf download Crobotia/iUMI_Inference --local-dir . --include "checkpoints/*"
| Backend | Default checkpoint directory | Model-file size |
|---|---|---|
| FLUX | checkpoints/flux_fold_clothes_bf16/ |
13,894,466,592 B |
| π0.5 | checkpoints/pi05_fold_clothes/epoch_2/ |
7,473,091,464 B |
The launchers use these directories automatically; override FLUX_EXPORT_DIR or PI05_CHECKPOINT_DIR if needed. Install the backend dependencies, then copy and edit the configuration:
cp configs/inference.env.example .env
# Set FLUX_RESOURCES, TOKENIZER_DIR, UMI_INFER_DATA and the Python executable paths.
set -a
source .env
set +a
bash scripts/run_flux_server.sh
# Or from the π0.5 environment: bash scripts/run_pi05_server.sh
Provide FLUX_RESOURCES/video_vae.safetensors and FLUX_RESOURCES/text_encoder/ separately. π0.5 requires TOKENIZER_DIR/paligemma-3b-pt-224/tokenizer.model. These external resources are not included in this weight release. Checkpoint-specific normalization statistics are included and loaded automatically.
The FLUX export has no saved sampling preset. The launcher uses Euler, 4 steps, shift 42 and video/action guidance 1.0; these are adjustable launcher defaults, with cloth-folding quality still unverified. π0.5 uses 10 denoising steps by default. Keep VLA_QUAT_PERM=0 for this checkpoint's xyz_quat_wxyz pose configuration.
Download integrity, file structure and sanitization have been checked. CPU checks do not establish GPU inference or physical-robot performance. See FLUX, π0.5 and validation.
许可证 / License
源码采用Apache-2.0许可，各子项目保留原有声明；Gemma、模型资源及厂商SDK遵循各自条款。详见LICENSES.md。
Source code is licensed under Apache-2.0, with upstream notices retained in each component. Gemma, model resources and vendor SDKs retain their respective terms. See LICENSES.md.
新增 demo 与 rollout 未附独立的媒体/数据许可声明；根目录 Apache-2.0 标签用于源码，不自动授予这些媒体、数据或模型资源的相同许可。使用或再分发前请向 Crobotia 确认适用条款；不将其他人工演示数据集的 CC BY-SA 声明套用于本次材料。
No separate media/data license declaration accompanies the added demos and rollouts. The root Apache-2.0 tag describes source code and does not automatically license media, data or model resources on the same terms. Confirm applicable use and redistribution terms with Crobotia; licenses of separate human-demonstration datasets are not applied here.