Fix 5 issues found in a fresh RTX 5090 rehearsal: git-lfs clone note; pinned snapshot_download in tools/prep_pi05_data.sh; install with [pi,dataset]; content-based dataset check (tools/verify_dataset_v30.py); image-only client-path PASS with chunk advisory, tolerance = 3 x max(H200 cross-precision, RTX 5090 measured) + deploy/tolerance.json + TOLERANCE_CHECK; RTX 5090 latency

HANDOVER.md

@@ -17,7 +17,7 @@
 ## 3. 上真机的顺序
 
 1. **回放测试**(`deploy/replay_demo.py`,与模型无关,`--dataset-root` 用你自己转换出来的目录):把一局示教的关节轨迹按 20 Hz 原样下发,看能否复现示教、夹爪停在约 −2.0 时夹不夹得住杯子、控制器滞后几帧。这一步不过,先查硬件和指令语义。
-2. **复现检查**(`deploy/fit_check_pi05_cups.py`):在你的 5090 上对 `deploy/expected/` 里的期望数字,按 `deploy/TOLERANCE.md` 判过不过;**一定要用你自己的「相机帧 → 张量」代码跑一遍客户端路径测试**(`--client-fn 你的模块:函数`)。
+2. **复现检查**(`deploy/fit_check_pi05_cups.py`):在你的 5090 上对 `deploy/expected/` 里的期望数字,按 `deploy/TOLERANCE.md` 判过不过(脚本末尾直接打印 `TOLERANCE_CHECK PASS/FAIL`);**一定要用你自己的「相机帧 → 张量」代码跑一遍客户端路径测试**(`--client-fn 你的模块:函数`)。这项测试只看图像是否一致;两条路径的块差只作提示,5090 上两次前向不逐位相同,块差不为 0 是正常的。
 3. **第一次上机**:先悬空或低速,只打印一块 50 步动作,核对关节范围、顺序、夹爪方向都对了再正式执行;确认急停可用。
 4. **按执行规则上机**:20 Hz;第 0 步约等于当前位置,跳过;执行第 1–25 步后重新推理;块起点渐入默认开(前 k = 5 步从实测关节线性过渡到预测),开 / 关可以对比。
 5. **训推一致性检查**(`deploy/rollout_consistency.py`):每次推理和每个控制步都记下来,跑完和离线参照对比(见第 7 节)。
@@ -32,7 +32,7 @@
 
 ## 5. 推理资源
 
-- bf16 推理显存约 9 GB;H200 上一次推理(出一整块 50 步)约 275 ms。5090 上的耗时请帮我们实测(见第 8 节)。
+- bf16 推理:我们在 RTX 5090 上实测一次推理(出一整块 50 步)中位 204 ms、最大 206 ms,显存峰值约 8.9 GB(nvidia-smi 10.6 GB,含 CUDA 上下文),同步循环(推理 + 执行 25 步)每块约 1.45 s;H200 上约 275 ms。你那台 5090 的实测也请发我们(见第 8 节)。
 
 ## 6. 离线数字(训练局拟合,不代表成功率)
 
@@ -45,7 +45,7 @@
 | 15k | 0.0074 rad | 0.0035 rad | 0.49 | 否 |
 | 20k | 0.0064 rad | 0.0034 rad | 0.47 | 是([仓](https://huggingface.co/spikefly/pi05-arx-stack-three-cups-step20k)) |
 
-推理延迟:H200 上 275 ms / 次,显存峰值 8.9 GB(2 万步档实测,各档结构相同)。
+推理延迟:H200 上 275 ms / 次(2 万步档)、RTX 5090 上 204 ms / 次(1 万步档);显存峰值都是约 8.9 GB(两档结构相同)。
 
 ## 7. 遇到问题先看哪几项
 

README.md

@@ -15,6 +15,8 @@ datasets:
 
 # π0.5 · ARX Lift2S 双臂叠三个杯子(1 万步存档)
 
+> **v3(10-09)**:修正数据下载 / 安装 / 校验 / 客户端判据,加 5090 实测;权重没变,已经下载过的请重新拉取 `README.md`、`HANDOVER.md`、`deploy/`、`tools/`。
+
 在 [`yaoxianze/arx_external_cam_cups`](https://huggingface.co/datasets/yaoxianze/arx_external_cam_cups)(50 局示教)上,从 [`lerobot/pi05_base`](https://huggingface.co/lerobot/pi05_base) 全参数微调 2 万步(全局批 32)的 π0.5,本仓是第 1 万步的存档。同一次训练的另一档:[`spikefly/pi05-arx-stack-three-cups-step20k`](https://huggingface.co/spikefly/pi05-arx-stack-three-cups-step20k)(2 万步)。
 本仓只有模型和部署工具,**不含数据**;复现检查和回放测试要用的数据按第 3 节从原数据仓下载、自己转换。来龙去脉和我们想要的反馈见 `HANDOVER.md`。
 
@@ -37,10 +39,14 @@ conda create -n pi05 python=3.12 -y && conda activate pi05
 pip install torch==2.7.1 torchvision==0.22.1 --index-url https://download.pytorch.org/whl/cu128
 git clone https://github.com/huggingface/lerobot.git && cd lerobot
 git checkout 89236ea0f4f81a81ca566081e20dd1ff5f823cbe
-pip install -e ".[pi]"
+pip install -e ".[pi,dataset]"
 ```
 
+> clone 卡在 git-lfs 上时:最常见的是机器上只配置了 lfs 过滤器、没装 git-lfs,用 `git -c filter.lfs.process= -c filter.lfs.smudge= -c filter.lfs.required=false clone https://github.com/huggingface/lerobot.git`;装了 git-lfs 的可以用 `GIT_LFS_SKIP_SMUDGE=1 git clone https://github.com/huggingface/lerobot.git`。推理不需要 LFS 文件。
+
 - 推理只要上游 lerobot 这个提交,不需要任何改动。
+- `[dataset]` 带 PyAV、torchcodec、jsonlines、datasets、pandas、pyarrow;只装 `[pi]` 的话,数据转换会因缺 jsonlines 直接 ImportError,环境里也没有 PyAV。
+- 看到 torchcodec 加载失败的提示可以忽略,复现检查固定用 pyav。
 - 不需要登录 HF:本仓和原数据仓都公开;分词器在 `tokenizer/` 里,不用另下 `google/paligemma`。
 - 我们训练和出期望值用的是 Python 3.12.14、torch 2.7.1(CUDA 12.6)、torchvision 0.22.1、transformers 5.5.4、PyAV 15.1.0、NVIDIA H200。版本不同,只要复现检查在容差内就行。
 
@@ -54,17 +60,19 @@ hf download spikefly/pi05-arx-stack-three-cups-step10k --local-dir pi05_step10k
 
 ## 3. 准备数据(复现检查和回放测试要用;推理不需要)
 
-从原数据仓(版本 `2d87a09`)下载,用仓里的 `tools/` 转成训练时用的 LeRobot v3.0 格式,再逐文件核对:
+从原数据仓下载**固定版本 `2d87a09`**(我们训练用的版本),用仓里的 `tools/` 转成训练时用的 LeRobot v3.0 格式,并自动校验:
 
 ```bash
 cd pi05_step10k
-REPO_DIR=$PWD
-bash tools/prep_pi05_data.sh ~/arx_data            # 只下 meta/、data/、videos/(约 1.1 GB);已有同版本的原始数据,可以作第二个参数传入,就不再下载
+bash tools/prep_pi05_data.sh ~/arx_data            # 只下 meta/、data/、videos/(约 1.15 GB),转换后自动跑校验;已有同版本的原始数据,可以作第二个参数传入,就不再下载
 DATA=~/arx_data/arx_cups_pi05                      # 你自己转换出来的目录,后面都用它
-(cd "$DATA" && sha256sum -c "$REPO_DIR/tools/dataset_v30_SHA256SUMS.txt")
+python tools/verify_dataset_v30.py "$DATA"         # 想单独再校验一次也可以
 ```
 
-我们用这套命令(上游 lerobot `89236ea`)从原数据仓 `2d87a09` 重建过一遍:13 个文件全部 OK,和训练用的数据逐字节一致;在重建的数据上跑第 4 节的复现检查,数字和 `deploy/expected/` 完全相同。`data/` 和 `videos/` 必须全部 OK;`meta/stats.json` 和 `meta/episodes/` 里有抽样算的统计量,换了 lerobot 版本可能对不上,这不影响推理和复现检查(归一化参数在本仓的处理器文件里)。
+- 下载后脚本会按目录数文件(meta 5、data 50、videos 200),和这个版本应有的数目对不上就停下。
+- 校验规则:**视频按字节比**(8 个 mp4 的 sha256,和 `tools/dataset_v30_SHA256SUMS.txt` 一致);**parquet 按内容比**(行数、列名、列类型、逐列内容哈希,参考值在 `tools/dataset_v30_reference.json`)。不同 pyarrow 版本写出的 parquet 字节会不同(文件头记录的写入版本不一样),这不算错。`meta/info.json`、`meta/stats.json` 只作提示,对不上不影响推理和复现检查。最后一行是 `VERIFY_DATASET_V30 PASS` 就行。
+
+- 我们用这套工具(上游 lerobot `89236ea`)从原数据仓 `2d87a09` 重建过:视频和 parquet 全部和训练用的一致,在重建的数据上跑第 4 节的复现检查,数字和 `deploy/expected/` 完全相同;parquet 的内容哈希在 pyarrow 25 和 26 下一致。
 
 ## 4. 复现检查(上真机前必须过)
 
@@ -73,9 +81,10 @@ python deploy/fit_check_pi05_cups.py --checkpoint . --dataset-root "$DATA" --out
     --episode-indices 0 25 --dtype bfloat16 --compare-to deploy/expected/action_fit_summary_bf16.json
 ```
 
-- 输出 `fitcheck_bf16/action_fit_summary.json`,��照 `deploy/TOLERANCE.md`:期望块最大差在容差内、拟合误差和块起点偏移的相对差 ≤ 10%、`client_path_test.pass` 为 true。
+- 输出 `fitcheck_bf16/action_fit_summary.json`;脚本会读 `deploy/tolerance.json`(和 `deploy/TOLERANCE.md` 同一组数)逐项判,末尾打印 `TOLERANCE_CHECK PASS/FAIL`:期望块最大差在容差内(执行段 ≤ 0.0509 rad、整块 ≤ 0.1216 rad;容差 = 3 × max(H200 跨精度差, 5090 实测差))、拟合误差和块起点偏移的相对差 ≤ 10%、`client_path_test.pass` 为 true(每帧图像差 ≤ 1/255)。
 - **再用你自己机器人端的取图代码跑一遍**:加 `--client-fn 你的模块:函数 --client-frame-format bgr`(或 `rgb`;模块要在 PYTHONPATH 上)。函数收相机原样交出的 uint8 HxWx3 帧,返回进模型的 float (3,H,W),取值 [0,1];模板 `deploy/example_client_fn.py`。BGR / RGB、缩放、裁剪接错,只有这样才抓得到。
-- 期望块用 CPU 上固定种子的噪声生成,所以换一块显卡也能逐块对比;5090 和 H200 之间有一点数值差,在容差内就正常。
+- 客户端路径测试的关卡只看图像:每帧图像差 ≤ 1/255 才算 PASS。两条路径预测出的块差只作提示(`client_path_test.chunk_advisory`,超过执行段容差才打 WARN):部分显卡上两次前向即使输入完全相同也不逐位相同(我们在 5090 上看到最大约 0.03 rad),块差不为 0 是正常的。想逐位复现可以加 `--deterministic`(默认关;会变慢,不支持确定性的算子只告警)。
+- 期望块用 CPU 上固定种子的噪声生成,所以换一块显卡也能逐块对比;5090 和 H200 之间有一点数值差,在容差内就正常(我们在 5090 上实测本档:执行段差 0.012–0.015 rad、整块差 0.037–0.041 rad)。
 
 ## 5. 回放测试(任何策略之前,与模型无关)
 
@@ -95,7 +104,7 @@ python deploy/replay_demo.py analyze --commands replay_ep000_commands.csv --meas
 - **执行**:20 Hz;第 0 步约等于「现在的位置」,不执行;执行第 1–25 步,然后重新推理。
 - **块起点渐入**:默认开——前 5 步(k 可调)从实测关节线性过渡到预测轨迹,缓解每次重推的起点跳变;开 / 关在真机上对比。
 - **夹爪**:直接下发模型输出值。
-- **资源**:bf16 推理显存约 9 GB;H200 上一次推理约 0.28 s。
+- **资源**(bf16):RTX 5090 上我们实测一次推理(出一整块 50 步)中位 204 ms、最大 206 ms,显存峰值约 8.9 GB(nvidia-smi 看到 10.6 GB,含 CUDA 上下文),同步循环(推理 + 执行 25 步)每块约 1.45 s;H200 上一次推理约 0.28 s。
 - 详细契约见 `deploy/IO_CONTRACT.md`。最小推理代码:
 
 ```python
@@ -154,13 +163,14 @@ python deploy/rollout_consistency.py run_*.jsonl --reference deploy/expected/act
 | 现象 | 多半是 | 怎么查 |
 |---|---|---|
 | 复现检查期望块差远超容差,数值接近 TOLERANCE.md 里的「送 BGR 时」 | 颜色通道接反(OpenCV 取图是 BGR) | 用 `--client-fn` 跑客户端路径测试 |
-| 复现检查不过,但客户端路径测试正常 | 数据没按第 3 节转换,或转换结果和我们的不一致 | 看 `dataset_v30_SHA256SUMS.txt` 的核对结果 |
+| 复现检查不过,但客户端路径测试正常 | 数据没按第 3 节转换,或转换结果和我们的不一致 | 看 `python tools/verify_dataset_v30.py "$DATA"` 的结果 |
 | 真机上块起点偏移是离线的好几倍 | 相机机位 / 曝光 / 光照变了,或三路相机接错位置 | 逐路存一帧图,和你转换出来的数据里的帧对比 |
 | 动作关节错位、方向反 | state 顺序或单位不对 | 打印 state,和你转换出来的数据里同一姿态的一帧对比 |
 | 动作幅度明显不对 | 客户端自己做了归一化(处理器里已经有) | 只送原始读数和 [0,1] 图像 |
 | 动作乱、不像在叠杯子 | 提示词不是逐字 `stack three cups` | |
 | 每块开头一跳 | 执行了第 0 步,或没开渐入 | 见第 6 节 |
 | 5090 上和期望值有小差 | 显卡 / 内核数值差 | bf16 下在容差内即正常 |
+| 客户端路径测试 PASS,但打了块差 WARN | 显卡内核非确定性(两次前向不逐位相同) | 图像差为 0 就不是取图的问题;想排除可加 `--deterministic` 再跑 |
 | `CUDA error: no kernel image is available` | torch 不是 CUDA 12.8 版 | 见第 1 节 |
 | 跟踪误差大、块起点偏移正常 | 控制器滞后 | 和回放测试的跟踪误差比 |
 
@@ -181,8 +191,8 @@ python deploy/rollout_consistency.py run_*.jsonl --reference deploy/expected/act
 | `model.safetensors`、`config.json` | π0.5 权重与配置 |
 | `policy_preprocessor*`、`policy_postprocessor*` | 相机改名、分位数归一化 / 反归一化参数 |
 | `tokenizer/` | 分词器 |
-| `deploy/` | 部署契约、容差、复现检查、客户端模板、回放测试、训推一致性工具、期望结果 |
-| `tools/` | 从源数据集重建 v3.0 数据的脚本(复现检查 / 回放测试要用),以及训练数据的逐文件校验和 |
+| `deploy/` | 部署契约、容差(`TOLERANCE.md` 和机器可读的 `tolerance.json`)、复现检���、客户端模板、回放测试、训推一致性工具、期望结果 |
+| `tools/` | 从源数据集重建 v3.0 数据的脚本(复现检查 / 回放测试要用);校验脚本 `verify_dataset_v30.py` 和参考值(`dataset_v30_reference.json`、`dataset_v30_SHA256SUMS.txt`) |
 | `README.md`、`HANDOVER.md` | 用法;来龙去脉与反馈 |
 | `SHA256SUMS` | 全部文件的校验和(`sha256sum -c SHA256SUMS`) |
 

SHA256SUMS

@@ -1,13 +1,14 @@
-f6a0f32b77f37f2fc29acff6a393f6cdbdc358abb8769e042da772bd441134da  ./HANDOVER.md
-0d0054e43b36ef83e80af1512139f59f67405d1287098be9b1cf3d1d1c866c02  ./README.md
+e71a5016382a71cf7a2de7450532097fb26cc2dccfd507647f5afcaafd24f406  ./HANDOVER.md
+dddc94ae240fd71a975d47ed22311e8ded9293d93ae37cf1a2756bfa1dfcd491  ./README.md
 d8580e6231b939328fb2280bea215f7228d0265ff41f33ce123525497c0aa7d5  ./config.json
-67552ba0c2bfdfec1b6ad292ee737c8ace8cb19eca9453f470c397c5518c7da9  ./deploy/IO_CONTRACT.md
-aa6f20df8665adb6f6feb4391dcb2dbcfa3cfc6fea14240a7730956da766983a  ./deploy/TOLERANCE.md
+28ddf474a5f5f93a404180fa4566a36976a386b893fd4abccdc1d82e489cb783  ./deploy/IO_CONTRACT.md
+f8576cd88f38a9d40782ede954e02f34f6d761c371f75b12037de69c1ee0a0dd  ./deploy/TOLERANCE.md
 bf1ac702571e77eed11a322f483491c6b23db4d183d31fa2186fa8f883adef26  ./deploy/example_client_fn.py
 abc75c6cf28a953f0608e148d4888d2e0289771b52a64ea578517c9e0030d099  ./deploy/expected/action_fit_summary_bf16.json
-8a1c11a28b64dfff70784867fbc67b522de728af5908696a56cfcda81bd61bee  ./deploy/fit_check_pi05_cups.py
+7066a5e01a29ba0e1318e541bbdfdafe99f8c3a106cc7ec4e583dc1eff4558cb  ./deploy/fit_check_pi05_cups.py
 bcac934f413fbda13e65fd9d9823a1bd81a084470c8815bddea164489004143a  ./deploy/replay_demo.py
 4457cefef028d50456004485c8455a61ad5e666149da4487c7d786006616a6e5  ./deploy/rollout_consistency.py
+75d6ca9db657889c67162b8eb819bc0cb82b2531021325d24738b9f2c4e4d6f7  ./deploy/tolerance.json
 6a91a92122563b9b117c937f4e42ea6f0af0b1f0f74e760157c383312a15c8b9  ./model.safetensors
 03af86cc58ef3893701df4d2fe8cd9dbd333fddfb9c0b69c7642ea1a9d984b1d  ./policy_postprocessor.json
 1b9e8e5e082d4e6c81d03d2c46136abd1271699ebf42fc11695e06316665bceb  ./policy_postprocessor_step_0_unnormalizer_processor.safetensors
@@ -16,5 +17,7 @@ bcac934f413fbda13e65fd9d9823a1bd81a084470c8815bddea164489004143a  ./deploy/repla
 7336bb4ce016bf8543de94e3a17c3d6ca502a6fc8628fec6ed88893639d7a296  ./tokenizer/tokenizer.json
 d36fcc5eed92f7b30976de93450dcb6229663e13c41be1f59ae54f2f20c9cc24  ./tokenizer/tokenizer_config.json
 906efb730220625609756c96ed6f08af7793643b00b559ace391efb6c9e49226  ./tools/dataset_v30_SHA256SUMS.txt
-ed9071e82b3a3992ec5d910a18b4dd04a773c340a13d5b24ae5949fd3511232a  ./tools/prep_pi05_data.sh
+6afaf67b01979a579fe26fb26f554f255266debb0cb901bd7ca780cc5673759e  ./tools/dataset_v30_reference.json
+42bbe62e159a47a4be04168df41dbbcb25d2fb958eee039d645434cdfac9ef89  ./tools/prep_pi05_data.sh
 0dfe405e9fcf2e13249633e9cc53ffdfa92800b694819f25bb608ef176150c8f  ./tools/quantile_stats_offline.py
+9fb5d685dbf78808b14781633a3da17f62533a9380d90df4e0a56276b0a71478  ./tools/verify_dataset_v30.py

deploy/IO_CONTRACT.md

@@ -2,7 +2,7 @@
 
 > 训练:LeRobot π0.5,上游 [huggingface/lerobot](https://github.com/huggingface/lerobot) 提交 `89236ea0f4f81a81ca566081e20dd1ff5f823cbe`(训练时另有两个只管存档的本地开关,与推理无关);底座 [`lerobot/pi05_base`](https://huggingface.co/lerobot/pi05_base);数据 [`yaoxianze/arx_external_cam_cups`](https://huggingface.co/datasets/yaoxianze/arx_external_cam_cups)(版本 `2d87a09`)转 LeRobot v3.0(转换与分位数统计脚本见 `tools/`);全局批 32、2 万步;绝对关节;分位数归一化。
 > 仓根 = 训练落盘的 `pretrained_model/`,`PI05Policy.from_pretrained(<仓名或本地目录>)` 直接可用。前后处理器(`policy_preprocessor*`、`policy_postprocessor*`)已经含相机改名和分位数归一化 / 反归一化参数,**客户端不要自己归一化**。
-> **推理代码**:上游 lerobot 提交 `89236ea` 即可(clone 后 `git checkout 89236ea && pip install -e ".[pi]"`)。
+> **推理代码**:上游 lerobot 提交 `89236ea` 即可(clone 后 `git checkout 89236ea && pip install -e ".[pi,dataset]"`;`[dataset]` 是复现检查和数据转换要用的 PyAV、jsonlines 等)。
 
 ## 输入(每次推理一帧)
 
@@ -34,7 +34,7 @@
 ## 上真机前
 
 0. **先做回放测试(与模型无关,任何策略之前)**:`python deploy/replay_demo.py export --dataset-root <数据目录> --episode 0 --out replay_ep000_commands.csv`(数据目录 = 按 README 第 3 节用 `tools/prep_pi05_data.sh` 转换出来的目录;本仓不含数据),按第 0 局的布局摆好杯子,机械臂**慢速**移到第 0 行,再按 20 Hz 逐行下发,记录实测关节(`RolloutLogger.state` 或 CSV)和视频;然后 `python deploy/replay_demo.py analyze --commands … --measured …`。它回答三件事:「拿实测位置当指令」能否复现示教;夹爪停在约 −2.0 时夹不夹得住杯子(看视频);控制器跟踪滞后几帧(作为真机执行段跟踪的基线)。回放都做不成,先查硬件 / 指令语义。
-1. 在部署机上跑 `deploy/fit_check_pi05_cups.py`(`--dataset-root` 用你自己转换出来的目录,`--compare-to deploy/expected/action_fit_summary_bf16.json`),对照 `deploy/TOLERANCE.md`:期望块最大差、聚合拟合误差相对差都要在容差内;`client_path_test.pass` 必须为 true。
+1. 在部署机上跑 `deploy/fit_check_pi05_cups.py`(`--dataset-root` 用你自己转换出来的目录,`--compare-to deploy/expected/action_fit_summary_bf16.json`),对照 `deploy/TOLERANCE.md`(脚本读 `deploy/tolerance.json`,末尾打印 `TOLERANCE_CHECK PASS/FAIL`):期望块最大差、聚合拟合误差相对差都要在容差内;`client_path_test.pass` 必须为 true(判据:每帧图像差 ≤ 1/255;两条路径的块差只作提示,部分显卡上两次前向不逐位相同)。
 2. **必须用你自己机器人端的「相机取图 → 张量」代码跑一遍客户端路径测试**:`--client-fn 你的模块:函数 --client-frame-format bgr|rgb`(函数收相机原样交出的 uint8 HxWx3 帧,返回进模型的 float (3,H,W) [0,1];客户端 / 服务端分开的话把两段拼起来)。模板见 `deploy/example_client_fn.py`。BGR、缩放、裁剪这类错只有这样才抓得到。
 3. 第一次上机先悬空或低速,只打印一块 50 步动作,核对关节范围、顺序、夹爪方向都对了再正式执行;确认急停可用。
 

deploy/TOLERANCE.md

@@ -1,8 +1,8 @@
 # π0.5 叠杯 1 万步档 · 复现检查容差(部署机对数用)
 
-来源:我们在 NVIDIA H200 上对本仓权重做的 bf16 复现检查(2026-10-09 13:40 UTC,`deploy/expected/action_fit_summary_bf16.json`);跨精度差取自同一存档早先在同类显卡上的 bf16 / fp32 两次复现检查。
-期望块 = 第 0, 25 局各 7 次规划(取在手臂运动段内),每次整块 50 × 14,CPU 固定种子噪声。
-**暂定值**:部署机(RTX 5090)上同为 bf16 实测后可以再收紧。
+**规则**:容差 = 3 × max(H200 上 bf16 / fp32 的跨精度差, RTX 5090 对 H200 的实测差),执行段(第 1–25 步)和整块分开算。
+来源:H200 上对本仓权重做的 bf16 与 fp32 两次复现检查(`deploy/expected/action_fit_summary_bf16.json` 是 bf16 那次);RTX 5090 对 H200 实测差取本档在 5090 上两次实测里较大的一次。
+期望块 = 第 0, 25 局各 7 次规划(取在手臂运动段内),每次整块 50 × 14,CPU 固定种子噪声。同样的数机器可读版在 `deploy/tolerance.json`,复现检查脚本会自动读它,并在末尾打印 `TOLERANCE_CHECK PASS/FAIL`。
 
 在部署机上运行(仓根目录;`$DATA` = 按 README 第 3 节自己转换出来的数据目录):
 
@@ -11,15 +11,18 @@ python deploy/fit_check_pi05_cups.py --checkpoint . --dataset-root "$DATA" --out
     --episode-indices 0 25 --dtype bfloat16 --compare-to deploy/expected/action_fit_summary_bf16.json
 ```
 
-| 检查项 | 我们这边的值(bf16) | 部署机通过条件 |
+| 检查项 | 我们这边的值(H200, bf16) | 部署机通过条件 |
 |---|---|---|
-| 期望块执行段(第 1–25 步)逐元素最大差 `compare_to.PRIMARY_golden_exec_max_abs_diff` | — | ≤ 0.1119 rad(沿用整块容差;执行段差不会大于整块差) |
-| 期望块整块逐元素最大差 `compare_to.golden_chunk_max_abs_diff` | — | ≤ 0.1119 rad(= 3 × 实测跨精度差 0.0373) |
-| 执行段拟合误差,关节平均 | 0.01067 rad | 相对差 ≤ 10%(实测跨精度差 0.3%) |
+| 期望块执行段(第 1–25 步)逐元素最大差 `compare_to.PRIMARY_golden_exec_max_abs_diff` | — | ≤ 0.0509 rad(= 3 × max(H200 跨精度 0.0170, 5090 实测 0.0150)) |
+| 期望块整块逐元素最大差 `compare_to.golden_chunk_max_abs_diff` | — | ≤ 0.1216 rad(= 3 × max(H200 跨精度 0.0373, 5090 实测 0.0405)) |
+| 执行段拟合误差,关节平均 | 0.01067 rad | 相对差 ≤ 10%(H200 跨精度差 0.3%) |
 | 执行段拟合误差,夹爪平均 | 0.0141 | 相对差 ≤ 10% |
-| 块起点偏移 \|â0 − s_t\|,关节中位 | 0.00445 rad | 相对差 ≤ 10%(实测跨精度差 0.8%) |
-| 客户端路径测试(用你自己的 `--client-fn`) | PASS | 必须 PASS |
+| 块起点偏移 \|â0 − s_t\|,关节中位 | 0.00445 rad | 相对差 ≤ 10%(= max(10%, 3 × H200 跨精度差 0.8%)) |
+| 客户端路径测试 `client_path_test.pass`(用你自己的 `--client-fn`) | PASS | 必须 PASS:每帧图像差 ≤ 1/255(BGR / RGB、缩放、裁剪接错都体现在这里) |
+| 客户端路径的块差 `client_path_test.chunk_advisory` | — | 只是提示,阈值 = 执行段容差 0.0509 rad |
 
-判别力核对:故意送 BGR 时,期望块整块最大差 = 4.6 × 整块容差(要求 ≥ 3 倍),即这个容差能抓出颜色接错。
+**为什么客户端路径的块差不算关卡**:部分显卡上两次前向即使输入完全相同也不逐位相同(我们在 RTX 5090 上看到最大约 0.03 rad;H200 上逐位相同),所以块差不为 0 是正常的。想要逐位可复现,可以加 `--deterministic`(默认关;会变慢,不支持确定性的算子只告警)。
+
+判别力核对:故意送 BGR 时,期望块执行段最大差 = 9.6 × 执行段容差、整块最大差 = 4.4 × 整块容差(要求 ≥ 3 倍),即这个容差能抓出颜色接错。
 
 真机上的块起点偏移离线基准(训练局、本档):关节中位 0.0045 rad;`deploy/rollout_consistency.py --reference deploy/expected/action_fit_summary_bf16.json` 会自动对比。

deploy/fit_check_pi05_cups.py

@@ -15,9 +15,15 @@ What it does:
        - the per-frame motion |s_{t+1} - s_t| as the yardstick.
   4. Client-path test: decodes the same frames straight from the mp4 with PyAV as uint8 RGB
      HxWx3 (what a camera driver hands over), converts them with `client_frame_to_tensor`
-     (the deployment contract) and checks the image tensors and the predicted chunk match the
-     dataset-tensor path; also feeds channel-swapped (BGR) frames to show how much a colour
-     mix-up moves the prediction.
+     (the deployment contract, or YOUR --client-fn) and checks that the image tensors match the
+     dataset-tensor path (PASS/FAIL: every frame within 1/255 - this is what catches BGR/RGB,
+     resize and crop mistakes). The predicted chunks of the two paths are compared too, but only
+     as an advisory: on some GPUs two forward passes on identical inputs are not bit-identical,
+     so a small non-zero chunk difference is normal (warned only above the executed-steps
+     tolerance). Also feeds channel-swapped (BGR) frames to show how much a colour mix-up moves
+     the prediction.
+  6. If deploy/tolerance.json is next to this script (or --tolerance-json is given) and
+     --compare-to is set, checks every item of deploy/TOLERANCE.md and prints TOLERANCE_CHECK PASS/FAIL.
   5. Writes action_fit_summary.json (+ golden chunks for 2 episodes) that the deployment
      machine should reproduce within tolerance before going on the robot.
 
@@ -30,6 +36,7 @@ from __future__ import annotations
 import argparse
 import json
 import math
+import os
 import sys
 from datetime import datetime, timezone
 from pathlib import Path
@@ -76,9 +83,44 @@ def parse_args() -> argparse.Namespace:
                    help="if > 0: after 3 warm-up calls, time N full requests (preprocess + predict + postprocess) on one "
                         "frame and report median / max latency, peak GPU memory, and the synchronous chunk cycle "
                         "(latency + execute_horizon x 50 ms at 20 Hz)")
+    p.add_argument("--tolerance-json", type=Path, default=None,
+                   help="machine-readable tolerances (default: tolerance.json next to this script, if present); used for "
+                        "the client-path chunk advisory threshold and the TOLERANCE_CHECK verdict")
+    p.add_argument("--client-chunk-advisory-tol", type=float, default=None,
+                   help="advisory threshold (rad) for |chunk(client path) - chunk(dataset path)|; default: the executed-steps "
+                        "tolerance from tolerance.json, else 0.05")
+    p.add_argument("--deterministic", action="store_true",
+                   help="off by default. Ask PyTorch / cuBLAS / cuDNN for deterministic kernels (slower; ops without a "
+                        "deterministic version only warn). Only for people who want bit-identical repeated forward passes.")
     return p.parse_args()
 
 
+def load_tolerance(args) -> tuple[dict | None, str | None]:
+    path = args.tolerance_json if args.tolerance_json is not None else Path(__file__).resolve().with_name("tolerance.json")
+    if path is not None and Path(path).is_file():
+        return json.loads(Path(path).read_text()), str(path)
+    return None, None
+
+
+def tolerance_check(summary: dict, other: dict, tol: dict) -> dict:
+    """Every row of deploy/TOLERANCE.md as a machine check (needs --compare-to against our expected json)."""
+    c = summary["compare_to"]
+    rel = lambda x, y: abs(x - y) / max(abs(y), 1e-9)
+    items = [
+        ("golden_exec_max_abs_diff_rad", c["PRIMARY_golden_exec_max_abs_diff"], tol["golden_exec_max_abs_diff_rad"]),
+        ("golden_chunk_max_abs_diff_rad", c["golden_chunk_max_abs_diff"], tol["golden_chunk_max_abs_diff_rad"]),
+        ("fit_joints_mean_rel_diff", c["fit_joints_rel_diff"], tol["fit_joints_rel_diff_max"]),
+        ("fit_grippers_mean_rel_diff", rel(summary["fit_executed_vs_recorded"]["grippers_mean"],
+                                           other["fit_executed_vs_recorded"]["grippers_mean"]), tol["fit_grippers_rel_diff_max"]),
+        ("block_start_joints_median_rel_diff", c["block_start_joints_median_rel_diff"],
+         tol["block_start_joints_median_rel_diff_max"]),
+    ]
+    rows = [{"item": n, "value": v, "limit": lim, "pass": (v is not None and v <= lim)} for n, v, lim in items]
+    rows.append({"item": "client_path_test.pass (images within 1/255)", "value": summary["client_path_test"]["pass"],
+                 "limit": True, "pass": bool(summary["client_path_test"]["pass"])})
+    return {"pass": all(r["pass"] for r in rows), "rows": rows}
+
+
 def measure_latency(policy, pre, post, ds, n_runs: int, execute_horizon: int) -> dict:
     import time
     sample = ds[100]
@@ -192,6 +234,12 @@ def dimwise(x: np.ndarray) -> dict:
 
 def main() -> int:
     args = parse_args()
+    if args.deterministic:
+        os.environ.setdefault("CUBLAS_WORKSPACE_CONFIG", ":4096:8")  # must be set before the first cuBLAS call
+        torch.use_deterministic_algorithms(True, warn_only=True)
+        torch.backends.cudnn.deterministic = True
+        torch.backends.cudnn.benchmark = False
+    tol_cfg, tol_path = load_tolerance(args)
     out_dir = args.output_dir.expanduser().resolve()
     out_dir.mkdir(parents=True, exist_ok=True)
 
@@ -306,14 +354,31 @@ def main() -> int:
         "vs_max_joint_speed": float(np.median(start0_abs[:, JOINT_IDX].max(axis=1)) /
                                     max(float(np.median(motion_moving[:, JOINT_IDX].max(axis=1))), 1e-9)),
     }
-    client_pass = all(r["image_max_abs_diff"] <= 1.0 / 255 + 1e-6 and r["chunk_max_abs_diff_client_vs_dataset"] <= 1e-4
-                      for r in client_rows)
+    # PASS/FAIL = the images: BGR/RGB, resize and crop mistakes all show up here.
+    client_pass = all(r["image_max_abs_diff"] <= 1.0 / 255 + 1e-6 for r in client_rows)
+    # Advisory only: two forward passes on identical inputs are not bit-identical on every GPU (seen on RTX 5090).
+    adv_thr = (args.client_chunk_advisory_tol if args.client_chunk_advisory_tol is not None
+               else (tol_cfg or {}).get("client_chunk_advisory_rad", 0.05))
+    chunk_diffs = [r["chunk_max_abs_diff_client_vs_dataset"] for r in client_rows]
+    n_over = sum(d > adv_thr for d in chunk_diffs)
+    chunk_advisory = {"threshold_rad": adv_thr,
+                      "threshold_source": ("--client-chunk-advisory-tol" if args.client_chunk_advisory_tol is not None
+                                           else (tol_path or "default 0.05")),
+                      "max_chunk_diff_rad": max(chunk_diffs) if chunk_diffs else None, "n_over": int(n_over),
+                      "status": "WARN" if n_over else "OK",
+                      "note": "advisory, not part of PASS/FAIL: some GPUs give slightly different chunks for identical inputs "
+                              "(non-deterministic kernels); a non-zero difference is normal"}
+    for r in client_rows:
+        if r["chunk_max_abs_diff_client_vs_dataset"] > adv_thr:
+            print(f"[WARN] client-path chunk diff {r['chunk_max_abs_diff_client_vs_dataset']:.4f} rad > advisory "
+                  f"{adv_thr:.4f} at ep{r['episode']} step {r['step']} (images differ by {r['image_max_abs_diff']:.2e})", flush=True)
     summary = {
         "metadata": {"created_utc": datetime.now(timezone.utc).isoformat(), "harness": "fit_check_pi05_cups.py",
                      "checkpoint": str(args.checkpoint), "dataset_root": str(args.dataset_root),
                      "episodes": args.episode_indices, "execute_horizon": args.execute_horizon,
                      "skip_first": args.skip_first, "seed": args.seed, "dtype": args.dtype,
                      "dim_names": DIM_NAMES, "task_prompt": TASK_PROMPT,
+                     "deterministic": bool(args.deterministic), "fit_check_version": "v2 2026-10-09 (image-only client PASS)",
                      "note": "all 50 episodes are training data: this measures fit + wiring, not generalisation"},
         "fit_executed_vs_recorded": dimwise(fit_abs),
         "block_start_offset_ahat0_vs_state": dimwise(start0_abs),
@@ -323,7 +388,10 @@ def main() -> int:
                                                      "moving_fraction": float(moving.mean())},
         "block_start_offset_in_frames_of_motion": frames_equiv,
         "episodes": episodes_out,
-        "client_path_test": {"pass": bool(client_pass), "client_fn": args.client_fn or "contract default (client_frame_to_tensor)",
+        "client_path_test": {"pass": bool(client_pass),
+                             "criterion": "every frame: image_max_abs_diff <= 1/255 (PASS/FAIL); chunk difference is advisory",
+                             "chunk_advisory": chunk_advisory,
+                             "client_fn": args.client_fn or "contract default (client_frame_to_tensor)",
                              "client_frame_format": args.client_frame_format, "rows": client_rows,
                              "bgr_chunk_max_abs_diff_min": min((r["chunk_max_abs_diff_bgr_vs_rgb"] for r in client_rows), default=None),
                              "bgr_chunk_max_abs_diff_median": (float(np.median([r["chunk_max_abs_diff_bgr_vs_rgb"] for r in client_rows]))
@@ -369,6 +437,12 @@ def main() -> int:
             "block_start_joints_median_rel_diff": rel(summary["block_start_offset_ahat0_vs_state"]["joints_median_rad"],
                                                       other["block_start_offset_ahat0_vs_state"]["joints_median_rad"]),
         }
+    if args.compare_to is not None and tol_cfg is not None:
+        summary["tolerance_check"] = {**tolerance_check(summary, other, tol_cfg), "tolerance_json": tol_path}
+        tc = summary["tolerance_check"]
+        for r in tc["rows"]:
+            print(f"[{'OK' if r['pass'] else 'FAIL'}] {r['item']}: {r['value']} (limit {r['limit']})", flush=True)
+        print(f"TOLERANCE_CHECK {'PASS' if tc['pass'] else 'FAIL'}", flush=True)
     if args.latency_runs > 0:
         summary["latency"] = measure_latency(policy, pre, post, ds, args.latency_runs, args.execute_horizon)
         print(f"[OK] latency median={summary['latency']['latency_median_s']*1000:.1f} ms max="
@@ -380,6 +454,7 @@ def main() -> int:
           f"= {frames_equiv['pooled_joints']:.2f} frames of per-joint motion, "
           f"{frames_equiv['vs_max_joint_speed']:.2f} frames of max-joint motion | "
           f"client_path={'PASS' if client_pass else 'FAIL'}"
+          f"{' (chunk advisory: ' + chunk_advisory['status'] + ')' if client_rows else ''}"
           + (f" | vs {summary['compare_to']['other_dtype']}: golden max diff={summary['compare_to']['golden_chunk_max_abs_diff']}"
              if "compare_to" in summary else ""), flush=True)
     print(f"[OK] wrote {out_dir / 'action_fit_summary.json'}", flush=True)

deploy/tolerance.json

@@ -0,0 +1,24 @@
+{
+  "checkpoint": "1 万步档",
+  "rule": "tolerance = 3 x max(H200 bf16-vs-fp32 cross-precision diff, RTX 5090-vs-H200 measured diff); executed steps 1..25 and full chunk separately",
+  "golden_exec_max_abs_diff_rad": 0.0509,
+  "golden_chunk_max_abs_diff_rad": 0.1216,
+  "fit_joints_rel_diff_max": 0.1,
+  "fit_grippers_rel_diff_max": 0.1,
+  "block_start_joints_median_rel_diff_max": 0.1,
+  "client_image_max_abs_diff_max": 0.003922,
+  "client_chunk_advisory_rad": 0.0509,
+  "sources": {
+    "h200_cross_precision_exec_rad": 0.016979,
+    "h200_cross_precision_full_rad": 0.037307,
+    "h200_cross_precision_fit_rel": 0.0033,
+    "h200_cross_precision_block_start_rel": 0.0082,
+    "rtx5090_vs_h200_exec_rad": 0.015049,
+    "rtx5090_vs_h200_full_rad": 0.040542,
+    "rtx5090_borrowed_from_other_checkpoint": false,
+    "bgr_exec_max_rad": 0.486992,
+    "bgr_full_max_rad": 0.537438,
+    "bgr_exec_over_tolerance": 9.56,
+    "bgr_full_over_tolerance": 4.42
+  }
+}

tools/dataset_v30_reference.json

@@ -0,0 +1,296 @@
+{
+ "about": "reference for tools/verify_dataset_v30.py: our training copy of yaoxianze/arx_external_cam_cups@2d87a09bdc0df9b02342fa89da3a375d74776841 converted to LeRobot v3.0",
+ "advisory": {
+  "meta/info.json": "f3e1fb26075cbc77f6baf8c173d9ce22194005733b8ba333465f313b0ee92055",
+  "meta/stats.json": "fb745fa48ed233ec7b9a60ba991dbd90a8f569f5ad0cec7c7eee874612159bfd"
+ },
+ "made_with": {
+  "numpy": "2.2.6",
+  "pyarrow": "25.0.1"
+ },
+ "parquet": {
+  "data/chunk-000/file-000.parquet": {
+   "columns": [
+    "observation.state",
+    "observation.qvel",
+    "observation.effort",
+    "observation.eef",
+    "observation.eef_tcp",
+    "observation.tcp_xyz_local",
+    "observation.left_tcp_pos",
+    "observation.right_tcp_pos",
+    "observation.tcp_camera_h_uvd",
+    "observation.tcp_camera_h_valid",
+    "action",
+    "observation.depth.image_m_path",
+    "observation.camera.params_path",
+    "timestamp",
+    "source_timestamp",
+    "frame_index",
+    "episode_index",
+    "index",
+    "task_index",
+    "observation.depth.camera_h_m_path",
+    "observation.depth.camera_l_m_path",
+    "observation.depth.camera_r_m_path",
+    "observation.depth.camera_external_m_path"
+   ],
+   "content_sha256": {
+    "action": "41c33c736779b1ed5e3bc4a461928e5abfa2809d9cf10b078c3401060bad286b",
+    "episode_index": "2b21e1780dadc37a8652a200a1b5e67e019b7af3c31195cd2e8d9a80a51a9e00",
+    "frame_index": "c9d047b8447bcdc3ebacbca14637cbeb07704c3b81f8b5f9b69193266ff2e818",
+    "index": "f0680514a7777f5c4d360c73aeee0b2cf90069207a5e9af82dd6e227dcbfe793",
+    "observation.camera.params_path": "0f80f74c768a08376f8d3708962b1f6da322f46e417727949e573552cfb96a0c",
+    "observation.depth.camera_external_m_path": "04252921002247f49ad87e53820bce3afc8208f3729af64d24d8c7ea9a0c9c6c",
+    "observation.depth.camera_h_m_path": "47894926c98962363402507d53c53ff230f5e0977395f33cafc0217ae77b017d",
+    "observation.depth.camera_l_m_path": "f2b3740868f3c102398c612dae69f261dcd4d3243c75f918d0abd401edb25626",
+    "observation.depth.camera_r_m_path": "2922cc89b347c755ea1de7660f774ddd2682dfd027cb199eca39e31aadf4da55",
+    "observation.depth.image_m_path": "47894926c98962363402507d53c53ff230f5e0977395f33cafc0217ae77b017d",
+    "observation.eef": "a9fa5edd89d609c3144cd360c593688ace8e782b2d8ef82abcca0ca5af753f24",
+    "observation.eef_tcp": "726659e02ff9488f2a46cff1d7802d36ae609eefe82f238f0aed7e7d71576fa1",
+    "observation.effort": "d79b9458a0fd896b71fca897c3b537ddad66af8cc13bc9f4d427c31511318c2c",
+    "observation.left_tcp_pos": "6bf2b3ac654c4f3f4cc0bb91403c309757a44ebe2892ff64ac465e058e4bbfa0",
+    "observation.qvel": "e559f5ce3041d9e876f6f68b2bf71ec6fe50be3aa08da4ca232621144c7ef3c9",
+    "observation.right_tcp_pos": "20755e0a597a6722f34f8c91c36404612303cf2cc217e32e9dadc1668df4f7ef",
+    "observation.state": "41c33c736779b1ed5e3bc4a461928e5abfa2809d9cf10b078c3401060bad286b",
+    "observation.tcp_camera_h_uvd": "abf0ecd9af4cf8b98d6af518e382f437aecb91d20dd1d911a577cc25f0cf7f4b",
+    "observation.tcp_camera_h_valid": "cfaf266561ecd6aae560e15f01520cd475cce011829915918125ac42d75fd8ab",
+    "observation.tcp_xyz_local": "516f6ba86fe7346fc4957161c333730fa8569426666ce9d8292b91836fec68ee",
+    "source_timestamp": "260536f6a199bab5ebf2c0708facdc6455f576e5caea5df67d847a85ba13fc14",
+    "task_index": "029d1cafffb2c55e77a80ae231476734daaa8d6840467262ec501504f3777b4f",
+    "timestamp": "be1493ae0c203621d8df8620fb0653d953ed21750444356f8e5977a75d09df9e"
+   },
+   "rows": 24003,
+   "types": {
+    "action": "list<element: float>",
+    "episode_index": "int64",
+    "frame_index": "int64",
+    "index": "int64",
+    "observation.camera.params_path": "string",
+    "observation.depth.camera_external_m_path": "string",
+    "observation.depth.camera_h_m_path": "string",
+    "observation.depth.camera_l_m_path": "string",
+    "observation.depth.camera_r_m_path": "string",
+    "observation.depth.image_m_path": "string",
+    "observation.eef": "list<element: float>",
+    "observation.eef_tcp": "list<element: float>",
+    "observation.effort": "list<element: float>",
+    "observation.left_tcp_pos": "list<element: float>",
+    "observation.qvel": "list<element: float>",
+    "observation.right_tcp_pos": "list<element: float>",
+    "observation.state": "list<element: float>",
+    "observation.tcp_camera_h_uvd": "list<element: float>",
+    "observation.tcp_camera_h_valid": "list<element: bool>",
+    "observation.tcp_xyz_local": "list<element: float>",
+    "source_timestamp": "double",
+    "task_index": "int64",
+    "timestamp": "double"
+   }
+  },
+  "meta/episodes/chunk-000/file-000.parquet": {
+   "columns": [
+    "episode_index",
+    "data/chunk_index",
+    "data/file_index",
+    "dataset_from_index",
+    "dataset_to_index",
+    "videos/observation.images.camera_external/chunk_index",
+    "videos/observation.images.camera_external/file_index",
+    "videos/observation.images.camera_external/from_timestamp",
+    "videos/observation.images.camera_external/to_timestamp",
+    "videos/observation.images.camera_h/chunk_index",
+    "videos/observation.images.camera_h/file_index",
+    "videos/observation.images.camera_h/from_timestamp",
+    "videos/observation.images.camera_h/to_timestamp",
+    "videos/observation.images.camera_l/chunk_index",
+    "videos/observation.images.camera_l/file_index",
+    "videos/observation.images.camera_l/from_timestamp",
+    "videos/observation.images.camera_l/to_timestamp",
+    "videos/observation.images.camera_r/chunk_index",
+    "videos/observation.images.camera_r/file_index",
+    "videos/observation.images.camera_r/from_timestamp",
+    "videos/observation.images.camera_r/to_timestamp",
+    "tasks",
+    "length",
+    "videos",
+    "tcp_offset_wrist_m_by_hand",
+    "stats/observation.state/min",
+    "stats/observation.state/max",
+    "stats/observation.state/mean",
+    "stats/observation.state/std",
+    "stats/observation.state/count",
+    "stats/observation.qvel/min",
+    "stats/observation.qvel/max",
+    "stats/observation.qvel/mean",
+    "stats/observation.qvel/std",
+    "stats/observation.qvel/count",
+    "stats/observation.effort/min",
+    "stats/observation.effort/max",
+    "stats/observation.effort/mean",
+    "stats/observation.effort/std",
+    "stats/observation.effort/count",
+    "stats/observation.eef/min",
+    "stats/observation.eef/max",
+    "stats/observation.eef/mean",
+    "stats/observation.eef/std",
+    "stats/observation.eef/count",
+    "stats/observation.eef_tcp/min",
+    "stats/observation.eef_tcp/max",
+    "stats/observation.eef_tcp/mean",
+    "stats/observation.eef_tcp/std",
+    "stats/observation.eef_tcp/count",
+    "stats/action/min",
+    "stats/action/max",
+    "stats/action/mean",
+    "stats/action/std",
+    "stats/action/count",
+    "meta/episodes/chunk_index",
+    "meta/episodes/file_index"
+   ],
+   "content_sha256": {
+    "data/chunk_index": "f7f1001c68b6e9da3dd6549fb12b728f8d89887caa3f9b94c99378d39d87825d",
+    "data/file_index": "f7f1001c68b6e9da3dd6549fb12b728f8d89887caa3f9b94c99378d39d87825d",
+    "dataset_from_index": "1e5789d4286a7cf2fba163060def2f522d944bfe84cb70b432a2328d6a346d0e",
+    "dataset_to_index": "15f0aa6073bc010b19a3c84b4eeeb67d1dd30a4e577e1b3e61be4ec75b5102d8",
+    "episode_index": "1f5af63fa5e6247d808eef59cd3fb5915e0a5ac2e605895643d748da1d5caa84",
+    "length": "0c7ca8615d570c277c57986eadb67d86c39954383ba77a0a7b736302362a8608",
+    "meta/episodes/chunk_index": "f7f1001c68b6e9da3dd6549fb12b728f8d89887caa3f9b94c99378d39d87825d",
+    "meta/episodes/file_index": "f7f1001c68b6e9da3dd6549fb12b728f8d89887caa3f9b94c99378d39d87825d",
+    "stats/action/count": "190d914d31da8fb50291acb0ecf7dff7941a3b58452f70a42fc49295bfb9ebf3",
+    "stats/action/max": "2a629d0bd6cc0a3a911b39bcaa989f1733917f58000810760f1da30ac98d168b",
+    "stats/action/mean": "ce40834fa5f88cab6cb6fb0aaa1aea33ddb88f4d461bcc72095700c2192c29c5",
+    "stats/action/min": "262b049912611b08b0231589cfe4e1aaed4918752632cc077d4fa339fca17daf",
+    "stats/action/std": "8626bbcea12f499d9d940ccdc06c520de03cc5de81a4a94d03da81eb435f22dc",
+    "stats/observation.eef/count": "190d914d31da8fb50291acb0ecf7dff7941a3b58452f70a42fc49295bfb9ebf3",
+    "stats/observation.eef/max": "92025f7bb27567b08e3fe45836f4d276335dcd5475a38272aeceeaf9863b0828",
+    "stats/observation.eef/mean": "7d029ad36956b90cdd22dc8eeccee0d438691ba5706bd29ffc500b8df535c142",
+    "stats/observation.eef/min": "d8d529f62e21ce495291faf3998f63c30f1aa78c97f0903455d5794ad5e0547a",
+    "stats/observation.eef/std": "9a03961f749887a023b63bae12013b9ade8c0e43e40e36d243031ffc89f7df72",
+    "stats/observation.eef_tcp/count": "190d914d31da8fb50291acb0ecf7dff7941a3b58452f70a42fc49295bfb9ebf3",
+    "stats/observation.eef_tcp/max": "349f134929e5fa5c2a7ddae75895ff69e186f37c3d36d568895e0215eeadbfef",
+    "stats/observation.eef_tcp/mean": "d006ef83fe8fee6f7ecc580d388e22532e0d859269a8710745eb4e2930e44981",
+    "stats/observation.eef_tcp/min": "7031ba620baf1ff83aa195df663ddab348370be199678402a842c99a9cade658",
+    "stats/observation.eef_tcp/std": "0a3949567b80999211b2d15dd459b03ada92f28a832ad468463835d7f6307c02",
+    "stats/observation.effort/count": "190d914d31da8fb50291acb0ecf7dff7941a3b58452f70a42fc49295bfb9ebf3",
+    "stats/observation.effort/max": "798b8ee654a607e88c9d6a32880b59a0b4a9f1e61c4401050cef283e497d76e8",
+    "stats/observation.effort/mean": "3843d9046830b7f2b0365393effe7b8f79d9704e6605eb0cbad4984ca8e0a3ea",
+    "stats/observation.effort/min": "bce672193d2ee517dd21c429f6723745693c1a406bdac26a732b93344888c7b3",
+    "stats/observation.effort/std": "551f73f423208bd4fd2b712a2f783dc8a747b67691f5bcf81edead7aa28d017e",
+    "stats/observation.qvel/count": "190d914d31da8fb50291acb0ecf7dff7941a3b58452f70a42fc49295bfb9ebf3",
+    "stats/observation.qvel/max": "3ec64c1472ed9519a9bc1db92b7b06a4448600d26aca0d116ad058d40552f67e",
+    "stats/observation.qvel/mean": "e74bfdcb0d4abada070a9ed57d3ff72ab3186928c1d30fb4ddbfd15d3eea504f",
+    "stats/observation.qvel/min": "45f2bfe92cc2bba0f4f471d22f6142190af0b020819115da9271eaf09334a6d8",
+    "stats/observation.qvel/std": "260020795f1e73efdcd595b4b3d5fbf536bed552e5ac7c821cfcac3b48df09af",
+    "stats/observation.state/count": "190d914d31da8fb50291acb0ecf7dff7941a3b58452f70a42fc49295bfb9ebf3",
+    "stats/observation.state/max": "2a629d0bd6cc0a3a911b39bcaa989f1733917f58000810760f1da30ac98d168b",
+    "stats/observation.state/mean": "ce40834fa5f88cab6cb6fb0aaa1aea33ddb88f4d461bcc72095700c2192c29c5",
+    "stats/observation.state/min": "262b049912611b08b0231589cfe4e1aaed4918752632cc077d4fa339fca17daf",
+    "stats/observation.state/std": "8626bbcea12f499d9d940ccdc06c520de03cc5de81a4a94d03da81eb435f22dc",
+    "tasks": "6c288624f0859b0f63d17e308ae95ec7c63488c7844b8326e3529a5ad5f65d35",
+    "tcp_offset_wrist_m_by_hand": "9e1db2aec24c0786755e58f2988fde136b6219d682580a96321acdc9330fceeb",
+    "videos": "cb72fc7cc086bf4308a682b8bcdec0531084f3f92dda2b53f379e9afd0eba86f",
+    "videos/observation.images.camera_external/chunk_index": "f7f1001c68b6e9da3dd6549fb12b728f8d89887caa3f9b94c99378d39d87825d",
+    "videos/observation.images.camera_external/file_index": "969f96280699242f0041717fdd335c157a2e445c2a75d94eeb60ca5fd48da3c2",
+    "videos/observation.images.camera_external/from_timestamp": "80bbac3ca5e80b4fd59e443200cc5177a7f80bbaa485651c635be2f9c974e0d9",
+    "videos/observation.images.camera_external/to_timestamp": "eb5593eac556d84090340bbec8bcd76c05ac80f3f5a4fb0f64f1ad5b42cd6c16",
+    "videos/observation.images.camera_h/chunk_index": "f7f1001c68b6e9da3dd6549fb12b728f8d89887caa3f9b94c99378d39d87825d",
+    "videos/observation.images.camera_h/file_index": "ab39ec175cb9e45a764e377c1ff808d5fa0ab6cdef86430b6e37e51ebf2c4450",
+    "videos/observation.images.camera_h/from_timestamp": "d20fc1de80380bf756f4d5fee2b7dbfbcb368bded4f5a4a8b53e175be0df563d",
+    "videos/observation.images.camera_h/to_timestamp": "7a5e503999ad68ff742fe84fd5273f5bf76f13ba163366b352d5dfc7c36cb717",
+    "videos/observation.images.camera_l/chunk_index": "f7f1001c68b6e9da3dd6549fb12b728f8d89887caa3f9b94c99378d39d87825d",
+    "videos/observation.images.camera_l/file_index": "fc5d64ccee781651d2e0db437dc60d4f526c210d3dcb3a13e0d21b92739a0548",
+    "videos/observation.images.camera_l/from_timestamp": "9096d8e8273ad808f9318d2347e921f297290d6847fec84c2a56a9636b54e6d4",
+    "videos/observation.images.camera_l/to_timestamp": "06505422653b77dbffb7f760c03f8f145809bc5d1deaadef5d494c086a6bd14c",
+    "videos/observation.images.camera_r/chunk_index": "f7f1001c68b6e9da3dd6549fb12b728f8d89887caa3f9b94c99378d39d87825d",
+    "videos/observation.images.camera_r/file_index": "fc5d64ccee781651d2e0db437dc60d4f526c210d3dcb3a13e0d21b92739a0548",
+    "videos/observation.images.camera_r/from_timestamp": "9096d8e8273ad808f9318d2347e921f297290d6847fec84c2a56a9636b54e6d4",
+    "videos/observation.images.camera_r/to_timestamp": "06505422653b77dbffb7f760c03f8f145809bc5d1deaadef5d494c086a6bd14c"
+   },
+   "rows": 50,
+   "types": {
+    "data/chunk_index": "int64",
+    "data/file_index": "int64",
+    "dataset_from_index": "int64",
+    "dataset_to_index": "int64",
+    "episode_index": "int64",
+    "length": "int64",
+    "meta/episodes/chunk_index": "int64",
+    "meta/episodes/file_index": "int64",
+    "stats/action/count": "list<element: int64>",
+    "stats/action/max": "list<element: double>",
+    "stats/action/mean": "list<element: double>",
+    "stats/action/min": "list<element: double>",
+    "stats/action/std": "list<element: double>",
+    "stats/observation.eef/count": "list<element: int64>",
+    "stats/observation.eef/max": "list<element: double>",
+    "stats/observation.eef/mean": "list<element: double>",
+    "stats/observation.eef/min": "list<element: double>",
+    "stats/observation.eef/std": "list<element: double>",
+    "stats/observation.eef_tcp/count": "list<element: int64>",
+    "stats/observation.eef_tcp/max": "list<element: double>",
+    "stats/observation.eef_tcp/mean": "list<element: double>",
+    "stats/observation.eef_tcp/min": "list<element: double>",
+    "stats/observation.eef_tcp/std": "list<element: double>",
+    "stats/observation.effort/count": "list<element: int64>",
+    "stats/observation.effort/max": "list<element: double>",
+    "stats/observation.effort/mean": "list<element: double>",
+    "stats/observation.effort/min": "list<element: double>",
+    "stats/observation.effort/std": "list<element: double>",
+    "stats/observation.qvel/count": "list<element: int64>",
+    "stats/observation.qvel/max": "list<element: double>",
+    "stats/observation.qvel/mean": "list<element: double>",
+    "stats/observation.qvel/min": "list<element: double>",
+    "stats/observation.qvel/std": "list<element: double>",
+    "stats/observation.state/count": "list<element: int64>",
+    "stats/observation.state/max": "list<element: double>",
+    "stats/observation.state/mean": "list<element: double>",
+    "stats/observation.state/min": "list<element: double>",
+    "stats/observation.state/std": "list<element: double>",
+    "tasks": "list<element: string>",
+    "tcp_offset_wrist_m_by_hand": "struct<left: list<element: double>, right: list<element: double>>",
+    "videos": "struct<observation.images.camera_h: string, observation.images.camera_l: string, observation.images.camera_r: string, observation.images.camera_external: string>",
+    "videos/observation.images.camera_external/chunk_index": "int64",
+    "videos/observation.images.camera_external/file_index": "int64",
+    "videos/observation.images.camera_external/from_timestamp": "double",
+    "videos/observation.images.camera_external/to_timestamp": "double",
+    "videos/observation.images.camera_h/chunk_index": "int64",
+    "videos/observation.images.camera_h/file_index": "int64",
+    "videos/observation.images.camera_h/from_timestamp": "double",
+    "videos/observation.images.camera_h/to_timestamp": "double",
+    "videos/observation.images.camera_l/chunk_index": "int64",
+    "videos/observation.images.camera_l/file_index": "int64",
+    "videos/observation.images.camera_l/from_timestamp": "double",
+    "videos/observation.images.camera_l/to_timestamp": "double",
+    "videos/observation.images.camera_r/chunk_index": "int64",
+    "videos/observation.images.camera_r/file_index": "int64",
+    "videos/observation.images.camera_r/from_timestamp": "double",
+    "videos/observation.images.camera_r/to_timestamp": "double"
+   }
+  },
+  "meta/tasks.parquet": {
+   "columns": [
+    "task_index",
+    "task"
+   ],
+   "content_sha256": {
+    "task": "74a3c8c3e8319791dc5ca939b32b835016e6538bdbea5af82acc3642515a436a",
+    "task_index": "7523bb6d8cee7db92c535809a6974a0343e5e9cfcbf151b995c602953fde96e0"
+   },
+   "rows": 1,
+   "types": {
+    "task": "string",
+    "task_index": "int64"
+   }
+  }
+ },
+ "videos": {
+  "videos/observation.images.camera_external/chunk-000/file-000.mp4": "da2abfb6a74ce2252f1e263660ec40e332c12b99c942e52b8630d570230c37c7",
+  "videos/observation.images.camera_external/chunk-000/file-001.mp4": "8c345f68b4511023316d2be22dd39a3c59fcfb2037d118837b9896f8ff36c530",
+  "videos/observation.images.camera_h/chunk-000/file-000.mp4": "3019abb0bc7ebaab64addcc0f08d9d835195663084f76d296f534cdf363129f4",
+  "videos/observation.images.camera_h/chunk-000/file-001.mp4": "0c03a34f2e4fb7fb2455859f4d60d0df4408edec04257978bf61215a5484b1c1",
+  "videos/observation.images.camera_l/chunk-000/file-000.mp4": "438b1da9dcf1c6d78acd0bed3ce278c8463a952d431af8db92baef9d73e75c6e",
+  "videos/observation.images.camera_l/chunk-000/file-001.mp4": "11625dbc198aedff9074a1af68d0e0d23105828b6e0b46c97019121a75d2859f",
+  "videos/observation.images.camera_r/chunk-000/file-000.mp4": "4ce3ce6f56f1d3e4b554c7051165792fb1b80360297b5af042017156e182fbf2",
+  "videos/observation.images.camera_r/chunk-000/file-001.mp4": "1e8f71f3378d10cbcc59228b82d323e37143e82028850eded4a7e90965635d04"
+ }
+}

tools/prep_pi05_data.sh

@@ -1,24 +1,38 @@
 #!/bin/bash
-# Rebuild the training dataset (LeRobot v3.0 copy + q01/q99 quantile stats) from the public source dataset.
+# Rebuild the LeRobot v3.0 data we trained on from the public source dataset (pinned version) and verify it.
 # NOT needed for inference: the processor files in the repo root already contain the normalisation parameters.
 # Needed for the reproduction check (deploy/fit_check_pi05_cups.py) and the replay test (deploy/replay_demo.py).
-# Needs upstream lerobot @89236ea (pip install -e ".[pi]") in the active env.
-# Check the result with tools/dataset_v30_SHA256SUMS.txt (see README). compute_quantile_stats_for_dataset samples frames,
-# so meta/stats.json (and the per-episode stats) can differ slightly from ours; inference never reads them.
-# Usage: bash tools/prep_pi05_data.sh <work_dir> [<existing LeRobot v2.1 snapshot of the source dataset @2d87a09>]
+# Needs upstream lerobot @89236ea installed with: pip install -e ".[pi,dataset]"
+# Usage: bash tools/prep_pi05_data.sh <work_dir> [<existing LeRobot v2.1 snapshot of the source dataset at this version>]
 set -euo pipefail
 WORK=${1:?usage: prep_pi05_data.sh <work_dir> [<existing v2.1 snapshot>]}
-REV=2d87a09bdc0df9b02342fa89da3a375d74776841
-SRC=${2:-$WORK/arx_external_cam_cups}   # LeRobot v2.1 snapshot of yaoxianze/arx_external_cam_cups@$REV
-DST="$WORK/arx_cups_pi05"               # v3.0 copy used for training
+REPO_ID=yaoxianze/arx_external_cam_cups
+REV=2d87a09bdc0df9b02342fa89da3a375d74776841     # the version we trained on
+EXPECT_META=5; EXPECT_DATA=50; EXPECT_VIDEOS=200  # files in meta/, data/, videos/ of that version (50 episodes x 4 cameras)
+SRC=${2:-$WORK/arx_external_cam_cups}
+DST="$WORK/arx_cups_pi05"
 TOOLS=$(cd "$(dirname "$0")" && pwd)
 [ ! -e "$DST" ] || { echo "target exists, refusing: $DST"; exit 2; }
-if [ -z "${2:-}" ]; then   # only meta/, data/ and videos/ are needed (~1.1 GB; depth/ is not used)
-  hf download yaoxianze/arx_external_cam_cups --repo-type dataset --revision "$REV" \
-     --include "meta/*" "data/*" "videos/*" --local-dir "$SRC"
+if [ -z "${2:-}" ]; then
+  echo "downloading $REPO_ID @ $REV (meta/, data/, videos/ only, ~1.15 GB) -> $SRC"
+  python - "$REPO_ID" "$REV" "$SRC" <<'PY'
+import sys
+from huggingface_hub import snapshot_download
+repo, rev, out = sys.argv[1:4]
+snapshot_download(repo, repo_type="dataset", revision=rev, allow_patterns=["meta/*", "data/*", "videos/*"], local_dir=out)
+PY
 fi
+bad=0
+for d in meta data videos; do
+  n=$(find "$SRC/$d" -type f 2>/dev/null | wc -l | tr -d ' ')
+  case $d in meta) e=$EXPECT_META ;; data) e=$EXPECT_DATA ;; videos) e=$EXPECT_VIDEOS ;; esac
+  echo "SOURCE_FILES $d=$n (expected $e)"
+  [ "$n" -eq "$e" ] || bad=1
+done
+[ "$bad" -eq 0 ] || { echo "source snapshot incomplete or not version $REV -> stop (re-run the download)"; exit 3; }
 mkdir -p "$DST"
-rsync -a "$SRC/meta" "$SRC/data" "$SRC/videos" "$DST/"
+cp -a "$SRC/meta" "$SRC/data" "$SRC/videos" "$DST/"
 python -m lerobot.scripts.convert_dataset_v21_to_v30 --repo-id=local/arx_cups_pi05 --root="$DST" --push-to-hub=false
 python "$TOOLS/quantile_stats_offline.py" "$DST" local/arx_cups_pi05
+python "$TOOLS/verify_dataset_v30.py" "$DST"
 echo "PREP_DONE $DST"

tools/verify_dataset_v30.py

@@ -0,0 +1,126 @@
+#!/usr/bin/env python3
+"""Check a rebuilt LeRobot v3.0 copy of yaoxianze/arx_external_cam_cups@2d87a09 against the copy we trained on.
+
+    python tools/verify_dataset_v30.py <DATA>       # DATA = .../arx_cups_pi05 made by tools/prep_pi05_data.sh
+
+Required (exit code 1 if anything fails):
+  - videos/: every mp4 byte-identical (sha256) to ours;
+  - the 3 parquet files: same row count, column names, column types and per-column CONTENT hash. The parquet bytes
+    themselves depend on the pyarrow version that wrote them (e.g. the created_by header), so they are NOT compared;
+    the content hashes are the same under pyarrow 25 and 26 (checked).
+Advisory only (printed, never fails): meta/info.json, meta/stats.json (stats are sampled, can differ across lerobot
+versions; inference and the reproduction check never read them).
+Reference values: tools/dataset_v30_reference.json, written with --make-reference on our training copy.
+"""
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import sys
+from pathlib import Path
+
+import numpy as np
+import pyarrow as pa
+import pyarrow.parquet as pq
+
+PARQUETS = ("data/chunk-000/file-000.parquet", "meta/episodes/chunk-000/file-000.parquet", "meta/tasks.parquet")
+ADVISORY = ("meta/info.json", "meta/stats.json")
+REF_DEFAULT = Path(__file__).resolve().parent / "dataset_v30_reference.json"
+
+
+def sha256_file(p: Path) -> str:
+    h = hashlib.sha256()
+    with open(p, "rb") as f:
+        for b in iter(lambda: f.read(16 << 20), b""):
+            h.update(b)
+    return h.hexdigest()
+
+
+def column_hash(col) -> str:
+    """Hash of a column's VALUES (independent of how the parquet file was encoded)."""
+    col = col.combine_chunks()
+    try:
+        arr = col.to_numpy(zero_copy_only=False)
+        if arr.dtype == object:
+            payload = json.dumps([x.tolist() if hasattr(x, "tolist") else x for x in arr], sort_keys=True, default=str).encode()
+        else:
+            payload = np.ascontiguousarray(arr).tobytes() + str(arr.dtype).encode()
+    except Exception:
+        payload = json.dumps(col.to_pylist(), sort_keys=True, default=str).encode()
+    return hashlib.sha256(payload).hexdigest()
+
+
+def parquet_signature(p: Path) -> dict:
+    t = pq.read_table(p)
+    return {"rows": t.num_rows, "columns": list(t.column_names),
+            "types": {f.name: str(f.type) for f in t.schema},
+            "content_sha256": {n: column_hash(t.column(n)) for n in t.column_names}}
+
+
+def make_reference(root: Path) -> dict:
+    return {"about": "reference for tools/verify_dataset_v30.py: our training copy of "
+                     "yaoxianze/arx_external_cam_cups@2d87a09bdc0df9b02342fa89da3a375d74776841 converted to LeRobot v3.0",
+            "made_with": {"pyarrow": pa.__version__, "numpy": np.__version__},
+            "videos": {str(p.relative_to(root)): sha256_file(p) for p in sorted((root / "videos").rglob("*.mp4"))},
+            "parquet": {rel: parquet_signature(root / rel) for rel in PARQUETS},
+            "advisory": {rel: sha256_file(root / rel) for rel in ADVISORY if (root / rel).exists()}}
+
+
+def verify(root: Path, ref: dict) -> int:
+    fails = []
+    have = {str(p.relative_to(root)) for p in (root / "videos").rglob("*.mp4")} if (root / "videos").exists() else set()
+    for rel, h in ref["videos"].items():
+        p = root / rel
+        if not p.exists():
+            fails.append(f"missing video {rel}")
+        elif sha256_file(p) != h:
+            fails.append(f"video bytes differ: {rel}")
+    extra = sorted(have - set(ref["videos"]))
+    print(f"VIDEOS ok={len(ref['videos']) - sum(1 for f in fails if 'video' in f)}/{len(ref['videos'])} extra={len(extra)}")
+    for rel, want in ref["parquet"].items():
+        p = root / rel
+        if not p.exists():
+            fails.append(f"missing parquet {rel}")
+            print(f"PARQUET {rel}: MISSING")
+            continue
+        got = parquet_signature(p)
+        problems = []
+        if got["rows"] != want["rows"]:
+            problems.append(f"rows {got['rows']} != {want['rows']}")
+        if got["columns"] != want["columns"]:
+            problems.append("column names / order differ")
+        if got["types"] != want["types"]:
+            problems.append("column types differ: " + ", ".join(sorted(k for k in set(got["types"]) | set(want["types"])
+                                                                       if got["types"].get(k) != want["types"].get(k))))
+        bad_cols = sorted(k for k in want["content_sha256"] if got["content_sha256"].get(k) != want["content_sha256"][k])
+        if bad_cols:
+            problems.append("content differs in columns: " + ", ".join(bad_cols[:10]))
+        print(f"PARQUET {rel}: {'OK' if not problems else 'FAIL'} rows={got['rows']} columns={len(got['columns'])}"
+              + ("" if not problems else " | " + "; ".join(problems)))
+        fails += [f"{rel}: {x}" for x in problems]
+    for rel, h in ref.get("advisory", {}).items():
+        p = root / rel
+        state = "missing" if not p.exists() else ("same" if sha256_file(p) == h else "differs (advisory only, not an error)")
+        print(f"ADVISORY {rel}: {state}")
+    if extra:
+        print("NOTE extra video files (not in our copy): " + ", ".join(extra[:5]))
+    print(f"VERIFY_DATASET_V30 {'PASS' if not fails else 'FAIL'}" + ("" if not fails else " | " + " | ".join(fails[:10])))
+    return 0 if not fails else 1
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("data", type=Path, help="LeRobot v3.0 dataset dir (arx_cups_pi05)")
+    ap.add_argument("--reference", type=Path, default=REF_DEFAULT)
+    ap.add_argument("--make-reference", action="store_true", help="write the reference from DATA instead of checking")
+    args = ap.parse_args()
+    if args.make_reference:
+        args.reference.write_text(json.dumps(make_reference(args.data), indent=1, sort_keys=True) + "\n")
+        print(f"REFERENCE_WRITTEN {args.reference}")
+        return 0
+    return verify(args.data, json.loads(args.reference.read_text()))
+
+
+if __name__ == "__main__":
+    sys.exit(main())