| # Dataset format (LeRobot v2.1, T-Rex force layout) |
|
|
| ``` |
| <root>/ |
| meta/info.json codebase_version "v2.1", fps, features, totals |
| meta/episodes.jsonl {"episode_index", "tasks": [str], "length"} |
| meta/tasks.jsonl {"task_index", "task"} |
| meta/stats.json mean/std/min/max/q01/q99 per feature |
| meta/relative_stats_dreamzero.json {"eef62": {q01, q99, ...}} over 20 Hz delta-base action chunks |
| meta/modality.json, embodiment.json GR00T/DreamZero modality description (state/action/video keys) |
| meta/dataset_ready.json written by scripts/check_dataset_ready.py |
| data/chunk-XXX/episode_XXXXXX.parquet |
| videos/chunk-XXX/<view>/episode_XXXXXX.mp4 |
| ``` |
|
|
| ## Parquet columns used by FACT |
|
|
| | column | shape | description | |
| |---|---|---| |
| | `timestamp` | float | native frame time (any rate; FACT resamples to a 20 Hz logical grid by nearest timestamp) | |
| | `observation.state_eef62` | [62] | left arm `[xyz(3), rot6d(6)]`, left hand (22), right arm (9), right hand (22) | |
| | `action.eef62_absolute` | [62] | commanded absolute EEF pose / hand targets in the same layout | |
| | `observation.tactile_force` | [60] | 10 F6 sensors × (Fx, Fy, Fz, Mx, My, Mz), finger order: left thumb..pinky, right thumb..pinky | |
| | `observation.state`, `action` | [58] | raw joint-space state / command (kept for reference, not used by FACT) | |
|
|
| `rot6d` is the first two columns of the rotation matrix. Delta-base actions |
| (`fact.data_utils.eef62_delta_base`): arm translation and rotation expressed in the frame of |
| the block reference state (`R_ref^T (t - t_ref)`, `R_ref^T R_t`); hand targets stay absolute. |
|
|
| ## Videos |
|
|
| Three RGB views are required: `observation.images.head_left`, `observation.images.left_wrist`, |
| `observation.images.right_wrist`. The offline tokenizer resizes each view to 160×320, tiles |
| them into a 2×2 grid `[head | right ; left | black]`, resizes back to 160×320 and encodes |
| single frames with the Wan2.2 VAE (`[48,10,20]` latent per frame). Deformation (tactile |
| camera) videos are never decoded. |
|
|
| ## Caches |
|
|
| ``` |
| <cache>/latents/episode_XXXXXX.npy float16 [N,48,10,20], one row per parquet frame (memory-mapped) |
| <cache>/text_cache.npz tokens [n_tasks,T,4096] float16, mask [n_tasks,T] bool, task_index [n_tasks] |
| ``` |
|
|
| Built by `scripts/build_cache_dataset.py --what latents|text` (automatically by |
| `scripts/train_full.sh`). |
|
|
| ## Sharpa north_ces datasets |
| |
| RoCo-TaskBoardAssembly, Robotic-Origami-Challenge and SharpaDex-v1.0 (Sharpa north_ces |
| humanoid, 65 joints, Sharpa Wave hands, 10-finger F6 tactile) were converted into this layout. |
| `observation.state_eef62` was obtained with a product-of-exponentials forward kinematics of the |
| torso + arm chain fitted to the recorded `tcp_pose` (0.00 mm residual; parameters in |
| `data_tools/north_ces_fk_params.json`), the 22 hand joints map one-to-one, and |
| `action.eef62_absolute` is the commanded `action.tcp_pose` / hand targets. Episodes without |
| tactile force were skipped. `data_tools/build_merged_dataset.py` merges such datasets with |
| the T-Rex data. |
|
|
