Dataset format (LeRobot v2.1, T-Rex force layout)
<root>/
meta/info.json codebase_version "v2.1", fps, features, totals
meta/episodes.jsonl {"episode_index", "tasks": [str], "length"}
meta/tasks.jsonl {"task_index", "task"}
meta/stats.json mean/std/min/max/q01/q99 per feature
meta/relative_stats_dreamzero.json {"eef62": {q01, q99, ...}} over 20 Hz delta-base action chunks
meta/modality.json, embodiment.json GR00T/DreamZero modality description (state/action/video keys)
meta/dataset_ready.json written by scripts/check_dataset_ready.py
data/chunk-XXX/episode_XXXXXX.parquet
videos/chunk-XXX/<view>/episode_XXXXXX.mp4
Parquet columns used by FACT
| column | shape | description |
|---|---|---|
timestamp |
float | native frame time (any rate; FACT resamples to a 20 Hz logical grid by nearest timestamp) |
observation.state_eef62 |
[62] | left arm [xyz(3), rot6d(6)], left hand (22), right arm (9), right hand (22) |
action.eef62_absolute |
[62] | commanded absolute EEF pose / hand targets in the same layout |
observation.tactile_force |
[60] | 10 F6 sensors × (Fx, Fy, Fz, Mx, My, Mz), finger order: left thumb..pinky, right thumb..pinky |
observation.state, action |
[58] | raw joint-space state / command (kept for reference, not used by FACT) |
rot6d is the first two columns of the rotation matrix. Delta-base actions
(fact.data_utils.eef62_delta_base): arm translation and rotation expressed in the frame of
the block reference state (R_ref^T (t - t_ref), R_ref^T R_t); hand targets stay absolute.
Videos
Three RGB views are required: observation.images.head_left, observation.images.left_wrist,
observation.images.right_wrist. The offline tokenizer resizes each view to 160×320, tiles
them into a 2×2 grid [head | right ; left | black], resizes back to 160×320 and encodes
single frames with the Wan2.2 VAE ([48,10,20] latent per frame). Deformation (tactile
camera) videos are never decoded.
Caches
<cache>/latents/episode_XXXXXX.npy float16 [N,48,10,20], one row per parquet frame (memory-mapped)
<cache>/text_cache.npz tokens [n_tasks,T,4096] float16, mask [n_tasks,T] bool, task_index [n_tasks]
Built by scripts/build_cache_dataset.py --what latents|text (automatically by
scripts/train_full.sh).
Sharpa north_ces datasets
RoCo-TaskBoardAssembly, Robotic-Origami-Challenge and SharpaDex-v1.0 (Sharpa north_ces
humanoid, 65 joints, Sharpa Wave hands, 10-finger F6 tactile) were converted into this layout.
observation.state_eef62 was obtained with a product-of-exponentials forward kinematics of the
torso + arm chain fitted to the recorded tcp_pose (0.00 mm residual; parameters in
data_tools/north_ces_fk_params.json), the 22 hand joints map one-to-one, and
action.eef62_absolute is the commanded action.tcp_pose / hand targets. Episodes without
tactile force were skipped. data_tools/build_merged_dataset.py merges such datasets with
the T-Rex data.