diff --git a/README.md b/README.md index b2e14d2175282ad41498bef06fca09f23e771aac..9ea537a306830bbbbcea050c6f9a552757e8116d 100644 --- a/README.md +++ b/README.md @@ -210,3 +210,21 @@ normalization statistics and the offline metrics, `protocol.json` with the weigh b6-pedal-bin/b512/SHA256SUMS` from the repository root). Load and resume as the A18 example above with `'b6-pedal-bin/b512/step-004000'`, `robot='g1'`, `--contract g1_sonic_dfx6` and the dataset `humanoid-benchmark/b6-pedal-bin` `v5/` with `--episodes 0:100` (the resume itself was not test-run from this layout). + +## B5, B8 and A20 paused continuation packages (2026-11-17) + +The following independent runs resumed the local step-4,000 baselines and were paused by the owner. +Each package contains a **step-6,000** inference checkpoint and full optimizer/EMA resume state, +configuration, logs, dataset revision, hashes and portable resume instructions. The intended total +was 20,000; later logged updates were not checkpointed. These are separate from the existing +A20 `b512-trimv2-cont` step-10,000/20,000/30,000 lineage. + +| Task | Continuation package | Saved step | +| --- | --- | --- | +| B5 | [b5-chair-push-standing-start/b512-cont-20261117/](b5-chair-push-standing-start/b512-cont-20261117/) | 6,000 | +| B8 | [b8-table-wipe/b512-cont-20261117/](b8-table-wipe/b512-cont-20261117/) | 6,000 | +| A20 | [a20-basket-transfer/b512-trimv2-cont-20261117/](a20-basket-transfer/b512-trimv2-cont-20261117/) | 6,000 | + +Shared frozen training code and the opt-in resume launcher: [`code/fmdit_continue_20261117/`](code/fmdit_continue_20261117/). +The three source datasets are referenced at pinned revisions and are not redistributed here. +Offline errors are training-frame checks, not held-out or closed-loop performance. diff --git a/a20-basket-transfer/b512-trimv2-cont-20261117/RESUME.md b/a20-basket-transfer/b512-trimv2-cont-20261117/RESUME.md new file mode 100644 index 0000000000000000000000000000000000000000..9dea36c4dbacd5fb51efbf1ad320eb4aa8d380e2 --- /dev/null +++ b/a20-basket-transfer/b512-trimv2-cont-20261117/RESUME.md @@ -0,0 +1,39 @@ +# A20 FM-DiT continuation, paused 2026-11-17 + +This is an independent continuation from the local step-4,000 baseline. +The portable training state is **step 6,000**; the last logged step is 6500. +Updates after step 6,000 remain only in the paused local process and are not in this release. +The training target was 20,000 total steps. This package is not a completed 20,000-step run. + +## Files + +- `train_state.pt`: raw policy, AdamW moments, LR scheduler, EMA, step and full config. +- `step-006000/`: EMA inference checkpoint, config and weight hash protocol. +- `run.json`, `resume.json`: recipe, data identity, preflight and pause details. +- `logs/`: current and original training logs. Log rows after 6,000 are history, not saved weights. +- `provenance/`: original run, original checkpoint config and hash protocol. +- `SHA256SUMS`: hashes of this package and its shared source snapshot, relative to repository root. + +## Resume + +Use Python 3.12 and install `code/fmdit_continue_20261117/baselines/requirements.txt`. +Download the original dataset separately (access to its private repository is required): + +```bash +hf download humanoid-benchmark/a20-basket-transfer-pre-reach-trim-expert-v2 --repo-type dataset --revision c4df4d40e51c439d7b41697d7c0d0184c844cb93 --local-dir /path/to/a20-basket-transfer-pre-reach-trim-expert-v2 +python code/fmdit_continue_20261117/resume.py --task a20 --dataset /path/to/a20-basket-transfer-pre-reach-trim-expert-v2 --out /path/to/new_a20_run --cache-dir /path/to/frame_cache --steps 20000 --device cuda:0 +``` + +Run these commands from the downloaded model repository root. The launcher resolves its source +and state relative to itself, copies the state to a new output directory and starts the frozen +trainer with `--resume`; add `--dry-run` to inspect without copying or training. +Confirm the trainer logs `resumed` at step 6,000. `--steps` means total steps, not additional steps; +use 16000 for 10,000 more steps from this checkpoint. Existing output state is never overwritten. +No training automatically starts when this repository is downloaded. + +Batch 512, seed 0, state44/action76, 16-row chunks, 168x224 head/chest RGB, hb-v2 augmentation, +500-step original warm-up, AdamW and EMA are unchanged. At the default total of 20,000, +the saved LR/scheduler continues consistently. Choosing a different total changes cosine decay +from the next scheduler update. Original RNG and DataLoader state were not recorded; this is +not a bit-exact continuation. Offline MSE uses 256 training frames, not held-out validation +or closed-loop success. Source snapshot: `code/fmdit_continue_20261117/` (original commit `028df1bb7024fb1e3d176de7e60c84d3333cd97f`). diff --git a/a20-basket-transfer/b512-trimv2-cont-20261117/SHA256SUMS b/a20-basket-transfer/b512-trimv2-cont-20261117/SHA256SUMS new file mode 100644 index 0000000000000000000000000000000000000000..82b206124546902fd86bbd6a289152160572e571 --- /dev/null +++ b/a20-basket-transfer/b512-trimv2-cont-20261117/SHA256SUMS @@ -0,0 +1,28 @@ +b9a7be40fc2c7aae17802fcaf44778a35421ed4736d83a82ba3339ea32122e1b a20-basket-transfer/b512-trimv2-cont-20261117/RESUME.md +dadb1cccff305eada783576cfb657c490f4a950c0960f693ea6da19db66bb29f a20-basket-transfer/b512-trimv2-cont-20261117/logs/parent_train_log.jsonl +40e848ed606c0bd8760a325bed9f30f57051b897aacbee4251826eefaf82226d a20-basket-transfer/b512-trimv2-cont-20261117/logs/train_log.jsonl +1f0193de17b46af4a81d456975d2b2644eb95831b2a8c370a0d1031bc5231ff0 a20-basket-transfer/b512-trimv2-cont-20261117/logs/train_stdout.log +636958d8eabe7dc28647a0100c346ef84f489b28da180a85d3386126b499a633 a20-basket-transfer/b512-trimv2-cont-20261117/provenance/parent_checkpoint_config.json +67840d199f99f3e23b7a7ab031268d5207abb1ceed99219610ee6c1127a1fe51 a20-basket-transfer/b512-trimv2-cont-20261117/provenance/parent_checkpoint_protocol.json +6c35080e05047caa0c14ba114078f106a9f6ae508dc48ba2f6ad42500e0c5ecc a20-basket-transfer/b512-trimv2-cont-20261117/provenance/parent_run.json +e107b7249679619e19029c5e27aa231aec703ed0fe0a3a9223de4fe424524d29 a20-basket-transfer/b512-trimv2-cont-20261117/resume.json +f23534ee016f8a4d3a2ea52ed651b695d9b5498d659d1ad49917e125767c38af a20-basket-transfer/b512-trimv2-cont-20261117/run.json +4b459c70526cd0e2c9c48b8d0d1e03975905c68a0454180b3f607a2d5b5968bb a20-basket-transfer/b512-trimv2-cont-20261117/step-006000/config.json +e6330966ba75b01c067645e78990edf9ad0c9d1bc25a27a62daf7b27ae9d86d2 a20-basket-transfer/b512-trimv2-cont-20261117/step-006000/model.pt +45929f8f33db892e017b7cad28c9a5981cb1f44608456171b125025e98d87e4d a20-basket-transfer/b512-trimv2-cont-20261117/step-006000/protocol.json +6d915877204726b60be63b55ae1b9714eb124f74236d86cd541195206297317c a20-basket-transfer/b512-trimv2-cont-20261117/train_state.pt +f7cf756cb60f0ce1da9b7ab8b1549b42f9a59d608bb06addd206d270b7c2d14c code/fmdit_continue_20261117/augmentation/__init__.py +96539b2fcf729917f668c9a776a0526c4b109611bf921dcbd753983c0f5813f2 code/fmdit_continue_20261117/augmentation/check_spec.py +9f7713cea4f846efcfc924234267ced48210357f419d67915c10b504316bedde code/fmdit_continue_20261117/augmentation/hb_aug_v2.py +815562186d3aaf87aba05085da6e951bc48b0b90a5d1c4910f2a3f625009bc52 code/fmdit_continue_20261117/augmentation/preview.py +0a8ee3ccab6e183804c08066a9e7eb207d9b13ca38c0e596ce5d58c8302d7929 code/fmdit_continue_20261117/baselines/__init__.py +3db95cd8e4e436875e11f1054a99d250ca7d94e0127909090b58730402340f19 code/fmdit_continue_20261117/baselines/act_nets.py +b537b03bc538910691b599862138494c11354147785a779a22ad83ff4b72a8d5 code/fmdit_continue_20261117/baselines/adapters.py +88c1971e18e9842136884f6a369468fe27f189b888b39870622304101a063b06 code/fmdit_continue_20261117/baselines/data.py +38a66937bb92393a3f1ce0644b249da28d582a4fd03d134b7743e6037abd88b9 code/fmdit_continue_20261117/baselines/nets.py +750d4c6193aabcacc0f53df26c2c51f64b3f836b940b32ccb5c69f5959419afa code/fmdit_continue_20261117/baselines/policies.py +bb211f66038e53efd08ee3ec03e133d8bb206ec36cc608d349b4fad3a61e874b code/fmdit_continue_20261117/baselines/requirements.txt +5d680d59124ca83f9c2088f8e1738395b87ddc58798acf7815e163102738b7ae code/fmdit_continue_20261117/baselines/runtime.py +b368d14aaab69d04f5372b1f3bb358c2a78f83134cc0e926faa8887e3ddbe297 code/fmdit_continue_20261117/baselines/train.py +fe9aaed24789080f05ce8f510d5a132bbd65e7c39b6de20500210742f0be4185 code/fmdit_continue_20261117/docs/augmentation_v2.md +5dbc2ec3f4f0638181eef9dca07c76f3594bdd5a24141f1a4b50d6532dbe4bba code/fmdit_continue_20261117/resume.py diff --git a/a20-basket-transfer/b512-trimv2-cont-20261117/logs/parent_train_log.jsonl b/a20-basket-transfer/b512-trimv2-cont-20261117/logs/parent_train_log.jsonl new file mode 100644 index 0000000000000000000000000000000000000000..3d26eaceac1f363e75332dec9011a1ee3f0b67b0 --- /dev/null +++ b/a20-basket-transfer/b512-trimv2-cont-20261117/logs/parent_train_log.jsonl @@ -0,0 +1,41 @@ +{"event": "train", "step": 100, "lr": 2.0200000000000003e-05, "backbone_lr": 2.02e-06, "ema_decay": 0.968612, "it_s": 0.39, "data_frac": 0.913, "fm": 1.8155318808555603, "loss": 1.8155318808555603} +{"event": "train", "step": 200, "lr": 4.02e-05, "backbone_lr": 4.0200000000000005e-06, "ema_decay": 0.981267, "it_s": 0.39, "data_frac": 0.919, "fm": 1.0865715634822846, "loss": 1.0865715634822846} +{"event": "train", "step": 300, "lr": 6.02e-05, "backbone_lr": 6.02e-06, "ema_decay": 0.986162, "it_s": 0.44, "data_frac": 0.91, "fm": 0.8929706597328186, "loss": 0.8929706597328186} +{"event": "train", "step": 400, "lr": 8.020000000000001e-05, "backbone_lr": 8.020000000000001e-06, "ema_decay": 0.988841, "it_s": 0.42, "data_frac": 0.915, "fm": 0.36802623093128206, "loss": 0.36802623093128206} +{"event": "train", "step": 500, "lr": 0.0001, "backbone_lr": 1e-05, "ema_decay": 0.990557, "it_s": 0.4, "data_frac": 0.919, "fm": 0.22475649788975716, "loss": 0.22475649788975716} +{"event": "train", "step": 600, "lr": 9.979871469976196e-05, "backbone_lr": 9.979871469976197e-06, "ema_decay": 0.991762, "it_s": 0.42, "data_frac": 0.914, "fm": 0.19103719875216485, "loss": 0.19103719875216485} +{"event": "train", "step": 700, "lr": 9.919647942993148e-05, "backbone_lr": 9.91964794299315e-06, "ema_decay": 0.99266, "it_s": 0.39, "data_frac": 0.921, "fm": 0.17412877932190896, "loss": 0.17412877932190896} +{"event": "train", "step": 800, "lr": 9.819814303479267e-05, "backbone_lr": 9.819814303479268e-06, "ema_decay": 0.993358, "it_s": 0.43, "data_frac": 0.911, "fm": 0.16297349587082863, "loss": 0.16297349587082863} +{"event": "train", "step": 900, "lr": 9.681174353198687e-05, "backbone_lr": 9.681174353198687e-06, "ema_decay": 0.993919, "it_s": 0.42, "data_frac": 0.912, "fm": 0.1552828885614872, "loss": 0.1552828885614872} +{"event": "train", "step": 1000, "lr": 9.504844339512095e-05, "backbone_lr": 9.504844339512096e-06, "ema_decay": 0.994381, "it_s": 0.4, "data_frac": 0.92, "fm": 0.14945878624916076, "loss": 0.14945878624916076} +{"event": "train", "step": 1100, "lr": 9.292243968009331e-05, "backbone_lr": 9.292243968009332e-06, "ema_decay": 0.994768, "it_s": 0.41, "data_frac": 0.916, "fm": 0.14591697677969934, "loss": 0.14591697677969934} +{"event": "train", "step": 1200, "lr": 9.045084971874738e-05, "backbone_lr": 9.045084971874738e-06, "ema_decay": 0.995098, "it_s": 0.4, "data_frac": 0.917, "fm": 0.14123237133026123, "loss": 0.14123237133026123} +{"event": "train", "step": 1300, "lr": 8.765357330018056e-05, "backbone_lr": 8.765357330018056e-06, "ema_decay": 0.995384, "it_s": 0.38, "data_frac": 0.922, "fm": 0.1380928837507963, "loss": 0.1380928837507963} +{"event": "train", "step": 1400, "lr": 8.455313244934324e-05, "backbone_lr": 8.455313244934324e-06, "ema_decay": 0.995633, "it_s": 0.44, "data_frac": 0.911, "fm": 0.1355636489391327, "loss": 0.1355636489391327} +{"event": "train", "step": 1500, "lr": 8.117449009293668e-05, "backbone_lr": 8.117449009293668e-06, "ema_decay": 0.995853, "it_s": 0.4, "data_frac": 0.915, "fm": 0.13267861403524875, "loss": 0.13267861403524875} +{"event": "train", "step": 1600, "lr": 7.754484907260513e-05, "backbone_lr": 7.754484907260513e-06, "ema_decay": 0.996049, "it_s": 0.42, "data_frac": 0.912, "fm": 0.13047040574252605, "loss": 0.13047040574252605} +{"event": "train", "step": 1700, "lr": 7.369343312364993e-05, "backbone_lr": 7.369343312364994e-06, "ema_decay": 0.996225, "it_s": 0.36, "data_frac": 0.925, "fm": 0.13081603579223156, "loss": 0.13081603579223156} +{"event": "train", "step": 1800, "lr": 6.965125158269619e-05, "backbone_lr": 6.965125158269619e-06, "ema_decay": 0.996383, "it_s": 0.48, "data_frac": 0.903, "fm": 0.12673527278006078, "loss": 0.12673527278006078} +{"event": "train", "step": 1900, "lr": 6.545084971874738e-05, "backbone_lr": 6.545084971874738e-06, "ema_decay": 0.996527, "it_s": 0.42, "data_frac": 0.911, "fm": 0.1263776320964098, "loss": 0.1263776320964098} +{"event": "train", "step": 2000, "lr": 6.112604669781572e-05, "backbone_lr": 6.112604669781572e-06, "ema_decay": 0.996658, "it_s": 0.38, "data_frac": 0.923, "fm": 0.1255116108059883, "loss": 0.1255116108059883} +{"event": "train", "step": 2100, "lr": 5.6711663290882776e-05, "backbone_lr": 5.671166329088278e-06, "ema_decay": 0.996778, "it_s": 0.45, "data_frac": 0.906, "fm": 0.12182304225862026, "loss": 0.12182304225862026} +{"event": "train", "step": 2200, "lr": 5.2243241517525754e-05, "backbone_lr": 5.224324151752575e-06, "ema_decay": 0.996888, "it_s": 0.38, "data_frac": 0.922, "fm": 0.12228876903653145, "loss": 0.12228876903653145} +{"event": "train", "step": 2300, "lr": 4.775675848247427e-05, "backbone_lr": 4.775675848247427e-06, "ema_decay": 0.99699, "it_s": 0.41, "data_frac": 0.913, "fm": 0.11980235382914543, "loss": 0.11980235382914543} +{"event": "train", "step": 2400, "lr": 4.328833670911724e-05, "backbone_lr": 4.3288336709117246e-06, "ema_decay": 0.997085, "it_s": 0.43, "data_frac": 0.915, "fm": 0.11944346323609352, "loss": 0.11944346323609352} +{"event": "train", "step": 2500, "lr": 3.887395330218429e-05, "backbone_lr": 3.887395330218429e-06, "ema_decay": 0.997172, "it_s": 0.39, "data_frac": 0.92, "fm": 0.11789588399231433, "loss": 0.11789588399231433} +{"event": "train", "step": 2600, "lr": 3.4549150281252636e-05, "backbone_lr": 3.4549150281252635e-06, "ema_decay": 0.997254, "it_s": 0.43, "data_frac": 0.912, "fm": 0.11748422630131244, "loss": 0.11748422630131244} +{"event": "train", "step": 2700, "lr": 3.0348748417303823e-05, "backbone_lr": 3.0348748417303826e-06, "ema_decay": 0.997331, "it_s": 0.43, "data_frac": 0.914, "fm": 0.1170263933390379, "loss": 0.1170263933390379} +{"event": "train", "step": 2800, "lr": 2.630656687635007e-05, "backbone_lr": 2.6306566876350072e-06, "ema_decay": 0.997403, "it_s": 0.4, "data_frac": 0.918, "fm": 0.11635425724089146, "loss": 0.11635425724089146} +{"event": "train", "step": 2900, "lr": 2.245515092739488e-05, "backbone_lr": 2.245515092739488e-06, "ema_decay": 0.99747, "it_s": 0.43, "data_frac": 0.911, "fm": 0.11531534694135188, "loss": 0.11531534694135188} +{"event": "train", "step": 3000, "lr": 1.8825509907063327e-05, "backbone_lr": 1.8825509907063328e-06, "ema_decay": 0.997534, "it_s": 0.45, "data_frac": 0.909, "fm": 0.11441182516515255, "loss": 0.11441182516515255} +{"event": "train", "step": 3100, "lr": 1.544686755065677e-05, "backbone_lr": 1.544686755065677e-06, "ema_decay": 0.997594, "it_s": 0.41, "data_frac": 0.916, "fm": 0.11447032988071441, "loss": 0.11447032988071441} +{"event": "train", "step": 3200, "lr": 1.2346426699819458e-05, "backbone_lr": 1.234642669981946e-06, "ema_decay": 0.99765, "it_s": 0.42, "data_frac": 0.911, "fm": 0.11343133807182312, "loss": 0.11343133807182312} +{"event": "train", "step": 3300, "lr": 9.549150281252633e-06, "backbone_lr": 9.549150281252633e-07, "ema_decay": 0.997704, "it_s": 0.43, "data_frac": 0.911, "fm": 0.11302094019949437, "loss": 0.11302094019949437} +{"event": "train", "step": 3400, "lr": 7.077560319906695e-06, "backbone_lr": 7.077560319906696e-07, "ema_decay": 0.997755, "it_s": 0.42, "data_frac": 0.913, "fm": 0.11316528491675854, "loss": 0.11316528491675854} +{"event": "train", "step": 3500, "lr": 4.951556604879048e-06, "backbone_lr": 4.951556604879049e-07, "ema_decay": 0.997803, "it_s": 0.41, "data_frac": 0.915, "fm": 0.11302028380334378, "loss": 0.11302028380334378} +{"event": "train", "step": 3600, "lr": 3.18825646801314e-06, "backbone_lr": 3.18825646801314e-07, "ema_decay": 0.997849, "it_s": 0.44, "data_frac": 0.91, "fm": 0.11321711145341397, "loss": 0.11321711145341397} +{"event": "train", "step": 3700, "lr": 1.8018569652073381e-06, "backbone_lr": 1.801856965207338e-07, "ema_decay": 0.997893, "it_s": 0.45, "data_frac": 0.906, "fm": 0.11361600130796433, "loss": 0.11361600130796433} +{"event": "train", "step": 3800, "lr": 8.035205700685167e-07, "backbone_lr": 8.035205700685167e-08, "ema_decay": 0.997934, "it_s": 0.42, "data_frac": 0.913, "fm": 0.11098091438412666, "loss": 0.11098091438412666} +{"event": "train", "step": 3900, "lr": 2.012853002380466e-07, "backbone_lr": 2.012853002380466e-08, "ema_decay": 0.997974, "it_s": 0.41, "data_frac": 0.915, "fm": 0.11410101048648358, "loss": 0.11410101048648358} +{"event": "train", "step": 4000, "lr": 0.0, "backbone_lr": 0.0, "ema_decay": 0.998012, "it_s": 0.5, "data_frac": 0.903, "fm": 0.11127129688858986, "loss": 0.11127129688858986} +{"event": "save", "step": 4000, "checkpoint": "/mnt/data0/zuo/Humanoid_benchmark/results/baselines_preflight_20261117/runs/a20-basket-transfer-pre-reach-trim-expert-v2/fm_dit/checkpoints/step-004000", "mse": 0.025112804025411606, "mse_row0": 0.025062359869480133, "mse_motion_token": 0.02457592636346817, "mse_inspire_left_hand_state": 0.0052911583334207535, "mse_inspire_right_hand_state": 0.0506611168384552} diff --git a/a20-basket-transfer/b512-trimv2-cont-20261117/logs/train_log.jsonl b/a20-basket-transfer/b512-trimv2-cont-20261117/logs/train_log.jsonl new file mode 100644 index 0000000000000000000000000000000000000000..945be10f19e052393957358804e22a3d4a6ddf4d --- /dev/null +++ b/a20-basket-transfer/b512-trimv2-cont-20261117/logs/train_log.jsonl @@ -0,0 +1,101 @@ +{"event": "train", "step": 4025, "lr": 9.215152885383474e-05, "backbone_lr": 9.215152885383473e-06, "ema_decay": 0.998021, "it_s": 0.57, "data_frac": 0.831, "fm": 0.27936236292123795, "loss": 0.27936236292123795} +{"event": "train", "step": 4050, "lr": 9.204286963963111e-05, "backbone_lr": 9.204286963963112e-06, "ema_decay": 0.998031, "it_s": 0.95, "data_frac": 0.808, "fm": 0.1851240873336792, "loss": 0.1851240873336792} +{"event": "train", "step": 4075, "lr": 9.193352839727121e-05, "backbone_lr": 9.193352839727122e-06, "ema_decay": 0.99804, "it_s": 1.04, "data_frac": 0.784, "fm": 0.1523358416557312, "loss": 0.1523358416557312} +{"event": "train", "step": 4100, "lr": 9.182350690051133e-05, "backbone_lr": 9.182350690051134e-06, "ema_decay": 0.998049, "it_s": 2.28, "data_frac": 0.577, "fm": 0.13585892975330352, "loss": 0.13585892975330352} +{"event": "train", "step": 4125, "lr": 9.171280693414307e-05, "backbone_lr": 9.171280693414307e-06, "ema_decay": 0.998058, "it_s": 0.66, "data_frac": 0.864, "fm": 0.12647916853427887, "loss": 0.12647916853427887} +{"event": "train", "step": 4150, "lr": 9.160143029396422e-05, "backbone_lr": 9.160143029396422e-06, "ema_decay": 0.998066, "it_s": 0.99, "data_frac": 0.799, "fm": 0.12340139746665954, "loss": 0.12340139746665954} +{"event": "train", "step": 4175, "lr": 9.148937878674976e-05, "backbone_lr": 9.148937878674975e-06, "ema_decay": 0.998075, "it_s": 1.39, "data_frac": 0.725, "fm": 0.12282259106636047, "loss": 0.12282259106636047} +{"event": "train", "step": 4200, "lr": 9.13766542302225e-05, "backbone_lr": 9.13766542302225e-06, "ema_decay": 0.998084, "it_s": 2.26, "data_frac": 0.559, "fm": 0.12243758767843246, "loss": 0.12243758767843246} +{"event": "train", "step": 4225, "lr": 9.126325845302357e-05, "backbone_lr": 9.126325845302357e-06, "ema_decay": 0.998092, "it_s": 0.69, "data_frac": 0.866, "fm": 0.11850759595632553, "loss": 0.11850759595632553} +{"event": "train", "step": 4250, "lr": 9.114919329468282e-05, "backbone_lr": 9.114919329468283e-06, "ema_decay": 0.998101, "it_s": 1.05, "data_frac": 0.781, "fm": 0.11744395792484283, "loss": 0.11744395792484283} +{"event": "train", "step": 4275, "lr": 9.103446060558892e-05, "backbone_lr": 9.103446060558892e-06, "ema_decay": 0.998109, "it_s": 2.47, "data_frac": 0.489, "fm": 0.11543788164854049, "loss": 0.11543788164854049} +{"event": "train", "step": 4300, "lr": 9.091906224695935e-05, "backbone_lr": 9.091906224695935e-06, "ema_decay": 0.998117, "it_s": 1.0, "data_frac": 0.801, "fm": 0.11410628646612167, "loss": 0.11410628646612167} +{"event": "train", "step": 4325, "lr": 9.080300009081024e-05, "backbone_lr": 9.080300009081025e-06, "ema_decay": 0.998125, "it_s": 0.76, "data_frac": 0.853, "fm": 0.11376235336065292, "loss": 0.11376235336065292} +{"event": "train", "step": 4350, "lr": 9.068627601992598e-05, "backbone_lr": 9.0686276019926e-06, "ema_decay": 0.998133, "it_s": 2.17, "data_frac": 0.538, "fm": 0.11388265669345855, "loss": 0.11388265669345855} +{"event": "train", "step": 4375, "lr": 9.056889192782866e-05, "backbone_lr": 9.056889192782865e-06, "ema_decay": 0.998141, "it_s": 1.14, "data_frac": 0.771, "fm": 0.11311278849840165, "loss": 0.11311278849840165} +{"event": "train", "step": 4400, "lr": 9.045084971874738e-05, "backbone_lr": 9.045084971874738e-06, "ema_decay": 0.998149, "it_s": 0.86, "data_frac": 0.829, "fm": 0.1135650286078453, "loss": 0.1135650286078453} +{"event": "train", "step": 4425, "lr": 9.033215130758735e-05, "backbone_lr": 9.033215130758735e-06, "ema_decay": 0.998157, "it_s": 0.85, "data_frac": 0.833, "fm": 0.11657118797302246, "loss": 0.11657118797302246} +{"event": "train", "step": 4450, "lr": 9.021279861989885e-05, "backbone_lr": 9.021279861989884e-06, "ema_decay": 0.998165, "it_s": 3.22, "data_frac": 0.377, "fm": 0.11299996435642243, "loss": 0.11299996435642243} +{"event": "train", "step": 4475, "lr": 9.009279359184593e-05, "backbone_lr": 9.009279359184594e-06, "ema_decay": 0.998173, "it_s": 0.85, "data_frac": 0.83, "fm": 0.10958211928606033, "loss": 0.10958211928606033} +{"event": "train", "step": 4500, "lr": 8.997213817017507e-05, "backbone_lr": 8.997213817017508e-06, "ema_decay": 0.99818, "it_s": 1.24, "data_frac": 0.752, "fm": 0.1114171290397644, "loss": 0.1114171290397644} +{"event": "train", "step": 4525, "lr": 8.985083431218354e-05, "backbone_lr": 8.985083431218354e-06, "ema_decay": 0.998188, "it_s": 1.41, "data_frac": 0.705, "fm": 0.10957210212945938, "loss": 0.10957210212945938} +{"event": "train", "step": 4550, "lr": 8.972888398568772e-05, "backbone_lr": 8.972888398568772e-06, "ema_decay": 0.998195, "it_s": 0.93, "data_frac": 0.824, "fm": 0.11293533921241761, "loss": 0.11293533921241761} +{"event": "train", "step": 4575, "lr": 8.96062891689911e-05, "backbone_lr": 8.96062891689911e-06, "ema_decay": 0.998203, "it_s": 1.19, "data_frac": 0.765, "fm": 0.11222538262605668, "loss": 0.11222538262605668} +{"event": "train", "step": 4600, "lr": 8.948305185085225e-05, "backbone_lr": 8.948305185085226e-06, "ema_decay": 0.99821, "it_s": 1.01, "data_frac": 0.795, "fm": 0.11155075877904892, "loss": 0.11155075877904892} +{"event": "train", "step": 4625, "lr": 8.935917403045251e-05, "backbone_lr": 8.935917403045251e-06, "ema_decay": 0.998217, "it_s": 5.14, "data_frac": 0.002, "fm": 0.10862381368875504, "loss": 0.10862381368875504} +{"event": "train", "step": 4650, "lr": 8.92346577173636e-05, "backbone_lr": 8.923465771736361e-06, "ema_decay": 0.998224, "it_s": 0.52, "data_frac": 0.895, "fm": 0.10961597681045532, "loss": 0.10961597681045532} +{"event": "train", "step": 4675, "lr": 8.910950493151498e-05, "backbone_lr": 8.910950493151498e-06, "ema_decay": 0.998232, "it_s": 1.65, "data_frac": 0.683, "fm": 0.1106258323788643, "loss": 0.1106258323788643} +{"event": "train", "step": 4700, "lr": 8.898371770316111e-05, "backbone_lr": 8.898371770316113e-06, "ema_decay": 0.998239, "it_s": 4.67, "data_frac": 0.005, "fm": 0.10965090662240982, "loss": 0.10965090662240982} +{"event": "train", "step": 4725, "lr": 8.885729807284856e-05, "backbone_lr": 8.885729807284855e-06, "ema_decay": 0.998246, "it_s": 0.88, "data_frac": 0.826, "fm": 0.10638074308633805, "loss": 0.10638074308633805} +{"event": "train", "step": 4750, "lr": 8.873024809138272e-05, "backbone_lr": 8.873024809138272e-06, "ema_decay": 0.998253, "it_s": 0.58, "data_frac": 0.887, "fm": 0.10897882670164108, "loss": 0.10897882670164108} +{"event": "train", "step": 4775, "lr": 8.860256981979484e-05, "backbone_lr": 8.860256981979485e-06, "ema_decay": 0.998259, "it_s": 2.27, "data_frac": 0.517, "fm": 0.10848556160926819, "loss": 0.10848556160926819} +{"event": "train", "step": 4800, "lr": 8.847426532930831e-05, "backbone_lr": 8.84742653293083e-06, "ema_decay": 0.998266, "it_s": 1.74, "data_frac": 0.674, "fm": 0.10899475157260895, "loss": 0.10899475157260895} +{"event": "train", "step": 4825, "lr": 8.834533670130518e-05, "backbone_lr": 8.834533670130518e-06, "ema_decay": 0.998273, "it_s": 1.27, "data_frac": 0.746, "fm": 0.1064724487066269, "loss": 0.1064724487066269} +{"event": "train", "step": 4850, "lr": 8.821578602729242e-05, "backbone_lr": 8.821578602729242e-06, "ema_decay": 0.99828, "it_s": 0.56, "data_frac": 0.882, "fm": 0.10697787135839462, "loss": 0.10697787135839462} +{"event": "train", "step": 4875, "lr": 8.808561540886796e-05, "backbone_lr": 8.808561540886796e-06, "ema_decay": 0.998286, "it_s": 4.25, "data_frac": 0.164, "fm": 0.10751055479049683, "loss": 0.10751055479049683} +{"event": "train", "step": 4900, "lr": 8.795482695768658e-05, "backbone_lr": 8.795482695768658e-06, "ema_decay": 0.998293, "it_s": 0.87, "data_frac": 0.83, "fm": 0.10378085047006608, "loss": 0.10378085047006608} +{"event": "train", "step": 4925, "lr": 8.782342279542568e-05, "backbone_lr": 8.782342279542569e-06, "ema_decay": 0.998299, "it_s": 1.5, "data_frac": 0.722, "fm": 0.10504777759313583, "loss": 0.10504777759313583} +{"event": "train", "step": 4950, "lr": 8.769140505375085e-05, "backbone_lr": 8.769140505375084e-06, "ema_decay": 0.998306, "it_s": 1.55, "data_frac": 0.704, "fm": 0.10637212932109832, "loss": 0.10637212932109832} +{"event": "train", "step": 4975, "lr": 8.755877587428128e-05, "backbone_lr": 8.755877587428127e-06, "ema_decay": 0.998312, "it_s": 0.68, "data_frac": 0.866, "fm": 0.10244016528129578, "loss": 0.10244016528129578} +{"event": "train", "step": 5000, "lr": 8.742553740855506e-05, "backbone_lr": 8.742553740855507e-06, "ema_decay": 0.998318, "it_s": 1.02, "data_frac": 0.794, "fm": 0.10446281790733337, "loss": 0.10446281790733337} +{"event": "train", "step": 5025, "lr": 8.729169181799422e-05, "backbone_lr": 8.729169181799423e-06, "ema_decay": 0.998325, "it_s": 1.53, "data_frac": 0.68, "fm": 0.10380947828292847, "loss": 0.10380947828292847} +{"event": "train", "step": 5050, "lr": 8.715724127386972e-05, "backbone_lr": 8.715724127386971e-06, "ema_decay": 0.998331, "it_s": 3.21, "data_frac": 0.387, "fm": 0.10200979679822922, "loss": 0.10200979679822922} +{"event": "train", "step": 5075, "lr": 8.702218795726617e-05, "backbone_lr": 8.702218795726619e-06, "ema_decay": 0.998337, "it_s": 0.53, "data_frac": 0.893, "fm": 0.10482226371765137, "loss": 0.10482226371765137} +{"event": "train", "step": 5100, "lr": 8.688653405904652e-05, "backbone_lr": 8.688653405904653e-06, "ema_decay": 0.998343, "it_s": 1.19, "data_frac": 0.757, "fm": 0.1049795463681221, "loss": 0.1049795463681221} +{"event": "train", "step": 5125, "lr": 8.675028177981643e-05, "backbone_lr": 8.675028177981643e-06, "ema_decay": 0.998349, "it_s": 5.17, "data_frac": 0.046, "fm": 0.103239366710186, "loss": 0.103239366710186} +{"event": "train", "step": 5150, "lr": 8.661343332988869e-05, "backbone_lr": 8.661343332988869e-06, "ema_decay": 0.998355, "it_s": 1.54, "data_frac": 0.71, "fm": 0.10244131833314896, "loss": 0.10244131833314896} +{"event": "train", "step": 5175, "lr": 8.647599092924721e-05, "backbone_lr": 8.64759909292472e-06, "ema_decay": 0.998361, "it_s": 0.48, "data_frac": 0.898, "fm": 0.1028094071149826, "loss": 0.1028094071149826} +{"event": "train", "step": 5200, "lr": 8.633795680751116e-05, "backbone_lr": 8.633795680751116e-06, "ema_decay": 0.998367, "it_s": 2.63, "data_frac": 0.413, "fm": 0.10102613866329194, "loss": 0.10102613866329194} +{"event": "train", "step": 5225, "lr": 8.619933320389872e-05, "backbone_lr": 8.619933320389872e-06, "ema_decay": 0.998373, "it_s": 1.02, "data_frac": 0.802, "fm": 0.10271453976631165, "loss": 0.10271453976631165} +{"event": "train", "step": 5250, "lr": 8.606012236719073e-05, "backbone_lr": 8.606012236719073e-06, "ema_decay": 0.998379, "it_s": 1.75, "data_frac": 0.658, "fm": 0.1013492727279663, "loss": 0.1013492727279663} +{"event": "train", "step": 5275, "lr": 8.592032655569429e-05, "backbone_lr": 8.59203265556943e-06, "ema_decay": 0.998385, "it_s": 0.55, "data_frac": 0.869, "fm": 0.09972631871700287, "loss": 0.09972631871700287} +{"event": "train", "step": 5300, "lr": 8.577994803720606e-05, "backbone_lr": 8.577994803720605e-06, "ema_decay": 0.99839, "it_s": 3.74, "data_frac": 0.205, "fm": 0.09866629302501678, "loss": 0.09866629302501678} +{"event": "train", "step": 5325, "lr": 8.56389890889755e-05, "backbone_lr": 8.56389890889755e-06, "ema_decay": 0.998396, "it_s": 0.8, "data_frac": 0.837, "fm": 0.10182167440652848, "loss": 0.10182167440652848} +{"event": "train", "step": 5350, "lr": 8.549745199766792e-05, "backbone_lr": 8.549745199766792e-06, "ema_decay": 0.998402, "it_s": 1.59, "data_frac": 0.697, "fm": 0.09962393790483474, "loss": 0.09962393790483474} +{"event": "train", "step": 5375, "lr": 8.535533905932738e-05, "backbone_lr": 8.535533905932739e-06, "ema_decay": 0.998407, "it_s": 1.6, "data_frac": 0.699, "fm": 0.10165397435426712, "loss": 0.10165397435426712} +{"event": "train", "step": 5400, "lr": 8.521265257933948e-05, "backbone_lr": 8.521265257933948e-06, "ema_decay": 0.998413, "it_s": 0.66, "data_frac": 0.862, "fm": 0.09791133671998978, "loss": 0.09791133671998978} +{"event": "train", "step": 5425, "lr": 8.506939487239388e-05, "backbone_lr": 8.506939487239389e-06, "ema_decay": 0.998418, "it_s": 1.0, "data_frac": 0.787, "fm": 0.09465882360935211, "loss": 0.09465882360935211} +{"event": "train", "step": 5450, "lr": 8.492556826244687e-05, "backbone_lr": 8.492556826244687e-06, "ema_decay": 0.998424, "it_s": 1.36, "data_frac": 0.728, "fm": 0.10158959805965423, "loss": 0.10158959805965423} +{"event": "train", "step": 5475, "lr": 8.47811750826835e-05, "backbone_lr": 8.478117508268352e-06, "ema_decay": 0.998429, "it_s": 3.59, "data_frac": 0.308, "fm": 0.10000236868858338, "loss": 0.10000236868858338} +{"event": "train", "step": 5500, "lr": 8.463621767547998e-05, "backbone_lr": 8.463621767547998e-06, "ema_decay": 0.998434, "it_s": 0.58, "data_frac": 0.883, "fm": 0.09906157135963439, "loss": 0.09906157135963439} +{"event": "train", "step": 5525, "lr": 8.449069839236538e-05, "backbone_lr": 8.44906983923654e-06, "ema_decay": 0.99844, "it_s": 0.91, "data_frac": 0.811, "fm": 0.09683233767747879, "loss": 0.09683233767747879} +{"event": "train", "step": 5550, "lr": 8.434461959398376e-05, "backbone_lr": 8.434461959398377e-06, "ema_decay": 0.998445, "it_s": 2.24, "data_frac": 0.577, "fm": 0.097518649995327, "loss": 0.097518649995327} +{"event": "train", "step": 5575, "lr": 8.419798365005569e-05, "backbone_lr": 8.419798365005569e-06, "ema_decay": 0.99845, "it_s": 1.95, "data_frac": 0.639, "fm": 0.1010474818944931, "loss": 0.1010474818944931} +{"event": "train", "step": 5600, "lr": 8.405079293933986e-05, "backbone_lr": 8.405079293933986e-06, "ema_decay": 0.998455, "it_s": 0.5, "data_frac": 0.898, "fm": 0.09772510349750518, "loss": 0.09772510349750518} +{"event": "train", "step": 5625, "lr": 8.390304984959454e-05, "backbone_lr": 8.390304984959455e-06, "ema_decay": 0.998461, "it_s": 0.96, "data_frac": 0.794, "fm": 0.0974524575471878, "loss": 0.0974524575471878} +{"event": "train", "step": 5650, "lr": 8.375475677753881e-05, "backbone_lr": 8.375475677753882e-06, "ema_decay": 0.998466, "it_s": 3.22, "data_frac": 0.386, "fm": 0.09912837445735931, "loss": 0.09912837445735931} +{"event": "train", "step": 5675, "lr": 8.360591612881364e-05, "backbone_lr": 8.360591612881363e-06, "ema_decay": 0.998471, "it_s": 1.74, "data_frac": 0.669, "fm": 0.09766773462295532, "loss": 0.09766773462295532} +{"event": "train", "step": 5700, "lr": 8.345653031794292e-05, "backbone_lr": 8.345653031794292e-06, "ema_decay": 0.998476, "it_s": 0.57, "data_frac": 0.879, "fm": 0.10231035619974137, "loss": 0.10231035619974137} +{"event": "train", "step": 5725, "lr": 8.330660176829428e-05, "backbone_lr": 8.330660176829427e-06, "ema_decay": 0.998481, "it_s": 4.26, "data_frac": 0.134, "fm": 0.09899430811405181, "loss": 0.09899430811405181} +{"event": "train", "step": 5750, "lr": 8.315613291203976e-05, "backbone_lr": 8.315613291203977e-06, "ema_decay": 0.998486, "it_s": 0.83, "data_frac": 0.832, "fm": 0.095704605281353, "loss": 0.095704605281353} +{"event": "train", "step": 5775, "lr": 8.30051261901164e-05, "backbone_lr": 8.30051261901164e-06, "ema_decay": 0.998491, "it_s": 1.58, "data_frac": 0.702, "fm": 0.09478480994701385, "loss": 0.09478480994701385} +{"event": "train", "step": 5800, "lr": 8.285358405218655e-05, "backbone_lr": 8.285358405218655e-06, "ema_decay": 0.998496, "it_s": 1.21, "data_frac": 0.769, "fm": 0.09800936311483383, "loss": 0.09800936311483383} +{"event": "train", "step": 5825, "lr": 8.270150895659823e-05, "backbone_lr": 8.270150895659824e-06, "ema_decay": 0.9985, "it_s": 0.69, "data_frac": 0.863, "fm": 0.09669087529182434, "loss": 0.09669087529182434} +{"event": "train", "step": 5850, "lr": 8.25489033703452e-05, "backbone_lr": 8.25489033703452e-06, "ema_decay": 0.998505, "it_s": 1.06, "data_frac": 0.773, "fm": 0.09527552545070649, "loss": 0.09527552545070649} +{"event": "train", "step": 5875, "lr": 8.239576976902695e-05, "backbone_lr": 8.239576976902694e-06, "ema_decay": 0.99851, "it_s": 1.22, "data_frac": 0.737, "fm": 0.09681353956460953, "loss": 0.09681353956460953} +{"event": "train", "step": 5900, "lr": 8.224211063680853e-05, "backbone_lr": 8.224211063680854e-06, "ema_decay": 0.998515, "it_s": 5.5, "data_frac": 0.001, "fm": 0.09730240762233734, "loss": 0.09730240762233734} +{"event": "train", "step": 5925, "lr": 8.208792846638029e-05, "backbone_lr": 8.20879284663803e-06, "ema_decay": 0.998519, "it_s": 0.54, "data_frac": 0.887, "fm": 0.09642397582530976, "loss": 0.09642397582530976} +{"event": "train", "step": 5950, "lr": 8.19332257589174e-05, "backbone_lr": 8.19332257589174e-06, "ema_decay": 0.998524, "it_s": 1.04, "data_frac": 0.787, "fm": 0.09538248509168624, "loss": 0.09538248509168624} +{"event": "train", "step": 5975, "lr": 8.177800502403928e-05, "backbone_lr": 8.177800502403928e-06, "ema_decay": 0.998529, "it_s": 0.99, "data_frac": 0.799, "fm": 0.0961689355969429, "loss": 0.0961689355969429} +{"event": "train", "step": 6000, "lr": 8.162226877976887e-05, "backbone_lr": 8.162226877976886e-06, "ema_decay": 0.998533, "it_s": 5.41, "data_frac": 0.001, "fm": 0.097640760242939, "loss": 0.097640760242939} +{"event": "save", "step": 6000, "checkpoint": "/mnt/data0/zuo/Humanoid_benchmark/results/fmdit_continue_20261117_01/runs/a20/fm_dit/checkpoints/step-006000", "mse": 0.021967994049191475, "mse_row0": 0.022920215502381325, "mse_motion_token": 0.02157071977853775, "mse_inspire_left_hand_state": 0.0040762959979474545, "mse_inspire_right_hand_state": 0.044097285717725754} +{"event": "train", "step": 6025, "lr": 8.146601955249188e-05, "backbone_lr": 8.146601955249187e-06, "ema_decay": 0.998538, "it_s": 0.53, "data_frac": 0.83, "fm": 0.09581303656101227, "loss": 0.09581303656101227} +{"event": "train", "step": 6050, "lr": 8.130925987691569e-05, "backbone_lr": 8.13092598769157e-06, "ema_decay": 0.998542, "it_s": 0.93, "data_frac": 0.814, "fm": 0.09383314609527588, "loss": 0.09383314609527588} +{"event": "train", "step": 6075, "lr": 8.11519922960283e-05, "backbone_lr": 8.115199229602831e-06, "ema_decay": 0.998547, "it_s": 5.29, "data_frac": 0.002, "fm": 0.09339079171419144, "loss": 0.09339079171419144} +{"event": "train", "step": 6100, "lr": 8.099421936105702e-05, "backbone_lr": 8.099421936105702e-06, "ema_decay": 0.998551, "it_s": 1.54, "data_frac": 0.68, "fm": 0.09210814416408539, "loss": 0.09210814416408539} +{"event": "train", "step": 6125, "lr": 8.083594363142717e-05, "backbone_lr": 8.083594363142717e-06, "ema_decay": 0.998556, "it_s": 0.53, "data_frac": 0.895, "fm": 0.09574653506278992, "loss": 0.09574653506278992} +{"event": "train", "step": 6150, "lr": 8.067716767472045e-05, "backbone_lr": 8.067716767472045e-06, "ema_decay": 0.99856, "it_s": 4.99, "data_frac": 0.004, "fm": 0.09378662049770355, "loss": 0.09378662049770355} +{"event": "train", "step": 6175, "lr": 8.051789406663342e-05, "backbone_lr": 8.051789406663342e-06, "ema_decay": 0.998565, "it_s": 0.9, "data_frac": 0.808, "fm": 0.09385949403047561, "loss": 0.09385949403047561} +{"event": "train", "step": 6200, "lr": 8.035812539093557e-05, "backbone_lr": 8.035812539093557e-06, "ema_decay": 0.998569, "it_s": 1.28, "data_frac": 0.746, "fm": 0.09663951128721238, "loss": 0.09663951128721238} +{"event": "train", "step": 6225, "lr": 8.019786423942752e-05, "backbone_lr": 8.019786423942752e-06, "ema_decay": 0.998573, "it_s": 1.92, "data_frac": 0.647, "fm": 0.09393858909606934, "loss": 0.09393858909606934} +{"event": "train", "step": 6250, "lr": 8.003711321189895e-05, "backbone_lr": 8.003711321189895e-06, "ema_decay": 0.998578, "it_s": 0.67, "data_frac": 0.868, "fm": 0.09255427211523055, "loss": 0.09255427211523055} +{"event": "train", "step": 6275, "lr": 7.987587491608637e-05, "backbone_lr": 7.987587491608636e-06, "ema_decay": 0.998582, "it_s": 0.89, "data_frac": 0.813, "fm": 0.09297254592180253, "loss": 0.09297254592180253} +{"event": "train", "step": 6300, "lr": 7.971415196763088e-05, "backbone_lr": 7.971415196763088e-06, "ema_decay": 0.998586, "it_s": 1.05, "data_frac": 0.79, "fm": 0.09344055145978927, "loss": 0.09344055145978927} +{"event": "train", "step": 6325, "lr": 7.95519469900357e-05, "backbone_lr": 7.95519469900357e-06, "ema_decay": 0.99859, "it_s": 2.77, "data_frac": 0.495, "fm": 0.0927633935213089, "loss": 0.0927633935213089} +{"event": "train", "step": 6350, "lr": 7.938926261462366e-05, "backbone_lr": 7.938926261462366e-06, "ema_decay": 0.998594, "it_s": 0.63, "data_frac": 0.867, "fm": 0.09345412462949752, "loss": 0.09345412462949752} +{"event": "train", "step": 6375, "lr": 7.922610148049445e-05, "backbone_lr": 7.922610148049445e-06, "ema_decay": 0.998599, "it_s": 1.06, "data_frac": 0.785, "fm": 0.09155107825994492, "loss": 0.09155107825994492} +{"event": "train", "step": 6400, "lr": 7.906246623448183e-05, "backbone_lr": 7.906246623448184e-06, "ema_decay": 0.998603, "it_s": 1.31, "data_frac": 0.739, "fm": 0.09374758034944534, "loss": 0.09374758034944534} +{"event": "train", "step": 6425, "lr": 7.889835953111075e-05, "backbone_lr": 7.889835953111075e-06, "ema_decay": 0.998607, "it_s": 2.04, "data_frac": 0.611, "fm": 0.09290882647037506, "loss": 0.09290882647037506} +{"event": "train", "step": 6450, "lr": 7.873378403255419e-05, "backbone_lr": 7.87337840325542e-06, "ema_decay": 0.998611, "it_s": 0.6, "data_frac": 0.88, "fm": 0.09184739232063294, "loss": 0.09184739232063294} +{"event": "train", "step": 6475, "lr": 7.856874240859005e-05, "backbone_lr": 7.856874240859006e-06, "ema_decay": 0.998615, "it_s": 0.99, "data_frac": 0.791, "fm": 0.09413167119026183, "loss": 0.09413167119026183} +{"event": "train", "step": 6500, "lr": 7.840323733655778e-05, "backbone_lr": 7.84032373365578e-06, "ema_decay": 0.998619, "it_s": 4.23, "data_frac": 0.185, "fm": 0.09035206019878388, "loss": 0.09035206019878388} diff --git a/a20-basket-transfer/b512-trimv2-cont-20261117/logs/train_stdout.log b/a20-basket-transfer/b512-trimv2-cont-20261117/logs/train_stdout.log new file mode 100644 index 0000000000000000000000000000000000000000..7c97f105feb6575f4a3a17b0dba80512c4b4ce1f --- /dev/null +++ b/a20-basket-transfer/b512-trimv2-cont-20261117/logs/train_stdout.log @@ -0,0 +1,104 @@ +{"event": "preflight", "augmentation": "hb-aug-v2-20261109", "samples": 64, "state_dim": 44, "body_dims": 29, "images_changed": 64, "body_values_moved": 1.0, "max_body_offset_deg": 2.5087, "actions_bit_identical": true, "non_body_state_bit_identical": true, "eval_items_deterministic": 8} +{"event": "resumed", "step": 4000} +{"event": "start", "policy": "fm_dit", "params": 22540684, "backbone_params": 11166912, "backbone_weights": "IMAGENET1K_V1", "backbone_norm": "frozen_bn", "train_frames": 54433, "steps": 20000, "from_step": 4000, "cameras": ["ego_view", "chest"], "hw": [168, 224]} +{"event": "train", "step": 4025, "lr": 9.215152885383474e-05, "backbone_lr": 9.215152885383473e-06, "ema_decay": 0.998021, "it_s": 0.57, "data_frac": 0.831, "fm": 0.27936236292123795, "loss": 0.27936236292123795} +{"event": "train", "step": 4050, "lr": 9.204286963963111e-05, "backbone_lr": 9.204286963963112e-06, "ema_decay": 0.998031, "it_s": 0.95, "data_frac": 0.808, "fm": 0.1851240873336792, "loss": 0.1851240873336792} +{"event": "train", "step": 4075, "lr": 9.193352839727121e-05, "backbone_lr": 9.193352839727122e-06, "ema_decay": 0.99804, "it_s": 1.04, "data_frac": 0.784, "fm": 0.1523358416557312, "loss": 0.1523358416557312} +{"event": "train", "step": 4100, "lr": 9.182350690051133e-05, "backbone_lr": 9.182350690051134e-06, "ema_decay": 0.998049, "it_s": 2.28, "data_frac": 0.577, "fm": 0.13585892975330352, "loss": 0.13585892975330352} +{"event": "train", "step": 4125, "lr": 9.171280693414307e-05, "backbone_lr": 9.171280693414307e-06, "ema_decay": 0.998058, "it_s": 0.66, "data_frac": 0.864, "fm": 0.12647916853427887, "loss": 0.12647916853427887} +{"event": "train", "step": 4150, "lr": 9.160143029396422e-05, "backbone_lr": 9.160143029396422e-06, "ema_decay": 0.998066, "it_s": 0.99, "data_frac": 0.799, "fm": 0.12340139746665954, "loss": 0.12340139746665954} +{"event": "train", "step": 4175, "lr": 9.148937878674976e-05, "backbone_lr": 9.148937878674975e-06, "ema_decay": 0.998075, "it_s": 1.39, "data_frac": 0.725, "fm": 0.12282259106636047, "loss": 0.12282259106636047} +{"event": "train", "step": 4200, "lr": 9.13766542302225e-05, "backbone_lr": 9.13766542302225e-06, "ema_decay": 0.998084, "it_s": 2.26, "data_frac": 0.559, "fm": 0.12243758767843246, "loss": 0.12243758767843246} +{"event": "train", "step": 4225, "lr": 9.126325845302357e-05, "backbone_lr": 9.126325845302357e-06, "ema_decay": 0.998092, "it_s": 0.69, "data_frac": 0.866, "fm": 0.11850759595632553, "loss": 0.11850759595632553} +{"event": "train", "step": 4250, "lr": 9.114919329468282e-05, "backbone_lr": 9.114919329468283e-06, "ema_decay": 0.998101, "it_s": 1.05, "data_frac": 0.781, "fm": 0.11744395792484283, "loss": 0.11744395792484283} +{"event": "train", "step": 4275, "lr": 9.103446060558892e-05, "backbone_lr": 9.103446060558892e-06, "ema_decay": 0.998109, "it_s": 2.47, "data_frac": 0.489, "fm": 0.11543788164854049, "loss": 0.11543788164854049} +{"event": "train", "step": 4300, "lr": 9.091906224695935e-05, "backbone_lr": 9.091906224695935e-06, "ema_decay": 0.998117, "it_s": 1.0, "data_frac": 0.801, "fm": 0.11410628646612167, "loss": 0.11410628646612167} +{"event": "train", "step": 4325, "lr": 9.080300009081024e-05, "backbone_lr": 9.080300009081025e-06, "ema_decay": 0.998125, "it_s": 0.76, "data_frac": 0.853, "fm": 0.11376235336065292, "loss": 0.11376235336065292} +{"event": "train", "step": 4350, "lr": 9.068627601992598e-05, "backbone_lr": 9.0686276019926e-06, "ema_decay": 0.998133, "it_s": 2.17, "data_frac": 0.538, "fm": 0.11388265669345855, "loss": 0.11388265669345855} +{"event": "train", "step": 4375, "lr": 9.056889192782866e-05, "backbone_lr": 9.056889192782865e-06, "ema_decay": 0.998141, "it_s": 1.14, "data_frac": 0.771, "fm": 0.11311278849840165, "loss": 0.11311278849840165} +{"event": "train", "step": 4400, "lr": 9.045084971874738e-05, "backbone_lr": 9.045084971874738e-06, "ema_decay": 0.998149, "it_s": 0.86, "data_frac": 0.829, "fm": 0.1135650286078453, "loss": 0.1135650286078453} +{"event": "train", "step": 4425, "lr": 9.033215130758735e-05, "backbone_lr": 9.033215130758735e-06, "ema_decay": 0.998157, "it_s": 0.85, "data_frac": 0.833, "fm": 0.11657118797302246, "loss": 0.11657118797302246} +{"event": "train", "step": 4450, "lr": 9.021279861989885e-05, "backbone_lr": 9.021279861989884e-06, "ema_decay": 0.998165, "it_s": 3.22, "data_frac": 0.377, "fm": 0.11299996435642243, "loss": 0.11299996435642243} +{"event": "train", "step": 4475, "lr": 9.009279359184593e-05, "backbone_lr": 9.009279359184594e-06, "ema_decay": 0.998173, "it_s": 0.85, "data_frac": 0.83, "fm": 0.10958211928606033, "loss": 0.10958211928606033} +{"event": "train", "step": 4500, "lr": 8.997213817017507e-05, "backbone_lr": 8.997213817017508e-06, "ema_decay": 0.99818, "it_s": 1.24, "data_frac": 0.752, "fm": 0.1114171290397644, "loss": 0.1114171290397644} +{"event": "train", "step": 4525, "lr": 8.985083431218354e-05, "backbone_lr": 8.985083431218354e-06, "ema_decay": 0.998188, "it_s": 1.41, "data_frac": 0.705, "fm": 0.10957210212945938, "loss": 0.10957210212945938} +{"event": "train", "step": 4550, "lr": 8.972888398568772e-05, "backbone_lr": 8.972888398568772e-06, "ema_decay": 0.998195, "it_s": 0.93, "data_frac": 0.824, "fm": 0.11293533921241761, "loss": 0.11293533921241761} +{"event": "train", "step": 4575, "lr": 8.96062891689911e-05, "backbone_lr": 8.96062891689911e-06, "ema_decay": 0.998203, "it_s": 1.19, "data_frac": 0.765, "fm": 0.11222538262605668, "loss": 0.11222538262605668} +{"event": "train", "step": 4600, "lr": 8.948305185085225e-05, "backbone_lr": 8.948305185085226e-06, "ema_decay": 0.99821, "it_s": 1.01, "data_frac": 0.795, "fm": 0.11155075877904892, "loss": 0.11155075877904892} +{"event": "train", "step": 4625, "lr": 8.935917403045251e-05, "backbone_lr": 8.935917403045251e-06, "ema_decay": 0.998217, "it_s": 5.14, "data_frac": 0.002, "fm": 0.10862381368875504, "loss": 0.10862381368875504} +{"event": "train", "step": 4650, "lr": 8.92346577173636e-05, "backbone_lr": 8.923465771736361e-06, "ema_decay": 0.998224, "it_s": 0.52, "data_frac": 0.895, "fm": 0.10961597681045532, "loss": 0.10961597681045532} +{"event": "train", "step": 4675, "lr": 8.910950493151498e-05, "backbone_lr": 8.910950493151498e-06, "ema_decay": 0.998232, "it_s": 1.65, "data_frac": 0.683, "fm": 0.1106258323788643, "loss": 0.1106258323788643} +{"event": "train", "step": 4700, "lr": 8.898371770316111e-05, "backbone_lr": 8.898371770316113e-06, "ema_decay": 0.998239, "it_s": 4.67, "data_frac": 0.005, "fm": 0.10965090662240982, "loss": 0.10965090662240982} +{"event": "train", "step": 4725, "lr": 8.885729807284856e-05, "backbone_lr": 8.885729807284855e-06, "ema_decay": 0.998246, "it_s": 0.88, "data_frac": 0.826, "fm": 0.10638074308633805, "loss": 0.10638074308633805} +{"event": "train", "step": 4750, "lr": 8.873024809138272e-05, "backbone_lr": 8.873024809138272e-06, "ema_decay": 0.998253, "it_s": 0.58, "data_frac": 0.887, "fm": 0.10897882670164108, "loss": 0.10897882670164108} +{"event": "train", "step": 4775, "lr": 8.860256981979484e-05, "backbone_lr": 8.860256981979485e-06, "ema_decay": 0.998259, "it_s": 2.27, "data_frac": 0.517, "fm": 0.10848556160926819, "loss": 0.10848556160926819} +{"event": "train", "step": 4800, "lr": 8.847426532930831e-05, "backbone_lr": 8.84742653293083e-06, "ema_decay": 0.998266, "it_s": 1.74, "data_frac": 0.674, "fm": 0.10899475157260895, "loss": 0.10899475157260895} +{"event": "train", "step": 4825, "lr": 8.834533670130518e-05, "backbone_lr": 8.834533670130518e-06, "ema_decay": 0.998273, "it_s": 1.27, "data_frac": 0.746, "fm": 0.1064724487066269, "loss": 0.1064724487066269} +{"event": "train", "step": 4850, "lr": 8.821578602729242e-05, "backbone_lr": 8.821578602729242e-06, "ema_decay": 0.99828, "it_s": 0.56, "data_frac": 0.882, "fm": 0.10697787135839462, "loss": 0.10697787135839462} +{"event": "train", "step": 4875, "lr": 8.808561540886796e-05, "backbone_lr": 8.808561540886796e-06, "ema_decay": 0.998286, "it_s": 4.25, "data_frac": 0.164, "fm": 0.10751055479049683, "loss": 0.10751055479049683} +{"event": "train", "step": 4900, "lr": 8.795482695768658e-05, "backbone_lr": 8.795482695768658e-06, "ema_decay": 0.998293, "it_s": 0.87, "data_frac": 0.83, "fm": 0.10378085047006608, "loss": 0.10378085047006608} +{"event": "train", "step": 4925, "lr": 8.782342279542568e-05, "backbone_lr": 8.782342279542569e-06, "ema_decay": 0.998299, "it_s": 1.5, "data_frac": 0.722, "fm": 0.10504777759313583, "loss": 0.10504777759313583} +{"event": "train", "step": 4950, "lr": 8.769140505375085e-05, "backbone_lr": 8.769140505375084e-06, "ema_decay": 0.998306, "it_s": 1.55, "data_frac": 0.704, "fm": 0.10637212932109832, "loss": 0.10637212932109832} +{"event": "train", "step": 4975, "lr": 8.755877587428128e-05, "backbone_lr": 8.755877587428127e-06, "ema_decay": 0.998312, "it_s": 0.68, "data_frac": 0.866, "fm": 0.10244016528129578, "loss": 0.10244016528129578} +{"event": "train", "step": 5000, "lr": 8.742553740855506e-05, "backbone_lr": 8.742553740855507e-06, "ema_decay": 0.998318, "it_s": 1.02, "data_frac": 0.794, "fm": 0.10446281790733337, "loss": 0.10446281790733337} +{"event": "train", "step": 5025, "lr": 8.729169181799422e-05, "backbone_lr": 8.729169181799423e-06, "ema_decay": 0.998325, "it_s": 1.53, "data_frac": 0.68, "fm": 0.10380947828292847, "loss": 0.10380947828292847} +{"event": "train", "step": 5050, "lr": 8.715724127386972e-05, "backbone_lr": 8.715724127386971e-06, "ema_decay": 0.998331, "it_s": 3.21, "data_frac": 0.387, "fm": 0.10200979679822922, "loss": 0.10200979679822922} +{"event": "train", "step": 5075, "lr": 8.702218795726617e-05, "backbone_lr": 8.702218795726619e-06, "ema_decay": 0.998337, "it_s": 0.53, "data_frac": 0.893, "fm": 0.10482226371765137, "loss": 0.10482226371765137} +{"event": "train", "step": 5100, "lr": 8.688653405904652e-05, "backbone_lr": 8.688653405904653e-06, "ema_decay": 0.998343, "it_s": 1.19, "data_frac": 0.757, "fm": 0.1049795463681221, "loss": 0.1049795463681221} +{"event": "train", "step": 5125, "lr": 8.675028177981643e-05, "backbone_lr": 8.675028177981643e-06, "ema_decay": 0.998349, "it_s": 5.17, "data_frac": 0.046, "fm": 0.103239366710186, "loss": 0.103239366710186} +{"event": "train", "step": 5150, "lr": 8.661343332988869e-05, "backbone_lr": 8.661343332988869e-06, "ema_decay": 0.998355, "it_s": 1.54, "data_frac": 0.71, "fm": 0.10244131833314896, "loss": 0.10244131833314896} +{"event": "train", "step": 5175, "lr": 8.647599092924721e-05, "backbone_lr": 8.64759909292472e-06, "ema_decay": 0.998361, "it_s": 0.48, "data_frac": 0.898, "fm": 0.1028094071149826, "loss": 0.1028094071149826} +{"event": "train", "step": 5200, "lr": 8.633795680751116e-05, "backbone_lr": 8.633795680751116e-06, "ema_decay": 0.998367, "it_s": 2.63, "data_frac": 0.413, "fm": 0.10102613866329194, "loss": 0.10102613866329194} +{"event": "train", "step": 5225, "lr": 8.619933320389872e-05, "backbone_lr": 8.619933320389872e-06, "ema_decay": 0.998373, "it_s": 1.02, "data_frac": 0.802, "fm": 0.10271453976631165, "loss": 0.10271453976631165} +{"event": "train", "step": 5250, "lr": 8.606012236719073e-05, "backbone_lr": 8.606012236719073e-06, "ema_decay": 0.998379, "it_s": 1.75, "data_frac": 0.658, "fm": 0.1013492727279663, "loss": 0.1013492727279663} +{"event": "train", "step": 5275, "lr": 8.592032655569429e-05, "backbone_lr": 8.59203265556943e-06, "ema_decay": 0.998385, "it_s": 0.55, "data_frac": 0.869, "fm": 0.09972631871700287, "loss": 0.09972631871700287} +{"event": "train", "step": 5300, "lr": 8.577994803720606e-05, "backbone_lr": 8.577994803720605e-06, "ema_decay": 0.99839, "it_s": 3.74, "data_frac": 0.205, "fm": 0.09866629302501678, "loss": 0.09866629302501678} +{"event": "train", "step": 5325, "lr": 8.56389890889755e-05, "backbone_lr": 8.56389890889755e-06, "ema_decay": 0.998396, "it_s": 0.8, "data_frac": 0.837, "fm": 0.10182167440652848, "loss": 0.10182167440652848} +{"event": "train", "step": 5350, "lr": 8.549745199766792e-05, "backbone_lr": 8.549745199766792e-06, "ema_decay": 0.998402, "it_s": 1.59, "data_frac": 0.697, "fm": 0.09962393790483474, "loss": 0.09962393790483474} +{"event": "train", "step": 5375, "lr": 8.535533905932738e-05, "backbone_lr": 8.535533905932739e-06, "ema_decay": 0.998407, "it_s": 1.6, "data_frac": 0.699, "fm": 0.10165397435426712, "loss": 0.10165397435426712} +{"event": "train", "step": 5400, "lr": 8.521265257933948e-05, "backbone_lr": 8.521265257933948e-06, "ema_decay": 0.998413, "it_s": 0.66, "data_frac": 0.862, "fm": 0.09791133671998978, "loss": 0.09791133671998978} +{"event": "train", "step": 5425, "lr": 8.506939487239388e-05, "backbone_lr": 8.506939487239389e-06, "ema_decay": 0.998418, "it_s": 1.0, "data_frac": 0.787, "fm": 0.09465882360935211, "loss": 0.09465882360935211} +{"event": "train", "step": 5450, "lr": 8.492556826244687e-05, "backbone_lr": 8.492556826244687e-06, "ema_decay": 0.998424, "it_s": 1.36, "data_frac": 0.728, "fm": 0.10158959805965423, "loss": 0.10158959805965423} +{"event": "train", "step": 5475, "lr": 8.47811750826835e-05, "backbone_lr": 8.478117508268352e-06, "ema_decay": 0.998429, "it_s": 3.59, "data_frac": 0.308, "fm": 0.10000236868858338, "loss": 0.10000236868858338} +{"event": "train", "step": 5500, "lr": 8.463621767547998e-05, "backbone_lr": 8.463621767547998e-06, "ema_decay": 0.998434, "it_s": 0.58, "data_frac": 0.883, "fm": 0.09906157135963439, "loss": 0.09906157135963439} +{"event": "train", "step": 5525, "lr": 8.449069839236538e-05, "backbone_lr": 8.44906983923654e-06, "ema_decay": 0.99844, "it_s": 0.91, "data_frac": 0.811, "fm": 0.09683233767747879, "loss": 0.09683233767747879} +{"event": "train", "step": 5550, "lr": 8.434461959398376e-05, "backbone_lr": 8.434461959398377e-06, "ema_decay": 0.998445, "it_s": 2.24, "data_frac": 0.577, "fm": 0.097518649995327, "loss": 0.097518649995327} +{"event": "train", "step": 5575, "lr": 8.419798365005569e-05, "backbone_lr": 8.419798365005569e-06, "ema_decay": 0.99845, "it_s": 1.95, "data_frac": 0.639, "fm": 0.1010474818944931, "loss": 0.1010474818944931} +{"event": "train", "step": 5600, "lr": 8.405079293933986e-05, "backbone_lr": 8.405079293933986e-06, "ema_decay": 0.998455, "it_s": 0.5, "data_frac": 0.898, "fm": 0.09772510349750518, "loss": 0.09772510349750518} +{"event": "train", "step": 5625, "lr": 8.390304984959454e-05, "backbone_lr": 8.390304984959455e-06, "ema_decay": 0.998461, "it_s": 0.96, "data_frac": 0.794, "fm": 0.0974524575471878, "loss": 0.0974524575471878} +{"event": "train", "step": 5650, "lr": 8.375475677753881e-05, "backbone_lr": 8.375475677753882e-06, "ema_decay": 0.998466, "it_s": 3.22, "data_frac": 0.386, "fm": 0.09912837445735931, "loss": 0.09912837445735931} +{"event": "train", "step": 5675, "lr": 8.360591612881364e-05, "backbone_lr": 8.360591612881363e-06, "ema_decay": 0.998471, "it_s": 1.74, "data_frac": 0.669, "fm": 0.09766773462295532, "loss": 0.09766773462295532} +{"event": "train", "step": 5700, "lr": 8.345653031794292e-05, "backbone_lr": 8.345653031794292e-06, "ema_decay": 0.998476, "it_s": 0.57, "data_frac": 0.879, "fm": 0.10231035619974137, "loss": 0.10231035619974137} +{"event": "train", "step": 5725, "lr": 8.330660176829428e-05, "backbone_lr": 8.330660176829427e-06, "ema_decay": 0.998481, "it_s": 4.26, "data_frac": 0.134, "fm": 0.09899430811405181, "loss": 0.09899430811405181} +{"event": "train", "step": 5750, "lr": 8.315613291203976e-05, "backbone_lr": 8.315613291203977e-06, "ema_decay": 0.998486, "it_s": 0.83, "data_frac": 0.832, "fm": 0.095704605281353, "loss": 0.095704605281353} +{"event": "train", "step": 5775, "lr": 8.30051261901164e-05, "backbone_lr": 8.30051261901164e-06, "ema_decay": 0.998491, "it_s": 1.58, "data_frac": 0.702, "fm": 0.09478480994701385, "loss": 0.09478480994701385} +{"event": "train", "step": 5800, "lr": 8.285358405218655e-05, "backbone_lr": 8.285358405218655e-06, "ema_decay": 0.998496, "it_s": 1.21, "data_frac": 0.769, "fm": 0.09800936311483383, "loss": 0.09800936311483383} +{"event": "train", "step": 5825, "lr": 8.270150895659823e-05, "backbone_lr": 8.270150895659824e-06, "ema_decay": 0.9985, "it_s": 0.69, "data_frac": 0.863, "fm": 0.09669087529182434, "loss": 0.09669087529182434} +{"event": "train", "step": 5850, "lr": 8.25489033703452e-05, "backbone_lr": 8.25489033703452e-06, "ema_decay": 0.998505, "it_s": 1.06, "data_frac": 0.773, "fm": 0.09527552545070649, "loss": 0.09527552545070649} +{"event": "train", "step": 5875, "lr": 8.239576976902695e-05, "backbone_lr": 8.239576976902694e-06, "ema_decay": 0.99851, "it_s": 1.22, "data_frac": 0.737, "fm": 0.09681353956460953, "loss": 0.09681353956460953} +{"event": "train", "step": 5900, "lr": 8.224211063680853e-05, "backbone_lr": 8.224211063680854e-06, "ema_decay": 0.998515, "it_s": 5.5, "data_frac": 0.001, "fm": 0.09730240762233734, "loss": 0.09730240762233734} +{"event": "train", "step": 5925, "lr": 8.208792846638029e-05, "backbone_lr": 8.20879284663803e-06, "ema_decay": 0.998519, "it_s": 0.54, "data_frac": 0.887, "fm": 0.09642397582530976, "loss": 0.09642397582530976} +{"event": "train", "step": 5950, "lr": 8.19332257589174e-05, "backbone_lr": 8.19332257589174e-06, "ema_decay": 0.998524, "it_s": 1.04, "data_frac": 0.787, "fm": 0.09538248509168624, "loss": 0.09538248509168624} +{"event": "train", "step": 5975, "lr": 8.177800502403928e-05, "backbone_lr": 8.177800502403928e-06, "ema_decay": 0.998529, "it_s": 0.99, "data_frac": 0.799, "fm": 0.0961689355969429, "loss": 0.0961689355969429} +{"event": "train", "step": 6000, "lr": 8.162226877976887e-05, "backbone_lr": 8.162226877976886e-06, "ema_decay": 0.998533, "it_s": 5.41, "data_frac": 0.001, "fm": 0.097640760242939, "loss": 0.097640760242939} +{"event": "save", "step": 6000, "checkpoint": "/mnt/data0/zuo/Humanoid_benchmark/results/fmdit_continue_20261117_01/runs/a20/fm_dit/checkpoints/step-006000", "mse": 0.021967994049191475, "mse_row0": 0.022920215502381325, "mse_motion_token": 0.02157071977853775, "mse_inspire_left_hand_state": 0.0040762959979474545, "mse_inspire_right_hand_state": 0.044097285717725754} +{"event": "train", "step": 6025, "lr": 8.146601955249188e-05, "backbone_lr": 8.146601955249187e-06, "ema_decay": 0.998538, "it_s": 0.53, "data_frac": 0.83, "fm": 0.09581303656101227, "loss": 0.09581303656101227} +{"event": "train", "step": 6050, "lr": 8.130925987691569e-05, "backbone_lr": 8.13092598769157e-06, "ema_decay": 0.998542, "it_s": 0.93, "data_frac": 0.814, "fm": 0.09383314609527588, "loss": 0.09383314609527588} +{"event": "train", "step": 6075, "lr": 8.11519922960283e-05, "backbone_lr": 8.115199229602831e-06, "ema_decay": 0.998547, "it_s": 5.29, "data_frac": 0.002, "fm": 0.09339079171419144, "loss": 0.09339079171419144} +{"event": "train", "step": 6100, "lr": 8.099421936105702e-05, "backbone_lr": 8.099421936105702e-06, "ema_decay": 0.998551, "it_s": 1.54, "data_frac": 0.68, "fm": 0.09210814416408539, "loss": 0.09210814416408539} +{"event": "train", "step": 6125, "lr": 8.083594363142717e-05, "backbone_lr": 8.083594363142717e-06, "ema_decay": 0.998556, "it_s": 0.53, "data_frac": 0.895, "fm": 0.09574653506278992, "loss": 0.09574653506278992} +{"event": "train", "step": 6150, "lr": 8.067716767472045e-05, "backbone_lr": 8.067716767472045e-06, "ema_decay": 0.99856, "it_s": 4.99, "data_frac": 0.004, "fm": 0.09378662049770355, "loss": 0.09378662049770355} +{"event": "train", "step": 6175, "lr": 8.051789406663342e-05, "backbone_lr": 8.051789406663342e-06, "ema_decay": 0.998565, "it_s": 0.9, "data_frac": 0.808, "fm": 0.09385949403047561, "loss": 0.09385949403047561} +{"event": "train", "step": 6200, "lr": 8.035812539093557e-05, "backbone_lr": 8.035812539093557e-06, "ema_decay": 0.998569, "it_s": 1.28, "data_frac": 0.746, "fm": 0.09663951128721238, "loss": 0.09663951128721238} +{"event": "train", "step": 6225, "lr": 8.019786423942752e-05, "backbone_lr": 8.019786423942752e-06, "ema_decay": 0.998573, "it_s": 1.92, "data_frac": 0.647, "fm": 0.09393858909606934, "loss": 0.09393858909606934} +{"event": "train", "step": 6250, "lr": 8.003711321189895e-05, "backbone_lr": 8.003711321189895e-06, "ema_decay": 0.998578, "it_s": 0.67, "data_frac": 0.868, "fm": 0.09255427211523055, "loss": 0.09255427211523055} +{"event": "train", "step": 6275, "lr": 7.987587491608637e-05, "backbone_lr": 7.987587491608636e-06, "ema_decay": 0.998582, "it_s": 0.89, "data_frac": 0.813, "fm": 0.09297254592180253, "loss": 0.09297254592180253} +{"event": "train", "step": 6300, "lr": 7.971415196763088e-05, "backbone_lr": 7.971415196763088e-06, "ema_decay": 0.998586, "it_s": 1.05, "data_frac": 0.79, "fm": 0.09344055145978927, "loss": 0.09344055145978927} +{"event": "train", "step": 6325, "lr": 7.95519469900357e-05, "backbone_lr": 7.95519469900357e-06, "ema_decay": 0.99859, "it_s": 2.77, "data_frac": 0.495, "fm": 0.0927633935213089, "loss": 0.0927633935213089} +{"event": "train", "step": 6350, "lr": 7.938926261462366e-05, "backbone_lr": 7.938926261462366e-06, "ema_decay": 0.998594, "it_s": 0.63, "data_frac": 0.867, "fm": 0.09345412462949752, "loss": 0.09345412462949752} +{"event": "train", "step": 6375, "lr": 7.922610148049445e-05, "backbone_lr": 7.922610148049445e-06, "ema_decay": 0.998599, "it_s": 1.06, "data_frac": 0.785, "fm": 0.09155107825994492, "loss": 0.09155107825994492} +{"event": "train", "step": 6400, "lr": 7.906246623448183e-05, "backbone_lr": 7.906246623448184e-06, "ema_decay": 0.998603, "it_s": 1.31, "data_frac": 0.739, "fm": 0.09374758034944534, "loss": 0.09374758034944534} +{"event": "train", "step": 6425, "lr": 7.889835953111075e-05, "backbone_lr": 7.889835953111075e-06, "ema_decay": 0.998607, "it_s": 2.04, "data_frac": 0.611, "fm": 0.09290882647037506, "loss": 0.09290882647037506} +{"event": "train", "step": 6450, "lr": 7.873378403255419e-05, "backbone_lr": 7.87337840325542e-06, "ema_decay": 0.998611, "it_s": 0.6, "data_frac": 0.88, "fm": 0.09184739232063294, "loss": 0.09184739232063294} +{"event": "train", "step": 6475, "lr": 7.856874240859005e-05, "backbone_lr": 7.856874240859006e-06, "ema_decay": 0.998615, "it_s": 0.99, "data_frac": 0.791, "fm": 0.09413167119026183, "loss": 0.09413167119026183} +{"event": "train", "step": 6500, "lr": 7.840323733655778e-05, "backbone_lr": 7.84032373365578e-06, "ema_decay": 0.998619, "it_s": 4.23, "data_frac": 0.185, "fm": 0.09035206019878388, "loss": 0.09035206019878388} diff --git a/a20-basket-transfer/b512-trimv2-cont-20261117/provenance/parent_checkpoint_config.json b/a20-basket-transfer/b512-trimv2-cont-20261117/provenance/parent_checkpoint_config.json new file mode 100644 index 0000000000000000000000000000000000000000..a5e9bdec3bdd116925c4a19433f0282017e0ea45 --- /dev/null +++ b/a20-basket-transfer/b512-trimv2-cont-20261117/provenance/parent_checkpoint_config.json @@ -0,0 +1,884 @@ +{ + "policy": { + "policy": "fm_dit", + "n_cameras": 2, + "state_dim": 44, + "action_dim": 76, + "horizon": 16, + "backbone": "resnet18", + "backbone_weights": "IMAGENET1K_V1", + "backbone_norm": "frozen_bn", + "hidden_dim": 384, + "num_layers": 4, + "num_heads": 8, + "mlp_ratio": 4.0, + "dropout": 0.1, + "time_embed_dim": 128, + "num_inference_steps": null, + "num_train_timesteps": 100, + "beta_schedule": "squaredcos_cap_v2", + "prediction_type": "epsilon", + "clip_sample": true, + "act_hidden_dim": 512, + "dim_feedforward": 3200, + "enc_layers": 4, + "dec_layers": 7, + "nheads": 8, + "latent_dim": 32, + "kl_weight": 10.0, + "extra": {} + }, + "contract": { + "name": "g1_sonic_dfx6", + "robot": "g1", + "cameras": [ + "ego_view", + "chest" + ], + "state": [ + [ + "observation.state", + 0, + 29 + ], + [ + "observation.inspire_left_hand_state", + 0, + 6 + ], + [ + "observation.inspire_right_hand_state", + 0, + 6 + ], + [ + "observation.projected_gravity", + 0, + 3 + ] + ], + "action": [ + [ + "action.motion_token", + 0, + 64 + ], + [ + "observation.inspire_left_hand_state", + 0, + 6 + ], + [ + "observation.inspire_right_hand_state", + 0, + 6 + ] + ], + "state_dim": 44, + "action_dim": 76, + "action_space": "sonic_dfx6", + "body_dims": 29 + }, + "image_hw": [ + 168, + 224 + ], + "task_text": "Grasp the basket handle with the right hand from the v10 round table, walk while raising it, place and release it on the long desk, then withdraw the open right hand diagonally backward and upward without an additional downward arm return.", + "dataset": { + "name": "a20-basket-transfer-pre-reach-trim-expert-v2", + "root": "/mnt/data0/zuo/Humanoid_benchmark/data/baselines/a20-basket-transfer-pre-reach-trim-expert-v2", + "robot_type": null, + "fps": 30.0, + "episodes": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 36, + 37, + 38, + 39, + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55, + 56, + 57, + 58, + 59, + 60, + 61, + 62, + 63, + 64, + 65, + 66, + 67, + 68, + 69, + 70, + 71, + 72, + 73, + 74, + 75, + 76, + 77, + 78, + 79, + 80, + 81, + 82, + 83, + 84, + 85, + 86, + 87, + 88, + 89, + 90, + 91, + 92, + 93, + 94, + 95, + 96, + 97, + 98, + 99 + ], + "train_episodes": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 36, + 37, + 38, + 39, + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55, + 56, + 57, + 58, + 59, + 60, + 61, + 62, + 63, + 64, + 65, + 66, + 67, + 68, + 69, + 70, + 71, + 72, + 73, + 74, + 75, + 76, + 77, + 78, + 79, + 80, + 81, + 82, + 83, + 84, + 85, + 86, + 87, + 88, + 89, + 90, + 91, + 92, + 93, + 94, + 95, + 96, + 97, + 98, + 99 + ], + "val_episodes": [], + "train_frames": 54433, + "git_commit": null + }, + "training": { + "steps": 4000, + "batch_size": 512, + "lr": 0.0001, + "betas": [ + 0.95, + 0.999 + ], + "weight_decay": 1e-06, + "warmup": 500, + "schedule": "linear warm-up + cosine", + "backbone_lr_scale": 0.1, + "ema": 0.9999, + "shift_px": 0, + "amp": true, + "seed": 0, + "augmentation": { + "spec": "docs/augmentation_v2.md", + "version": "hb-aug-v2-20261109", + "implementation": "evaluation/augmentation/hb_aug_v2.py", + "visual": { + "effects": [ + "exposure", + "contrast", + "saturation", + "hue", + "gamma", + "temperature", + "rotation_crop", + "noise", + "gaussian_blur", + "jpeg", + "motion_blur", + "sharpen", + "defocus", + "light_shadow", + "wb_channel" + ], + "boosted_per_sample": 1, + "boost": 1.5, + "pixel_scale": "H / 224", + "future_targets": "none (one observation frame)" + }, + "proprio": { + "robot": "g1", + "body_dims": 29, + "bias_deg": 2.0, + "bias_scope": "per_sample", + "noise_deg": 0.25, + "noise_clip_sigma": 3, + "delay_probability": 0.0, + "history_drop_probability": 0.0, + "velocity": false + }, + "native_image_augmentation": "off", + "evaluation": "off" + }, + "preflight": { + "samples": 64, + "state_dim": 44, + "body_dims": 29, + "images_changed": 64, + "body_values_moved": 1.0, + "max_body_offset_deg": 2.5853, + "actions_bit_identical": true, + "non_body_state_bit_identical": true, + "eval_items_deterministic": 8 + } + }, + "stats": { + "state": { + "min": [ + -0.6335107088088989, + -0.26151371002197266, + -0.09143047779798508, + 0.13051827251911163, + -0.41364213824272156, + -0.17374181747436523, + -0.5706103444099426, + -0.4567093253135681, + -0.5945078730583191, + 0.07693764567375183, + -0.479587197303772, + -0.23942920565605164, + -0.1249256581068039, + -0.061801809817552567, + 0.03293042629957199, + -0.15965673327445984, + 0.12444422394037247, + -1.0301204919815063, + 0.8110364079475403, + -0.42373916506767273, + -0.2393926978111267, + -0.1671462506055832, + -1.8559280633926392, + -2.1003689765930176, + -0.44611114263534546, + -0.86998051404953, + -1.5242679119110107, + -0.9486506581306458, + -0.4465007185935974, + 0.9997791647911072, + 0.9987788796424866, + 0.9989608526229858, + 0.9997321963310242, + 0.8891316056251526, + 0.9935978651046753, + 0.3987571895122528, + 0.39748135209083557, + 0.3998531103134155, + 0.47865480184555054, + 0.43634313344955444, + 0.0, + -0.1312161237001419, + -0.10554340481758118, + -1.0 + ], + "max": [ + 0.18651676177978516, + 0.2880381643772125, + 0.3991033434867859, + 1.1868021488189697, + 0.30963531136512756, + 0.25351187586784363, + 0.13313187658786774, + 0.14919418096542358, + 0.09004810452461243, + 1.1367807388305664, + 0.47997909784317017, + 0.1897573471069336, + 0.12687286734580994, + 0.0903186947107315, + 0.2110750675201416, + 0.1764557808637619, + 0.3972950279712677, + -0.03375900536775589, + 1.248238205909729, + 0.5087170004844666, + 0.16197925806045532, + 0.19902198016643524, + 0.7476708292961121, + -0.14924407005310059, + 1.3937729597091675, + 1.4777758121490479, + 1.449870228767395, + 0.9068933725357056, + 1.5030572414398193, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.1443370282649994, + 0.07133471965789795, + -0.9889927506446838 + ], + "mean": [ + -0.1197492680031697, + 0.07687621598947185, + 0.13623676943730387, + 0.27067508540983654, + -0.09364010490166123, + -0.029264338937400346, + -0.0877177892591824, + -0.09172362643767557, + -0.21603681153788365, + 0.27020221177162185, + -0.13314246080400155, + 0.01521604484593622, + -0.007014541196278249, + 0.023597627586334522, + 0.1362211304918607, + 0.0044590163081436405, + 0.24020097902181153, + -0.5506130059849405, + 0.986759154493458, + -0.11353825983460156, + -0.09008970046721287, + 0.00793797976488734, + -0.7956446111257796, + -0.6433268364566035, + 0.35943391980855405, + 0.4128302732079997, + -0.6341984046200151, + 0.17366417734106818, + 0.31817584543570415, + 0.9999920738393157, + 0.9999916277741105, + 0.9999916507452171, + 0.9999910946907014, + 0.9975380262153739, + 0.9997842168412904, + 0.7431049855982732, + 0.743061001937713, + 0.7431612987122399, + 0.7812160503121215, + 0.7544858488771042, + 0.1884228588481359, + -0.018600662889153487, + -0.001923185563505277, + -0.9992875029526394 + ], + "std": [ + 0.1225902736805545, + 0.0667315519538753, + 0.06966628355722553, + 0.1683343132651939, + 0.11676670853146011, + 0.06393707008498987, + 0.14262675997248406, + 0.07183651035230894, + 0.07077858012951907, + 0.1583747884788023, + 0.09609718753190293, + 0.06937572380392895, + 0.038858334068535276, + 0.01853271444550659, + 0.019156535689633315, + 0.04755913941308977, + 0.041089601727671066, + 0.14354888870689778, + 0.053966848435702465, + 0.13684428975351073, + 0.05453949581291963, + 0.06076597347825228, + 0.6370602259123855, + 0.4013113120474627, + 0.24111485368872668, + 0.5381053688248473, + 0.48085827035516315, + 0.1527991524681821, + 0.44871163018722837, + 1.2824078270394207e-05, + 2.1003630589963823e-05, + 1.8574155779181667e-05, + 1.5382720229173254e-05, + 0.0031309164055460465, + 0.0001241612656891375, + 0.2938442161025196, + 0.29381668166139846, + 0.29367033348300187, + 0.25040401472179635, + 0.2708879748387615, + 0.3732468841625382, + 0.027734659598288042, + 0.01745881392269428, + 0.0008838855519788744 + ] + }, + "action": { + "min": [ + -0.125, + -0.5, + -0.375, + -0.375, + -0.625, + -0.5625, + -0.4375, + -0.25, + -0.375, + -0.1875, + -0.625, + -0.25, + -0.25, + -0.375, + -0.3125, + -0.5, + -0.1875, + -0.25, + -0.5, + -0.375, + -0.4375, + -0.5, + -0.3125, + -0.5, + -0.5, + 0.0, + -0.4375, + -0.5, + -0.5, + -0.375, + -0.0625, + -0.125, + -0.375, + -0.1875, + 0.0625, + -0.6875, + -0.1875, + -0.5, + -0.375, + -0.3125, + -0.4375, + -0.5625, + -0.375, + -0.125, + -0.5625, + -0.4375, + -0.375, + -0.25, + -0.1875, + -0.5625, + -0.375, + -0.1875, + -0.5, + -0.5625, + -0.3125, + -0.3125, + -0.5, + -0.125, + -0.625, + -0.3125, + -0.5625, + -0.1875, + -0.3125, + -0.125, + 0.9997791647911072, + 0.9987788796424866, + 0.9989608526229858, + 0.9997321963310242, + 0.8891316056251526, + 0.9935978651046753, + 0.3987571895122528, + 0.39748135209083557, + 0.3998531103134155, + 0.47865480184555054, + 0.43634313344955444, + 0.0 + ], + "max": [ + 0.5625, + 0.375, + 0.375, + 0.4375, + 0.25, + 0.25, + 0.1875, + 0.25, + 0.3125, + 0.1875, + 0.125, + 0.375, + 0.5, + 0.25, + 0.3125, + 0.25, + 0.375, + 0.5625, + 0.1875, + 0.25, + 0.25, + 0.1875, + 0.5, + 0.3125, + 0.0625, + 0.5, + 0.0, + 0.1875, + 0.3125, + 0.375, + 0.6875, + 0.5, + 0.25, + 0.5, + 0.375, + 0.125, + 0.6875, + 0.1875, + 0.375, + 0.25, + 0.125, + 0.375, + 0.1875, + 0.5, + 0.3125, + 0.5, + 0.5, + 0.4375, + 0.5, + 0.1875, + 0.5, + 0.3125, + 0.625, + 0.125, + 0.25, + 0.5625, + 0.5, + 0.3125, + 0.125, + 0.1875, + 0.25, + 0.625, + 0.4375, + 0.6875, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0 + ], + "mean": [ + 0.28530257380633073, + 0.0012216853746808002, + 0.02519611265225139, + 0.0920742013117043, + -0.18735762313302592, + -0.18916948358532507, + -0.10082578582844966, + -0.04547218599011629, + 0.009826300222291624, + -0.09046442415446512, + -0.3906258611504051, + 0.07456643947605313, + 0.06761982620836625, + 0.016528346774934324, + 0.02783008469127184, + -0.15926115591644774, + 0.137793250417945, + 0.13872558925651718, + -0.17794123050355484, + -0.04058314809031286, + -0.1081088218543898, + -0.27256788161593154, + 0.0295489408998218, + -0.12704953796410265, + -0.2065647217680451, + 0.257109657744383, + -0.2610296143883306, + -0.11359147943343192, + -0.08978239303363768, + 0.042958775007807766, + 0.3583063123472893, + 0.2295861425973215, + 0.005530882001726894, + 0.18030882001726894, + 0.22869399077765326, + -0.3416688865210442, + 0.16757757242849006, + -0.24212104789374092, + -0.08830580713905167, + 0.03962554883985817, + -0.24867382837616886, + -0.16334300883655137, + -0.10593298183087466, + 0.29390833685448164, + -0.17902857641504236, + -0.02676455459004648, + -0.010649560009553029, + 0.006596412102952253, + 0.19015693604982273, + -0.28608220197306783, + 0.0342829717267099, + 0.05214208292763581, + 0.2372308617933974, + -0.27289282236878365, + -0.07418638509727556, + 0.21441496886080136, + -0.07936362133264747, + 0.07053166277809417, + -0.2592969797732993, + -0.04738279168886521, + -0.25777561405764887, + 0.3476016387118108, + 0.0341980048867415, + 0.4095493542520162, + 0.9999920738393157, + 0.9999916277741105, + 0.9999916507452171, + 0.9999910946907014, + 0.9975380262153739, + 0.9997842168412904, + 0.7431049855982732, + 0.743061001937713, + 0.7431612987122399, + 0.7812160503121215, + 0.7544858488771042, + 0.1884228588481359 + ], + "std": [ + 0.13976769962393334, + 0.2042750041966315, + 0.11249063031363368, + 0.11082072512754977, + 0.1564752219884858, + 0.17244514813212275, + 0.10524673666978752, + 0.09606056085676273, + 0.16248529891535804, + 0.06517700714473815, + 0.12267993314050843, + 0.09188274452085451, + 0.1817845505649146, + 0.13895894808224718, + 0.09471602567899642, + 0.143892040083983, + 0.13123846053152619, + 0.17169478968550717, + 0.099377781857111, + 0.12335366669054255, + 0.09243992450340514, + 0.12474594719512408, + 0.1211368699762344, + 0.17473771543792177, + 0.12139112019256279, + 0.08058301151688785, + 0.08181288643740524, + 0.12245444611527911, + 0.15642136427791328, + 0.16023434329274994, + 0.1749952692685118, + 0.10352440920325795, + 0.07368975064774456, + 0.13144910448634894, + 0.04419890783153379, + 0.18157602689607938, + 0.15703202636240324, + 0.1308473547559705, + 0.10877569321547828, + 0.06509046280547963, + 0.08347560937172435, + 0.19809736217235097, + 0.06719381956707862, + 0.09127596366580104, + 0.13233099033714146, + 0.25952623468134783, + 0.21598709847307782, + 0.10931842674892184, + 0.11294459446909082, + 0.1403015515249487, + 0.10231818070157317, + 0.10139944441893624, + 0.288021039372521, + 0.11780041132456433, + 0.11650297357712408, + 0.21262101617713844, + 0.28703233994958033, + 0.09814627765272926, + 0.21616444128412057, + 0.06262758949048293, + 0.1745534315389492, + 0.17455467068976818, + 0.10468632106020986, + 0.18477106893909007, + 1.2824078270394207e-05, + 2.1003630589963823e-05, + 1.8574155779181667e-05, + 1.5382720229173254e-05, + 0.0031309164055460465, + 0.0001241612656891375, + 0.2938442161025196, + 0.29381668166139846, + 0.29367033348300187, + 0.25040401472179635, + 0.2708879748387615, + 0.3732468841625382 + ] + } + }, + "code_commit": "028df1bb7024fb1e3d176de7e60c84d3333cd97f", + "step": 4000, + "weights": "ema", + "offline_eval": { + "mse": 0.025112804025411606, + "mse_row0": 0.025062359869480133, + "mse_motion_token": 0.02457592636346817, + "mse_inspire_left_hand_state": 0.0052911583334207535, + "mse_inspire_right_hand_state": 0.0506611168384552, + "samples": 256, + "frames": "train" + }, + "format": "hb_baselines/1" +} \ No newline at end of file diff --git a/a20-basket-transfer/b512-trimv2-cont-20261117/provenance/parent_checkpoint_protocol.json b/a20-basket-transfer/b512-trimv2-cont-20261117/provenance/parent_checkpoint_protocol.json new file mode 100644 index 0000000000000000000000000000000000000000..99d7e645cf3e4b15581828fe218301d767c2722b --- /dev/null +++ b/a20-basket-transfer/b512-trimv2-cont-20261117/provenance/parent_checkpoint_protocol.json @@ -0,0 +1,11 @@ +{ + "format": "hb_baselines/1", + "step": 4000, + "policy": "fm_dit", + "files": { + "model.pt": { + "bytes": 90311637, + "sha256": "4c4ae7d2d53b4f17346dc75adf8c5d47d2cd8e63d099c54022b868499f0b0a4c" + } + } +} \ No newline at end of file diff --git a/a20-basket-transfer/b512-trimv2-cont-20261117/provenance/parent_run.json b/a20-basket-transfer/b512-trimv2-cont-20261117/provenance/parent_run.json new file mode 100644 index 0000000000000000000000000000000000000000..330c0327f32ccf8f3ccef7a867f670bd8e26865d --- /dev/null +++ b/a20-basket-transfer/b512-trimv2-cont-20261117/provenance/parent_run.json @@ -0,0 +1,894 @@ +{ + "argv": [ + "/mnt/data0/zuo/Humanoid_benchmark/results/baselines_preflight_20261117/main/evaluation/baselines/train.py", + "--policy", + "fm_dit", + "--contract", + "g1_sonic_dfx6", + "--dataset", + "/mnt/data0/zuo/Humanoid_benchmark/data/baselines/a20-basket-transfer-pre-reach-trim-expert-v2", + "--cache-dir", + "/mnt/data0/zuo/Humanoid_benchmark/cache/baselines/frames", + "--out", + "/mnt/data0/zuo/Humanoid_benchmark/results/baselines_preflight_20261117/runs/a20-basket-transfer-pre-reach-trim-expert-v2/fm_dit", + "--batch-size", + "512", + "--steps", + "4000", + "--workers", + "12" + ], + "started": "2026-11-17T04:07:19", + "host": "xt-p-A100-004", + "torch": "2.9.0+cu128", + "policy": { + "policy": "fm_dit", + "n_cameras": 2, + "state_dim": 44, + "action_dim": 76, + "horizon": 16, + "backbone": "resnet18", + "backbone_weights": "IMAGENET1K_V1", + "backbone_norm": "frozen_bn", + "hidden_dim": 384, + "num_layers": 4, + "num_heads": 8, + "mlp_ratio": 4.0, + "dropout": 0.1, + "time_embed_dim": 128, + "num_inference_steps": null, + "num_train_timesteps": 100, + "beta_schedule": "squaredcos_cap_v2", + "prediction_type": "epsilon", + "clip_sample": true, + "act_hidden_dim": 512, + "dim_feedforward": 3200, + "enc_layers": 4, + "dec_layers": 7, + "nheads": 8, + "latent_dim": 32, + "kl_weight": 10.0, + "extra": {} + }, + "contract": { + "name": "g1_sonic_dfx6", + "robot": "g1", + "cameras": [ + "ego_view", + "chest" + ], + "state": [ + [ + "observation.state", + 0, + 29 + ], + [ + "observation.inspire_left_hand_state", + 0, + 6 + ], + [ + "observation.inspire_right_hand_state", + 0, + 6 + ], + [ + "observation.projected_gravity", + 0, + 3 + ] + ], + "action": [ + [ + "action.motion_token", + 0, + 64 + ], + [ + "observation.inspire_left_hand_state", + 0, + 6 + ], + [ + "observation.inspire_right_hand_state", + 0, + 6 + ] + ], + "state_dim": 44, + "action_dim": 76, + "action_space": "sonic_dfx6", + "body_dims": 29 + }, + "image_hw": [ + 168, + 224 + ], + "task_text": "Grasp the basket handle with the right hand from the v10 round table, walk while raising it, place and release it on the long desk, then withdraw the open right hand diagonally backward and upward without an additional downward arm return.", + "dataset": { + "name": "a20-basket-transfer-pre-reach-trim-expert-v2", + "root": "/mnt/data0/zuo/Humanoid_benchmark/data/baselines/a20-basket-transfer-pre-reach-trim-expert-v2", + "robot_type": null, + "fps": 30.0, + "episodes": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 36, + 37, + 38, + 39, + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55, + 56, + 57, + 58, + 59, + 60, + 61, + 62, + 63, + 64, + 65, + 66, + 67, + 68, + 69, + 70, + 71, + 72, + 73, + 74, + 75, + 76, + 77, + 78, + 79, + 80, + 81, + 82, + 83, + 84, + 85, + 86, + 87, + 88, + 89, + 90, + 91, + 92, + 93, + 94, + 95, + 96, + 97, + 98, + 99 + ], + "train_episodes": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 36, + 37, + 38, + 39, + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55, + 56, + 57, + 58, + 59, + 60, + 61, + 62, + 63, + 64, + 65, + 66, + 67, + 68, + 69, + 70, + 71, + 72, + 73, + 74, + 75, + 76, + 77, + 78, + 79, + 80, + 81, + 82, + 83, + 84, + 85, + 86, + 87, + 88, + 89, + 90, + 91, + 92, + 93, + 94, + 95, + 96, + 97, + 98, + 99 + ], + "val_episodes": [], + "train_frames": 54433, + "git_commit": null + }, + "training": { + "steps": 4000, + "batch_size": 512, + "lr": 0.0001, + "betas": [ + 0.95, + 0.999 + ], + "weight_decay": 1e-06, + "warmup": 500, + "schedule": "linear warm-up + cosine", + "backbone_lr_scale": 0.1, + "ema": 0.9999, + "shift_px": 0, + "amp": true, + "seed": 0, + "augmentation": { + "spec": "docs/augmentation_v2.md", + "version": "hb-aug-v2-20261109", + "implementation": "evaluation/augmentation/hb_aug_v2.py", + "visual": { + "effects": [ + "exposure", + "contrast", + "saturation", + "hue", + "gamma", + "temperature", + "rotation_crop", + "noise", + "gaussian_blur", + "jpeg", + "motion_blur", + "sharpen", + "defocus", + "light_shadow", + "wb_channel" + ], + "boosted_per_sample": 1, + "boost": 1.5, + "pixel_scale": "H / 224", + "future_targets": "none (one observation frame)" + }, + "proprio": { + "robot": "g1", + "body_dims": 29, + "bias_deg": 2.0, + "bias_scope": "per_sample", + "noise_deg": 0.25, + "noise_clip_sigma": 3, + "delay_probability": 0.0, + "history_drop_probability": 0.0, + "velocity": false + }, + "native_image_augmentation": "off", + "evaluation": "off" + }, + "preflight": { + "samples": 64, + "state_dim": 44, + "body_dims": 29, + "images_changed": 64, + "body_values_moved": 1.0, + "max_body_offset_deg": 2.5853, + "actions_bit_identical": true, + "non_body_state_bit_identical": true, + "eval_items_deterministic": 8 + } + }, + "stats": { + "state": { + "min": [ + -0.6335107088088989, + -0.26151371002197266, + -0.09143047779798508, + 0.13051827251911163, + -0.41364213824272156, + -0.17374181747436523, + -0.5706103444099426, + -0.4567093253135681, + -0.5945078730583191, + 0.07693764567375183, + -0.479587197303772, + -0.23942920565605164, + -0.1249256581068039, + -0.061801809817552567, + 0.03293042629957199, + -0.15965673327445984, + 0.12444422394037247, + -1.0301204919815063, + 0.8110364079475403, + -0.42373916506767273, + -0.2393926978111267, + -0.1671462506055832, + -1.8559280633926392, + -2.1003689765930176, + -0.44611114263534546, + -0.86998051404953, + -1.5242679119110107, + -0.9486506581306458, + -0.4465007185935974, + 0.9997791647911072, + 0.9987788796424866, + 0.9989608526229858, + 0.9997321963310242, + 0.8891316056251526, + 0.9935978651046753, + 0.3987571895122528, + 0.39748135209083557, + 0.3998531103134155, + 0.47865480184555054, + 0.43634313344955444, + 0.0, + -0.1312161237001419, + -0.10554340481758118, + -1.0 + ], + "max": [ + 0.18651676177978516, + 0.2880381643772125, + 0.3991033434867859, + 1.1868021488189697, + 0.30963531136512756, + 0.25351187586784363, + 0.13313187658786774, + 0.14919418096542358, + 0.09004810452461243, + 1.1367807388305664, + 0.47997909784317017, + 0.1897573471069336, + 0.12687286734580994, + 0.0903186947107315, + 0.2110750675201416, + 0.1764557808637619, + 0.3972950279712677, + -0.03375900536775589, + 1.248238205909729, + 0.5087170004844666, + 0.16197925806045532, + 0.19902198016643524, + 0.7476708292961121, + -0.14924407005310059, + 1.3937729597091675, + 1.4777758121490479, + 1.449870228767395, + 0.9068933725357056, + 1.5030572414398193, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.1443370282649994, + 0.07133471965789795, + -0.9889927506446838 + ], + "mean": [ + -0.1197492680031697, + 0.07687621598947185, + 0.13623676943730387, + 0.27067508540983654, + -0.09364010490166123, + -0.029264338937400346, + -0.0877177892591824, + -0.09172362643767557, + -0.21603681153788365, + 0.27020221177162185, + -0.13314246080400155, + 0.01521604484593622, + -0.007014541196278249, + 0.023597627586334522, + 0.1362211304918607, + 0.0044590163081436405, + 0.24020097902181153, + -0.5506130059849405, + 0.986759154493458, + -0.11353825983460156, + -0.09008970046721287, + 0.00793797976488734, + -0.7956446111257796, + -0.6433268364566035, + 0.35943391980855405, + 0.4128302732079997, + -0.6341984046200151, + 0.17366417734106818, + 0.31817584543570415, + 0.9999920738393157, + 0.9999916277741105, + 0.9999916507452171, + 0.9999910946907014, + 0.9975380262153739, + 0.9997842168412904, + 0.7431049855982732, + 0.743061001937713, + 0.7431612987122399, + 0.7812160503121215, + 0.7544858488771042, + 0.1884228588481359, + -0.018600662889153487, + -0.001923185563505277, + -0.9992875029526394 + ], + "std": [ + 0.1225902736805545, + 0.0667315519538753, + 0.06966628355722553, + 0.1683343132651939, + 0.11676670853146011, + 0.06393707008498987, + 0.14262675997248406, + 0.07183651035230894, + 0.07077858012951907, + 0.1583747884788023, + 0.09609718753190293, + 0.06937572380392895, + 0.038858334068535276, + 0.01853271444550659, + 0.019156535689633315, + 0.04755913941308977, + 0.041089601727671066, + 0.14354888870689778, + 0.053966848435702465, + 0.13684428975351073, + 0.05453949581291963, + 0.06076597347825228, + 0.6370602259123855, + 0.4013113120474627, + 0.24111485368872668, + 0.5381053688248473, + 0.48085827035516315, + 0.1527991524681821, + 0.44871163018722837, + 1.2824078270394207e-05, + 2.1003630589963823e-05, + 1.8574155779181667e-05, + 1.5382720229173254e-05, + 0.0031309164055460465, + 0.0001241612656891375, + 0.2938442161025196, + 0.29381668166139846, + 0.29367033348300187, + 0.25040401472179635, + 0.2708879748387615, + 0.3732468841625382, + 0.027734659598288042, + 0.01745881392269428, + 0.0008838855519788744 + ] + }, + "action": { + "min": [ + -0.125, + -0.5, + -0.375, + -0.375, + -0.625, + -0.5625, + -0.4375, + -0.25, + -0.375, + -0.1875, + -0.625, + -0.25, + -0.25, + -0.375, + -0.3125, + -0.5, + -0.1875, + -0.25, + -0.5, + -0.375, + -0.4375, + -0.5, + -0.3125, + -0.5, + -0.5, + 0.0, + -0.4375, + -0.5, + -0.5, + -0.375, + -0.0625, + -0.125, + -0.375, + -0.1875, + 0.0625, + -0.6875, + -0.1875, + -0.5, + -0.375, + -0.3125, + -0.4375, + -0.5625, + -0.375, + -0.125, + -0.5625, + -0.4375, + -0.375, + -0.25, + -0.1875, + -0.5625, + -0.375, + -0.1875, + -0.5, + -0.5625, + -0.3125, + -0.3125, + -0.5, + -0.125, + -0.625, + -0.3125, + -0.5625, + -0.1875, + -0.3125, + -0.125, + 0.9997791647911072, + 0.9987788796424866, + 0.9989608526229858, + 0.9997321963310242, + 0.8891316056251526, + 0.9935978651046753, + 0.3987571895122528, + 0.39748135209083557, + 0.3998531103134155, + 0.47865480184555054, + 0.43634313344955444, + 0.0 + ], + "max": [ + 0.5625, + 0.375, + 0.375, + 0.4375, + 0.25, + 0.25, + 0.1875, + 0.25, + 0.3125, + 0.1875, + 0.125, + 0.375, + 0.5, + 0.25, + 0.3125, + 0.25, + 0.375, + 0.5625, + 0.1875, + 0.25, + 0.25, + 0.1875, + 0.5, + 0.3125, + 0.0625, + 0.5, + 0.0, + 0.1875, + 0.3125, + 0.375, + 0.6875, + 0.5, + 0.25, + 0.5, + 0.375, + 0.125, + 0.6875, + 0.1875, + 0.375, + 0.25, + 0.125, + 0.375, + 0.1875, + 0.5, + 0.3125, + 0.5, + 0.5, + 0.4375, + 0.5, + 0.1875, + 0.5, + 0.3125, + 0.625, + 0.125, + 0.25, + 0.5625, + 0.5, + 0.3125, + 0.125, + 0.1875, + 0.25, + 0.625, + 0.4375, + 0.6875, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0 + ], + "mean": [ + 0.28530257380633073, + 0.0012216853746808002, + 0.02519611265225139, + 0.0920742013117043, + -0.18735762313302592, + -0.18916948358532507, + -0.10082578582844966, + -0.04547218599011629, + 0.009826300222291624, + -0.09046442415446512, + -0.3906258611504051, + 0.07456643947605313, + 0.06761982620836625, + 0.016528346774934324, + 0.02783008469127184, + -0.15926115591644774, + 0.137793250417945, + 0.13872558925651718, + -0.17794123050355484, + -0.04058314809031286, + -0.1081088218543898, + -0.27256788161593154, + 0.0295489408998218, + -0.12704953796410265, + -0.2065647217680451, + 0.257109657744383, + -0.2610296143883306, + -0.11359147943343192, + -0.08978239303363768, + 0.042958775007807766, + 0.3583063123472893, + 0.2295861425973215, + 0.005530882001726894, + 0.18030882001726894, + 0.22869399077765326, + -0.3416688865210442, + 0.16757757242849006, + -0.24212104789374092, + -0.08830580713905167, + 0.03962554883985817, + -0.24867382837616886, + -0.16334300883655137, + -0.10593298183087466, + 0.29390833685448164, + -0.17902857641504236, + -0.02676455459004648, + -0.010649560009553029, + 0.006596412102952253, + 0.19015693604982273, + -0.28608220197306783, + 0.0342829717267099, + 0.05214208292763581, + 0.2372308617933974, + -0.27289282236878365, + -0.07418638509727556, + 0.21441496886080136, + -0.07936362133264747, + 0.07053166277809417, + -0.2592969797732993, + -0.04738279168886521, + -0.25777561405764887, + 0.3476016387118108, + 0.0341980048867415, + 0.4095493542520162, + 0.9999920738393157, + 0.9999916277741105, + 0.9999916507452171, + 0.9999910946907014, + 0.9975380262153739, + 0.9997842168412904, + 0.7431049855982732, + 0.743061001937713, + 0.7431612987122399, + 0.7812160503121215, + 0.7544858488771042, + 0.1884228588481359 + ], + "std": [ + 0.13976769962393334, + 0.2042750041966315, + 0.11249063031363368, + 0.11082072512754977, + 0.1564752219884858, + 0.17244514813212275, + 0.10524673666978752, + 0.09606056085676273, + 0.16248529891535804, + 0.06517700714473815, + 0.12267993314050843, + 0.09188274452085451, + 0.1817845505649146, + 0.13895894808224718, + 0.09471602567899642, + 0.143892040083983, + 0.13123846053152619, + 0.17169478968550717, + 0.099377781857111, + 0.12335366669054255, + 0.09243992450340514, + 0.12474594719512408, + 0.1211368699762344, + 0.17473771543792177, + 0.12139112019256279, + 0.08058301151688785, + 0.08181288643740524, + 0.12245444611527911, + 0.15642136427791328, + 0.16023434329274994, + 0.1749952692685118, + 0.10352440920325795, + 0.07368975064774456, + 0.13144910448634894, + 0.04419890783153379, + 0.18157602689607938, + 0.15703202636240324, + 0.1308473547559705, + 0.10877569321547828, + 0.06509046280547963, + 0.08347560937172435, + 0.19809736217235097, + 0.06719381956707862, + 0.09127596366580104, + 0.13233099033714146, + 0.25952623468134783, + 0.21598709847307782, + 0.10931842674892184, + 0.11294459446909082, + 0.1403015515249487, + 0.10231818070157317, + 0.10139944441893624, + 0.288021039372521, + 0.11780041132456433, + 0.11650297357712408, + 0.21262101617713844, + 0.28703233994958033, + 0.09814627765272926, + 0.21616444128412057, + 0.06262758949048293, + 0.1745534315389492, + 0.17455467068976818, + 0.10468632106020986, + 0.18477106893909007, + 1.2824078270394207e-05, + 2.1003630589963823e-05, + 1.8574155779181667e-05, + 1.5382720229173254e-05, + 0.0031309164055460465, + 0.0001241612656891375, + 0.2938442161025196, + 0.29381668166139846, + 0.29367033348300187, + 0.25040401472179635, + 0.2708879748387615, + 0.3732468841625382 + ] + } + }, + "code_commit": "028df1bb7024fb1e3d176de7e60c84d3333cd97f" +} \ No newline at end of file diff --git a/a20-basket-transfer/b512-trimv2-cont-20261117/resume.json b/a20-basket-transfer/b512-trimv2-cont-20261117/resume.json new file mode 100644 index 0000000000000000000000000000000000000000..7219599c854402de0c9ff24ef871a4d2e45ad9c9 --- /dev/null +++ b/a20-basket-transfer/b512-trimv2-cont-20261117/resume.json @@ -0,0 +1,26 @@ +{ + "task": "a20", + "status": "paused_by_owner", + "saved_step": 6000, + "last_logged_step": 6500, + "paused_at": "2026-11-17T08:36:34.220194+00:00", + "planned_total_steps": 20000, + "dataset_repo": "humanoid-benchmark/a20-basket-transfer-pre-reach-trim-expert-v2", + "dataset_revision": "c4df4d40e51c439d7b41697d7c0d0184c844cb93", + "dataset_train_frames": 54433, + "dataset_episodes": 100, + "code_directory": "code/fmdit_continue_20261117", + "source_commit": "028df1bb7024fb1e3d176de7e60c84d3333cd97f", + "parent_local_checkpoint": "/mnt/data0/zuo/Humanoid_benchmark/results/baselines_preflight_20261117/runs/a20-basket-transfer-pre-reach-trim-expert-v2/fm_dit/checkpoints/step-004000", + "parent_state_sha256": "748979aefef3ddd72ced7072be1579603d027436c85731335a7c134ad93fb79d", + "offline_eval": { + "mse": 0.021967994049191475, + "mse_row0": 0.022920215502381325, + "mse_motion_token": 0.02157071977853775, + "mse_inspire_left_hand_state": 0.0040762959979474545, + "mse_inspire_right_hand_state": 0.044097285717725754, + "samples": 256, + "frames": "train" + }, + "limitation": "Portable resume starts at step 6000. Later logged updates are not in the saved state. RNG and DataLoader states were not saved by the original trainer; resume is not bit-exact." +} diff --git a/a20-basket-transfer/b512-trimv2-cont-20261117/run.json b/a20-basket-transfer/b512-trimv2-cont-20261117/run.json new file mode 100644 index 0000000000000000000000000000000000000000..bb0647fdd1732ebf22b24023c4035d9928780e6c --- /dev/null +++ b/a20-basket-transfer/b512-trimv2-cont-20261117/run.json @@ -0,0 +1,899 @@ +{ + "argv": [ + "/mnt/data0/zuo/Humanoid_benchmark/results/baselines_preflight_20261117/main/evaluation/baselines/train.py", + "--policy", + "fm_dit", + "--contract", + "g1_sonic_dfx6", + "--dataset", + "/mnt/data0/zuo/Humanoid_benchmark/data/baselines/a20-basket-transfer-pre-reach-trim-expert-v2", + "--cache-dir", + "/mnt/data0/zuo/Humanoid_benchmark/cache/baselines/frames", + "--out", + "/mnt/data0/zuo/Humanoid_benchmark/results/fmdit_continue_20261117_01/runs/a20/fm_dit", + "--batch-size", + "512", + "--steps", + "20000", + "--save-every", + "2000", + "--log-every", + "25", + "--workers", + "32", + "--resume" + ], + "started": "2026-11-17T15:57:43", + "host": "xt-p-A100-004", + "torch": "2.9.0+cu128", + "policy": { + "policy": "fm_dit", + "n_cameras": 2, + "state_dim": 44, + "action_dim": 76, + "horizon": 16, + "backbone": "resnet18", + "backbone_weights": "IMAGENET1K_V1", + "backbone_norm": "frozen_bn", + "hidden_dim": 384, + "num_layers": 4, + "num_heads": 8, + "mlp_ratio": 4.0, + "dropout": 0.1, + "time_embed_dim": 128, + "num_inference_steps": null, + "num_train_timesteps": 100, + "beta_schedule": "squaredcos_cap_v2", + "prediction_type": "epsilon", + "clip_sample": true, + "act_hidden_dim": 512, + "dim_feedforward": 3200, + "enc_layers": 4, + "dec_layers": 7, + "nheads": 8, + "latent_dim": 32, + "kl_weight": 10.0, + "extra": {} + }, + "contract": { + "name": "g1_sonic_dfx6", + "robot": "g1", + "cameras": [ + "ego_view", + "chest" + ], + "state": [ + [ + "observation.state", + 0, + 29 + ], + [ + "observation.inspire_left_hand_state", + 0, + 6 + ], + [ + "observation.inspire_right_hand_state", + 0, + 6 + ], + [ + "observation.projected_gravity", + 0, + 3 + ] + ], + "action": [ + [ + "action.motion_token", + 0, + 64 + ], + [ + "observation.inspire_left_hand_state", + 0, + 6 + ], + [ + "observation.inspire_right_hand_state", + 0, + 6 + ] + ], + "state_dim": 44, + "action_dim": 76, + "action_space": "sonic_dfx6", + "body_dims": 29 + }, + "image_hw": [ + 168, + 224 + ], + "task_text": "Grasp the basket handle with the right hand from the v10 round table, walk while raising it, place and release it on the long desk, then withdraw the open right hand diagonally backward and upward without an additional downward arm return.", + "dataset": { + "name": "a20-basket-transfer-pre-reach-trim-expert-v2", + "root": "/mnt/data0/zuo/Humanoid_benchmark/data/baselines/a20-basket-transfer-pre-reach-trim-expert-v2", + "robot_type": null, + "fps": 30.0, + "episodes": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 36, + 37, + 38, + 39, + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55, + 56, + 57, + 58, + 59, + 60, + 61, + 62, + 63, + 64, + 65, + 66, + 67, + 68, + 69, + 70, + 71, + 72, + 73, + 74, + 75, + 76, + 77, + 78, + 79, + 80, + 81, + 82, + 83, + 84, + 85, + 86, + 87, + 88, + 89, + 90, + 91, + 92, + 93, + 94, + 95, + 96, + 97, + 98, + 99 + ], + "train_episodes": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 36, + 37, + 38, + 39, + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55, + 56, + 57, + 58, + 59, + 60, + 61, + 62, + 63, + 64, + 65, + 66, + 67, + 68, + 69, + 70, + 71, + 72, + 73, + 74, + 75, + 76, + 77, + 78, + 79, + 80, + 81, + 82, + 83, + 84, + 85, + 86, + 87, + 88, + 89, + 90, + 91, + 92, + 93, + 94, + 95, + 96, + 97, + 98, + 99 + ], + "val_episodes": [], + "train_frames": 54433, + "git_commit": null + }, + "training": { + "steps": 20000, + "batch_size": 512, + "lr": 0.0001, + "betas": [ + 0.95, + 0.999 + ], + "weight_decay": 1e-06, + "warmup": 500, + "schedule": "linear warm-up + cosine", + "backbone_lr_scale": 0.1, + "ema": 0.9999, + "shift_px": 0, + "amp": true, + "seed": 0, + "augmentation": { + "spec": "docs/augmentation_v2.md", + "version": "hb-aug-v2-20261109", + "implementation": "evaluation/augmentation/hb_aug_v2.py", + "visual": { + "effects": [ + "exposure", + "contrast", + "saturation", + "hue", + "gamma", + "temperature", + "rotation_crop", + "noise", + "gaussian_blur", + "jpeg", + "motion_blur", + "sharpen", + "defocus", + "light_shadow", + "wb_channel" + ], + "boosted_per_sample": 1, + "boost": 1.5, + "pixel_scale": "H / 224", + "future_targets": "none (one observation frame)" + }, + "proprio": { + "robot": "g1", + "body_dims": 29, + "bias_deg": 2.0, + "bias_scope": "per_sample", + "noise_deg": 0.25, + "noise_clip_sigma": 3, + "delay_probability": 0.0, + "history_drop_probability": 0.0, + "velocity": false + }, + "native_image_augmentation": "off", + "evaluation": "off" + }, + "preflight": { + "samples": 64, + "state_dim": 44, + "body_dims": 29, + "images_changed": 64, + "body_values_moved": 1.0, + "max_body_offset_deg": 2.5087, + "actions_bit_identical": true, + "non_body_state_bit_identical": true, + "eval_items_deterministic": 8 + } + }, + "stats": { + "state": { + "min": [ + -0.6335107088088989, + -0.26151371002197266, + -0.09143047779798508, + 0.13051827251911163, + -0.41364213824272156, + -0.17374181747436523, + -0.5706103444099426, + -0.4567093253135681, + -0.5945078730583191, + 0.07693764567375183, + -0.479587197303772, + -0.23942920565605164, + -0.1249256581068039, + -0.061801809817552567, + 0.03293042629957199, + -0.15965673327445984, + 0.12444422394037247, + -1.0301204919815063, + 0.8110364079475403, + -0.42373916506767273, + -0.2393926978111267, + -0.1671462506055832, + -1.8559280633926392, + -2.1003689765930176, + -0.44611114263534546, + -0.86998051404953, + -1.5242679119110107, + -0.9486506581306458, + -0.4465007185935974, + 0.9997791647911072, + 0.9987788796424866, + 0.9989608526229858, + 0.9997321963310242, + 0.8891316056251526, + 0.9935978651046753, + 0.3987571895122528, + 0.39748135209083557, + 0.3998531103134155, + 0.47865480184555054, + 0.43634313344955444, + 0.0, + -0.1312161237001419, + -0.10554340481758118, + -1.0 + ], + "max": [ + 0.18651676177978516, + 0.2880381643772125, + 0.3991033434867859, + 1.1868021488189697, + 0.30963531136512756, + 0.25351187586784363, + 0.13313187658786774, + 0.14919418096542358, + 0.09004810452461243, + 1.1367807388305664, + 0.47997909784317017, + 0.1897573471069336, + 0.12687286734580994, + 0.0903186947107315, + 0.2110750675201416, + 0.1764557808637619, + 0.3972950279712677, + -0.03375900536775589, + 1.248238205909729, + 0.5087170004844666, + 0.16197925806045532, + 0.19902198016643524, + 0.7476708292961121, + -0.14924407005310059, + 1.3937729597091675, + 1.4777758121490479, + 1.449870228767395, + 0.9068933725357056, + 1.5030572414398193, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.1443370282649994, + 0.07133471965789795, + -0.9889927506446838 + ], + "mean": [ + -0.1197492680031697, + 0.07687621598947185, + 0.13623676943730387, + 0.27067508540983654, + -0.09364010490166123, + -0.029264338937400346, + -0.0877177892591824, + -0.09172362643767557, + -0.21603681153788365, + 0.27020221177162185, + -0.13314246080400155, + 0.01521604484593622, + -0.007014541196278249, + 0.023597627586334522, + 0.1362211304918607, + 0.0044590163081436405, + 0.24020097902181153, + -0.5506130059849405, + 0.986759154493458, + -0.11353825983460156, + -0.09008970046721287, + 0.00793797976488734, + -0.7956446111257796, + -0.6433268364566035, + 0.35943391980855405, + 0.4128302732079997, + -0.6341984046200151, + 0.17366417734106818, + 0.31817584543570415, + 0.9999920738393157, + 0.9999916277741105, + 0.9999916507452171, + 0.9999910946907014, + 0.9975380262153739, + 0.9997842168412904, + 0.7431049855982732, + 0.743061001937713, + 0.7431612987122399, + 0.7812160503121215, + 0.7544858488771042, + 0.1884228588481359, + -0.018600662889153487, + -0.001923185563505277, + -0.9992875029526394 + ], + "std": [ + 0.1225902736805545, + 0.0667315519538753, + 0.06966628355722553, + 0.1683343132651939, + 0.11676670853146011, + 0.06393707008498987, + 0.14262675997248406, + 0.07183651035230894, + 0.07077858012951907, + 0.1583747884788023, + 0.09609718753190293, + 0.06937572380392895, + 0.038858334068535276, + 0.01853271444550659, + 0.019156535689633315, + 0.04755913941308977, + 0.041089601727671066, + 0.14354888870689778, + 0.053966848435702465, + 0.13684428975351073, + 0.05453949581291963, + 0.06076597347825228, + 0.6370602259123855, + 0.4013113120474627, + 0.24111485368872668, + 0.5381053688248473, + 0.48085827035516315, + 0.1527991524681821, + 0.44871163018722837, + 1.2824078270394207e-05, + 2.1003630589963823e-05, + 1.8574155779181667e-05, + 1.5382720229173254e-05, + 0.0031309164055460465, + 0.0001241612656891375, + 0.2938442161025196, + 0.29381668166139846, + 0.29367033348300187, + 0.25040401472179635, + 0.2708879748387615, + 0.3732468841625382, + 0.027734659598288042, + 0.01745881392269428, + 0.0008838855519788744 + ] + }, + "action": { + "min": [ + -0.125, + -0.5, + -0.375, + -0.375, + -0.625, + -0.5625, + -0.4375, + -0.25, + -0.375, + -0.1875, + -0.625, + -0.25, + -0.25, + -0.375, + -0.3125, + -0.5, + -0.1875, + -0.25, + -0.5, + -0.375, + -0.4375, + -0.5, + -0.3125, + -0.5, + -0.5, + 0.0, + -0.4375, + -0.5, + -0.5, + -0.375, + -0.0625, + -0.125, + -0.375, + -0.1875, + 0.0625, + -0.6875, + -0.1875, + -0.5, + -0.375, + -0.3125, + -0.4375, + -0.5625, + -0.375, + -0.125, + -0.5625, + -0.4375, + -0.375, + -0.25, + -0.1875, + -0.5625, + -0.375, + -0.1875, + -0.5, + -0.5625, + -0.3125, + -0.3125, + -0.5, + -0.125, + -0.625, + -0.3125, + -0.5625, + -0.1875, + -0.3125, + -0.125, + 0.9997791647911072, + 0.9987788796424866, + 0.9989608526229858, + 0.9997321963310242, + 0.8891316056251526, + 0.9935978651046753, + 0.3987571895122528, + 0.39748135209083557, + 0.3998531103134155, + 0.47865480184555054, + 0.43634313344955444, + 0.0 + ], + "max": [ + 0.5625, + 0.375, + 0.375, + 0.4375, + 0.25, + 0.25, + 0.1875, + 0.25, + 0.3125, + 0.1875, + 0.125, + 0.375, + 0.5, + 0.25, + 0.3125, + 0.25, + 0.375, + 0.5625, + 0.1875, + 0.25, + 0.25, + 0.1875, + 0.5, + 0.3125, + 0.0625, + 0.5, + 0.0, + 0.1875, + 0.3125, + 0.375, + 0.6875, + 0.5, + 0.25, + 0.5, + 0.375, + 0.125, + 0.6875, + 0.1875, + 0.375, + 0.25, + 0.125, + 0.375, + 0.1875, + 0.5, + 0.3125, + 0.5, + 0.5, + 0.4375, + 0.5, + 0.1875, + 0.5, + 0.3125, + 0.625, + 0.125, + 0.25, + 0.5625, + 0.5, + 0.3125, + 0.125, + 0.1875, + 0.25, + 0.625, + 0.4375, + 0.6875, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0 + ], + "mean": [ + 0.28530257380633073, + 0.0012216853746808002, + 0.02519611265225139, + 0.0920742013117043, + -0.18735762313302592, + -0.18916948358532507, + -0.10082578582844966, + -0.04547218599011629, + 0.009826300222291624, + -0.09046442415446512, + -0.3906258611504051, + 0.07456643947605313, + 0.06761982620836625, + 0.016528346774934324, + 0.02783008469127184, + -0.15926115591644774, + 0.137793250417945, + 0.13872558925651718, + -0.17794123050355484, + -0.04058314809031286, + -0.1081088218543898, + -0.27256788161593154, + 0.0295489408998218, + -0.12704953796410265, + -0.2065647217680451, + 0.257109657744383, + -0.2610296143883306, + -0.11359147943343192, + -0.08978239303363768, + 0.042958775007807766, + 0.3583063123472893, + 0.2295861425973215, + 0.005530882001726894, + 0.18030882001726894, + 0.22869399077765326, + -0.3416688865210442, + 0.16757757242849006, + -0.24212104789374092, + -0.08830580713905167, + 0.03962554883985817, + -0.24867382837616886, + -0.16334300883655137, + -0.10593298183087466, + 0.29390833685448164, + -0.17902857641504236, + -0.02676455459004648, + -0.010649560009553029, + 0.006596412102952253, + 0.19015693604982273, + -0.28608220197306783, + 0.0342829717267099, + 0.05214208292763581, + 0.2372308617933974, + -0.27289282236878365, + -0.07418638509727556, + 0.21441496886080136, + -0.07936362133264747, + 0.07053166277809417, + -0.2592969797732993, + -0.04738279168886521, + -0.25777561405764887, + 0.3476016387118108, + 0.0341980048867415, + 0.4095493542520162, + 0.9999920738393157, + 0.9999916277741105, + 0.9999916507452171, + 0.9999910946907014, + 0.9975380262153739, + 0.9997842168412904, + 0.7431049855982732, + 0.743061001937713, + 0.7431612987122399, + 0.7812160503121215, + 0.7544858488771042, + 0.1884228588481359 + ], + "std": [ + 0.13976769962393334, + 0.2042750041966315, + 0.11249063031363368, + 0.11082072512754977, + 0.1564752219884858, + 0.17244514813212275, + 0.10524673666978752, + 0.09606056085676273, + 0.16248529891535804, + 0.06517700714473815, + 0.12267993314050843, + 0.09188274452085451, + 0.1817845505649146, + 0.13895894808224718, + 0.09471602567899642, + 0.143892040083983, + 0.13123846053152619, + 0.17169478968550717, + 0.099377781857111, + 0.12335366669054255, + 0.09243992450340514, + 0.12474594719512408, + 0.1211368699762344, + 0.17473771543792177, + 0.12139112019256279, + 0.08058301151688785, + 0.08181288643740524, + 0.12245444611527911, + 0.15642136427791328, + 0.16023434329274994, + 0.1749952692685118, + 0.10352440920325795, + 0.07368975064774456, + 0.13144910448634894, + 0.04419890783153379, + 0.18157602689607938, + 0.15703202636240324, + 0.1308473547559705, + 0.10877569321547828, + 0.06509046280547963, + 0.08347560937172435, + 0.19809736217235097, + 0.06719381956707862, + 0.09127596366580104, + 0.13233099033714146, + 0.25952623468134783, + 0.21598709847307782, + 0.10931842674892184, + 0.11294459446909082, + 0.1403015515249487, + 0.10231818070157317, + 0.10139944441893624, + 0.288021039372521, + 0.11780041132456433, + 0.11650297357712408, + 0.21262101617713844, + 0.28703233994958033, + 0.09814627765272926, + 0.21616444128412057, + 0.06262758949048293, + 0.1745534315389492, + 0.17455467068976818, + 0.10468632106020986, + 0.18477106893909007, + 1.2824078270394207e-05, + 2.1003630589963823e-05, + 1.8574155779181667e-05, + 1.5382720229173254e-05, + 0.0031309164055460465, + 0.0001241612656891375, + 0.2938442161025196, + 0.29381668166139846, + 0.29367033348300187, + 0.25040401472179635, + 0.2708879748387615, + 0.3732468841625382 + ] + } + }, + "code_commit": "028df1bb7024fb1e3d176de7e60c84d3333cd97f" +} \ No newline at end of file diff --git a/a20-basket-transfer/b512-trimv2-cont-20261117/step-006000/config.json b/a20-basket-transfer/b512-trimv2-cont-20261117/step-006000/config.json new file mode 100644 index 0000000000000000000000000000000000000000..a37f09cf16e3d5638fe83c7faee035f3c53fcf45 --- /dev/null +++ b/a20-basket-transfer/b512-trimv2-cont-20261117/step-006000/config.json @@ -0,0 +1,884 @@ +{ + "policy": { + "policy": "fm_dit", + "n_cameras": 2, + "state_dim": 44, + "action_dim": 76, + "horizon": 16, + "backbone": "resnet18", + "backbone_weights": "IMAGENET1K_V1", + "backbone_norm": "frozen_bn", + "hidden_dim": 384, + "num_layers": 4, + "num_heads": 8, + "mlp_ratio": 4.0, + "dropout": 0.1, + "time_embed_dim": 128, + "num_inference_steps": null, + "num_train_timesteps": 100, + "beta_schedule": "squaredcos_cap_v2", + "prediction_type": "epsilon", + "clip_sample": true, + "act_hidden_dim": 512, + "dim_feedforward": 3200, + "enc_layers": 4, + "dec_layers": 7, + "nheads": 8, + "latent_dim": 32, + "kl_weight": 10.0, + "extra": {} + }, + "contract": { + "name": "g1_sonic_dfx6", + "robot": "g1", + "cameras": [ + "ego_view", + "chest" + ], + "state": [ + [ + "observation.state", + 0, + 29 + ], + [ + "observation.inspire_left_hand_state", + 0, + 6 + ], + [ + "observation.inspire_right_hand_state", + 0, + 6 + ], + [ + "observation.projected_gravity", + 0, + 3 + ] + ], + "action": [ + [ + "action.motion_token", + 0, + 64 + ], + [ + "observation.inspire_left_hand_state", + 0, + 6 + ], + [ + "observation.inspire_right_hand_state", + 0, + 6 + ] + ], + "state_dim": 44, + "action_dim": 76, + "action_space": "sonic_dfx6", + "body_dims": 29 + }, + "image_hw": [ + 168, + 224 + ], + "task_text": "Grasp the basket handle with the right hand from the v10 round table, walk while raising it, place and release it on the long desk, then withdraw the open right hand diagonally backward and upward without an additional downward arm return.", + "dataset": { + "name": "a20-basket-transfer-pre-reach-trim-expert-v2", + "root": "/mnt/data0/zuo/Humanoid_benchmark/data/baselines/a20-basket-transfer-pre-reach-trim-expert-v2", + "robot_type": null, + "fps": 30.0, + "episodes": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 36, + 37, + 38, + 39, + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55, + 56, + 57, + 58, + 59, + 60, + 61, + 62, + 63, + 64, + 65, + 66, + 67, + 68, + 69, + 70, + 71, + 72, + 73, + 74, + 75, + 76, + 77, + 78, + 79, + 80, + 81, + 82, + 83, + 84, + 85, + 86, + 87, + 88, + 89, + 90, + 91, + 92, + 93, + 94, + 95, + 96, + 97, + 98, + 99 + ], + "train_episodes": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 36, + 37, + 38, + 39, + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55, + 56, + 57, + 58, + 59, + 60, + 61, + 62, + 63, + 64, + 65, + 66, + 67, + 68, + 69, + 70, + 71, + 72, + 73, + 74, + 75, + 76, + 77, + 78, + 79, + 80, + 81, + 82, + 83, + 84, + 85, + 86, + 87, + 88, + 89, + 90, + 91, + 92, + 93, + 94, + 95, + 96, + 97, + 98, + 99 + ], + "val_episodes": [], + "train_frames": 54433, + "git_commit": null + }, + "training": { + "steps": 20000, + "batch_size": 512, + "lr": 0.0001, + "betas": [ + 0.95, + 0.999 + ], + "weight_decay": 1e-06, + "warmup": 500, + "schedule": "linear warm-up + cosine", + "backbone_lr_scale": 0.1, + "ema": 0.9999, + "shift_px": 0, + "amp": true, + "seed": 0, + "augmentation": { + "spec": "docs/augmentation_v2.md", + "version": "hb-aug-v2-20261109", + "implementation": "evaluation/augmentation/hb_aug_v2.py", + "visual": { + "effects": [ + "exposure", + "contrast", + "saturation", + "hue", + "gamma", + "temperature", + "rotation_crop", + "noise", + "gaussian_blur", + "jpeg", + "motion_blur", + "sharpen", + "defocus", + "light_shadow", + "wb_channel" + ], + "boosted_per_sample": 1, + "boost": 1.5, + "pixel_scale": "H / 224", + "future_targets": "none (one observation frame)" + }, + "proprio": { + "robot": "g1", + "body_dims": 29, + "bias_deg": 2.0, + "bias_scope": "per_sample", + "noise_deg": 0.25, + "noise_clip_sigma": 3, + "delay_probability": 0.0, + "history_drop_probability": 0.0, + "velocity": false + }, + "native_image_augmentation": "off", + "evaluation": "off" + }, + "preflight": { + "samples": 64, + "state_dim": 44, + "body_dims": 29, + "images_changed": 64, + "body_values_moved": 1.0, + "max_body_offset_deg": 2.5087, + "actions_bit_identical": true, + "non_body_state_bit_identical": true, + "eval_items_deterministic": 8 + } + }, + "stats": { + "state": { + "min": [ + -0.6335107088088989, + -0.26151371002197266, + -0.09143047779798508, + 0.13051827251911163, + -0.41364213824272156, + -0.17374181747436523, + -0.5706103444099426, + -0.4567093253135681, + -0.5945078730583191, + 0.07693764567375183, + -0.479587197303772, + -0.23942920565605164, + -0.1249256581068039, + -0.061801809817552567, + 0.03293042629957199, + -0.15965673327445984, + 0.12444422394037247, + -1.0301204919815063, + 0.8110364079475403, + -0.42373916506767273, + -0.2393926978111267, + -0.1671462506055832, + -1.8559280633926392, + -2.1003689765930176, + -0.44611114263534546, + -0.86998051404953, + -1.5242679119110107, + -0.9486506581306458, + -0.4465007185935974, + 0.9997791647911072, + 0.9987788796424866, + 0.9989608526229858, + 0.9997321963310242, + 0.8891316056251526, + 0.9935978651046753, + 0.3987571895122528, + 0.39748135209083557, + 0.3998531103134155, + 0.47865480184555054, + 0.43634313344955444, + 0.0, + -0.1312161237001419, + -0.10554340481758118, + -1.0 + ], + "max": [ + 0.18651676177978516, + 0.2880381643772125, + 0.3991033434867859, + 1.1868021488189697, + 0.30963531136512756, + 0.25351187586784363, + 0.13313187658786774, + 0.14919418096542358, + 0.09004810452461243, + 1.1367807388305664, + 0.47997909784317017, + 0.1897573471069336, + 0.12687286734580994, + 0.0903186947107315, + 0.2110750675201416, + 0.1764557808637619, + 0.3972950279712677, + -0.03375900536775589, + 1.248238205909729, + 0.5087170004844666, + 0.16197925806045532, + 0.19902198016643524, + 0.7476708292961121, + -0.14924407005310059, + 1.3937729597091675, + 1.4777758121490479, + 1.449870228767395, + 0.9068933725357056, + 1.5030572414398193, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.1443370282649994, + 0.07133471965789795, + -0.9889927506446838 + ], + "mean": [ + -0.1197492680031697, + 0.07687621598947185, + 0.13623676943730387, + 0.27067508540983654, + -0.09364010490166123, + -0.029264338937400346, + -0.0877177892591824, + -0.09172362643767557, + -0.21603681153788365, + 0.27020221177162185, + -0.13314246080400155, + 0.01521604484593622, + -0.007014541196278249, + 0.023597627586334522, + 0.1362211304918607, + 0.0044590163081436405, + 0.24020097902181153, + -0.5506130059849405, + 0.986759154493458, + -0.11353825983460156, + -0.09008970046721287, + 0.00793797976488734, + -0.7956446111257796, + -0.6433268364566035, + 0.35943391980855405, + 0.4128302732079997, + -0.6341984046200151, + 0.17366417734106818, + 0.31817584543570415, + 0.9999920738393157, + 0.9999916277741105, + 0.9999916507452171, + 0.9999910946907014, + 0.9975380262153739, + 0.9997842168412904, + 0.7431049855982732, + 0.743061001937713, + 0.7431612987122399, + 0.7812160503121215, + 0.7544858488771042, + 0.1884228588481359, + -0.018600662889153487, + -0.001923185563505277, + -0.9992875029526394 + ], + "std": [ + 0.1225902736805545, + 0.0667315519538753, + 0.06966628355722553, + 0.1683343132651939, + 0.11676670853146011, + 0.06393707008498987, + 0.14262675997248406, + 0.07183651035230894, + 0.07077858012951907, + 0.1583747884788023, + 0.09609718753190293, + 0.06937572380392895, + 0.038858334068535276, + 0.01853271444550659, + 0.019156535689633315, + 0.04755913941308977, + 0.041089601727671066, + 0.14354888870689778, + 0.053966848435702465, + 0.13684428975351073, + 0.05453949581291963, + 0.06076597347825228, + 0.6370602259123855, + 0.4013113120474627, + 0.24111485368872668, + 0.5381053688248473, + 0.48085827035516315, + 0.1527991524681821, + 0.44871163018722837, + 1.2824078270394207e-05, + 2.1003630589963823e-05, + 1.8574155779181667e-05, + 1.5382720229173254e-05, + 0.0031309164055460465, + 0.0001241612656891375, + 0.2938442161025196, + 0.29381668166139846, + 0.29367033348300187, + 0.25040401472179635, + 0.2708879748387615, + 0.3732468841625382, + 0.027734659598288042, + 0.01745881392269428, + 0.0008838855519788744 + ] + }, + "action": { + "min": [ + -0.125, + -0.5, + -0.375, + -0.375, + -0.625, + -0.5625, + -0.4375, + -0.25, + -0.375, + -0.1875, + -0.625, + -0.25, + -0.25, + -0.375, + -0.3125, + -0.5, + -0.1875, + -0.25, + -0.5, + -0.375, + -0.4375, + -0.5, + -0.3125, + -0.5, + -0.5, + 0.0, + -0.4375, + -0.5, + -0.5, + -0.375, + -0.0625, + -0.125, + -0.375, + -0.1875, + 0.0625, + -0.6875, + -0.1875, + -0.5, + -0.375, + -0.3125, + -0.4375, + -0.5625, + -0.375, + -0.125, + -0.5625, + -0.4375, + -0.375, + -0.25, + -0.1875, + -0.5625, + -0.375, + -0.1875, + -0.5, + -0.5625, + -0.3125, + -0.3125, + -0.5, + -0.125, + -0.625, + -0.3125, + -0.5625, + -0.1875, + -0.3125, + -0.125, + 0.9997791647911072, + 0.9987788796424866, + 0.9989608526229858, + 0.9997321963310242, + 0.8891316056251526, + 0.9935978651046753, + 0.3987571895122528, + 0.39748135209083557, + 0.3998531103134155, + 0.47865480184555054, + 0.43634313344955444, + 0.0 + ], + "max": [ + 0.5625, + 0.375, + 0.375, + 0.4375, + 0.25, + 0.25, + 0.1875, + 0.25, + 0.3125, + 0.1875, + 0.125, + 0.375, + 0.5, + 0.25, + 0.3125, + 0.25, + 0.375, + 0.5625, + 0.1875, + 0.25, + 0.25, + 0.1875, + 0.5, + 0.3125, + 0.0625, + 0.5, + 0.0, + 0.1875, + 0.3125, + 0.375, + 0.6875, + 0.5, + 0.25, + 0.5, + 0.375, + 0.125, + 0.6875, + 0.1875, + 0.375, + 0.25, + 0.125, + 0.375, + 0.1875, + 0.5, + 0.3125, + 0.5, + 0.5, + 0.4375, + 0.5, + 0.1875, + 0.5, + 0.3125, + 0.625, + 0.125, + 0.25, + 0.5625, + 0.5, + 0.3125, + 0.125, + 0.1875, + 0.25, + 0.625, + 0.4375, + 0.6875, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0 + ], + "mean": [ + 0.28530257380633073, + 0.0012216853746808002, + 0.02519611265225139, + 0.0920742013117043, + -0.18735762313302592, + -0.18916948358532507, + -0.10082578582844966, + -0.04547218599011629, + 0.009826300222291624, + -0.09046442415446512, + -0.3906258611504051, + 0.07456643947605313, + 0.06761982620836625, + 0.016528346774934324, + 0.02783008469127184, + -0.15926115591644774, + 0.137793250417945, + 0.13872558925651718, + -0.17794123050355484, + -0.04058314809031286, + -0.1081088218543898, + -0.27256788161593154, + 0.0295489408998218, + -0.12704953796410265, + -0.2065647217680451, + 0.257109657744383, + -0.2610296143883306, + -0.11359147943343192, + -0.08978239303363768, + 0.042958775007807766, + 0.3583063123472893, + 0.2295861425973215, + 0.005530882001726894, + 0.18030882001726894, + 0.22869399077765326, + -0.3416688865210442, + 0.16757757242849006, + -0.24212104789374092, + -0.08830580713905167, + 0.03962554883985817, + -0.24867382837616886, + -0.16334300883655137, + -0.10593298183087466, + 0.29390833685448164, + -0.17902857641504236, + -0.02676455459004648, + -0.010649560009553029, + 0.006596412102952253, + 0.19015693604982273, + -0.28608220197306783, + 0.0342829717267099, + 0.05214208292763581, + 0.2372308617933974, + -0.27289282236878365, + -0.07418638509727556, + 0.21441496886080136, + -0.07936362133264747, + 0.07053166277809417, + -0.2592969797732993, + -0.04738279168886521, + -0.25777561405764887, + 0.3476016387118108, + 0.0341980048867415, + 0.4095493542520162, + 0.9999920738393157, + 0.9999916277741105, + 0.9999916507452171, + 0.9999910946907014, + 0.9975380262153739, + 0.9997842168412904, + 0.7431049855982732, + 0.743061001937713, + 0.7431612987122399, + 0.7812160503121215, + 0.7544858488771042, + 0.1884228588481359 + ], + "std": [ + 0.13976769962393334, + 0.2042750041966315, + 0.11249063031363368, + 0.11082072512754977, + 0.1564752219884858, + 0.17244514813212275, + 0.10524673666978752, + 0.09606056085676273, + 0.16248529891535804, + 0.06517700714473815, + 0.12267993314050843, + 0.09188274452085451, + 0.1817845505649146, + 0.13895894808224718, + 0.09471602567899642, + 0.143892040083983, + 0.13123846053152619, + 0.17169478968550717, + 0.099377781857111, + 0.12335366669054255, + 0.09243992450340514, + 0.12474594719512408, + 0.1211368699762344, + 0.17473771543792177, + 0.12139112019256279, + 0.08058301151688785, + 0.08181288643740524, + 0.12245444611527911, + 0.15642136427791328, + 0.16023434329274994, + 0.1749952692685118, + 0.10352440920325795, + 0.07368975064774456, + 0.13144910448634894, + 0.04419890783153379, + 0.18157602689607938, + 0.15703202636240324, + 0.1308473547559705, + 0.10877569321547828, + 0.06509046280547963, + 0.08347560937172435, + 0.19809736217235097, + 0.06719381956707862, + 0.09127596366580104, + 0.13233099033714146, + 0.25952623468134783, + 0.21598709847307782, + 0.10931842674892184, + 0.11294459446909082, + 0.1403015515249487, + 0.10231818070157317, + 0.10139944441893624, + 0.288021039372521, + 0.11780041132456433, + 0.11650297357712408, + 0.21262101617713844, + 0.28703233994958033, + 0.09814627765272926, + 0.21616444128412057, + 0.06262758949048293, + 0.1745534315389492, + 0.17455467068976818, + 0.10468632106020986, + 0.18477106893909007, + 1.2824078270394207e-05, + 2.1003630589963823e-05, + 1.8574155779181667e-05, + 1.5382720229173254e-05, + 0.0031309164055460465, + 0.0001241612656891375, + 0.2938442161025196, + 0.29381668166139846, + 0.29367033348300187, + 0.25040401472179635, + 0.2708879748387615, + 0.3732468841625382 + ] + } + }, + "code_commit": "028df1bb7024fb1e3d176de7e60c84d3333cd97f", + "step": 6000, + "weights": "ema", + "offline_eval": { + "mse": 0.021967994049191475, + "mse_row0": 0.022920215502381325, + "mse_motion_token": 0.02157071977853775, + "mse_inspire_left_hand_state": 0.0040762959979474545, + "mse_inspire_right_hand_state": 0.044097285717725754, + "samples": 256, + "frames": "train" + }, + "format": "hb_baselines/1" +} \ No newline at end of file diff --git a/a20-basket-transfer/b512-trimv2-cont-20261117/step-006000/model.pt b/a20-basket-transfer/b512-trimv2-cont-20261117/step-006000/model.pt new file mode 100644 index 0000000000000000000000000000000000000000..c68070b6670825ba924d16d88f1b99e67c35cd4e --- /dev/null +++ b/a20-basket-transfer/b512-trimv2-cont-20261117/step-006000/model.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e6330966ba75b01c067645e78990edf9ad0c9d1bc25a27a62daf7b27ae9d86d2 +size 90311637 diff --git a/a20-basket-transfer/b512-trimv2-cont-20261117/step-006000/protocol.json b/a20-basket-transfer/b512-trimv2-cont-20261117/step-006000/protocol.json new file mode 100644 index 0000000000000000000000000000000000000000..e9e6b0f2b81f6456b59c8e6286c7a498e46dd372 --- /dev/null +++ b/a20-basket-transfer/b512-trimv2-cont-20261117/step-006000/protocol.json @@ -0,0 +1,11 @@ +{ + "format": "hb_baselines/1", + "step": 6000, + "policy": "fm_dit", + "files": { + "model.pt": { + "bytes": 90311637, + "sha256": "e6330966ba75b01c067645e78990edf9ad0c9d1bc25a27a62daf7b27ae9d86d2" + } + } +} \ No newline at end of file diff --git a/a20-basket-transfer/b512-trimv2-cont-20261117/train_state.pt b/a20-basket-transfer/b512-trimv2-cont-20261117/train_state.pt new file mode 100644 index 0000000000000000000000000000000000000000..b7153143f7391e03cf2e74430a132f1934a6a7eb --- /dev/null +++ b/a20-basket-transfer/b512-trimv2-cont-20261117/train_state.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6d915877204726b60be63b55ae1b9714eb124f74236d86cd541195206297317c +size 361027953 diff --git a/b5-chair-push-standing-start/b512-cont-20261117/RESUME.md b/b5-chair-push-standing-start/b512-cont-20261117/RESUME.md new file mode 100644 index 0000000000000000000000000000000000000000..eebdd4eb6df74546efd5f055a9eba95dc94191a0 --- /dev/null +++ b/b5-chair-push-standing-start/b512-cont-20261117/RESUME.md @@ -0,0 +1,39 @@ +# B5 FM-DiT continuation, paused 2026-11-17 + +This is an independent continuation from the local step-4,000 baseline. +The portable training state is **step 6,000**; the last logged step is 6450. +Updates after step 6,000 remain only in the paused local process and are not in this release. +The training target was 20,000 total steps. This package is not a completed 20,000-step run. + +## Files + +- `train_state.pt`: raw policy, AdamW moments, LR scheduler, EMA, step and full config. +- `step-006000/`: EMA inference checkpoint, config and weight hash protocol. +- `run.json`, `resume.json`: recipe, data identity, preflight and pause details. +- `logs/`: current and original training logs. Log rows after 6,000 are history, not saved weights. +- `provenance/`: original run, original checkpoint config and hash protocol. +- `SHA256SUMS`: hashes of this package and its shared source snapshot, relative to repository root. + +## Resume + +Use Python 3.12 and install `code/fmdit_continue_20261117/baselines/requirements.txt`. +Download the original dataset separately (access to its private repository is required): + +```bash +hf download humanoid-benchmark/b5-chair-push-standing-start-expert-v1 --repo-type dataset --revision 913f5a16863c360cf2211c83bf98839d3f411293 --local-dir /path/to/b5-chair-push-standing-start-expert-v1 +python code/fmdit_continue_20261117/resume.py --task b5 --dataset /path/to/b5-chair-push-standing-start-expert-v1 --out /path/to/new_b5_run --cache-dir /path/to/frame_cache --steps 20000 --device cuda:0 +``` + +Run these commands from the downloaded model repository root. The launcher resolves its source +and state relative to itself, copies the state to a new output directory and starts the frozen +trainer with `--resume`; add `--dry-run` to inspect without copying or training. +Confirm the trainer logs `resumed` at step 6,000. `--steps` means total steps, not additional steps; +use 16000 for 10,000 more steps from this checkpoint. Existing output state is never overwritten. +No training automatically starts when this repository is downloaded. + +Batch 512, seed 0, state44/action76, 16-row chunks, 168x224 head/chest RGB, hb-v2 augmentation, +500-step original warm-up, AdamW and EMA are unchanged. At the default total of 20,000, +the saved LR/scheduler continues consistently. Choosing a different total changes cosine decay +from the next scheduler update. Original RNG and DataLoader state were not recorded; this is +not a bit-exact continuation. Offline MSE uses 256 training frames, not held-out validation +or closed-loop success. Source snapshot: `code/fmdit_continue_20261117/` (original commit `028df1bb7024fb1e3d176de7e60c84d3333cd97f`). diff --git a/b5-chair-push-standing-start/b512-cont-20261117/SHA256SUMS b/b5-chair-push-standing-start/b512-cont-20261117/SHA256SUMS new file mode 100644 index 0000000000000000000000000000000000000000..ccd06e7e6d3fe355e37f99193a0eea0a4326fc42 --- /dev/null +++ b/b5-chair-push-standing-start/b512-cont-20261117/SHA256SUMS @@ -0,0 +1,28 @@ +efd80e6edf87d36f99fd94bbe9b96d6b35d167ecde703487e319fbfa543fdd82 b5-chair-push-standing-start/b512-cont-20261117/RESUME.md +ab9a586c2b4e915ea22ee113593f94aaaa0450a29aecf25e591098a57cec5b84 b5-chair-push-standing-start/b512-cont-20261117/logs/parent_train_log.jsonl +8407c661bee0cf984f86c896d602aebcdd5bf5c825ffe5e4a694ac98c485fc2e b5-chair-push-standing-start/b512-cont-20261117/logs/train_log.jsonl +511d8933819ee51b21a2708fdd309a383f1bcce03bde0a34c658d47a33b2cff0 b5-chair-push-standing-start/b512-cont-20261117/logs/train_stdout.log +e0b6ab57b0ce166912bd79f487b087e6d82e852672dc4cbc8ffb2c6080ec9e7b b5-chair-push-standing-start/b512-cont-20261117/provenance/parent_checkpoint_config.json +98b407d4ce9da2f79c7ed8112d6318b1dd10941b0431fe18b789045f46be8579 b5-chair-push-standing-start/b512-cont-20261117/provenance/parent_checkpoint_protocol.json +cdd12f6b34c320fa2b98c4b9d3e7763b00b47a3cb2f08b3edfbaaa604d16c08d b5-chair-push-standing-start/b512-cont-20261117/provenance/parent_run.json +cc637e45fc7a755dc7fe107fec52d58ccf54cca2428ef6ea3824cc3ef49e21ab b5-chair-push-standing-start/b512-cont-20261117/resume.json +b6947d418f47afba62436535b134da472a1b7ef460157c8ad5996b967bcab151 b5-chair-push-standing-start/b512-cont-20261117/run.json +5577ebd4a19504480823331dcafb095e600fe6daf697ca5b144880cbfcfd2ec5 b5-chair-push-standing-start/b512-cont-20261117/step-006000/config.json +5178087fc50ffb7c5b233bae26ba6bc083c037bcd611d62a9f573166c8b564b3 b5-chair-push-standing-start/b512-cont-20261117/step-006000/model.pt +0a7efc01e3d1a6d2114ad7afba4f3820cfb13142dbcd80800c8604309a61933f b5-chair-push-standing-start/b512-cont-20261117/step-006000/protocol.json +74b1d066a512bb535c3e87ad34636723b05507ef591530ae59f80b45e20f0797 b5-chair-push-standing-start/b512-cont-20261117/train_state.pt +f7cf756cb60f0ce1da9b7ab8b1549b42f9a59d608bb06addd206d270b7c2d14c code/fmdit_continue_20261117/augmentation/__init__.py +96539b2fcf729917f668c9a776a0526c4b109611bf921dcbd753983c0f5813f2 code/fmdit_continue_20261117/augmentation/check_spec.py +9f7713cea4f846efcfc924234267ced48210357f419d67915c10b504316bedde code/fmdit_continue_20261117/augmentation/hb_aug_v2.py +815562186d3aaf87aba05085da6e951bc48b0b90a5d1c4910f2a3f625009bc52 code/fmdit_continue_20261117/augmentation/preview.py +0a8ee3ccab6e183804c08066a9e7eb207d9b13ca38c0e596ce5d58c8302d7929 code/fmdit_continue_20261117/baselines/__init__.py +3db95cd8e4e436875e11f1054a99d250ca7d94e0127909090b58730402340f19 code/fmdit_continue_20261117/baselines/act_nets.py +b537b03bc538910691b599862138494c11354147785a779a22ad83ff4b72a8d5 code/fmdit_continue_20261117/baselines/adapters.py +88c1971e18e9842136884f6a369468fe27f189b888b39870622304101a063b06 code/fmdit_continue_20261117/baselines/data.py +38a66937bb92393a3f1ce0644b249da28d582a4fd03d134b7743e6037abd88b9 code/fmdit_continue_20261117/baselines/nets.py +750d4c6193aabcacc0f53df26c2c51f64b3f836b940b32ccb5c69f5959419afa code/fmdit_continue_20261117/baselines/policies.py +bb211f66038e53efd08ee3ec03e133d8bb206ec36cc608d349b4fad3a61e874b code/fmdit_continue_20261117/baselines/requirements.txt +5d680d59124ca83f9c2088f8e1738395b87ddc58798acf7815e163102738b7ae code/fmdit_continue_20261117/baselines/runtime.py +b368d14aaab69d04f5372b1f3bb358c2a78f83134cc0e926faa8887e3ddbe297 code/fmdit_continue_20261117/baselines/train.py +fe9aaed24789080f05ce8f510d5a132bbd65e7c39b6de20500210742f0be4185 code/fmdit_continue_20261117/docs/augmentation_v2.md +5dbc2ec3f4f0638181eef9dca07c76f3594bdd5a24141f1a4b50d6532dbe4bba code/fmdit_continue_20261117/resume.py diff --git a/b5-chair-push-standing-start/b512-cont-20261117/logs/parent_train_log.jsonl b/b5-chair-push-standing-start/b512-cont-20261117/logs/parent_train_log.jsonl new file mode 100644 index 0000000000000000000000000000000000000000..2cfe619d42a88c77164c3b1a460d58c4c99b9ed0 --- /dev/null +++ b/b5-chair-push-standing-start/b512-cont-20261117/logs/parent_train_log.jsonl @@ -0,0 +1,41 @@ +{"event": "train", "step": 100, "lr": 2.0200000000000003e-05, "backbone_lr": 2.02e-06, "ema_decay": 0.968612, "it_s": 0.4, "data_frac": 0.916, "fm": 1.7907884335517883, "loss": 1.7907884335517883} +{"event": "train", "step": 200, "lr": 4.02e-05, "backbone_lr": 4.0200000000000005e-06, "ema_decay": 0.981267, "it_s": 0.45, "data_frac": 0.91, "fm": 1.084095389842987, "loss": 1.084095389842987} +{"event": "train", "step": 300, "lr": 6.02e-05, "backbone_lr": 6.02e-06, "ema_decay": 0.986162, "it_s": 0.48, "data_frac": 0.905, "fm": 0.8346792155504227, "loss": 0.8346792155504227} +{"event": "train", "step": 400, "lr": 8.020000000000001e-05, "backbone_lr": 8.020000000000001e-06, "ema_decay": 0.988841, "it_s": 0.41, "data_frac": 0.919, "fm": 0.33616609528660774, "loss": 0.33616609528660774} +{"event": "train", "step": 500, "lr": 0.0001, "backbone_lr": 1e-05, "ema_decay": 0.990557, "it_s": 0.42, "data_frac": 0.917, "fm": 0.22414932146668434, "loss": 0.22414932146668434} +{"event": "train", "step": 600, "lr": 9.979871469976196e-05, "backbone_lr": 9.979871469976197e-06, "ema_decay": 0.991762, "it_s": 0.47, "data_frac": 0.906, "fm": 0.19345682978630066, "loss": 0.19345682978630066} +{"event": "train", "step": 700, "lr": 9.919647942993148e-05, "backbone_lr": 9.91964794299315e-06, "ema_decay": 0.99266, "it_s": 0.39, "data_frac": 0.921, "fm": 0.17725971847772598, "loss": 0.17725971847772598} +{"event": "train", "step": 800, "lr": 9.819814303479267e-05, "backbone_lr": 9.819814303479268e-06, "ema_decay": 0.993358, "it_s": 0.43, "data_frac": 0.915, "fm": 0.16511793345212936, "loss": 0.16511793345212936} +{"event": "train", "step": 900, "lr": 9.681174353198687e-05, "backbone_lr": 9.681174353198687e-06, "ema_decay": 0.993919, "it_s": 0.45, "data_frac": 0.91, "fm": 0.15741885900497438, "loss": 0.15741885900497438} +{"event": "train", "step": 1000, "lr": 9.504844339512095e-05, "backbone_lr": 9.504844339512096e-06, "ema_decay": 0.994381, "it_s": 0.43, "data_frac": 0.912, "fm": 0.15153889253735542, "loss": 0.15153889253735542} +{"event": "train", "step": 1100, "lr": 9.292243968009331e-05, "backbone_lr": 9.292243968009332e-06, "ema_decay": 0.994768, "it_s": 0.42, "data_frac": 0.916, "fm": 0.14672573275864123, "loss": 0.14672573275864123} +{"event": "train", "step": 1200, "lr": 9.045084971874738e-05, "backbone_lr": 9.045084971874738e-06, "ema_decay": 0.995098, "it_s": 0.48, "data_frac": 0.903, "fm": 0.14304948396980763, "loss": 0.14304948396980763} +{"event": "train", "step": 1300, "lr": 8.765357330018056e-05, "backbone_lr": 8.765357330018056e-06, "ema_decay": 0.995384, "it_s": 0.41, "data_frac": 0.918, "fm": 0.13919761806726455, "loss": 0.13919761806726455} +{"event": "train", "step": 1400, "lr": 8.455313244934324e-05, "backbone_lr": 8.455313244934324e-06, "ema_decay": 0.995633, "it_s": 0.45, "data_frac": 0.91, "fm": 0.13632708579301833, "loss": 0.13632708579301833} +{"event": "train", "step": 1500, "lr": 8.117449009293668e-05, "backbone_lr": 8.117449009293668e-06, "ema_decay": 0.995853, "it_s": 0.48, "data_frac": 0.905, "fm": 0.13352992624044419, "loss": 0.13352992624044419} +{"event": "train", "step": 1600, "lr": 7.754484907260513e-05, "backbone_lr": 7.754484907260513e-06, "ema_decay": 0.996049, "it_s": 0.38, "data_frac": 0.924, "fm": 0.1312162097543478, "loss": 0.1312162097543478} +{"event": "train", "step": 1700, "lr": 7.369343312364993e-05, "backbone_lr": 7.369343312364994e-06, "ema_decay": 0.996225, "it_s": 0.47, "data_frac": 0.907, "fm": 0.13106721945106983, "loss": 0.13106721945106983} +{"event": "train", "step": 1800, "lr": 6.965125158269619e-05, "backbone_lr": 6.965125158269619e-06, "ema_decay": 0.996383, "it_s": 0.45, "data_frac": 0.911, "fm": 0.1269450791180134, "loss": 0.1269450791180134} +{"event": "train", "step": 1900, "lr": 6.545084971874738e-05, "backbone_lr": 6.545084971874738e-06, "ema_decay": 0.996527, "it_s": 0.42, "data_frac": 0.916, "fm": 0.12604121074080468, "loss": 0.12604121074080468} +{"event": "train", "step": 2000, "lr": 6.112604669781572e-05, "backbone_lr": 6.112604669781572e-06, "ema_decay": 0.996658, "it_s": 0.4, "data_frac": 0.92, "fm": 0.1254367930442095, "loss": 0.1254367930442095} +{"event": "train", "step": 2100, "lr": 5.6711663290882776e-05, "backbone_lr": 5.671166329088278e-06, "ema_decay": 0.996778, "it_s": 0.44, "data_frac": 0.912, "fm": 0.12186870411038399, "loss": 0.12186870411038399} +{"event": "train", "step": 2200, "lr": 5.2243241517525754e-05, "backbone_lr": 5.224324151752575e-06, "ema_decay": 0.996888, "it_s": 0.4, "data_frac": 0.92, "fm": 0.12166027002036571, "loss": 0.12166027002036571} +{"event": "train", "step": 2300, "lr": 4.775675848247427e-05, "backbone_lr": 4.775675848247427e-06, "ema_decay": 0.99699, "it_s": 0.43, "data_frac": 0.914, "fm": 0.11966084256768227, "loss": 0.11966084256768227} +{"event": "train", "step": 2400, "lr": 4.328833670911724e-05, "backbone_lr": 4.3288336709117246e-06, "ema_decay": 0.997085, "it_s": 0.46, "data_frac": 0.909, "fm": 0.11889605417847633, "loss": 0.11889605417847633} +{"event": "train", "step": 2500, "lr": 3.887395330218429e-05, "backbone_lr": 3.887395330218429e-06, "ema_decay": 0.997172, "it_s": 0.39, "data_frac": 0.922, "fm": 0.11775025725364685, "loss": 0.11775025725364685} +{"event": "train", "step": 2600, "lr": 3.4549150281252636e-05, "backbone_lr": 3.4549150281252635e-06, "ema_decay": 0.997254, "it_s": 0.44, "data_frac": 0.912, "fm": 0.11720367975533008, "loss": 0.11720367975533008} +{"event": "train", "step": 2700, "lr": 3.0348748417303823e-05, "backbone_lr": 3.0348748417303826e-06, "ema_decay": 0.997331, "it_s": 0.45, "data_frac": 0.91, "fm": 0.11644909419119358, "loss": 0.11644909419119358} +{"event": "train", "step": 2800, "lr": 2.630656687635007e-05, "backbone_lr": 2.6306566876350072e-06, "ema_decay": 0.997403, "it_s": 0.38, "data_frac": 0.923, "fm": 0.11563613310456276, "loss": 0.11563613310456276} +{"event": "train", "step": 2900, "lr": 2.245515092739488e-05, "backbone_lr": 2.245515092739488e-06, "ema_decay": 0.99747, "it_s": 0.45, "data_frac": 0.911, "fm": 0.11447477258741856, "loss": 0.11447477258741856} +{"event": "train", "step": 3000, "lr": 1.8825509907063327e-05, "backbone_lr": 1.8825509907063328e-06, "ema_decay": 0.997534, "it_s": 0.43, "data_frac": 0.914, "fm": 0.11389160357415676, "loss": 0.11389160357415676} +{"event": "train", "step": 3100, "lr": 1.544686755065677e-05, "backbone_lr": 1.544686755065677e-06, "ema_decay": 0.997594, "it_s": 0.39, "data_frac": 0.921, "fm": 0.11373364746570587, "loss": 0.11373364746570587} +{"event": "train", "step": 3200, "lr": 1.2346426699819458e-05, "backbone_lr": 1.234642669981946e-06, "ema_decay": 0.99765, "it_s": 0.42, "data_frac": 0.917, "fm": 0.11283902548253537, "loss": 0.11283902548253537} +{"event": "train", "step": 3300, "lr": 9.549150281252633e-06, "backbone_lr": 9.549150281252633e-07, "ema_decay": 0.997704, "it_s": 0.43, "data_frac": 0.913, "fm": 0.11270486310124397, "loss": 0.11270486310124397} +{"event": "train", "step": 3400, "lr": 7.077560319906695e-06, "backbone_lr": 7.077560319906696e-07, "ema_decay": 0.997755, "it_s": 0.39, "data_frac": 0.92, "fm": 0.11248455762863159, "loss": 0.11248455762863159} +{"event": "train", "step": 3500, "lr": 4.951556604879048e-06, "backbone_lr": 4.951556604879049e-07, "ema_decay": 0.997803, "it_s": 0.44, "data_frac": 0.913, "fm": 0.11236242108047008, "loss": 0.11236242108047008} +{"event": "train", "step": 3600, "lr": 3.18825646801314e-06, "backbone_lr": 3.18825646801314e-07, "ema_decay": 0.997849, "it_s": 0.49, "data_frac": 0.903, "fm": 0.11242782689630985, "loss": 0.11242782689630985} +{"event": "train", "step": 3700, "lr": 1.8018569652073381e-06, "backbone_lr": 1.801856965207338e-07, "ema_decay": 0.997893, "it_s": 0.42, "data_frac": 0.916, "fm": 0.11282645657658577, "loss": 0.11282645657658577} +{"event": "train", "step": 3800, "lr": 8.035205700685167e-07, "backbone_lr": 8.035205700685167e-08, "ema_decay": 0.997934, "it_s": 0.44, "data_frac": 0.913, "fm": 0.11049417972564697, "loss": 0.11049417972564697} +{"event": "train", "step": 3900, "lr": 2.012853002380466e-07, "backbone_lr": 2.012853002380466e-08, "ema_decay": 0.997974, "it_s": 0.46, "data_frac": 0.907, "fm": 0.11322126060724258, "loss": 0.11322126060724258} +{"event": "train", "step": 4000, "lr": 0.0, "backbone_lr": 0.0, "ema_decay": 0.998012, "it_s": 0.39, "data_frac": 0.922, "fm": 0.11050335392355919, "loss": 0.11050335392355919} +{"event": "save", "step": 4000, "checkpoint": "/mnt/data0/zuo/Humanoid_benchmark/results/baselines_preflight_20261117/runs/b5_chair_push_standing_start_v1/fm_dit/checkpoints/step-004000", "mse": 0.03382289409637451, "mse_row0": 0.03190438449382782, "mse_motion_token": 0.030815009027719498, "mse_inspire_left_hand_state": 0.04951713606715202, "mse_inspire_right_hand_state": 0.0502127967774868} diff --git a/b5-chair-push-standing-start/b512-cont-20261117/logs/train_log.jsonl b/b5-chair-push-standing-start/b512-cont-20261117/logs/train_log.jsonl new file mode 100644 index 0000000000000000000000000000000000000000..76bcd1b366a520e73969ce6e11535ab3c772013b --- /dev/null +++ b/b5-chair-push-standing-start/b512-cont-20261117/logs/train_log.jsonl @@ -0,0 +1,99 @@ +{"event": "train", "step": 4025, "lr": 9.215152885383474e-05, "backbone_lr": 9.215152885383473e-06, "ema_decay": 0.998021, "it_s": 0.61, "data_frac": 0.84, "fm": 0.30036773085594176, "loss": 0.30036773085594176} +{"event": "train", "step": 4050, "lr": 9.204286963963111e-05, "backbone_lr": 9.204286963963112e-06, "ema_decay": 0.998031, "it_s": 0.88, "data_frac": 0.823, "fm": 0.20012644588947295, "loss": 0.20012644588947295} +{"event": "train", "step": 4075, "lr": 9.193352839727121e-05, "backbone_lr": 9.193352839727122e-06, "ema_decay": 0.99804, "it_s": 0.79, "data_frac": 0.85, "fm": 0.16311768770217897, "loss": 0.16311768770217897} +{"event": "train", "step": 4100, "lr": 9.182350690051133e-05, "backbone_lr": 9.182350690051134e-06, "ema_decay": 0.998049, "it_s": 2.23, "data_frac": 0.587, "fm": 0.1437873661518097, "loss": 0.1437873661518097} +{"event": "train", "step": 4125, "lr": 9.171280693414307e-05, "backbone_lr": 9.171280693414307e-06, "ema_decay": 0.998058, "it_s": 0.59, "data_frac": 0.885, "fm": 0.13227831691503525, "loss": 0.13227831691503525} +{"event": "train", "step": 4150, "lr": 9.160143029396422e-05, "backbone_lr": 9.160143029396422e-06, "ema_decay": 0.998066, "it_s": 4.72, "data_frac": 0.002, "fm": 0.12756982773542405, "loss": 0.12756982773542405} +{"event": "train", "step": 4175, "lr": 9.148937878674976e-05, "backbone_lr": 9.148937878674975e-06, "ema_decay": 0.998075, "it_s": 0.92, "data_frac": 0.826, "fm": 0.12656888127326965, "loss": 0.12656888127326965} +{"event": "train", "step": 4200, "lr": 9.13766542302225e-05, "backbone_lr": 9.13766542302225e-06, "ema_decay": 0.998084, "it_s": 1.02, "data_frac": 0.794, "fm": 0.1255243808031082, "loss": 0.1255243808031082} +{"event": "train", "step": 4225, "lr": 9.126325845302357e-05, "backbone_lr": 9.126325845302357e-06, "ema_decay": 0.998092, "it_s": 1.34, "data_frac": 0.73, "fm": 0.12075435340404511, "loss": 0.12075435340404511} +{"event": "train", "step": 4250, "lr": 9.114919329468282e-05, "backbone_lr": 9.114919329468283e-06, "ema_decay": 0.998101, "it_s": 0.6, "data_frac": 0.874, "fm": 0.11895803481340408, "loss": 0.11895803481340408} +{"event": "train", "step": 4275, "lr": 9.103446060558892e-05, "backbone_lr": 9.103446060558892e-06, "ema_decay": 0.998109, "it_s": 1.11, "data_frac": 0.774, "fm": 0.11751056581735611, "loss": 0.11751056581735611} +{"event": "train", "step": 4300, "lr": 9.091906224695935e-05, "backbone_lr": 9.091906224695935e-06, "ema_decay": 0.998117, "it_s": 5.33, "data_frac": 0.001, "fm": 0.11636869221925736, "loss": 0.11636869221925736} +{"event": "train", "step": 4325, "lr": 9.080300009081024e-05, "backbone_lr": 9.080300009081025e-06, "ema_decay": 0.998125, "it_s": 0.94, "data_frac": 0.801, "fm": 0.11493111044168472, "loss": 0.11493111044168472} +{"event": "train", "step": 4350, "lr": 9.068627601992598e-05, "backbone_lr": 9.0686276019926e-06, "ema_decay": 0.998133, "it_s": 0.92, "data_frac": 0.808, "fm": 0.11559643805027008, "loss": 0.11559643805027008} +{"event": "train", "step": 4375, "lr": 9.056889192782866e-05, "backbone_lr": 9.056889192782865e-06, "ema_decay": 0.998141, "it_s": 0.7, "data_frac": 0.861, "fm": 0.11376965045928955, "loss": 0.11376965045928955} +{"event": "train", "step": 4400, "lr": 9.045084971874738e-05, "backbone_lr": 9.045084971874738e-06, "ema_decay": 0.998149, "it_s": 1.79, "data_frac": 0.654, "fm": 0.1141718265414238, "loss": 0.1141718265414238} +{"event": "train", "step": 4425, "lr": 9.033215130758735e-05, "backbone_lr": 9.033215130758735e-06, "ema_decay": 0.998157, "it_s": 0.93, "data_frac": 0.812, "fm": 0.11762478679418564, "loss": 0.11762478679418564} +{"event": "train", "step": 4450, "lr": 9.021279861989885e-05, "backbone_lr": 9.021279861989884e-06, "ema_decay": 0.998165, "it_s": 2.38, "data_frac": 0.498, "fm": 0.1135200247168541, "loss": 0.1135200247168541} +{"event": "train", "step": 4475, "lr": 9.009279359184593e-05, "backbone_lr": 9.009279359184594e-06, "ema_decay": 0.998173, "it_s": 0.93, "data_frac": 0.81, "fm": 0.11074629306793213, "loss": 0.11074629306793213} +{"event": "train", "step": 4500, "lr": 8.997213817017507e-05, "backbone_lr": 8.997213817017508e-06, "ema_decay": 0.99818, "it_s": 0.76, "data_frac": 0.837, "fm": 0.11229030340909958, "loss": 0.11229030340909958} +{"event": "train", "step": 4525, "lr": 8.985083431218354e-05, "backbone_lr": 8.985083431218354e-06, "ema_decay": 0.998188, "it_s": 1.22, "data_frac": 0.739, "fm": 0.10931755155324936, "loss": 0.10931755155324936} +{"event": "train", "step": 4550, "lr": 8.972888398568772e-05, "backbone_lr": 8.972888398568772e-06, "ema_decay": 0.998195, "it_s": 0.72, "data_frac": 0.862, "fm": 0.11274766504764556, "loss": 0.11274766504764556} +{"event": "train", "step": 4575, "lr": 8.96062891689911e-05, "backbone_lr": 8.96062891689911e-06, "ema_decay": 0.998203, "it_s": 1.19, "data_frac": 0.766, "fm": 0.11252273738384247, "loss": 0.11252273738384247} +{"event": "train", "step": 4600, "lr": 8.948305185085225e-05, "backbone_lr": 8.948305185085226e-06, "ema_decay": 0.99821, "it_s": 5.44, "data_frac": 0.001, "fm": 0.11177674174308777, "loss": 0.11177674174308777} +{"event": "train", "step": 4625, "lr": 8.935917403045251e-05, "backbone_lr": 8.935917403045251e-06, "ema_decay": 0.998217, "it_s": 0.69, "data_frac": 0.864, "fm": 0.10977381765842438, "loss": 0.10977381765842438} +{"event": "train", "step": 4650, "lr": 8.92346577173636e-05, "backbone_lr": 8.923465771736361e-06, "ema_decay": 0.998224, "it_s": 1.39, "data_frac": 0.739, "fm": 0.11017541021108627, "loss": 0.11017541021108627} +{"event": "train", "step": 4675, "lr": 8.910950493151498e-05, "backbone_lr": 8.910950493151498e-06, "ema_decay": 0.998232, "it_s": 0.71, "data_frac": 0.861, "fm": 0.11157936811447143, "loss": 0.11157936811447143} +{"event": "train", "step": 4700, "lr": 8.898371770316111e-05, "backbone_lr": 8.898371770316113e-06, "ema_decay": 0.998239, "it_s": 1.64, "data_frac": 0.685, "fm": 0.11001256704330445, "loss": 0.11001256704330445} +{"event": "train", "step": 4725, "lr": 8.885729807284856e-05, "backbone_lr": 8.885729807284855e-06, "ema_decay": 0.998246, "it_s": 0.65, "data_frac": 0.874, "fm": 0.1063080707192421, "loss": 0.1063080707192421} +{"event": "train", "step": 4750, "lr": 8.873024809138272e-05, "backbone_lr": 8.873024809138272e-06, "ema_decay": 0.998253, "it_s": 5.07, "data_frac": 0.022, "fm": 0.10854193478822709, "loss": 0.10854193478822709} +{"event": "train", "step": 4775, "lr": 8.860256981979484e-05, "backbone_lr": 8.860256981979485e-06, "ema_decay": 0.998259, "it_s": 1.34, "data_frac": 0.739, "fm": 0.10757887542247772, "loss": 0.10757887542247772} +{"event": "train", "step": 4800, "lr": 8.847426532930831e-05, "backbone_lr": 8.84742653293083e-06, "ema_decay": 0.998266, "it_s": 0.84, "data_frac": 0.83, "fm": 0.10880939394235611, "loss": 0.10880939394235611} +{"event": "train", "step": 4825, "lr": 8.834533670130518e-05, "backbone_lr": 8.834533670130518e-06, "ema_decay": 0.998273, "it_s": 1.09, "data_frac": 0.783, "fm": 0.10648771852254868, "loss": 0.10648771852254868} +{"event": "train", "step": 4850, "lr": 8.821578602729242e-05, "backbone_lr": 8.821578602729242e-06, "ema_decay": 0.99828, "it_s": 0.71, "data_frac": 0.861, "fm": 0.10719650745391845, "loss": 0.10719650745391845} +{"event": "train", "step": 4875, "lr": 8.808561540886796e-05, "backbone_lr": 8.808561540886796e-06, "ema_decay": 0.998286, "it_s": 1.78, "data_frac": 0.662, "fm": 0.10717803657054902, "loss": 0.10717803657054902} +{"event": "train", "step": 4900, "lr": 8.795482695768658e-05, "backbone_lr": 8.795482695768658e-06, "ema_decay": 0.998293, "it_s": 3.88, "data_frac": 0.275, "fm": 0.10350908666849136, "loss": 0.10350908666849136} +{"event": "train", "step": 4925, "lr": 8.782342279542568e-05, "backbone_lr": 8.782342279542569e-06, "ema_decay": 0.998299, "it_s": 0.88, "data_frac": 0.808, "fm": 0.10498440027236938, "loss": 0.10498440027236938} +{"event": "train", "step": 4950, "lr": 8.769140505375085e-05, "backbone_lr": 8.769140505375084e-06, "ema_decay": 0.998306, "it_s": 0.96, "data_frac": 0.813, "fm": 0.1064383864402771, "loss": 0.1064383864402771} +{"event": "train", "step": 4975, "lr": 8.755877587428128e-05, "backbone_lr": 8.755877587428127e-06, "ema_decay": 0.998312, "it_s": 0.85, "data_frac": 0.837, "fm": 0.10203986793756485, "loss": 0.10203986793756485} +{"event": "train", "step": 5000, "lr": 8.742553740855506e-05, "backbone_lr": 8.742553740855507e-06, "ema_decay": 0.998318, "it_s": 1.69, "data_frac": 0.668, "fm": 0.10572098553180695, "loss": 0.10572098553180695} +{"event": "train", "step": 5025, "lr": 8.729169181799422e-05, "backbone_lr": 8.729169181799423e-06, "ema_decay": 0.998325, "it_s": 0.6, "data_frac": 0.885, "fm": 0.10311138957738876, "loss": 0.10311138957738876} +{"event": "train", "step": 5050, "lr": 8.715724127386972e-05, "backbone_lr": 8.715724127386971e-06, "ema_decay": 0.998331, "it_s": 4.75, "data_frac": 0.002, "fm": 0.10220446288585663, "loss": 0.10220446288585663} +{"event": "train", "step": 5075, "lr": 8.702218795726617e-05, "backbone_lr": 8.702218795726619e-06, "ema_decay": 0.998337, "it_s": 1.02, "data_frac": 0.807, "fm": 0.10448342114686966, "loss": 0.10448342114686966} +{"event": "train", "step": 5100, "lr": 8.688653405904652e-05, "backbone_lr": 8.688653405904653e-06, "ema_decay": 0.998343, "it_s": 0.97, "data_frac": 0.793, "fm": 0.1041489601135254, "loss": 0.1041489601135254} +{"event": "train", "step": 5125, "lr": 8.675028177981643e-05, "backbone_lr": 8.675028177981643e-06, "ema_decay": 0.998349, "it_s": 1.31, "data_frac": 0.727, "fm": 0.10290780544281006, "loss": 0.10290780544281006} +{"event": "train", "step": 5150, "lr": 8.661343332988869e-05, "backbone_lr": 8.661343332988869e-06, "ema_decay": 0.998355, "it_s": 0.64, "data_frac": 0.874, "fm": 0.10130853086709976, "loss": 0.10130853086709976} +{"event": "train", "step": 5175, "lr": 8.647599092924721e-05, "backbone_lr": 8.64759909292472e-06, "ema_decay": 0.998361, "it_s": 1.04, "data_frac": 0.796, "fm": 0.1019318300485611, "loss": 0.1019318300485611} +{"event": "train", "step": 5200, "lr": 8.633795680751116e-05, "backbone_lr": 8.633795680751116e-06, "ema_decay": 0.998367, "it_s": 5.48, "data_frac": 0.001, "fm": 0.10094406306743622, "loss": 0.10094406306743622} +{"event": "train", "step": 5225, "lr": 8.619933320389872e-05, "backbone_lr": 8.619933320389872e-06, "ema_decay": 0.998373, "it_s": 0.99, "data_frac": 0.807, "fm": 0.10107415467500687, "loss": 0.10107415467500687} +{"event": "train", "step": 5250, "lr": 8.606012236719073e-05, "backbone_lr": 8.606012236719073e-06, "ema_decay": 0.998379, "it_s": 1.0, "data_frac": 0.8, "fm": 0.10106273412704468, "loss": 0.10106273412704468} +{"event": "train", "step": 5275, "lr": 8.592032655569429e-05, "backbone_lr": 8.59203265556943e-06, "ema_decay": 0.998385, "it_s": 0.68, "data_frac": 0.864, "fm": 0.09865057528018951, "loss": 0.09865057528018951} +{"event": "train", "step": 5300, "lr": 8.577994803720606e-05, "backbone_lr": 8.577994803720605e-06, "ema_decay": 0.99839, "it_s": 1.64, "data_frac": 0.684, "fm": 0.09809650510549545, "loss": 0.09809650510549545} +{"event": "train", "step": 5325, "lr": 8.56389890889755e-05, "backbone_lr": 8.56389890889755e-06, "ema_decay": 0.998396, "it_s": 0.84, "data_frac": 0.841, "fm": 0.09978836447000504, "loss": 0.09978836447000504} +{"event": "train", "step": 5350, "lr": 8.549745199766792e-05, "backbone_lr": 8.549745199766792e-06, "ema_decay": 0.998402, "it_s": 2.81, "data_frac": 0.437, "fm": 0.09904367864131927, "loss": 0.09904367864131927} +{"event": "train", "step": 5375, "lr": 8.535533905932738e-05, "backbone_lr": 8.535533905932739e-06, "ema_decay": 0.998407, "it_s": 1.06, "data_frac": 0.779, "fm": 0.10063733458518982, "loss": 0.10063733458518982} +{"event": "train", "step": 5400, "lr": 8.521265257933948e-05, "backbone_lr": 8.521265257933948e-06, "ema_decay": 0.998413, "it_s": 0.74, "data_frac": 0.857, "fm": 0.09784005373716355, "loss": 0.09784005373716355} +{"event": "train", "step": 5425, "lr": 8.506939487239388e-05, "backbone_lr": 8.506939487239389e-06, "ema_decay": 0.998418, "it_s": 1.83, "data_frac": 0.655, "fm": 0.09449451565742492, "loss": 0.09449451565742492} +{"event": "train", "step": 5450, "lr": 8.492556826244687e-05, "backbone_lr": 8.492556826244687e-06, "ema_decay": 0.998424, "it_s": 0.66, "data_frac": 0.874, "fm": 0.10104174077510834, "loss": 0.10104174077510834} +{"event": "train", "step": 5475, "lr": 8.47811750826835e-05, "backbone_lr": 8.478117508268352e-06, "ema_decay": 0.998429, "it_s": 1.01, "data_frac": 0.791, "fm": 0.09966644018888474, "loss": 0.09966644018888474} +{"event": "train", "step": 5500, "lr": 8.463621767547998e-05, "backbone_lr": 8.463621767547998e-06, "ema_decay": 0.998434, "it_s": 3.1, "data_frac": 0.411, "fm": 0.09821988344192505, "loss": 0.09821988344192505} +{"event": "train", "step": 5525, "lr": 8.449069839236538e-05, "backbone_lr": 8.44906983923654e-06, "ema_decay": 0.99844, "it_s": 1.03, "data_frac": 0.793, "fm": 0.09577731519937516, "loss": 0.09577731519937516} +{"event": "train", "step": 5550, "lr": 8.434461959398376e-05, "backbone_lr": 8.434461959398377e-06, "ema_decay": 0.998445, "it_s": 1.66, "data_frac": 0.663, "fm": 0.09619454890489579, "loss": 0.09619454890489579} +{"event": "train", "step": 5575, "lr": 8.419798365005569e-05, "backbone_lr": 8.419798365005569e-06, "ema_decay": 0.99845, "it_s": 0.62, "data_frac": 0.875, "fm": 0.10012628436088562, "loss": 0.10012628436088562} +{"event": "train", "step": 5600, "lr": 8.405079293933986e-05, "backbone_lr": 8.405079293933986e-06, "ema_decay": 0.998455, "it_s": 0.99, "data_frac": 0.795, "fm": 0.09661119401454926, "loss": 0.09661119401454926} +{"event": "train", "step": 5625, "lr": 8.390304984959454e-05, "backbone_lr": 8.390304984959455e-06, "ema_decay": 0.998461, "it_s": 0.99, "data_frac": 0.808, "fm": 0.09651722699403763, "loss": 0.09651722699403763} +{"event": "train", "step": 5650, "lr": 8.375475677753881e-05, "backbone_lr": 8.375475677753882e-06, "ema_decay": 0.998466, "it_s": 4.8, "data_frac": 0.002, "fm": 0.09850124537944793, "loss": 0.09850124537944793} +{"event": "train", "step": 5675, "lr": 8.360591612881364e-05, "backbone_lr": 8.360591612881363e-06, "ema_decay": 0.998471, "it_s": 1.13, "data_frac": 0.78, "fm": 0.09692844301462174, "loss": 0.09692844301462174} +{"event": "train", "step": 5700, "lr": 8.345653031794292e-05, "backbone_lr": 8.345653031794292e-06, "ema_decay": 0.998476, "it_s": 0.67, "data_frac": 0.868, "fm": 0.09970933407545089, "loss": 0.09970933407545089} +{"event": "train", "step": 5725, "lr": 8.330660176829428e-05, "backbone_lr": 8.330660176829427e-06, "ema_decay": 0.998481, "it_s": 1.31, "data_frac": 0.75, "fm": 0.09610348284244537, "loss": 0.09610348284244537} +{"event": "train", "step": 5750, "lr": 8.315613291203976e-05, "backbone_lr": 8.315613291203977e-06, "ema_decay": 0.998486, "it_s": 0.98, "data_frac": 0.813, "fm": 0.09469598233699798, "loss": 0.09469598233699798} +{"event": "train", "step": 5775, "lr": 8.30051261901164e-05, "backbone_lr": 8.30051261901164e-06, "ema_decay": 0.998491, "it_s": 1.14, "data_frac": 0.762, "fm": 0.09305141806602478, "loss": 0.09305141806602478} +{"event": "train", "step": 5800, "lr": 8.285358405218655e-05, "backbone_lr": 8.285358405218655e-06, "ema_decay": 0.998496, "it_s": 2.0, "data_frac": 0.608, "fm": 0.09563789576292038, "loss": 0.09563789576292038} +{"event": "train", "step": 5825, "lr": 8.270150895659823e-05, "backbone_lr": 8.270150895659824e-06, "ema_decay": 0.9985, "it_s": 0.68, "data_frac": 0.869, "fm": 0.0947405183315277, "loss": 0.0947405183315277} +{"event": "train", "step": 5850, "lr": 8.25489033703452e-05, "backbone_lr": 8.25489033703452e-06, "ema_decay": 0.998505, "it_s": 1.85, "data_frac": 0.658, "fm": 0.09316308856010437, "loss": 0.09316308856010437} +{"event": "train", "step": 5875, "lr": 8.239576976902695e-05, "backbone_lr": 8.239576976902694e-06, "ema_decay": 0.99851, "it_s": 0.69, "data_frac": 0.856, "fm": 0.0948196342587471, "loss": 0.0948196342587471} +{"event": "train", "step": 5900, "lr": 8.224211063680853e-05, "backbone_lr": 8.224211063680854e-06, "ema_decay": 0.998515, "it_s": 1.04, "data_frac": 0.785, "fm": 0.09499820053577424, "loss": 0.09499820053577424} +{"event": "train", "step": 5925, "lr": 8.208792846638029e-05, "backbone_lr": 8.20879284663803e-06, "ema_decay": 0.998519, "it_s": 0.76, "data_frac": 0.854, "fm": 0.0940330410003662, "loss": 0.0940330410003662} +{"event": "train", "step": 5950, "lr": 8.19332257589174e-05, "backbone_lr": 8.19332257589174e-06, "ema_decay": 0.998524, "it_s": 5.41, "data_frac": 0.001, "fm": 0.09308128386735916, "loss": 0.09308128386735916} +{"event": "train", "step": 5975, "lr": 8.177800502403928e-05, "backbone_lr": 8.177800502403928e-06, "ema_decay": 0.998529, "it_s": 1.89, "data_frac": 0.615, "fm": 0.09462020754814147, "loss": 0.09462020754814147} +{"event": "train", "step": 6000, "lr": 8.162226877976887e-05, "backbone_lr": 8.162226877976886e-06, "ema_decay": 0.998533, "it_s": 0.63, "data_frac": 0.876, "fm": 0.0966397100687027, "loss": 0.0966397100687027} +{"event": "save", "step": 6000, "checkpoint": "/mnt/data0/zuo/Humanoid_benchmark/results/fmdit_continue_20261117_01/runs/b5/fm_dit/checkpoints/step-006000", "mse": 0.030160661786794662, "mse_row0": 0.02748788893222809, "mse_motion_token": 0.027840631082654, "mse_inspire_left_hand_state": 0.04214290902018547, "mse_inspire_right_hand_state": 0.04292541742324829} +{"event": "train", "step": 6025, "lr": 8.146601955249188e-05, "backbone_lr": 8.146601955249187e-06, "ema_decay": 0.998538, "it_s": 0.93, "data_frac": 0.653, "fm": 0.09339529901742935, "loss": 0.09339529901742935} +{"event": "train", "step": 6050, "lr": 8.130925987691569e-05, "backbone_lr": 8.13092598769157e-06, "ema_decay": 0.998542, "it_s": 1.1, "data_frac": 0.796, "fm": 0.09227325141429901, "loss": 0.09227325141429901} +{"event": "train", "step": 6075, "lr": 8.11519922960283e-05, "backbone_lr": 8.115199229602831e-06, "ema_decay": 0.998547, "it_s": 1.59, "data_frac": 0.702, "fm": 0.09111574530601502, "loss": 0.09111574530601502} +{"event": "train", "step": 6100, "lr": 8.099421936105702e-05, "backbone_lr": 8.099421936105702e-06, "ema_decay": 0.998551, "it_s": 2.78, "data_frac": 0.464, "fm": 0.09019713640213013, "loss": 0.09019713640213013} +{"event": "train", "step": 6125, "lr": 8.083594363142717e-05, "backbone_lr": 8.083594363142717e-06, "ema_decay": 0.998556, "it_s": 0.63, "data_frac": 0.869, "fm": 0.09284347414970398, "loss": 0.09284347414970398} +{"event": "train", "step": 6150, "lr": 8.067716767472045e-05, "backbone_lr": 8.067716767472045e-06, "ema_decay": 0.99856, "it_s": 1.11, "data_frac": 0.787, "fm": 0.09122987002134324, "loss": 0.09122987002134324} +{"event": "train", "step": 6175, "lr": 8.051789406663342e-05, "backbone_lr": 8.051789406663342e-06, "ema_decay": 0.998565, "it_s": 1.2, "data_frac": 0.77, "fm": 0.09065371483564377, "loss": 0.09065371483564377} +{"event": "train", "step": 6200, "lr": 8.035812539093557e-05, "backbone_lr": 8.035812539093557e-06, "ema_decay": 0.998569, "it_s": 0.87, "data_frac": 0.827, "fm": 0.09362118899822235, "loss": 0.09362118899822235} +{"event": "train", "step": 6225, "lr": 8.019786423942752e-05, "backbone_lr": 8.019786423942752e-06, "ema_decay": 0.998573, "it_s": 0.66, "data_frac": 0.871, "fm": 0.09150694876909256, "loss": 0.09150694876909256} +{"event": "train", "step": 6250, "lr": 8.003711321189895e-05, "backbone_lr": 8.003711321189895e-06, "ema_decay": 0.998578, "it_s": 4.4, "data_frac": 0.003, "fm": 0.09103299051523209, "loss": 0.09103299051523209} +{"event": "train", "step": 6275, "lr": 7.987587491608637e-05, "backbone_lr": 7.987587491608636e-06, "ema_decay": 0.998582, "it_s": 1.51, "data_frac": 0.722, "fm": 0.08997465133666992, "loss": 0.08997465133666992} +{"event": "train", "step": 6300, "lr": 7.971415196763088e-05, "backbone_lr": 7.971415196763088e-06, "ema_decay": 0.998586, "it_s": 0.88, "data_frac": 0.818, "fm": 0.08997664302587509, "loss": 0.08997664302587509} +{"event": "train", "step": 6325, "lr": 7.95519469900357e-05, "backbone_lr": 7.95519469900357e-06, "ema_decay": 0.99859, "it_s": 0.89, "data_frac": 0.823, "fm": 0.08965153753757477, "loss": 0.08965153753757477} +{"event": "train", "step": 6350, "lr": 7.938926261462366e-05, "backbone_lr": 7.938926261462366e-06, "ema_decay": 0.998594, "it_s": 0.67, "data_frac": 0.864, "fm": 0.09144886314868927, "loss": 0.09144886314868927} +{"event": "train", "step": 6375, "lr": 7.922610148049445e-05, "backbone_lr": 7.922610148049445e-06, "ema_decay": 0.998599, "it_s": 1.9, "data_frac": 0.642, "fm": 0.0890133461356163, "loss": 0.0890133461356163} +{"event": "train", "step": 6400, "lr": 7.906246623448183e-05, "backbone_lr": 7.906246623448184e-06, "ema_decay": 0.998603, "it_s": 5.5, "data_frac": 0.001, "fm": 0.09129035502672195, "loss": 0.09129035502672195} +{"event": "train", "step": 6425, "lr": 7.889835953111075e-05, "backbone_lr": 7.889835953111075e-06, "ema_decay": 0.998607, "it_s": 0.67, "data_frac": 0.861, "fm": 0.08976365983486176, "loss": 0.08976365983486176} +{"event": "train", "step": 6450, "lr": 7.873378403255419e-05, "backbone_lr": 7.87337840325542e-06, "ema_decay": 0.998611, "it_s": 0.99, "data_frac": 0.806, "fm": 0.08863012373447418, "loss": 0.08863012373447418} diff --git a/b5-chair-push-standing-start/b512-cont-20261117/logs/train_stdout.log b/b5-chair-push-standing-start/b512-cont-20261117/logs/train_stdout.log new file mode 100644 index 0000000000000000000000000000000000000000..eca6f889f58e76341fe9bd42d25c637e98e5c2c5 --- /dev/null +++ b/b5-chair-push-standing-start/b512-cont-20261117/logs/train_stdout.log @@ -0,0 +1,102 @@ +{"event": "preflight", "augmentation": "hb-aug-v2-20261109", "samples": 64, "state_dim": 44, "body_dims": 29, "images_changed": 64, "body_values_moved": 1.0, "max_body_offset_deg": 2.552, "actions_bit_identical": true, "non_body_state_bit_identical": true, "eval_items_deterministic": 8} +{"event": "resumed", "step": 4000} +{"event": "start", "policy": "fm_dit", "params": 22540684, "backbone_params": 11166912, "backbone_weights": "IMAGENET1K_V1", "backbone_norm": "frozen_bn", "train_frames": 30908, "steps": 20000, "from_step": 4000, "cameras": ["ego_view", "chest"], "hw": [168, 224]} +{"event": "train", "step": 4025, "lr": 9.215152885383474e-05, "backbone_lr": 9.215152885383473e-06, "ema_decay": 0.998021, "it_s": 0.61, "data_frac": 0.84, "fm": 0.30036773085594176, "loss": 0.30036773085594176} +{"event": "train", "step": 4050, "lr": 9.204286963963111e-05, "backbone_lr": 9.204286963963112e-06, "ema_decay": 0.998031, "it_s": 0.88, "data_frac": 0.823, "fm": 0.20012644588947295, "loss": 0.20012644588947295} +{"event": "train", "step": 4075, "lr": 9.193352839727121e-05, "backbone_lr": 9.193352839727122e-06, "ema_decay": 0.99804, "it_s": 0.79, "data_frac": 0.85, "fm": 0.16311768770217897, "loss": 0.16311768770217897} +{"event": "train", "step": 4100, "lr": 9.182350690051133e-05, "backbone_lr": 9.182350690051134e-06, "ema_decay": 0.998049, "it_s": 2.23, "data_frac": 0.587, "fm": 0.1437873661518097, "loss": 0.1437873661518097} +{"event": "train", "step": 4125, "lr": 9.171280693414307e-05, "backbone_lr": 9.171280693414307e-06, "ema_decay": 0.998058, "it_s": 0.59, "data_frac": 0.885, "fm": 0.13227831691503525, "loss": 0.13227831691503525} +{"event": "train", "step": 4150, "lr": 9.160143029396422e-05, "backbone_lr": 9.160143029396422e-06, "ema_decay": 0.998066, "it_s": 4.72, "data_frac": 0.002, "fm": 0.12756982773542405, "loss": 0.12756982773542405} +{"event": "train", "step": 4175, "lr": 9.148937878674976e-05, "backbone_lr": 9.148937878674975e-06, "ema_decay": 0.998075, "it_s": 0.92, "data_frac": 0.826, "fm": 0.12656888127326965, "loss": 0.12656888127326965} +{"event": "train", "step": 4200, "lr": 9.13766542302225e-05, "backbone_lr": 9.13766542302225e-06, "ema_decay": 0.998084, "it_s": 1.02, "data_frac": 0.794, "fm": 0.1255243808031082, "loss": 0.1255243808031082} +{"event": "train", "step": 4225, "lr": 9.126325845302357e-05, "backbone_lr": 9.126325845302357e-06, "ema_decay": 0.998092, "it_s": 1.34, "data_frac": 0.73, "fm": 0.12075435340404511, "loss": 0.12075435340404511} +{"event": "train", "step": 4250, "lr": 9.114919329468282e-05, "backbone_lr": 9.114919329468283e-06, "ema_decay": 0.998101, "it_s": 0.6, "data_frac": 0.874, "fm": 0.11895803481340408, "loss": 0.11895803481340408} +{"event": "train", "step": 4275, "lr": 9.103446060558892e-05, "backbone_lr": 9.103446060558892e-06, "ema_decay": 0.998109, "it_s": 1.11, "data_frac": 0.774, "fm": 0.11751056581735611, "loss": 0.11751056581735611} +{"event": "train", "step": 4300, "lr": 9.091906224695935e-05, "backbone_lr": 9.091906224695935e-06, "ema_decay": 0.998117, "it_s": 5.33, "data_frac": 0.001, "fm": 0.11636869221925736, "loss": 0.11636869221925736} +{"event": "train", "step": 4325, "lr": 9.080300009081024e-05, "backbone_lr": 9.080300009081025e-06, "ema_decay": 0.998125, "it_s": 0.94, "data_frac": 0.801, "fm": 0.11493111044168472, "loss": 0.11493111044168472} +{"event": "train", "step": 4350, "lr": 9.068627601992598e-05, "backbone_lr": 9.0686276019926e-06, "ema_decay": 0.998133, "it_s": 0.92, "data_frac": 0.808, "fm": 0.11559643805027008, "loss": 0.11559643805027008} +{"event": "train", "step": 4375, "lr": 9.056889192782866e-05, "backbone_lr": 9.056889192782865e-06, "ema_decay": 0.998141, "it_s": 0.7, "data_frac": 0.861, "fm": 0.11376965045928955, "loss": 0.11376965045928955} +{"event": "train", "step": 4400, "lr": 9.045084971874738e-05, "backbone_lr": 9.045084971874738e-06, "ema_decay": 0.998149, "it_s": 1.79, "data_frac": 0.654, "fm": 0.1141718265414238, "loss": 0.1141718265414238} +{"event": "train", "step": 4425, "lr": 9.033215130758735e-05, "backbone_lr": 9.033215130758735e-06, "ema_decay": 0.998157, "it_s": 0.93, "data_frac": 0.812, "fm": 0.11762478679418564, "loss": 0.11762478679418564} +{"event": "train", "step": 4450, "lr": 9.021279861989885e-05, "backbone_lr": 9.021279861989884e-06, "ema_decay": 0.998165, "it_s": 2.38, "data_frac": 0.498, "fm": 0.1135200247168541, "loss": 0.1135200247168541} +{"event": "train", "step": 4475, "lr": 9.009279359184593e-05, "backbone_lr": 9.009279359184594e-06, "ema_decay": 0.998173, "it_s": 0.93, "data_frac": 0.81, "fm": 0.11074629306793213, "loss": 0.11074629306793213} +{"event": "train", "step": 4500, "lr": 8.997213817017507e-05, "backbone_lr": 8.997213817017508e-06, "ema_decay": 0.99818, "it_s": 0.76, "data_frac": 0.837, "fm": 0.11229030340909958, "loss": 0.11229030340909958} +{"event": "train", "step": 4525, "lr": 8.985083431218354e-05, "backbone_lr": 8.985083431218354e-06, "ema_decay": 0.998188, "it_s": 1.22, "data_frac": 0.739, "fm": 0.10931755155324936, "loss": 0.10931755155324936} +{"event": "train", "step": 4550, "lr": 8.972888398568772e-05, "backbone_lr": 8.972888398568772e-06, "ema_decay": 0.998195, "it_s": 0.72, "data_frac": 0.862, "fm": 0.11274766504764556, "loss": 0.11274766504764556} +{"event": "train", "step": 4575, "lr": 8.96062891689911e-05, "backbone_lr": 8.96062891689911e-06, "ema_decay": 0.998203, "it_s": 1.19, "data_frac": 0.766, "fm": 0.11252273738384247, "loss": 0.11252273738384247} +{"event": "train", "step": 4600, "lr": 8.948305185085225e-05, "backbone_lr": 8.948305185085226e-06, "ema_decay": 0.99821, "it_s": 5.44, "data_frac": 0.001, "fm": 0.11177674174308777, "loss": 0.11177674174308777} +{"event": "train", "step": 4625, "lr": 8.935917403045251e-05, "backbone_lr": 8.935917403045251e-06, "ema_decay": 0.998217, "it_s": 0.69, "data_frac": 0.864, "fm": 0.10977381765842438, "loss": 0.10977381765842438} +{"event": "train", "step": 4650, "lr": 8.92346577173636e-05, "backbone_lr": 8.923465771736361e-06, "ema_decay": 0.998224, "it_s": 1.39, "data_frac": 0.739, "fm": 0.11017541021108627, "loss": 0.11017541021108627} +{"event": "train", "step": 4675, "lr": 8.910950493151498e-05, "backbone_lr": 8.910950493151498e-06, "ema_decay": 0.998232, "it_s": 0.71, "data_frac": 0.861, "fm": 0.11157936811447143, "loss": 0.11157936811447143} +{"event": "train", "step": 4700, "lr": 8.898371770316111e-05, "backbone_lr": 8.898371770316113e-06, "ema_decay": 0.998239, "it_s": 1.64, "data_frac": 0.685, "fm": 0.11001256704330445, "loss": 0.11001256704330445} +{"event": "train", "step": 4725, "lr": 8.885729807284856e-05, "backbone_lr": 8.885729807284855e-06, "ema_decay": 0.998246, "it_s": 0.65, "data_frac": 0.874, "fm": 0.1063080707192421, "loss": 0.1063080707192421} +{"event": "train", "step": 4750, "lr": 8.873024809138272e-05, "backbone_lr": 8.873024809138272e-06, "ema_decay": 0.998253, "it_s": 5.07, "data_frac": 0.022, "fm": 0.10854193478822709, "loss": 0.10854193478822709} +{"event": "train", "step": 4775, "lr": 8.860256981979484e-05, "backbone_lr": 8.860256981979485e-06, "ema_decay": 0.998259, "it_s": 1.34, "data_frac": 0.739, "fm": 0.10757887542247772, "loss": 0.10757887542247772} +{"event": "train", "step": 4800, "lr": 8.847426532930831e-05, "backbone_lr": 8.84742653293083e-06, "ema_decay": 0.998266, "it_s": 0.84, "data_frac": 0.83, "fm": 0.10880939394235611, "loss": 0.10880939394235611} +{"event": "train", "step": 4825, "lr": 8.834533670130518e-05, "backbone_lr": 8.834533670130518e-06, "ema_decay": 0.998273, "it_s": 1.09, "data_frac": 0.783, "fm": 0.10648771852254868, "loss": 0.10648771852254868} +{"event": "train", "step": 4850, "lr": 8.821578602729242e-05, "backbone_lr": 8.821578602729242e-06, "ema_decay": 0.99828, "it_s": 0.71, "data_frac": 0.861, "fm": 0.10719650745391845, "loss": 0.10719650745391845} +{"event": "train", "step": 4875, "lr": 8.808561540886796e-05, "backbone_lr": 8.808561540886796e-06, "ema_decay": 0.998286, "it_s": 1.78, "data_frac": 0.662, "fm": 0.10717803657054902, "loss": 0.10717803657054902} +{"event": "train", "step": 4900, "lr": 8.795482695768658e-05, "backbone_lr": 8.795482695768658e-06, "ema_decay": 0.998293, "it_s": 3.88, "data_frac": 0.275, "fm": 0.10350908666849136, "loss": 0.10350908666849136} +{"event": "train", "step": 4925, "lr": 8.782342279542568e-05, "backbone_lr": 8.782342279542569e-06, "ema_decay": 0.998299, "it_s": 0.88, "data_frac": 0.808, "fm": 0.10498440027236938, "loss": 0.10498440027236938} +{"event": "train", "step": 4950, "lr": 8.769140505375085e-05, "backbone_lr": 8.769140505375084e-06, "ema_decay": 0.998306, "it_s": 0.96, "data_frac": 0.813, "fm": 0.1064383864402771, "loss": 0.1064383864402771} +{"event": "train", "step": 4975, "lr": 8.755877587428128e-05, "backbone_lr": 8.755877587428127e-06, "ema_decay": 0.998312, "it_s": 0.85, "data_frac": 0.837, "fm": 0.10203986793756485, "loss": 0.10203986793756485} +{"event": "train", "step": 5000, "lr": 8.742553740855506e-05, "backbone_lr": 8.742553740855507e-06, "ema_decay": 0.998318, "it_s": 1.69, "data_frac": 0.668, "fm": 0.10572098553180695, "loss": 0.10572098553180695} +{"event": "train", "step": 5025, "lr": 8.729169181799422e-05, "backbone_lr": 8.729169181799423e-06, "ema_decay": 0.998325, "it_s": 0.6, "data_frac": 0.885, "fm": 0.10311138957738876, "loss": 0.10311138957738876} +{"event": "train", "step": 5050, "lr": 8.715724127386972e-05, "backbone_lr": 8.715724127386971e-06, "ema_decay": 0.998331, "it_s": 4.75, "data_frac": 0.002, "fm": 0.10220446288585663, "loss": 0.10220446288585663} +{"event": "train", "step": 5075, "lr": 8.702218795726617e-05, "backbone_lr": 8.702218795726619e-06, "ema_decay": 0.998337, "it_s": 1.02, "data_frac": 0.807, "fm": 0.10448342114686966, "loss": 0.10448342114686966} +{"event": "train", "step": 5100, "lr": 8.688653405904652e-05, "backbone_lr": 8.688653405904653e-06, "ema_decay": 0.998343, "it_s": 0.97, "data_frac": 0.793, "fm": 0.1041489601135254, "loss": 0.1041489601135254} +{"event": "train", "step": 5125, "lr": 8.675028177981643e-05, "backbone_lr": 8.675028177981643e-06, "ema_decay": 0.998349, "it_s": 1.31, "data_frac": 0.727, "fm": 0.10290780544281006, "loss": 0.10290780544281006} +{"event": "train", "step": 5150, "lr": 8.661343332988869e-05, "backbone_lr": 8.661343332988869e-06, "ema_decay": 0.998355, "it_s": 0.64, "data_frac": 0.874, "fm": 0.10130853086709976, "loss": 0.10130853086709976} +{"event": "train", "step": 5175, "lr": 8.647599092924721e-05, "backbone_lr": 8.64759909292472e-06, "ema_decay": 0.998361, "it_s": 1.04, "data_frac": 0.796, "fm": 0.1019318300485611, "loss": 0.1019318300485611} +{"event": "train", "step": 5200, "lr": 8.633795680751116e-05, "backbone_lr": 8.633795680751116e-06, "ema_decay": 0.998367, "it_s": 5.48, "data_frac": 0.001, "fm": 0.10094406306743622, "loss": 0.10094406306743622} +{"event": "train", "step": 5225, "lr": 8.619933320389872e-05, "backbone_lr": 8.619933320389872e-06, "ema_decay": 0.998373, "it_s": 0.99, "data_frac": 0.807, "fm": 0.10107415467500687, "loss": 0.10107415467500687} +{"event": "train", "step": 5250, "lr": 8.606012236719073e-05, "backbone_lr": 8.606012236719073e-06, "ema_decay": 0.998379, "it_s": 1.0, "data_frac": 0.8, "fm": 0.10106273412704468, "loss": 0.10106273412704468} +{"event": "train", "step": 5275, "lr": 8.592032655569429e-05, "backbone_lr": 8.59203265556943e-06, "ema_decay": 0.998385, "it_s": 0.68, "data_frac": 0.864, "fm": 0.09865057528018951, "loss": 0.09865057528018951} +{"event": "train", "step": 5300, "lr": 8.577994803720606e-05, "backbone_lr": 8.577994803720605e-06, "ema_decay": 0.99839, "it_s": 1.64, "data_frac": 0.684, "fm": 0.09809650510549545, "loss": 0.09809650510549545} +{"event": "train", "step": 5325, "lr": 8.56389890889755e-05, "backbone_lr": 8.56389890889755e-06, "ema_decay": 0.998396, "it_s": 0.84, "data_frac": 0.841, "fm": 0.09978836447000504, "loss": 0.09978836447000504} +{"event": "train", "step": 5350, "lr": 8.549745199766792e-05, "backbone_lr": 8.549745199766792e-06, "ema_decay": 0.998402, "it_s": 2.81, "data_frac": 0.437, "fm": 0.09904367864131927, "loss": 0.09904367864131927} +{"event": "train", "step": 5375, "lr": 8.535533905932738e-05, "backbone_lr": 8.535533905932739e-06, "ema_decay": 0.998407, "it_s": 1.06, "data_frac": 0.779, "fm": 0.10063733458518982, "loss": 0.10063733458518982} +{"event": "train", "step": 5400, "lr": 8.521265257933948e-05, "backbone_lr": 8.521265257933948e-06, "ema_decay": 0.998413, "it_s": 0.74, "data_frac": 0.857, "fm": 0.09784005373716355, "loss": 0.09784005373716355} +{"event": "train", "step": 5425, "lr": 8.506939487239388e-05, "backbone_lr": 8.506939487239389e-06, "ema_decay": 0.998418, "it_s": 1.83, "data_frac": 0.655, "fm": 0.09449451565742492, "loss": 0.09449451565742492} +{"event": "train", "step": 5450, "lr": 8.492556826244687e-05, "backbone_lr": 8.492556826244687e-06, "ema_decay": 0.998424, "it_s": 0.66, "data_frac": 0.874, "fm": 0.10104174077510834, "loss": 0.10104174077510834} +{"event": "train", "step": 5475, "lr": 8.47811750826835e-05, "backbone_lr": 8.478117508268352e-06, "ema_decay": 0.998429, "it_s": 1.01, "data_frac": 0.791, "fm": 0.09966644018888474, "loss": 0.09966644018888474} +{"event": "train", "step": 5500, "lr": 8.463621767547998e-05, "backbone_lr": 8.463621767547998e-06, "ema_decay": 0.998434, "it_s": 3.1, "data_frac": 0.411, "fm": 0.09821988344192505, "loss": 0.09821988344192505} +{"event": "train", "step": 5525, "lr": 8.449069839236538e-05, "backbone_lr": 8.44906983923654e-06, "ema_decay": 0.99844, "it_s": 1.03, "data_frac": 0.793, "fm": 0.09577731519937516, "loss": 0.09577731519937516} +{"event": "train", "step": 5550, "lr": 8.434461959398376e-05, "backbone_lr": 8.434461959398377e-06, "ema_decay": 0.998445, "it_s": 1.66, "data_frac": 0.663, "fm": 0.09619454890489579, "loss": 0.09619454890489579} +{"event": "train", "step": 5575, "lr": 8.419798365005569e-05, "backbone_lr": 8.419798365005569e-06, "ema_decay": 0.99845, "it_s": 0.62, "data_frac": 0.875, "fm": 0.10012628436088562, "loss": 0.10012628436088562} +{"event": "train", "step": 5600, "lr": 8.405079293933986e-05, "backbone_lr": 8.405079293933986e-06, "ema_decay": 0.998455, "it_s": 0.99, "data_frac": 0.795, "fm": 0.09661119401454926, "loss": 0.09661119401454926} +{"event": "train", "step": 5625, "lr": 8.390304984959454e-05, "backbone_lr": 8.390304984959455e-06, "ema_decay": 0.998461, "it_s": 0.99, "data_frac": 0.808, "fm": 0.09651722699403763, "loss": 0.09651722699403763} +{"event": "train", "step": 5650, "lr": 8.375475677753881e-05, "backbone_lr": 8.375475677753882e-06, "ema_decay": 0.998466, "it_s": 4.8, "data_frac": 0.002, "fm": 0.09850124537944793, "loss": 0.09850124537944793} +{"event": "train", "step": 5675, "lr": 8.360591612881364e-05, "backbone_lr": 8.360591612881363e-06, "ema_decay": 0.998471, "it_s": 1.13, "data_frac": 0.78, "fm": 0.09692844301462174, "loss": 0.09692844301462174} +{"event": "train", "step": 5700, "lr": 8.345653031794292e-05, "backbone_lr": 8.345653031794292e-06, "ema_decay": 0.998476, "it_s": 0.67, "data_frac": 0.868, "fm": 0.09970933407545089, "loss": 0.09970933407545089} +{"event": "train", "step": 5725, "lr": 8.330660176829428e-05, "backbone_lr": 8.330660176829427e-06, "ema_decay": 0.998481, "it_s": 1.31, "data_frac": 0.75, "fm": 0.09610348284244537, "loss": 0.09610348284244537} +{"event": "train", "step": 5750, "lr": 8.315613291203976e-05, "backbone_lr": 8.315613291203977e-06, "ema_decay": 0.998486, "it_s": 0.98, "data_frac": 0.813, "fm": 0.09469598233699798, "loss": 0.09469598233699798} +{"event": "train", "step": 5775, "lr": 8.30051261901164e-05, "backbone_lr": 8.30051261901164e-06, "ema_decay": 0.998491, "it_s": 1.14, "data_frac": 0.762, "fm": 0.09305141806602478, "loss": 0.09305141806602478} +{"event": "train", "step": 5800, "lr": 8.285358405218655e-05, "backbone_lr": 8.285358405218655e-06, "ema_decay": 0.998496, "it_s": 2.0, "data_frac": 0.608, "fm": 0.09563789576292038, "loss": 0.09563789576292038} +{"event": "train", "step": 5825, "lr": 8.270150895659823e-05, "backbone_lr": 8.270150895659824e-06, "ema_decay": 0.9985, "it_s": 0.68, "data_frac": 0.869, "fm": 0.0947405183315277, "loss": 0.0947405183315277} +{"event": "train", "step": 5850, "lr": 8.25489033703452e-05, "backbone_lr": 8.25489033703452e-06, "ema_decay": 0.998505, "it_s": 1.85, "data_frac": 0.658, "fm": 0.09316308856010437, "loss": 0.09316308856010437} +{"event": "train", "step": 5875, "lr": 8.239576976902695e-05, "backbone_lr": 8.239576976902694e-06, "ema_decay": 0.99851, "it_s": 0.69, "data_frac": 0.856, "fm": 0.0948196342587471, "loss": 0.0948196342587471} +{"event": "train", "step": 5900, "lr": 8.224211063680853e-05, "backbone_lr": 8.224211063680854e-06, "ema_decay": 0.998515, "it_s": 1.04, "data_frac": 0.785, "fm": 0.09499820053577424, "loss": 0.09499820053577424} +{"event": "train", "step": 5925, "lr": 8.208792846638029e-05, "backbone_lr": 8.20879284663803e-06, "ema_decay": 0.998519, "it_s": 0.76, "data_frac": 0.854, "fm": 0.0940330410003662, "loss": 0.0940330410003662} +{"event": "train", "step": 5950, "lr": 8.19332257589174e-05, "backbone_lr": 8.19332257589174e-06, "ema_decay": 0.998524, "it_s": 5.41, "data_frac": 0.001, "fm": 0.09308128386735916, "loss": 0.09308128386735916} +{"event": "train", "step": 5975, "lr": 8.177800502403928e-05, "backbone_lr": 8.177800502403928e-06, "ema_decay": 0.998529, "it_s": 1.89, "data_frac": 0.615, "fm": 0.09462020754814147, "loss": 0.09462020754814147} +{"event": "train", "step": 6000, "lr": 8.162226877976887e-05, "backbone_lr": 8.162226877976886e-06, "ema_decay": 0.998533, "it_s": 0.63, "data_frac": 0.876, "fm": 0.0966397100687027, "loss": 0.0966397100687027} +{"event": "save", "step": 6000, "checkpoint": "/mnt/data0/zuo/Humanoid_benchmark/results/fmdit_continue_20261117_01/runs/b5/fm_dit/checkpoints/step-006000", "mse": 0.030160661786794662, "mse_row0": 0.02748788893222809, "mse_motion_token": 0.027840631082654, "mse_inspire_left_hand_state": 0.04214290902018547, "mse_inspire_right_hand_state": 0.04292541742324829} +{"event": "train", "step": 6025, "lr": 8.146601955249188e-05, "backbone_lr": 8.146601955249187e-06, "ema_decay": 0.998538, "it_s": 0.93, "data_frac": 0.653, "fm": 0.09339529901742935, "loss": 0.09339529901742935} +{"event": "train", "step": 6050, "lr": 8.130925987691569e-05, "backbone_lr": 8.13092598769157e-06, "ema_decay": 0.998542, "it_s": 1.1, "data_frac": 0.796, "fm": 0.09227325141429901, "loss": 0.09227325141429901} +{"event": "train", "step": 6075, "lr": 8.11519922960283e-05, "backbone_lr": 8.115199229602831e-06, "ema_decay": 0.998547, "it_s": 1.59, "data_frac": 0.702, "fm": 0.09111574530601502, "loss": 0.09111574530601502} +{"event": "train", "step": 6100, "lr": 8.099421936105702e-05, "backbone_lr": 8.099421936105702e-06, "ema_decay": 0.998551, "it_s": 2.78, "data_frac": 0.464, "fm": 0.09019713640213013, "loss": 0.09019713640213013} +{"event": "train", "step": 6125, "lr": 8.083594363142717e-05, "backbone_lr": 8.083594363142717e-06, "ema_decay": 0.998556, "it_s": 0.63, "data_frac": 0.869, "fm": 0.09284347414970398, "loss": 0.09284347414970398} +{"event": "train", "step": 6150, "lr": 8.067716767472045e-05, "backbone_lr": 8.067716767472045e-06, "ema_decay": 0.99856, "it_s": 1.11, "data_frac": 0.787, "fm": 0.09122987002134324, "loss": 0.09122987002134324} +{"event": "train", "step": 6175, "lr": 8.051789406663342e-05, "backbone_lr": 8.051789406663342e-06, "ema_decay": 0.998565, "it_s": 1.2, "data_frac": 0.77, "fm": 0.09065371483564377, "loss": 0.09065371483564377} +{"event": "train", "step": 6200, "lr": 8.035812539093557e-05, "backbone_lr": 8.035812539093557e-06, "ema_decay": 0.998569, "it_s": 0.87, "data_frac": 0.827, "fm": 0.09362118899822235, "loss": 0.09362118899822235} +{"event": "train", "step": 6225, "lr": 8.019786423942752e-05, "backbone_lr": 8.019786423942752e-06, "ema_decay": 0.998573, "it_s": 0.66, "data_frac": 0.871, "fm": 0.09150694876909256, "loss": 0.09150694876909256} +{"event": "train", "step": 6250, "lr": 8.003711321189895e-05, "backbone_lr": 8.003711321189895e-06, "ema_decay": 0.998578, "it_s": 4.4, "data_frac": 0.003, "fm": 0.09103299051523209, "loss": 0.09103299051523209} +{"event": "train", "step": 6275, "lr": 7.987587491608637e-05, "backbone_lr": 7.987587491608636e-06, "ema_decay": 0.998582, "it_s": 1.51, "data_frac": 0.722, "fm": 0.08997465133666992, "loss": 0.08997465133666992} +{"event": "train", "step": 6300, "lr": 7.971415196763088e-05, "backbone_lr": 7.971415196763088e-06, "ema_decay": 0.998586, "it_s": 0.88, "data_frac": 0.818, "fm": 0.08997664302587509, "loss": 0.08997664302587509} +{"event": "train", "step": 6325, "lr": 7.95519469900357e-05, "backbone_lr": 7.95519469900357e-06, "ema_decay": 0.99859, "it_s": 0.89, "data_frac": 0.823, "fm": 0.08965153753757477, "loss": 0.08965153753757477} +{"event": "train", "step": 6350, "lr": 7.938926261462366e-05, "backbone_lr": 7.938926261462366e-06, "ema_decay": 0.998594, "it_s": 0.67, "data_frac": 0.864, "fm": 0.09144886314868927, "loss": 0.09144886314868927} +{"event": "train", "step": 6375, "lr": 7.922610148049445e-05, "backbone_lr": 7.922610148049445e-06, "ema_decay": 0.998599, "it_s": 1.9, "data_frac": 0.642, "fm": 0.0890133461356163, "loss": 0.0890133461356163} +{"event": "train", "step": 6400, "lr": 7.906246623448183e-05, "backbone_lr": 7.906246623448184e-06, "ema_decay": 0.998603, "it_s": 5.5, "data_frac": 0.001, "fm": 0.09129035502672195, "loss": 0.09129035502672195} +{"event": "train", "step": 6425, "lr": 7.889835953111075e-05, "backbone_lr": 7.889835953111075e-06, "ema_decay": 0.998607, "it_s": 0.67, "data_frac": 0.861, "fm": 0.08976365983486176, "loss": 0.08976365983486176} +{"event": "train", "step": 6450, "lr": 7.873378403255419e-05, "backbone_lr": 7.87337840325542e-06, "ema_decay": 0.998611, "it_s": 0.99, "data_frac": 0.806, "fm": 0.08863012373447418, "loss": 0.08863012373447418} diff --git a/b5-chair-push-standing-start/b512-cont-20261117/provenance/parent_checkpoint_config.json b/b5-chair-push-standing-start/b512-cont-20261117/provenance/parent_checkpoint_config.json new file mode 100644 index 0000000000000000000000000000000000000000..e4d1b5badac94e87748a559792f1cef23bc467e4 --- /dev/null +++ b/b5-chair-push-standing-start/b512-cont-20261117/provenance/parent_checkpoint_config.json @@ -0,0 +1,884 @@ +{ + "policy": { + "policy": "fm_dit", + "n_cameras": 2, + "state_dim": 44, + "action_dim": 76, + "horizon": 16, + "backbone": "resnet18", + "backbone_weights": "IMAGENET1K_V1", + "backbone_norm": "frozen_bn", + "hidden_dim": 384, + "num_layers": 4, + "num_heads": 8, + "mlp_ratio": 4.0, + "dropout": 0.1, + "time_embed_dim": 128, + "num_inference_steps": null, + "num_train_timesteps": 100, + "beta_schedule": "squaredcos_cap_v2", + "prediction_type": "epsilon", + "clip_sample": true, + "act_hidden_dim": 512, + "dim_feedforward": 3200, + "enc_layers": 4, + "dec_layers": 7, + "nheads": 8, + "latent_dim": 32, + "kl_weight": 10.0, + "extra": {} + }, + "contract": { + "name": "g1_sonic_dfx6", + "robot": "g1", + "cameras": [ + "ego_view", + "chest" + ], + "state": [ + [ + "observation.state", + 0, + 29 + ], + [ + "observation.inspire_left_hand_state", + 0, + 6 + ], + [ + "observation.inspire_right_hand_state", + 0, + 6 + ], + [ + "observation.projected_gravity", + 0, + 3 + ] + ], + "action": [ + [ + "action.motion_token", + 0, + 64 + ], + [ + "observation.inspire_left_hand_state", + 0, + 6 + ], + [ + "observation.inspire_right_hand_state", + 0, + 6 + ] + ], + "state_dim": 44, + "action_dim": 76, + "action_space": "sonic_dfx6", + "body_dims": 29 + }, + "image_hw": [ + 168, + 224 + ], + "task_text": "Establish and verify a stable two-hand grasp around the chair back, straighten it by stepping into position behind it, then walk forward to push it under the table.", + "dataset": { + "name": "b5-chair-push-standing-start-expert-v1", + "root": "/mnt/data0/zuo/Humanoid_benchmark/data/baselines/b5-chair-push-standing-start-expert-v1", + "robot_type": null, + "fps": 30.0, + "episodes": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 36, + 37, + 38, + 39, + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55, + 56, + 57, + 58, + 59, + 60, + 61, + 62, + 63, + 64, + 65, + 66, + 67, + 68, + 69, + 70, + 71, + 72, + 73, + 74, + 75, + 76, + 77, + 78, + 79, + 80, + 81, + 82, + 83, + 84, + 85, + 86, + 87, + 88, + 89, + 90, + 91, + 92, + 93, + 94, + 95, + 96, + 97, + 98, + 99 + ], + "train_episodes": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 36, + 37, + 38, + 39, + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55, + 56, + 57, + 58, + 59, + 60, + 61, + 62, + 63, + 64, + 65, + 66, + 67, + 68, + 69, + 70, + 71, + 72, + 73, + 74, + 75, + 76, + 77, + 78, + 79, + 80, + 81, + 82, + 83, + 84, + 85, + 86, + 87, + 88, + 89, + 90, + 91, + 92, + 93, + 94, + 95, + 96, + 97, + 98, + 99 + ], + "val_episodes": [], + "train_frames": 30908, + "git_commit": null + }, + "training": { + "steps": 4000, + "batch_size": 512, + "lr": 0.0001, + "betas": [ + 0.95, + 0.999 + ], + "weight_decay": 1e-06, + "warmup": 500, + "schedule": "linear warm-up + cosine", + "backbone_lr_scale": 0.1, + "ema": 0.9999, + "shift_px": 0, + "amp": true, + "seed": 0, + "augmentation": { + "spec": "docs/augmentation_v2.md", + "version": "hb-aug-v2-20261109", + "implementation": "evaluation/augmentation/hb_aug_v2.py", + "visual": { + "effects": [ + "exposure", + "contrast", + "saturation", + "hue", + "gamma", + "temperature", + "rotation_crop", + "noise", + "gaussian_blur", + "jpeg", + "motion_blur", + "sharpen", + "defocus", + "light_shadow", + "wb_channel" + ], + "boosted_per_sample": 1, + "boost": 1.5, + "pixel_scale": "H / 224", + "future_targets": "none (one observation frame)" + }, + "proprio": { + "robot": "g1", + "body_dims": 29, + "bias_deg": 2.0, + "bias_scope": "per_sample", + "noise_deg": 0.25, + "noise_clip_sigma": 3, + "delay_probability": 0.0, + "history_drop_probability": 0.0, + "velocity": false + }, + "native_image_augmentation": "off", + "evaluation": "off" + }, + "preflight": { + "samples": 64, + "state_dim": 44, + "body_dims": 29, + "images_changed": 64, + "body_values_moved": 1.0, + "max_body_offset_deg": 2.6143, + "actions_bit_identical": true, + "non_body_state_bit_identical": true, + "eval_items_deterministic": 8 + } + }, + "stats": { + "state": { + "min": [ + -0.5093753933906555, + -0.12233220040798187, + -0.12761788070201874, + 0.10250309854745865, + -0.8273318409919739, + -0.15657654404640198, + -0.4967050552368164, + -0.30181893706321716, + -0.5301669836044312, + 0.09314415603876114, + -0.8944554924964905, + -0.14240269362926483, + -0.10451218485832214, + -0.0940587967634201, + -0.13018359243869781, + -1.393661618232727, + -0.020927773788571358, + -0.7382517457008362, + -0.8502405285835266, + -0.11091316491365433, + -0.24967733025550842, + -1.2172361612319946, + -1.5036237239837646, + -1.1288223266601562, + -0.02860570326447487, + -0.8521270155906677, + -1.870940923690796, + -0.095971018075943, + -0.1065346822142601, + 0.0, + 0.5949721336364746, + 0.6192536950111389, + 0.6028932332992554, + 0.3511531352996826, + 0.0, + 0.0, + 0.5953419804573059, + 0.6113170981407166, + 0.5994111895561218, + 0.03766731545329094, + 0.0, + -0.3042502999305725, + -0.098258838057518, + -0.9999995827674866 + ], + "max": [ + 0.42436036467552185, + 0.2632223069667816, + 0.4142952561378479, + 1.3769621849060059, + 0.3906438946723938, + 0.12971684336662292, + 0.28290697932243347, + 0.07751978188753128, + 0.012389450334012508, + 1.309631109237671, + 0.5117528438568115, + 0.14354851841926575, + 0.09299838542938232, + 0.1345142424106598, + 0.16879205405712128, + 0.26465392112731934, + 1.0681228637695312, + 0.368639200925827, + 1.2249290943145752, + 1.7016332149505615, + 0.73288893699646, + 0.4079762399196625, + 0.20960678160190582, + 0.1851842999458313, + 0.9496553540229797, + 1.338996171951294, + 0.04667161777615547, + 1.0116424560546875, + 1.5066314935684204, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.9997431039810181, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.9997568130493164, + 0.011030972935259342, + 0.10935939103364944, + -0.9520710110664368 + ], + "mean": [ + -0.015364952224250912, + 0.081615613869982, + 0.12889738646888202, + 0.27149564947104204, + -0.14955734140047547, + -0.03703078041298108, + -0.0667806330627162, + -0.07314824380692604, + -0.2412484148070091, + 0.25491780809063996, + -0.10854783438820112, + -0.005161355253899878, + 0.003673423170905123, + 0.02969264617779002, + 0.08950587210737222, + -0.47956076188028, + 0.42631070073819, + -0.27129812784427826, + 0.16324256069371254, + 1.0385976806191235, + 0.32744283883844555, + -0.3669884986385585, + -0.5611831615193444, + -0.3750104856050489, + 0.3347049141703264, + 0.09427332727041168, + -1.2676208074144515, + 0.44498776552803626, + 0.6750011010472442, + 0.8714133721444814, + 0.8704363160829457, + 0.871101992679426, + 0.8709877289072792, + 0.9374116361295481, + 0.000909107080241183, + 0.8686004803123465, + 0.8727763422085179, + 0.8736698483416437, + 0.873315653691274, + 0.9243064344480587, + 0.0010054465065242088, + -0.07857978353459398, + 0.005790211846556424, + -0.9955768040204798 + ], + "std": [ + 0.12856331774418192, + 0.037393329668315815, + 0.05114975202975518, + 0.19438684278818802, + 0.12199680903864253, + 0.042366122817800854, + 0.1197195537504191, + 0.041234348615534516, + 0.06264541254527864, + 0.19296680414798664, + 0.14698603073664226, + 0.03873242198119102, + 0.022580757805711404, + 0.02236052295122295, + 0.03593445604445683, + 0.3256732586786403, + 0.19224309523346822, + 0.1334741564088957, + 0.48949046035954324, + 0.3411011488066223, + 0.17404862210424266, + 0.28167826002687124, + 0.32010586210868225, + 0.19978493123512664, + 0.12314692856716673, + 0.4856849744536303, + 0.4150175273557762, + 0.18968756344956073, + 0.36121237047109955, + 0.16123118351068702, + 0.16110483963491448, + 0.16022198436241275, + 0.16099862679020233, + 0.07811886944382597, + 0.019824963023862628, + 0.1676569977855643, + 0.15816317480687228, + 0.15654935499028427, + 0.15744207009759817, + 0.1022531519695512, + 0.019857930854210358, + 0.048533023773255274, + 0.015253372177349003, + 0.005513492459359003 + ] + }, + "action": { + "min": [ + -0.4375, + -0.375, + -0.5, + -0.4375, + -0.4375, + -0.1875, + -0.3125, + -0.4375, + -0.1875, + -0.5, + -0.5, + -0.25, + -0.625, + -0.125, + -0.3125, + -0.25, + -0.125, + -0.1875, + -0.5, + -0.3125, + -0.5625, + -0.5, + -0.0625, + -0.1875, + -0.5, + -0.4375, + -0.375, + -0.1875, + -0.4375, + -0.3125, + -0.3125, + -0.1875, + -0.0625, + -0.1875, + -0.5, + -0.5, + -0.125, + -0.5625, + -0.3125, + -0.25, + -0.375, + -0.5, + -0.5625, + -0.0625, + -0.5, + -0.625, + -0.6875, + -0.1875, + 0.0, + -0.375, + -0.5, + -0.3125, + -0.375, + -0.4375, + -0.5, + -0.25, + -0.6875, + -0.1875, + -0.3125, + -0.5, + -0.625, + -0.3125, + -0.4375, + -0.1875, + 0.0, + 0.5949721336364746, + 0.6192536950111389, + 0.6028932332992554, + 0.3511531352996826, + 0.0, + 0.0, + 0.5953419804573059, + 0.6113170981407166, + 0.5994111895561218, + 0.03766731545329094, + 0.0 + ], + "max": [ + 0.1875, + 0.5, + 0.375, + 0.25, + 0.4375, + 0.5625, + 0.3125, + 0.3125, + 0.5, + 0.25, + 0.125, + 0.3125, + 0.5, + 0.6875, + 0.4375, + 0.3125, + 0.4375, + 0.375, + 0.25, + 0.5, + 0.0625, + 0.1875, + 0.4375, + 0.5625, + 0.5625, + 0.375, + 0.0, + 0.4375, + 0.3125, + 0.625, + 0.625, + 0.625, + 0.6875, + 0.5, + 0.3125, + 0.0625, + 0.6875, + 0.1875, + 0.4375, + 0.1875, + 0.375, + 0.125, + 0.25, + 0.5, + 0.3125, + 0.4375, + 0.4375, + 0.375, + 0.8125, + 0.4375, + 0.5, + 0.5, + 0.5, + 0.375, + 0.25, + 0.1875, + 0.5, + 0.375, + 0.1875, + 0.25, + -0.0625, + 0.375, + 0.1875, + 0.5625, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.9997431039810181, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.9997568130493164 + ], + "mean": [ + -0.1328256438462534, + 0.14030752555972564, + -0.07960115504076615, + 0.016393409473275528, + 0.02376811828652776, + 0.2037397275786204, + -0.08447448880548725, + -0.022285896855183125, + 0.2105644169794228, + -0.20490043030930502, + -0.20489031965834087, + -0.02232229519865407, + -0.24486985570078945, + 0.35233394266856477, + 0.11206645528665718, + -0.004509350330011648, + 0.19562694124498511, + 0.047886065096415166, + -0.09148723631422286, + 0.09802276109745049, + -0.2587922220784263, + -0.20438680924032612, + 0.22753211142746216, + 0.0762383525300893, + 0.16564683900608257, + -0.046375533842370906, + -0.1699741976187395, + 0.0878838003105992, + -0.034000097062249254, + 0.36507336288339587, + 0.20983847224019672, + 0.22507724537336612, + 0.2735578167464734, + 0.12311941892066779, + -0.15756236249514688, + -0.23991563672835511, + 0.16439918467710626, + -0.22254553837194255, + -0.02215243626245632, + -0.09517155752555972, + -0.06882926750355894, + -0.21601203571890773, + -0.13072465057590268, + 0.23748099197618738, + -0.1322998899961175, + -0.21782386437168372, + -0.3010628316293516, + 0.09485812734567102, + 0.45991531318752427, + 0.008933771191924421, + -0.006416219101850654, + 0.09854042642681507, + 0.20279539277856864, + -0.047738449592338555, + -0.16898537595444546, + -0.035510628316293515, + -0.2886550407661447, + 0.056002895690436136, + -0.09289868318881843, + -0.14478856606703766, + -0.38754125145593377, + 0.07912797657564384, + -0.14342565031707002, + 0.2597203798369354, + 0.8714133721444814, + 0.8704363160829457, + 0.871101992679426, + 0.8709877289072792, + 0.9374116361295481, + 0.000909107080241183, + 0.8686004803123465, + 0.8727763422085179, + 0.8736698483416437, + 0.873315653691274, + 0.9243064344480587, + 0.0010054465065242088 + ], + "std": [ + 0.07789955998211606, + 0.15884626071604924, + 0.1581233979968548, + 0.09502975183202761, + 0.10422883895758395, + 0.13493252972835035, + 0.09668550524041442, + 0.10216842451806651, + 0.13706091772556664, + 0.1357826846018742, + 0.10177050699336576, + 0.08465054607636743, + 0.20205230739695762, + 0.12706670920463609, + 0.15069616802995173, + 0.06293637535646919, + 0.08382843427524267, + 0.07581424073507208, + 0.13568732985173562, + 0.14033116801561796, + 0.09932655110968273, + 0.09121062873145251, + 0.07208299484685422, + 0.1299125790615487, + 0.17977553384414668, + 0.1506115834117787, + 0.06899038047757403, + 0.08011449013545052, + 0.12308408904450967, + 0.16677321150773933, + 0.15547954149229914, + 0.10494672115866652, + 0.09867708152909252, + 0.10241109878343188, + 0.13636784165465646, + 0.07477624483874172, + 0.12988677428759557, + 0.13445998836373205, + 0.09184543622018963, + 0.0721655820519191, + 0.08807969513597146, + 0.11157826948556746, + 0.14881728477040088, + 0.09995603247268962, + 0.11745649633145971, + 0.19648932081391213, + 0.1997733493935003, + 0.07585100812231088, + 0.12983318820006998, + 0.13470484176152134, + 0.12727505199554814, + 0.1148687607803913, + 0.16623945022132947, + 0.0903189189593882, + 0.11630889040778654, + 0.06267302238221951, + 0.2107078066502835, + 0.0918116315905211, + 0.07687874194300864, + 0.1305883985144006, + 0.11356755494785113, + 0.0892572154462054, + 0.1057525617384013, + 0.12177318529042777, + 0.16123118351068702, + 0.16110483963491448, + 0.16022198436241275, + 0.16099862679020233, + 0.07811886944382597, + 0.019824963023862628, + 0.1676569977855643, + 0.15816317480687228, + 0.15654935499028427, + 0.15744207009759817, + 0.1022531519695512, + 0.019857930854210358 + ] + } + }, + "code_commit": "028df1bb7024fb1e3d176de7e60c84d3333cd97f", + "step": 4000, + "weights": "ema", + "offline_eval": { + "mse": 0.03382289409637451, + "mse_row0": 0.03190438449382782, + "mse_motion_token": 0.030815009027719498, + "mse_inspire_left_hand_state": 0.04951713606715202, + "mse_inspire_right_hand_state": 0.0502127967774868, + "samples": 256, + "frames": "train" + }, + "format": "hb_baselines/1" +} \ No newline at end of file diff --git a/b5-chair-push-standing-start/b512-cont-20261117/provenance/parent_checkpoint_protocol.json b/b5-chair-push-standing-start/b512-cont-20261117/provenance/parent_checkpoint_protocol.json new file mode 100644 index 0000000000000000000000000000000000000000..11b312faa69cf00e24120708702ec641a5d6827e --- /dev/null +++ b/b5-chair-push-standing-start/b512-cont-20261117/provenance/parent_checkpoint_protocol.json @@ -0,0 +1,11 @@ +{ + "format": "hb_baselines/1", + "step": 4000, + "policy": "fm_dit", + "files": { + "model.pt": { + "bytes": 90311509, + "sha256": "d258772759c5954d4d649766d1827f49c9b1627119151ca407e4c44eda9fdec6" + } + } +} \ No newline at end of file diff --git a/b5-chair-push-standing-start/b512-cont-20261117/provenance/parent_run.json b/b5-chair-push-standing-start/b512-cont-20261117/provenance/parent_run.json new file mode 100644 index 0000000000000000000000000000000000000000..8f9e59c4849f90d66f6512fe997cf63a2b990885 --- /dev/null +++ b/b5-chair-push-standing-start/b512-cont-20261117/provenance/parent_run.json @@ -0,0 +1,894 @@ +{ + "argv": [ + "/mnt/data0/zuo/Humanoid_benchmark/results/baselines_preflight_20261117/main/evaluation/baselines/train.py", + "--policy", + "fm_dit", + "--contract", + "g1_sonic_dfx6", + "--dataset", + "/mnt/data0/zuo/Humanoid_benchmark/data/baselines/b5-chair-push-standing-start-expert-v1", + "--cache-dir", + "/mnt/data0/zuo/Humanoid_benchmark/cache/baselines/frames", + "--out", + "/mnt/data0/zuo/Humanoid_benchmark/results/baselines_preflight_20261117/runs/b5_chair_push_standing_start_v1/fm_dit", + "--batch-size", + "512", + "--steps", + "4000", + "--workers", + "12" + ], + "started": "2026-11-17T04:07:16", + "host": "xt-p-A100-004", + "torch": "2.9.0+cu128", + "policy": { + "policy": "fm_dit", + "n_cameras": 2, + "state_dim": 44, + "action_dim": 76, + "horizon": 16, + "backbone": "resnet18", + "backbone_weights": "IMAGENET1K_V1", + "backbone_norm": "frozen_bn", + "hidden_dim": 384, + "num_layers": 4, + "num_heads": 8, + "mlp_ratio": 4.0, + "dropout": 0.1, + "time_embed_dim": 128, + "num_inference_steps": null, + "num_train_timesteps": 100, + "beta_schedule": "squaredcos_cap_v2", + "prediction_type": "epsilon", + "clip_sample": true, + "act_hidden_dim": 512, + "dim_feedforward": 3200, + "enc_layers": 4, + "dec_layers": 7, + "nheads": 8, + "latent_dim": 32, + "kl_weight": 10.0, + "extra": {} + }, + "contract": { + "name": "g1_sonic_dfx6", + "robot": "g1", + "cameras": [ + "ego_view", + "chest" + ], + "state": [ + [ + "observation.state", + 0, + 29 + ], + [ + "observation.inspire_left_hand_state", + 0, + 6 + ], + [ + "observation.inspire_right_hand_state", + 0, + 6 + ], + [ + "observation.projected_gravity", + 0, + 3 + ] + ], + "action": [ + [ + "action.motion_token", + 0, + 64 + ], + [ + "observation.inspire_left_hand_state", + 0, + 6 + ], + [ + "observation.inspire_right_hand_state", + 0, + 6 + ] + ], + "state_dim": 44, + "action_dim": 76, + "action_space": "sonic_dfx6", + "body_dims": 29 + }, + "image_hw": [ + 168, + 224 + ], + "task_text": "Establish and verify a stable two-hand grasp around the chair back, straighten it by stepping into position behind it, then walk forward to push it under the table.", + "dataset": { + "name": "b5-chair-push-standing-start-expert-v1", + "root": "/mnt/data0/zuo/Humanoid_benchmark/data/baselines/b5-chair-push-standing-start-expert-v1", + "robot_type": null, + "fps": 30.0, + "episodes": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 36, + 37, + 38, + 39, + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55, + 56, + 57, + 58, + 59, + 60, + 61, + 62, + 63, + 64, + 65, + 66, + 67, + 68, + 69, + 70, + 71, + 72, + 73, + 74, + 75, + 76, + 77, + 78, + 79, + 80, + 81, + 82, + 83, + 84, + 85, + 86, + 87, + 88, + 89, + 90, + 91, + 92, + 93, + 94, + 95, + 96, + 97, + 98, + 99 + ], + "train_episodes": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 36, + 37, + 38, + 39, + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55, + 56, + 57, + 58, + 59, + 60, + 61, + 62, + 63, + 64, + 65, + 66, + 67, + 68, + 69, + 70, + 71, + 72, + 73, + 74, + 75, + 76, + 77, + 78, + 79, + 80, + 81, + 82, + 83, + 84, + 85, + 86, + 87, + 88, + 89, + 90, + 91, + 92, + 93, + 94, + 95, + 96, + 97, + 98, + 99 + ], + "val_episodes": [], + "train_frames": 30908, + "git_commit": null + }, + "training": { + "steps": 4000, + "batch_size": 512, + "lr": 0.0001, + "betas": [ + 0.95, + 0.999 + ], + "weight_decay": 1e-06, + "warmup": 500, + "schedule": "linear warm-up + cosine", + "backbone_lr_scale": 0.1, + "ema": 0.9999, + "shift_px": 0, + "amp": true, + "seed": 0, + "augmentation": { + "spec": "docs/augmentation_v2.md", + "version": "hb-aug-v2-20261109", + "implementation": "evaluation/augmentation/hb_aug_v2.py", + "visual": { + "effects": [ + "exposure", + "contrast", + "saturation", + "hue", + "gamma", + "temperature", + "rotation_crop", + "noise", + "gaussian_blur", + "jpeg", + "motion_blur", + "sharpen", + "defocus", + "light_shadow", + "wb_channel" + ], + "boosted_per_sample": 1, + "boost": 1.5, + "pixel_scale": "H / 224", + "future_targets": "none (one observation frame)" + }, + "proprio": { + "robot": "g1", + "body_dims": 29, + "bias_deg": 2.0, + "bias_scope": "per_sample", + "noise_deg": 0.25, + "noise_clip_sigma": 3, + "delay_probability": 0.0, + "history_drop_probability": 0.0, + "velocity": false + }, + "native_image_augmentation": "off", + "evaluation": "off" + }, + "preflight": { + "samples": 64, + "state_dim": 44, + "body_dims": 29, + "images_changed": 64, + "body_values_moved": 1.0, + "max_body_offset_deg": 2.6143, + "actions_bit_identical": true, + "non_body_state_bit_identical": true, + "eval_items_deterministic": 8 + } + }, + "stats": { + "state": { + "min": [ + -0.5093753933906555, + -0.12233220040798187, + -0.12761788070201874, + 0.10250309854745865, + -0.8273318409919739, + -0.15657654404640198, + -0.4967050552368164, + -0.30181893706321716, + -0.5301669836044312, + 0.09314415603876114, + -0.8944554924964905, + -0.14240269362926483, + -0.10451218485832214, + -0.0940587967634201, + -0.13018359243869781, + -1.393661618232727, + -0.020927773788571358, + -0.7382517457008362, + -0.8502405285835266, + -0.11091316491365433, + -0.24967733025550842, + -1.2172361612319946, + -1.5036237239837646, + -1.1288223266601562, + -0.02860570326447487, + -0.8521270155906677, + -1.870940923690796, + -0.095971018075943, + -0.1065346822142601, + 0.0, + 0.5949721336364746, + 0.6192536950111389, + 0.6028932332992554, + 0.3511531352996826, + 0.0, + 0.0, + 0.5953419804573059, + 0.6113170981407166, + 0.5994111895561218, + 0.03766731545329094, + 0.0, + -0.3042502999305725, + -0.098258838057518, + -0.9999995827674866 + ], + "max": [ + 0.42436036467552185, + 0.2632223069667816, + 0.4142952561378479, + 1.3769621849060059, + 0.3906438946723938, + 0.12971684336662292, + 0.28290697932243347, + 0.07751978188753128, + 0.012389450334012508, + 1.309631109237671, + 0.5117528438568115, + 0.14354851841926575, + 0.09299838542938232, + 0.1345142424106598, + 0.16879205405712128, + 0.26465392112731934, + 1.0681228637695312, + 0.368639200925827, + 1.2249290943145752, + 1.7016332149505615, + 0.73288893699646, + 0.4079762399196625, + 0.20960678160190582, + 0.1851842999458313, + 0.9496553540229797, + 1.338996171951294, + 0.04667161777615547, + 1.0116424560546875, + 1.5066314935684204, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.9997431039810181, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.9997568130493164, + 0.011030972935259342, + 0.10935939103364944, + -0.9520710110664368 + ], + "mean": [ + -0.015364952224250912, + 0.081615613869982, + 0.12889738646888202, + 0.27149564947104204, + -0.14955734140047547, + -0.03703078041298108, + -0.0667806330627162, + -0.07314824380692604, + -0.2412484148070091, + 0.25491780809063996, + -0.10854783438820112, + -0.005161355253899878, + 0.003673423170905123, + 0.02969264617779002, + 0.08950587210737222, + -0.47956076188028, + 0.42631070073819, + -0.27129812784427826, + 0.16324256069371254, + 1.0385976806191235, + 0.32744283883844555, + -0.3669884986385585, + -0.5611831615193444, + -0.3750104856050489, + 0.3347049141703264, + 0.09427332727041168, + -1.2676208074144515, + 0.44498776552803626, + 0.6750011010472442, + 0.8714133721444814, + 0.8704363160829457, + 0.871101992679426, + 0.8709877289072792, + 0.9374116361295481, + 0.000909107080241183, + 0.8686004803123465, + 0.8727763422085179, + 0.8736698483416437, + 0.873315653691274, + 0.9243064344480587, + 0.0010054465065242088, + -0.07857978353459398, + 0.005790211846556424, + -0.9955768040204798 + ], + "std": [ + 0.12856331774418192, + 0.037393329668315815, + 0.05114975202975518, + 0.19438684278818802, + 0.12199680903864253, + 0.042366122817800854, + 0.1197195537504191, + 0.041234348615534516, + 0.06264541254527864, + 0.19296680414798664, + 0.14698603073664226, + 0.03873242198119102, + 0.022580757805711404, + 0.02236052295122295, + 0.03593445604445683, + 0.3256732586786403, + 0.19224309523346822, + 0.1334741564088957, + 0.48949046035954324, + 0.3411011488066223, + 0.17404862210424266, + 0.28167826002687124, + 0.32010586210868225, + 0.19978493123512664, + 0.12314692856716673, + 0.4856849744536303, + 0.4150175273557762, + 0.18968756344956073, + 0.36121237047109955, + 0.16123118351068702, + 0.16110483963491448, + 0.16022198436241275, + 0.16099862679020233, + 0.07811886944382597, + 0.019824963023862628, + 0.1676569977855643, + 0.15816317480687228, + 0.15654935499028427, + 0.15744207009759817, + 0.1022531519695512, + 0.019857930854210358, + 0.048533023773255274, + 0.015253372177349003, + 0.005513492459359003 + ] + }, + "action": { + "min": [ + -0.4375, + -0.375, + -0.5, + -0.4375, + -0.4375, + -0.1875, + -0.3125, + -0.4375, + -0.1875, + -0.5, + -0.5, + -0.25, + -0.625, + -0.125, + -0.3125, + -0.25, + -0.125, + -0.1875, + -0.5, + -0.3125, + -0.5625, + -0.5, + -0.0625, + -0.1875, + -0.5, + -0.4375, + -0.375, + -0.1875, + -0.4375, + -0.3125, + -0.3125, + -0.1875, + -0.0625, + -0.1875, + -0.5, + -0.5, + -0.125, + -0.5625, + -0.3125, + -0.25, + -0.375, + -0.5, + -0.5625, + -0.0625, + -0.5, + -0.625, + -0.6875, + -0.1875, + 0.0, + -0.375, + -0.5, + -0.3125, + -0.375, + -0.4375, + -0.5, + -0.25, + -0.6875, + -0.1875, + -0.3125, + -0.5, + -0.625, + -0.3125, + -0.4375, + -0.1875, + 0.0, + 0.5949721336364746, + 0.6192536950111389, + 0.6028932332992554, + 0.3511531352996826, + 0.0, + 0.0, + 0.5953419804573059, + 0.6113170981407166, + 0.5994111895561218, + 0.03766731545329094, + 0.0 + ], + "max": [ + 0.1875, + 0.5, + 0.375, + 0.25, + 0.4375, + 0.5625, + 0.3125, + 0.3125, + 0.5, + 0.25, + 0.125, + 0.3125, + 0.5, + 0.6875, + 0.4375, + 0.3125, + 0.4375, + 0.375, + 0.25, + 0.5, + 0.0625, + 0.1875, + 0.4375, + 0.5625, + 0.5625, + 0.375, + 0.0, + 0.4375, + 0.3125, + 0.625, + 0.625, + 0.625, + 0.6875, + 0.5, + 0.3125, + 0.0625, + 0.6875, + 0.1875, + 0.4375, + 0.1875, + 0.375, + 0.125, + 0.25, + 0.5, + 0.3125, + 0.4375, + 0.4375, + 0.375, + 0.8125, + 0.4375, + 0.5, + 0.5, + 0.5, + 0.375, + 0.25, + 0.1875, + 0.5, + 0.375, + 0.1875, + 0.25, + -0.0625, + 0.375, + 0.1875, + 0.5625, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.9997431039810181, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.9997568130493164 + ], + "mean": [ + -0.1328256438462534, + 0.14030752555972564, + -0.07960115504076615, + 0.016393409473275528, + 0.02376811828652776, + 0.2037397275786204, + -0.08447448880548725, + -0.022285896855183125, + 0.2105644169794228, + -0.20490043030930502, + -0.20489031965834087, + -0.02232229519865407, + -0.24486985570078945, + 0.35233394266856477, + 0.11206645528665718, + -0.004509350330011648, + 0.19562694124498511, + 0.047886065096415166, + -0.09148723631422286, + 0.09802276109745049, + -0.2587922220784263, + -0.20438680924032612, + 0.22753211142746216, + 0.0762383525300893, + 0.16564683900608257, + -0.046375533842370906, + -0.1699741976187395, + 0.0878838003105992, + -0.034000097062249254, + 0.36507336288339587, + 0.20983847224019672, + 0.22507724537336612, + 0.2735578167464734, + 0.12311941892066779, + -0.15756236249514688, + -0.23991563672835511, + 0.16439918467710626, + -0.22254553837194255, + -0.02215243626245632, + -0.09517155752555972, + -0.06882926750355894, + -0.21601203571890773, + -0.13072465057590268, + 0.23748099197618738, + -0.1322998899961175, + -0.21782386437168372, + -0.3010628316293516, + 0.09485812734567102, + 0.45991531318752427, + 0.008933771191924421, + -0.006416219101850654, + 0.09854042642681507, + 0.20279539277856864, + -0.047738449592338555, + -0.16898537595444546, + -0.035510628316293515, + -0.2886550407661447, + 0.056002895690436136, + -0.09289868318881843, + -0.14478856606703766, + -0.38754125145593377, + 0.07912797657564384, + -0.14342565031707002, + 0.2597203798369354, + 0.8714133721444814, + 0.8704363160829457, + 0.871101992679426, + 0.8709877289072792, + 0.9374116361295481, + 0.000909107080241183, + 0.8686004803123465, + 0.8727763422085179, + 0.8736698483416437, + 0.873315653691274, + 0.9243064344480587, + 0.0010054465065242088 + ], + "std": [ + 0.07789955998211606, + 0.15884626071604924, + 0.1581233979968548, + 0.09502975183202761, + 0.10422883895758395, + 0.13493252972835035, + 0.09668550524041442, + 0.10216842451806651, + 0.13706091772556664, + 0.1357826846018742, + 0.10177050699336576, + 0.08465054607636743, + 0.20205230739695762, + 0.12706670920463609, + 0.15069616802995173, + 0.06293637535646919, + 0.08382843427524267, + 0.07581424073507208, + 0.13568732985173562, + 0.14033116801561796, + 0.09932655110968273, + 0.09121062873145251, + 0.07208299484685422, + 0.1299125790615487, + 0.17977553384414668, + 0.1506115834117787, + 0.06899038047757403, + 0.08011449013545052, + 0.12308408904450967, + 0.16677321150773933, + 0.15547954149229914, + 0.10494672115866652, + 0.09867708152909252, + 0.10241109878343188, + 0.13636784165465646, + 0.07477624483874172, + 0.12988677428759557, + 0.13445998836373205, + 0.09184543622018963, + 0.0721655820519191, + 0.08807969513597146, + 0.11157826948556746, + 0.14881728477040088, + 0.09995603247268962, + 0.11745649633145971, + 0.19648932081391213, + 0.1997733493935003, + 0.07585100812231088, + 0.12983318820006998, + 0.13470484176152134, + 0.12727505199554814, + 0.1148687607803913, + 0.16623945022132947, + 0.0903189189593882, + 0.11630889040778654, + 0.06267302238221951, + 0.2107078066502835, + 0.0918116315905211, + 0.07687874194300864, + 0.1305883985144006, + 0.11356755494785113, + 0.0892572154462054, + 0.1057525617384013, + 0.12177318529042777, + 0.16123118351068702, + 0.16110483963491448, + 0.16022198436241275, + 0.16099862679020233, + 0.07811886944382597, + 0.019824963023862628, + 0.1676569977855643, + 0.15816317480687228, + 0.15654935499028427, + 0.15744207009759817, + 0.1022531519695512, + 0.019857930854210358 + ] + } + }, + "code_commit": "028df1bb7024fb1e3d176de7e60c84d3333cd97f" +} \ No newline at end of file diff --git a/b5-chair-push-standing-start/b512-cont-20261117/resume.json b/b5-chair-push-standing-start/b512-cont-20261117/resume.json new file mode 100644 index 0000000000000000000000000000000000000000..6385b7133495b3dfcb8487d3bc2c1f882ec0a232 --- /dev/null +++ b/b5-chair-push-standing-start/b512-cont-20261117/resume.json @@ -0,0 +1,26 @@ +{ + "task": "b5", + "status": "paused_by_owner", + "saved_step": 6000, + "last_logged_step": 6450, + "paused_at": "2026-11-17T08:36:34.220194+00:00", + "planned_total_steps": 20000, + "dataset_repo": "humanoid-benchmark/b5-chair-push-standing-start-expert-v1", + "dataset_revision": "913f5a16863c360cf2211c83bf98839d3f411293", + "dataset_train_frames": 30908, + "dataset_episodes": 100, + "code_directory": "code/fmdit_continue_20261117", + "source_commit": "028df1bb7024fb1e3d176de7e60c84d3333cd97f", + "parent_local_checkpoint": "/mnt/data0/zuo/Humanoid_benchmark/results/baselines_preflight_20261117/runs/b5_chair_push_standing_start_v1/fm_dit/checkpoints/step-004000", + "parent_state_sha256": "2b54ad4b94436bbaa0502c23ed3eb173efbf6b9a8f63d0bd2f1886c9c48115e6", + "offline_eval": { + "mse": 0.030160661786794662, + "mse_row0": 0.02748788893222809, + "mse_motion_token": 0.027840631082654, + "mse_inspire_left_hand_state": 0.04214290902018547, + "mse_inspire_right_hand_state": 0.04292541742324829, + "samples": 256, + "frames": "train" + }, + "limitation": "Portable resume starts at step 6000. Later logged updates are not in the saved state. RNG and DataLoader states were not saved by the original trainer; resume is not bit-exact." +} diff --git a/b5-chair-push-standing-start/b512-cont-20261117/run.json b/b5-chair-push-standing-start/b512-cont-20261117/run.json new file mode 100644 index 0000000000000000000000000000000000000000..7ec2833f168879a538010a5c7073a62ad0fe56ac --- /dev/null +++ b/b5-chair-push-standing-start/b512-cont-20261117/run.json @@ -0,0 +1,899 @@ +{ + "argv": [ + "/mnt/data0/zuo/Humanoid_benchmark/results/baselines_preflight_20261117/main/evaluation/baselines/train.py", + "--policy", + "fm_dit", + "--contract", + "g1_sonic_dfx6", + "--dataset", + "/mnt/data0/zuo/Humanoid_benchmark/data/baselines/b5-chair-push-standing-start-expert-v1", + "--cache-dir", + "/mnt/data0/zuo/Humanoid_benchmark/cache/baselines/frames", + "--out", + "/mnt/data0/zuo/Humanoid_benchmark/results/fmdit_continue_20261117_01/runs/b5/fm_dit", + "--batch-size", + "512", + "--steps", + "20000", + "--save-every", + "2000", + "--log-every", + "25", + "--workers", + "32", + "--resume" + ], + "started": "2026-11-17T15:57:41", + "host": "xt-p-A100-004", + "torch": "2.9.0+cu128", + "policy": { + "policy": "fm_dit", + "n_cameras": 2, + "state_dim": 44, + "action_dim": 76, + "horizon": 16, + "backbone": "resnet18", + "backbone_weights": "IMAGENET1K_V1", + "backbone_norm": "frozen_bn", + "hidden_dim": 384, + "num_layers": 4, + "num_heads": 8, + "mlp_ratio": 4.0, + "dropout": 0.1, + "time_embed_dim": 128, + "num_inference_steps": null, + "num_train_timesteps": 100, + "beta_schedule": "squaredcos_cap_v2", + "prediction_type": "epsilon", + "clip_sample": true, + "act_hidden_dim": 512, + "dim_feedforward": 3200, + "enc_layers": 4, + "dec_layers": 7, + "nheads": 8, + "latent_dim": 32, + "kl_weight": 10.0, + "extra": {} + }, + "contract": { + "name": "g1_sonic_dfx6", + "robot": "g1", + "cameras": [ + "ego_view", + "chest" + ], + "state": [ + [ + "observation.state", + 0, + 29 + ], + [ + "observation.inspire_left_hand_state", + 0, + 6 + ], + [ + "observation.inspire_right_hand_state", + 0, + 6 + ], + [ + "observation.projected_gravity", + 0, + 3 + ] + ], + "action": [ + [ + "action.motion_token", + 0, + 64 + ], + [ + "observation.inspire_left_hand_state", + 0, + 6 + ], + [ + "observation.inspire_right_hand_state", + 0, + 6 + ] + ], + "state_dim": 44, + "action_dim": 76, + "action_space": "sonic_dfx6", + "body_dims": 29 + }, + "image_hw": [ + 168, + 224 + ], + "task_text": "Establish and verify a stable two-hand grasp around the chair back, straighten it by stepping into position behind it, then walk forward to push it under the table.", + "dataset": { + "name": "b5-chair-push-standing-start-expert-v1", + "root": "/mnt/data0/zuo/Humanoid_benchmark/data/baselines/b5-chair-push-standing-start-expert-v1", + "robot_type": null, + "fps": 30.0, + "episodes": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 36, + 37, + 38, + 39, + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55, + 56, + 57, + 58, + 59, + 60, + 61, + 62, + 63, + 64, + 65, + 66, + 67, + 68, + 69, + 70, + 71, + 72, + 73, + 74, + 75, + 76, + 77, + 78, + 79, + 80, + 81, + 82, + 83, + 84, + 85, + 86, + 87, + 88, + 89, + 90, + 91, + 92, + 93, + 94, + 95, + 96, + 97, + 98, + 99 + ], + "train_episodes": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 36, + 37, + 38, + 39, + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55, + 56, + 57, + 58, + 59, + 60, + 61, + 62, + 63, + 64, + 65, + 66, + 67, + 68, + 69, + 70, + 71, + 72, + 73, + 74, + 75, + 76, + 77, + 78, + 79, + 80, + 81, + 82, + 83, + 84, + 85, + 86, + 87, + 88, + 89, + 90, + 91, + 92, + 93, + 94, + 95, + 96, + 97, + 98, + 99 + ], + "val_episodes": [], + "train_frames": 30908, + "git_commit": null + }, + "training": { + "steps": 20000, + "batch_size": 512, + "lr": 0.0001, + "betas": [ + 0.95, + 0.999 + ], + "weight_decay": 1e-06, + "warmup": 500, + "schedule": "linear warm-up + cosine", + "backbone_lr_scale": 0.1, + "ema": 0.9999, + "shift_px": 0, + "amp": true, + "seed": 0, + "augmentation": { + "spec": "docs/augmentation_v2.md", + "version": "hb-aug-v2-20261109", + "implementation": "evaluation/augmentation/hb_aug_v2.py", + "visual": { + "effects": [ + "exposure", + "contrast", + "saturation", + "hue", + "gamma", + "temperature", + "rotation_crop", + "noise", + "gaussian_blur", + "jpeg", + "motion_blur", + "sharpen", + "defocus", + "light_shadow", + "wb_channel" + ], + "boosted_per_sample": 1, + "boost": 1.5, + "pixel_scale": "H / 224", + "future_targets": "none (one observation frame)" + }, + "proprio": { + "robot": "g1", + "body_dims": 29, + "bias_deg": 2.0, + "bias_scope": "per_sample", + "noise_deg": 0.25, + "noise_clip_sigma": 3, + "delay_probability": 0.0, + "history_drop_probability": 0.0, + "velocity": false + }, + "native_image_augmentation": "off", + "evaluation": "off" + }, + "preflight": { + "samples": 64, + "state_dim": 44, + "body_dims": 29, + "images_changed": 64, + "body_values_moved": 1.0, + "max_body_offset_deg": 2.552, + "actions_bit_identical": true, + "non_body_state_bit_identical": true, + "eval_items_deterministic": 8 + } + }, + "stats": { + "state": { + "min": [ + -0.5093753933906555, + -0.12233220040798187, + -0.12761788070201874, + 0.10250309854745865, + -0.8273318409919739, + -0.15657654404640198, + -0.4967050552368164, + -0.30181893706321716, + -0.5301669836044312, + 0.09314415603876114, + -0.8944554924964905, + -0.14240269362926483, + -0.10451218485832214, + -0.0940587967634201, + -0.13018359243869781, + -1.393661618232727, + -0.020927773788571358, + -0.7382517457008362, + -0.8502405285835266, + -0.11091316491365433, + -0.24967733025550842, + -1.2172361612319946, + -1.5036237239837646, + -1.1288223266601562, + -0.02860570326447487, + -0.8521270155906677, + -1.870940923690796, + -0.095971018075943, + -0.1065346822142601, + 0.0, + 0.5949721336364746, + 0.6192536950111389, + 0.6028932332992554, + 0.3511531352996826, + 0.0, + 0.0, + 0.5953419804573059, + 0.6113170981407166, + 0.5994111895561218, + 0.03766731545329094, + 0.0, + -0.3042502999305725, + -0.098258838057518, + -0.9999995827674866 + ], + "max": [ + 0.42436036467552185, + 0.2632223069667816, + 0.4142952561378479, + 1.3769621849060059, + 0.3906438946723938, + 0.12971684336662292, + 0.28290697932243347, + 0.07751978188753128, + 0.012389450334012508, + 1.309631109237671, + 0.5117528438568115, + 0.14354851841926575, + 0.09299838542938232, + 0.1345142424106598, + 0.16879205405712128, + 0.26465392112731934, + 1.0681228637695312, + 0.368639200925827, + 1.2249290943145752, + 1.7016332149505615, + 0.73288893699646, + 0.4079762399196625, + 0.20960678160190582, + 0.1851842999458313, + 0.9496553540229797, + 1.338996171951294, + 0.04667161777615547, + 1.0116424560546875, + 1.5066314935684204, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.9997431039810181, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.9997568130493164, + 0.011030972935259342, + 0.10935939103364944, + -0.9520710110664368 + ], + "mean": [ + -0.015364952224250912, + 0.081615613869982, + 0.12889738646888202, + 0.27149564947104204, + -0.14955734140047547, + -0.03703078041298108, + -0.0667806330627162, + -0.07314824380692604, + -0.2412484148070091, + 0.25491780809063996, + -0.10854783438820112, + -0.005161355253899878, + 0.003673423170905123, + 0.02969264617779002, + 0.08950587210737222, + -0.47956076188028, + 0.42631070073819, + -0.27129812784427826, + 0.16324256069371254, + 1.0385976806191235, + 0.32744283883844555, + -0.3669884986385585, + -0.5611831615193444, + -0.3750104856050489, + 0.3347049141703264, + 0.09427332727041168, + -1.2676208074144515, + 0.44498776552803626, + 0.6750011010472442, + 0.8714133721444814, + 0.8704363160829457, + 0.871101992679426, + 0.8709877289072792, + 0.9374116361295481, + 0.000909107080241183, + 0.8686004803123465, + 0.8727763422085179, + 0.8736698483416437, + 0.873315653691274, + 0.9243064344480587, + 0.0010054465065242088, + -0.07857978353459398, + 0.005790211846556424, + -0.9955768040204798 + ], + "std": [ + 0.12856331774418192, + 0.037393329668315815, + 0.05114975202975518, + 0.19438684278818802, + 0.12199680903864253, + 0.042366122817800854, + 0.1197195537504191, + 0.041234348615534516, + 0.06264541254527864, + 0.19296680414798664, + 0.14698603073664226, + 0.03873242198119102, + 0.022580757805711404, + 0.02236052295122295, + 0.03593445604445683, + 0.3256732586786403, + 0.19224309523346822, + 0.1334741564088957, + 0.48949046035954324, + 0.3411011488066223, + 0.17404862210424266, + 0.28167826002687124, + 0.32010586210868225, + 0.19978493123512664, + 0.12314692856716673, + 0.4856849744536303, + 0.4150175273557762, + 0.18968756344956073, + 0.36121237047109955, + 0.16123118351068702, + 0.16110483963491448, + 0.16022198436241275, + 0.16099862679020233, + 0.07811886944382597, + 0.019824963023862628, + 0.1676569977855643, + 0.15816317480687228, + 0.15654935499028427, + 0.15744207009759817, + 0.1022531519695512, + 0.019857930854210358, + 0.048533023773255274, + 0.015253372177349003, + 0.005513492459359003 + ] + }, + "action": { + "min": [ + -0.4375, + -0.375, + -0.5, + -0.4375, + -0.4375, + -0.1875, + -0.3125, + -0.4375, + -0.1875, + -0.5, + -0.5, + -0.25, + -0.625, + -0.125, + -0.3125, + -0.25, + -0.125, + -0.1875, + -0.5, + -0.3125, + -0.5625, + -0.5, + -0.0625, + -0.1875, + -0.5, + -0.4375, + -0.375, + -0.1875, + -0.4375, + -0.3125, + -0.3125, + -0.1875, + -0.0625, + -0.1875, + -0.5, + -0.5, + -0.125, + -0.5625, + -0.3125, + -0.25, + -0.375, + -0.5, + -0.5625, + -0.0625, + -0.5, + -0.625, + -0.6875, + -0.1875, + 0.0, + -0.375, + -0.5, + -0.3125, + -0.375, + -0.4375, + -0.5, + -0.25, + -0.6875, + -0.1875, + -0.3125, + -0.5, + -0.625, + -0.3125, + -0.4375, + -0.1875, + 0.0, + 0.5949721336364746, + 0.6192536950111389, + 0.6028932332992554, + 0.3511531352996826, + 0.0, + 0.0, + 0.5953419804573059, + 0.6113170981407166, + 0.5994111895561218, + 0.03766731545329094, + 0.0 + ], + "max": [ + 0.1875, + 0.5, + 0.375, + 0.25, + 0.4375, + 0.5625, + 0.3125, + 0.3125, + 0.5, + 0.25, + 0.125, + 0.3125, + 0.5, + 0.6875, + 0.4375, + 0.3125, + 0.4375, + 0.375, + 0.25, + 0.5, + 0.0625, + 0.1875, + 0.4375, + 0.5625, + 0.5625, + 0.375, + 0.0, + 0.4375, + 0.3125, + 0.625, + 0.625, + 0.625, + 0.6875, + 0.5, + 0.3125, + 0.0625, + 0.6875, + 0.1875, + 0.4375, + 0.1875, + 0.375, + 0.125, + 0.25, + 0.5, + 0.3125, + 0.4375, + 0.4375, + 0.375, + 0.8125, + 0.4375, + 0.5, + 0.5, + 0.5, + 0.375, + 0.25, + 0.1875, + 0.5, + 0.375, + 0.1875, + 0.25, + -0.0625, + 0.375, + 0.1875, + 0.5625, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.9997431039810181, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.9997568130493164 + ], + "mean": [ + -0.1328256438462534, + 0.14030752555972564, + -0.07960115504076615, + 0.016393409473275528, + 0.02376811828652776, + 0.2037397275786204, + -0.08447448880548725, + -0.022285896855183125, + 0.2105644169794228, + -0.20490043030930502, + -0.20489031965834087, + -0.02232229519865407, + -0.24486985570078945, + 0.35233394266856477, + 0.11206645528665718, + -0.004509350330011648, + 0.19562694124498511, + 0.047886065096415166, + -0.09148723631422286, + 0.09802276109745049, + -0.2587922220784263, + -0.20438680924032612, + 0.22753211142746216, + 0.0762383525300893, + 0.16564683900608257, + -0.046375533842370906, + -0.1699741976187395, + 0.0878838003105992, + -0.034000097062249254, + 0.36507336288339587, + 0.20983847224019672, + 0.22507724537336612, + 0.2735578167464734, + 0.12311941892066779, + -0.15756236249514688, + -0.23991563672835511, + 0.16439918467710626, + -0.22254553837194255, + -0.02215243626245632, + -0.09517155752555972, + -0.06882926750355894, + -0.21601203571890773, + -0.13072465057590268, + 0.23748099197618738, + -0.1322998899961175, + -0.21782386437168372, + -0.3010628316293516, + 0.09485812734567102, + 0.45991531318752427, + 0.008933771191924421, + -0.006416219101850654, + 0.09854042642681507, + 0.20279539277856864, + -0.047738449592338555, + -0.16898537595444546, + -0.035510628316293515, + -0.2886550407661447, + 0.056002895690436136, + -0.09289868318881843, + -0.14478856606703766, + -0.38754125145593377, + 0.07912797657564384, + -0.14342565031707002, + 0.2597203798369354, + 0.8714133721444814, + 0.8704363160829457, + 0.871101992679426, + 0.8709877289072792, + 0.9374116361295481, + 0.000909107080241183, + 0.8686004803123465, + 0.8727763422085179, + 0.8736698483416437, + 0.873315653691274, + 0.9243064344480587, + 0.0010054465065242088 + ], + "std": [ + 0.07789955998211606, + 0.15884626071604924, + 0.1581233979968548, + 0.09502975183202761, + 0.10422883895758395, + 0.13493252972835035, + 0.09668550524041442, + 0.10216842451806651, + 0.13706091772556664, + 0.1357826846018742, + 0.10177050699336576, + 0.08465054607636743, + 0.20205230739695762, + 0.12706670920463609, + 0.15069616802995173, + 0.06293637535646919, + 0.08382843427524267, + 0.07581424073507208, + 0.13568732985173562, + 0.14033116801561796, + 0.09932655110968273, + 0.09121062873145251, + 0.07208299484685422, + 0.1299125790615487, + 0.17977553384414668, + 0.1506115834117787, + 0.06899038047757403, + 0.08011449013545052, + 0.12308408904450967, + 0.16677321150773933, + 0.15547954149229914, + 0.10494672115866652, + 0.09867708152909252, + 0.10241109878343188, + 0.13636784165465646, + 0.07477624483874172, + 0.12988677428759557, + 0.13445998836373205, + 0.09184543622018963, + 0.0721655820519191, + 0.08807969513597146, + 0.11157826948556746, + 0.14881728477040088, + 0.09995603247268962, + 0.11745649633145971, + 0.19648932081391213, + 0.1997733493935003, + 0.07585100812231088, + 0.12983318820006998, + 0.13470484176152134, + 0.12727505199554814, + 0.1148687607803913, + 0.16623945022132947, + 0.0903189189593882, + 0.11630889040778654, + 0.06267302238221951, + 0.2107078066502835, + 0.0918116315905211, + 0.07687874194300864, + 0.1305883985144006, + 0.11356755494785113, + 0.0892572154462054, + 0.1057525617384013, + 0.12177318529042777, + 0.16123118351068702, + 0.16110483963491448, + 0.16022198436241275, + 0.16099862679020233, + 0.07811886944382597, + 0.019824963023862628, + 0.1676569977855643, + 0.15816317480687228, + 0.15654935499028427, + 0.15744207009759817, + 0.1022531519695512, + 0.019857930854210358 + ] + } + }, + "code_commit": "028df1bb7024fb1e3d176de7e60c84d3333cd97f" +} \ No newline at end of file diff --git a/b5-chair-push-standing-start/b512-cont-20261117/step-006000/config.json b/b5-chair-push-standing-start/b512-cont-20261117/step-006000/config.json new file mode 100644 index 0000000000000000000000000000000000000000..0f8ae078aae889d442e502e0a9fe6619dc40f6ef --- /dev/null +++ b/b5-chair-push-standing-start/b512-cont-20261117/step-006000/config.json @@ -0,0 +1,884 @@ +{ + "policy": { + "policy": "fm_dit", + "n_cameras": 2, + "state_dim": 44, + "action_dim": 76, + "horizon": 16, + "backbone": "resnet18", + "backbone_weights": "IMAGENET1K_V1", + "backbone_norm": "frozen_bn", + "hidden_dim": 384, + "num_layers": 4, + "num_heads": 8, + "mlp_ratio": 4.0, + "dropout": 0.1, + "time_embed_dim": 128, + "num_inference_steps": null, + "num_train_timesteps": 100, + "beta_schedule": "squaredcos_cap_v2", + "prediction_type": "epsilon", + "clip_sample": true, + "act_hidden_dim": 512, + "dim_feedforward": 3200, + "enc_layers": 4, + "dec_layers": 7, + "nheads": 8, + "latent_dim": 32, + "kl_weight": 10.0, + "extra": {} + }, + "contract": { + "name": "g1_sonic_dfx6", + "robot": "g1", + "cameras": [ + "ego_view", + "chest" + ], + "state": [ + [ + "observation.state", + 0, + 29 + ], + [ + "observation.inspire_left_hand_state", + 0, + 6 + ], + [ + "observation.inspire_right_hand_state", + 0, + 6 + ], + [ + "observation.projected_gravity", + 0, + 3 + ] + ], + "action": [ + [ + "action.motion_token", + 0, + 64 + ], + [ + "observation.inspire_left_hand_state", + 0, + 6 + ], + [ + "observation.inspire_right_hand_state", + 0, + 6 + ] + ], + "state_dim": 44, + "action_dim": 76, + "action_space": "sonic_dfx6", + "body_dims": 29 + }, + "image_hw": [ + 168, + 224 + ], + "task_text": "Establish and verify a stable two-hand grasp around the chair back, straighten it by stepping into position behind it, then walk forward to push it under the table.", + "dataset": { + "name": "b5-chair-push-standing-start-expert-v1", + "root": "/mnt/data0/zuo/Humanoid_benchmark/data/baselines/b5-chair-push-standing-start-expert-v1", + "robot_type": null, + "fps": 30.0, + "episodes": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 36, + 37, + 38, + 39, + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55, + 56, + 57, + 58, + 59, + 60, + 61, + 62, + 63, + 64, + 65, + 66, + 67, + 68, + 69, + 70, + 71, + 72, + 73, + 74, + 75, + 76, + 77, + 78, + 79, + 80, + 81, + 82, + 83, + 84, + 85, + 86, + 87, + 88, + 89, + 90, + 91, + 92, + 93, + 94, + 95, + 96, + 97, + 98, + 99 + ], + "train_episodes": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 36, + 37, + 38, + 39, + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55, + 56, + 57, + 58, + 59, + 60, + 61, + 62, + 63, + 64, + 65, + 66, + 67, + 68, + 69, + 70, + 71, + 72, + 73, + 74, + 75, + 76, + 77, + 78, + 79, + 80, + 81, + 82, + 83, + 84, + 85, + 86, + 87, + 88, + 89, + 90, + 91, + 92, + 93, + 94, + 95, + 96, + 97, + 98, + 99 + ], + "val_episodes": [], + "train_frames": 30908, + "git_commit": null + }, + "training": { + "steps": 20000, + "batch_size": 512, + "lr": 0.0001, + "betas": [ + 0.95, + 0.999 + ], + "weight_decay": 1e-06, + "warmup": 500, + "schedule": "linear warm-up + cosine", + "backbone_lr_scale": 0.1, + "ema": 0.9999, + "shift_px": 0, + "amp": true, + "seed": 0, + "augmentation": { + "spec": "docs/augmentation_v2.md", + "version": "hb-aug-v2-20261109", + "implementation": "evaluation/augmentation/hb_aug_v2.py", + "visual": { + "effects": [ + "exposure", + "contrast", + "saturation", + "hue", + "gamma", + "temperature", + "rotation_crop", + "noise", + "gaussian_blur", + "jpeg", + "motion_blur", + "sharpen", + "defocus", + "light_shadow", + "wb_channel" + ], + "boosted_per_sample": 1, + "boost": 1.5, + "pixel_scale": "H / 224", + "future_targets": "none (one observation frame)" + }, + "proprio": { + "robot": "g1", + "body_dims": 29, + "bias_deg": 2.0, + "bias_scope": "per_sample", + "noise_deg": 0.25, + "noise_clip_sigma": 3, + "delay_probability": 0.0, + "history_drop_probability": 0.0, + "velocity": false + }, + "native_image_augmentation": "off", + "evaluation": "off" + }, + "preflight": { + "samples": 64, + "state_dim": 44, + "body_dims": 29, + "images_changed": 64, + "body_values_moved": 1.0, + "max_body_offset_deg": 2.552, + "actions_bit_identical": true, + "non_body_state_bit_identical": true, + "eval_items_deterministic": 8 + } + }, + "stats": { + "state": { + "min": [ + -0.5093753933906555, + -0.12233220040798187, + -0.12761788070201874, + 0.10250309854745865, + -0.8273318409919739, + -0.15657654404640198, + -0.4967050552368164, + -0.30181893706321716, + -0.5301669836044312, + 0.09314415603876114, + -0.8944554924964905, + -0.14240269362926483, + -0.10451218485832214, + -0.0940587967634201, + -0.13018359243869781, + -1.393661618232727, + -0.020927773788571358, + -0.7382517457008362, + -0.8502405285835266, + -0.11091316491365433, + -0.24967733025550842, + -1.2172361612319946, + -1.5036237239837646, + -1.1288223266601562, + -0.02860570326447487, + -0.8521270155906677, + -1.870940923690796, + -0.095971018075943, + -0.1065346822142601, + 0.0, + 0.5949721336364746, + 0.6192536950111389, + 0.6028932332992554, + 0.3511531352996826, + 0.0, + 0.0, + 0.5953419804573059, + 0.6113170981407166, + 0.5994111895561218, + 0.03766731545329094, + 0.0, + -0.3042502999305725, + -0.098258838057518, + -0.9999995827674866 + ], + "max": [ + 0.42436036467552185, + 0.2632223069667816, + 0.4142952561378479, + 1.3769621849060059, + 0.3906438946723938, + 0.12971684336662292, + 0.28290697932243347, + 0.07751978188753128, + 0.012389450334012508, + 1.309631109237671, + 0.5117528438568115, + 0.14354851841926575, + 0.09299838542938232, + 0.1345142424106598, + 0.16879205405712128, + 0.26465392112731934, + 1.0681228637695312, + 0.368639200925827, + 1.2249290943145752, + 1.7016332149505615, + 0.73288893699646, + 0.4079762399196625, + 0.20960678160190582, + 0.1851842999458313, + 0.9496553540229797, + 1.338996171951294, + 0.04667161777615547, + 1.0116424560546875, + 1.5066314935684204, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.9997431039810181, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.9997568130493164, + 0.011030972935259342, + 0.10935939103364944, + -0.9520710110664368 + ], + "mean": [ + -0.015364952224250912, + 0.081615613869982, + 0.12889738646888202, + 0.27149564947104204, + -0.14955734140047547, + -0.03703078041298108, + -0.0667806330627162, + -0.07314824380692604, + -0.2412484148070091, + 0.25491780809063996, + -0.10854783438820112, + -0.005161355253899878, + 0.003673423170905123, + 0.02969264617779002, + 0.08950587210737222, + -0.47956076188028, + 0.42631070073819, + -0.27129812784427826, + 0.16324256069371254, + 1.0385976806191235, + 0.32744283883844555, + -0.3669884986385585, + -0.5611831615193444, + -0.3750104856050489, + 0.3347049141703264, + 0.09427332727041168, + -1.2676208074144515, + 0.44498776552803626, + 0.6750011010472442, + 0.8714133721444814, + 0.8704363160829457, + 0.871101992679426, + 0.8709877289072792, + 0.9374116361295481, + 0.000909107080241183, + 0.8686004803123465, + 0.8727763422085179, + 0.8736698483416437, + 0.873315653691274, + 0.9243064344480587, + 0.0010054465065242088, + -0.07857978353459398, + 0.005790211846556424, + -0.9955768040204798 + ], + "std": [ + 0.12856331774418192, + 0.037393329668315815, + 0.05114975202975518, + 0.19438684278818802, + 0.12199680903864253, + 0.042366122817800854, + 0.1197195537504191, + 0.041234348615534516, + 0.06264541254527864, + 0.19296680414798664, + 0.14698603073664226, + 0.03873242198119102, + 0.022580757805711404, + 0.02236052295122295, + 0.03593445604445683, + 0.3256732586786403, + 0.19224309523346822, + 0.1334741564088957, + 0.48949046035954324, + 0.3411011488066223, + 0.17404862210424266, + 0.28167826002687124, + 0.32010586210868225, + 0.19978493123512664, + 0.12314692856716673, + 0.4856849744536303, + 0.4150175273557762, + 0.18968756344956073, + 0.36121237047109955, + 0.16123118351068702, + 0.16110483963491448, + 0.16022198436241275, + 0.16099862679020233, + 0.07811886944382597, + 0.019824963023862628, + 0.1676569977855643, + 0.15816317480687228, + 0.15654935499028427, + 0.15744207009759817, + 0.1022531519695512, + 0.019857930854210358, + 0.048533023773255274, + 0.015253372177349003, + 0.005513492459359003 + ] + }, + "action": { + "min": [ + -0.4375, + -0.375, + -0.5, + -0.4375, + -0.4375, + -0.1875, + -0.3125, + -0.4375, + -0.1875, + -0.5, + -0.5, + -0.25, + -0.625, + -0.125, + -0.3125, + -0.25, + -0.125, + -0.1875, + -0.5, + -0.3125, + -0.5625, + -0.5, + -0.0625, + -0.1875, + -0.5, + -0.4375, + -0.375, + -0.1875, + -0.4375, + -0.3125, + -0.3125, + -0.1875, + -0.0625, + -0.1875, + -0.5, + -0.5, + -0.125, + -0.5625, + -0.3125, + -0.25, + -0.375, + -0.5, + -0.5625, + -0.0625, + -0.5, + -0.625, + -0.6875, + -0.1875, + 0.0, + -0.375, + -0.5, + -0.3125, + -0.375, + -0.4375, + -0.5, + -0.25, + -0.6875, + -0.1875, + -0.3125, + -0.5, + -0.625, + -0.3125, + -0.4375, + -0.1875, + 0.0, + 0.5949721336364746, + 0.6192536950111389, + 0.6028932332992554, + 0.3511531352996826, + 0.0, + 0.0, + 0.5953419804573059, + 0.6113170981407166, + 0.5994111895561218, + 0.03766731545329094, + 0.0 + ], + "max": [ + 0.1875, + 0.5, + 0.375, + 0.25, + 0.4375, + 0.5625, + 0.3125, + 0.3125, + 0.5, + 0.25, + 0.125, + 0.3125, + 0.5, + 0.6875, + 0.4375, + 0.3125, + 0.4375, + 0.375, + 0.25, + 0.5, + 0.0625, + 0.1875, + 0.4375, + 0.5625, + 0.5625, + 0.375, + 0.0, + 0.4375, + 0.3125, + 0.625, + 0.625, + 0.625, + 0.6875, + 0.5, + 0.3125, + 0.0625, + 0.6875, + 0.1875, + 0.4375, + 0.1875, + 0.375, + 0.125, + 0.25, + 0.5, + 0.3125, + 0.4375, + 0.4375, + 0.375, + 0.8125, + 0.4375, + 0.5, + 0.5, + 0.5, + 0.375, + 0.25, + 0.1875, + 0.5, + 0.375, + 0.1875, + 0.25, + -0.0625, + 0.375, + 0.1875, + 0.5625, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.9997431039810181, + 1.0, + 1.0, + 1.0, + 1.0, + 1.0, + 0.9997568130493164 + ], + "mean": [ + -0.1328256438462534, + 0.14030752555972564, + -0.07960115504076615, + 0.016393409473275528, + 0.02376811828652776, + 0.2037397275786204, + -0.08447448880548725, + -0.022285896855183125, + 0.2105644169794228, + -0.20490043030930502, + -0.20489031965834087, + -0.02232229519865407, + -0.24486985570078945, + 0.35233394266856477, + 0.11206645528665718, + -0.004509350330011648, + 0.19562694124498511, + 0.047886065096415166, + -0.09148723631422286, + 0.09802276109745049, + -0.2587922220784263, + -0.20438680924032612, + 0.22753211142746216, + 0.0762383525300893, + 0.16564683900608257, + -0.046375533842370906, + -0.1699741976187395, + 0.0878838003105992, + -0.034000097062249254, + 0.36507336288339587, + 0.20983847224019672, + 0.22507724537336612, + 0.2735578167464734, + 0.12311941892066779, + -0.15756236249514688, + -0.23991563672835511, + 0.16439918467710626, + -0.22254553837194255, + -0.02215243626245632, + -0.09517155752555972, + -0.06882926750355894, + -0.21601203571890773, + -0.13072465057590268, + 0.23748099197618738, + -0.1322998899961175, + -0.21782386437168372, + -0.3010628316293516, + 0.09485812734567102, + 0.45991531318752427, + 0.008933771191924421, + -0.006416219101850654, + 0.09854042642681507, + 0.20279539277856864, + -0.047738449592338555, + -0.16898537595444546, + -0.035510628316293515, + -0.2886550407661447, + 0.056002895690436136, + -0.09289868318881843, + -0.14478856606703766, + -0.38754125145593377, + 0.07912797657564384, + -0.14342565031707002, + 0.2597203798369354, + 0.8714133721444814, + 0.8704363160829457, + 0.871101992679426, + 0.8709877289072792, + 0.9374116361295481, + 0.000909107080241183, + 0.8686004803123465, + 0.8727763422085179, + 0.8736698483416437, + 0.873315653691274, + 0.9243064344480587, + 0.0010054465065242088 + ], + "std": [ + 0.07789955998211606, + 0.15884626071604924, + 0.1581233979968548, + 0.09502975183202761, + 0.10422883895758395, + 0.13493252972835035, + 0.09668550524041442, + 0.10216842451806651, + 0.13706091772556664, + 0.1357826846018742, + 0.10177050699336576, + 0.08465054607636743, + 0.20205230739695762, + 0.12706670920463609, + 0.15069616802995173, + 0.06293637535646919, + 0.08382843427524267, + 0.07581424073507208, + 0.13568732985173562, + 0.14033116801561796, + 0.09932655110968273, + 0.09121062873145251, + 0.07208299484685422, + 0.1299125790615487, + 0.17977553384414668, + 0.1506115834117787, + 0.06899038047757403, + 0.08011449013545052, + 0.12308408904450967, + 0.16677321150773933, + 0.15547954149229914, + 0.10494672115866652, + 0.09867708152909252, + 0.10241109878343188, + 0.13636784165465646, + 0.07477624483874172, + 0.12988677428759557, + 0.13445998836373205, + 0.09184543622018963, + 0.0721655820519191, + 0.08807969513597146, + 0.11157826948556746, + 0.14881728477040088, + 0.09995603247268962, + 0.11745649633145971, + 0.19648932081391213, + 0.1997733493935003, + 0.07585100812231088, + 0.12983318820006998, + 0.13470484176152134, + 0.12727505199554814, + 0.1148687607803913, + 0.16623945022132947, + 0.0903189189593882, + 0.11630889040778654, + 0.06267302238221951, + 0.2107078066502835, + 0.0918116315905211, + 0.07687874194300864, + 0.1305883985144006, + 0.11356755494785113, + 0.0892572154462054, + 0.1057525617384013, + 0.12177318529042777, + 0.16123118351068702, + 0.16110483963491448, + 0.16022198436241275, + 0.16099862679020233, + 0.07811886944382597, + 0.019824963023862628, + 0.1676569977855643, + 0.15816317480687228, + 0.15654935499028427, + 0.15744207009759817, + 0.1022531519695512, + 0.019857930854210358 + ] + } + }, + "code_commit": "028df1bb7024fb1e3d176de7e60c84d3333cd97f", + "step": 6000, + "weights": "ema", + "offline_eval": { + "mse": 0.030160661786794662, + "mse_row0": 0.02748788893222809, + "mse_motion_token": 0.027840631082654, + "mse_inspire_left_hand_state": 0.04214290902018547, + "mse_inspire_right_hand_state": 0.04292541742324829, + "samples": 256, + "frames": "train" + }, + "format": "hb_baselines/1" +} \ No newline at end of file diff --git a/b5-chair-push-standing-start/b512-cont-20261117/step-006000/model.pt b/b5-chair-push-standing-start/b512-cont-20261117/step-006000/model.pt new file mode 100644 index 0000000000000000000000000000000000000000..aa9c7882d4f588d941b0e8c28b06cbdf71534daf --- /dev/null +++ b/b5-chair-push-standing-start/b512-cont-20261117/step-006000/model.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:5178087fc50ffb7c5b233bae26ba6bc083c037bcd611d62a9f573166c8b564b3 +size 90311509 diff --git a/b5-chair-push-standing-start/b512-cont-20261117/step-006000/protocol.json b/b5-chair-push-standing-start/b512-cont-20261117/step-006000/protocol.json new file mode 100644 index 0000000000000000000000000000000000000000..c10b1319081d887e82faf28d30563ec427d8f0a5 --- /dev/null +++ b/b5-chair-push-standing-start/b512-cont-20261117/step-006000/protocol.json @@ -0,0 +1,11 @@ +{ + "format": "hb_baselines/1", + "step": 6000, + "policy": "fm_dit", + "files": { + "model.pt": { + "bytes": 90311509, + "sha256": "5178087fc50ffb7c5b233bae26ba6bc083c037bcd611d62a9f573166c8b564b3" + } + } +} \ No newline at end of file diff --git a/b5-chair-push-standing-start/b512-cont-20261117/train_state.pt b/b5-chair-push-standing-start/b512-cont-20261117/train_state.pt new file mode 100644 index 0000000000000000000000000000000000000000..f6db2037351753ff7be92176169d21814fbaa5f5 --- /dev/null +++ b/b5-chair-push-standing-start/b512-cont-20261117/train_state.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:74b1d066a512bb535c3e87ad34636723b05507ef591530ae59f80b45e20f0797 +size 361027889 diff --git a/b8-table-wipe/b512-cont-20261117/RESUME.md b/b8-table-wipe/b512-cont-20261117/RESUME.md new file mode 100644 index 0000000000000000000000000000000000000000..531ff4001189dfa6d3db6448ca2233a893b98638 --- /dev/null +++ b/b8-table-wipe/b512-cont-20261117/RESUME.md @@ -0,0 +1,39 @@ +# B8 FM-DiT continuation, paused 2026-11-17 + +This is an independent continuation from the local step-4,000 baseline. +The portable training state is **step 6,000**; the last logged step is 6700. +Updates after step 6,000 remain only in the paused local process and are not in this release. +The training target was 20,000 total steps. This package is not a completed 20,000-step run. + +## Files + +- `train_state.pt`: raw policy, AdamW moments, LR scheduler, EMA, step and full config. +- `step-006000/`: EMA inference checkpoint, config and weight hash protocol. +- `run.json`, `resume.json`: recipe, data identity, preflight and pause details. +- `logs/`: current and original training logs. Log rows after 6,000 are history, not saved weights. +- `provenance/`: original run, original checkpoint config and hash protocol. +- `SHA256SUMS`: hashes of this package and its shared source snapshot, relative to repository root. + +## Resume + +Use Python 3.12 and install `code/fmdit_continue_20261117/baselines/requirements.txt`. +Download the original dataset separately (access to its private repository is required): + +```bash +hf download humanoid-benchmark/b8-table-wipe-expert-v1 --repo-type dataset --revision 3b2e6180508f84f70fda8b33cd3e41c14225f0b5 --local-dir /path/to/b8-table-wipe-expert-v1 +python code/fmdit_continue_20261117/resume.py --task b8 --dataset /path/to/b8-table-wipe-expert-v1 --out /path/to/new_b8_run --cache-dir /path/to/frame_cache --steps 20000 --device cuda:0 +``` + +Run these commands from the downloaded model repository root. The launcher resolves its source +and state relative to itself, copies the state to a new output directory and starts the frozen +trainer with `--resume`; add `--dry-run` to inspect without copying or training. +Confirm the trainer logs `resumed` at step 6,000. `--steps` means total steps, not additional steps; +use 16000 for 10,000 more steps from this checkpoint. Existing output state is never overwritten. +No training automatically starts when this repository is downloaded. + +Batch 512, seed 0, state44/action76, 16-row chunks, 168x224 head/chest RGB, hb-v2 augmentation, +500-step original warm-up, AdamW and EMA are unchanged. At the default total of 20,000, +the saved LR/scheduler continues consistently. Choosing a different total changes cosine decay +from the next scheduler update. Original RNG and DataLoader state were not recorded; this is +not a bit-exact continuation. Offline MSE uses 256 training frames, not held-out validation +or closed-loop success. Source snapshot: `code/fmdit_continue_20261117/` (original commit `028df1bb7024fb1e3d176de7e60c84d3333cd97f`). diff --git a/b8-table-wipe/b512-cont-20261117/SHA256SUMS b/b8-table-wipe/b512-cont-20261117/SHA256SUMS new file mode 100644 index 0000000000000000000000000000000000000000..87cd38a9c0b4a146a604df4017da5e6ea01045de --- /dev/null +++ b/b8-table-wipe/b512-cont-20261117/SHA256SUMS @@ -0,0 +1,28 @@ +4aa90cf211b68317b81112a0c06950fb8a463bdca6730235e65e3eff8f8a7109 b8-table-wipe/b512-cont-20261117/RESUME.md +5dda5685e9d78c71c5bbd6431ce8201dfddd0f392db514f82e7b61d0befc8889 b8-table-wipe/b512-cont-20261117/logs/parent_train_log.jsonl +9ef1481dbfb382d3ca1277eb0988fe18212734ab948ebbadc9d52dc384f366e9 b8-table-wipe/b512-cont-20261117/logs/train_log.jsonl +97f34e2086db3604f836f41881111fae411c385d253a0c48fe6b58472a7b9f45 b8-table-wipe/b512-cont-20261117/logs/train_stdout.log +47ea3ef34090e5f3ebac9cf47be4361c4548ace3e1380befecbbf87212858d11 b8-table-wipe/b512-cont-20261117/provenance/parent_checkpoint_config.json +b9e9013369f8b288ab9ca65652339796afdc817eaa42803fd49af7399975ec35 b8-table-wipe/b512-cont-20261117/provenance/parent_checkpoint_protocol.json +7aeb0b1f1ec0512dbae9b56e53397172acdb4ed5771d893e05bb5760d3cf7a4e b8-table-wipe/b512-cont-20261117/provenance/parent_run.json +73e8644e1a219adcaf7c03129622fd31500a8ede285dc52d34eab76b8b4828c1 b8-table-wipe/b512-cont-20261117/resume.json +49a658077102eda30b4220f93ffbe6a8c8230d3a3a02eb9720b6f725ab3bfdc8 b8-table-wipe/b512-cont-20261117/run.json +9005976e6e78ac4658569fc6e10074a357e8ba257894f0db755a69f2531e8870 b8-table-wipe/b512-cont-20261117/step-006000/config.json +49f877e3a020061ca3fd91b94f178596a9348d02548e116ea261eeecb5675ed8 b8-table-wipe/b512-cont-20261117/step-006000/model.pt +7f3c8de81798a4a0626d88e50d8448736f72bd100410c995ad9fda7cf35d6db8 b8-table-wipe/b512-cont-20261117/step-006000/protocol.json +defddc0d62576449aad58f234a7023e3e21eb0289831e03dac2c19ffd888f097 b8-table-wipe/b512-cont-20261117/train_state.pt +f7cf756cb60f0ce1da9b7ab8b1549b42f9a59d608bb06addd206d270b7c2d14c code/fmdit_continue_20261117/augmentation/__init__.py +96539b2fcf729917f668c9a776a0526c4b109611bf921dcbd753983c0f5813f2 code/fmdit_continue_20261117/augmentation/check_spec.py +9f7713cea4f846efcfc924234267ced48210357f419d67915c10b504316bedde code/fmdit_continue_20261117/augmentation/hb_aug_v2.py +815562186d3aaf87aba05085da6e951bc48b0b90a5d1c4910f2a3f625009bc52 code/fmdit_continue_20261117/augmentation/preview.py +0a8ee3ccab6e183804c08066a9e7eb207d9b13ca38c0e596ce5d58c8302d7929 code/fmdit_continue_20261117/baselines/__init__.py +3db95cd8e4e436875e11f1054a99d250ca7d94e0127909090b58730402340f19 code/fmdit_continue_20261117/baselines/act_nets.py +b537b03bc538910691b599862138494c11354147785a779a22ad83ff4b72a8d5 code/fmdit_continue_20261117/baselines/adapters.py +88c1971e18e9842136884f6a369468fe27f189b888b39870622304101a063b06 code/fmdit_continue_20261117/baselines/data.py +38a66937bb92393a3f1ce0644b249da28d582a4fd03d134b7743e6037abd88b9 code/fmdit_continue_20261117/baselines/nets.py +750d4c6193aabcacc0f53df26c2c51f64b3f836b940b32ccb5c69f5959419afa code/fmdit_continue_20261117/baselines/policies.py +bb211f66038e53efd08ee3ec03e133d8bb206ec36cc608d349b4fad3a61e874b code/fmdit_continue_20261117/baselines/requirements.txt +5d680d59124ca83f9c2088f8e1738395b87ddc58798acf7815e163102738b7ae code/fmdit_continue_20261117/baselines/runtime.py +b368d14aaab69d04f5372b1f3bb358c2a78f83134cc0e926faa8887e3ddbe297 code/fmdit_continue_20261117/baselines/train.py +fe9aaed24789080f05ce8f510d5a132bbd65e7c39b6de20500210742f0be4185 code/fmdit_continue_20261117/docs/augmentation_v2.md +5dbc2ec3f4f0638181eef9dca07c76f3594bdd5a24141f1a4b50d6532dbe4bba code/fmdit_continue_20261117/resume.py diff --git a/b8-table-wipe/b512-cont-20261117/logs/parent_train_log.jsonl b/b8-table-wipe/b512-cont-20261117/logs/parent_train_log.jsonl new file mode 100644 index 0000000000000000000000000000000000000000..04cbdb66b32d66ed417bb3f4b73ee9067fc633dc --- /dev/null +++ b/b8-table-wipe/b512-cont-20261117/logs/parent_train_log.jsonl @@ -0,0 +1,41 @@ +{"event": "train", "step": 100, "lr": 2.0200000000000003e-05, "backbone_lr": 2.02e-06, "ema_decay": 0.968612, "it_s": 0.39, "data_frac": 0.912, "fm": 1.8642640209197998, "loss": 1.8642640209197998} +{"event": "train", "step": 200, "lr": 4.02e-05, "backbone_lr": 4.0200000000000005e-06, "ema_decay": 0.981267, "it_s": 0.43, "data_frac": 0.913, "fm": 1.0814206337928771, "loss": 1.0814206337928771} +{"event": "train", "step": 300, "lr": 6.02e-05, "backbone_lr": 6.02e-06, "ema_decay": 0.986162, "it_s": 0.45, "data_frac": 0.907, "fm": 0.7938938274979591, "loss": 0.7938938274979591} +{"event": "train", "step": 400, "lr": 8.020000000000001e-05, "backbone_lr": 8.020000000000001e-06, "ema_decay": 0.988841, "it_s": 0.41, "data_frac": 0.915, "fm": 0.322593115568161, "loss": 0.322593115568161} +{"event": "train", "step": 500, "lr": 0.0001, "backbone_lr": 1e-05, "ema_decay": 0.990557, "it_s": 0.4, "data_frac": 0.92, "fm": 0.2253869031369686, "loss": 0.2253869031369686} +{"event": "train", "step": 600, "lr": 9.979871469976196e-05, "backbone_lr": 9.979871469976197e-06, "ema_decay": 0.991762, "it_s": 0.43, "data_frac": 0.912, "fm": 0.19886314541101455, "loss": 0.19886314541101455} +{"event": "train", "step": 700, "lr": 9.919647942993148e-05, "backbone_lr": 9.91964794299315e-06, "ema_decay": 0.99266, "it_s": 0.4, "data_frac": 0.918, "fm": 0.18408803522586822, "loss": 0.18408803522586822} +{"event": "train", "step": 800, "lr": 9.819814303479267e-05, "backbone_lr": 9.819814303479268e-06, "ema_decay": 0.993358, "it_s": 0.41, "data_frac": 0.916, "fm": 0.17290466755628586, "loss": 0.17290466755628586} +{"event": "train", "step": 900, "lr": 9.681174353198687e-05, "backbone_lr": 9.681174353198687e-06, "ema_decay": 0.993919, "it_s": 0.44, "data_frac": 0.909, "fm": 0.16579829320311545, "loss": 0.16579829320311545} +{"event": "train", "step": 1000, "lr": 9.504844339512095e-05, "backbone_lr": 9.504844339512096e-06, "ema_decay": 0.994381, "it_s": 0.4, "data_frac": 0.918, "fm": 0.15981068179011346, "loss": 0.15981068179011346} +{"event": "train", "step": 1100, "lr": 9.292243968009331e-05, "backbone_lr": 9.292243968009332e-06, "ema_decay": 0.994768, "it_s": 0.42, "data_frac": 0.913, "fm": 0.15560077399015426, "loss": 0.15560077399015426} +{"event": "train", "step": 1200, "lr": 9.045084971874738e-05, "backbone_lr": 9.045084971874738e-06, "ema_decay": 0.995098, "it_s": 0.44, "data_frac": 0.909, "fm": 0.15098393872380256, "loss": 0.15098393872380256} +{"event": "train", "step": 1300, "lr": 8.765357330018056e-05, "backbone_lr": 8.765357330018056e-06, "ema_decay": 0.995384, "it_s": 0.37, "data_frac": 0.926, "fm": 0.14670720651745797, "loss": 0.14670720651745797} +{"event": "train", "step": 1400, "lr": 8.455313244934324e-05, "backbone_lr": 8.455313244934324e-06, "ema_decay": 0.995633, "it_s": 0.47, "data_frac": 0.904, "fm": 0.14375186525285244, "loss": 0.14375186525285244} +{"event": "train", "step": 1500, "lr": 8.117449009293668e-05, "backbone_lr": 8.117449009293668e-06, "ema_decay": 0.995853, "it_s": 0.42, "data_frac": 0.914, "fm": 0.14028333067893983, "loss": 0.14028333067893983} +{"event": "train", "step": 1600, "lr": 7.754484907260513e-05, "backbone_lr": 7.754484907260513e-06, "ema_decay": 0.996049, "it_s": 0.39, "data_frac": 0.921, "fm": 0.1390533269941807, "loss": 0.1390533269941807} +{"event": "train", "step": 1700, "lr": 7.369343312364993e-05, "backbone_lr": 7.369343312364994e-06, "ema_decay": 0.996225, "it_s": 0.48, "data_frac": 0.901, "fm": 0.1383609838038683, "loss": 0.1383609838038683} +{"event": "train", "step": 1800, "lr": 6.965125158269619e-05, "backbone_lr": 6.965125158269619e-06, "ema_decay": 0.996383, "it_s": 0.45, "data_frac": 0.908, "fm": 0.1336882469058037, "loss": 0.1336882469058037} +{"event": "train", "step": 1900, "lr": 6.545084971874738e-05, "backbone_lr": 6.545084971874738e-06, "ema_decay": 0.996527, "it_s": 0.38, "data_frac": 0.921, "fm": 0.13274769395589828, "loss": 0.13274769395589828} +{"event": "train", "step": 2000, "lr": 6.112604669781572e-05, "backbone_lr": 6.112604669781572e-06, "ema_decay": 0.996658, "it_s": 0.44, "data_frac": 0.908, "fm": 0.1316966664791107, "loss": 0.1316966664791107} +{"event": "train", "step": 2100, "lr": 5.6711663290882776e-05, "backbone_lr": 5.671166329088278e-06, "ema_decay": 0.996778, "it_s": 0.4, "data_frac": 0.917, "fm": 0.12813001714646816, "loss": 0.12813001714646816} +{"event": "train", "step": 2200, "lr": 5.2243241517525754e-05, "backbone_lr": 5.224324151752575e-06, "ema_decay": 0.996888, "it_s": 0.39, "data_frac": 0.92, "fm": 0.12834149174392223, "loss": 0.12834149174392223} +{"event": "train", "step": 2300, "lr": 4.775675848247427e-05, "backbone_lr": 4.775675848247427e-06, "ema_decay": 0.99699, "it_s": 0.43, "data_frac": 0.912, "fm": 0.12588734790682793, "loss": 0.12588734790682793} +{"event": "train", "step": 2400, "lr": 4.328833670911724e-05, "backbone_lr": 4.3288336709117246e-06, "ema_decay": 0.997085, "it_s": 0.47, "data_frac": 0.905, "fm": 0.12559257812798022, "loss": 0.12559257812798022} +{"event": "train", "step": 2500, "lr": 3.887395330218429e-05, "backbone_lr": 3.887395330218429e-06, "ema_decay": 0.997172, "it_s": 0.42, "data_frac": 0.913, "fm": 0.124132684096694, "loss": 0.124132684096694} +{"event": "train", "step": 2600, "lr": 3.4549150281252636e-05, "backbone_lr": 3.4549150281252635e-06, "ema_decay": 0.997254, "it_s": 0.42, "data_frac": 0.915, "fm": 0.12321630582213401, "loss": 0.12321630582213401} +{"event": "train", "step": 2700, "lr": 3.0348748417303823e-05, "backbone_lr": 3.0348748417303826e-06, "ema_decay": 0.997331, "it_s": 0.46, "data_frac": 0.906, "fm": 0.12238777838647366, "loss": 0.12238777838647366} +{"event": "train", "step": 2800, "lr": 2.630656687635007e-05, "backbone_lr": 2.6306566876350072e-06, "ema_decay": 0.997403, "it_s": 0.37, "data_frac": 0.924, "fm": 0.12237035527825356, "loss": 0.12237035527825356} +{"event": "train", "step": 2900, "lr": 2.245515092739488e-05, "backbone_lr": 2.245515092739488e-06, "ema_decay": 0.99747, "it_s": 0.47, "data_frac": 0.901, "fm": 0.12099793791770935, "loss": 0.12099793791770935} +{"event": "train", "step": 3000, "lr": 1.8825509907063327e-05, "backbone_lr": 1.8825509907063328e-06, "ema_decay": 0.997534, "it_s": 0.45, "data_frac": 0.908, "fm": 0.12019743457436562, "loss": 0.12019743457436562} +{"event": "train", "step": 3100, "lr": 1.544686755065677e-05, "backbone_lr": 1.544686755065677e-06, "ema_decay": 0.997594, "it_s": 0.4, "data_frac": 0.919, "fm": 0.11983832776546478, "loss": 0.11983832776546478} +{"event": "train", "step": 3200, "lr": 1.2346426699819458e-05, "backbone_lr": 1.234642669981946e-06, "ema_decay": 0.99765, "it_s": 0.44, "data_frac": 0.907, "fm": 0.11914024040102959, "loss": 0.11914024040102959} +{"event": "train", "step": 3300, "lr": 9.549150281252633e-06, "backbone_lr": 9.549150281252633e-07, "ema_decay": 0.997704, "it_s": 0.39, "data_frac": 0.918, "fm": 0.11888715997338295, "loss": 0.11888715997338295} +{"event": "train", "step": 3400, "lr": 7.077560319906695e-06, "backbone_lr": 7.077560319906696e-07, "ema_decay": 0.997755, "it_s": 0.4, "data_frac": 0.919, "fm": 0.11843070551753045, "loss": 0.11843070551753045} +{"event": "train", "step": 3500, "lr": 4.951556604879048e-06, "backbone_lr": 4.951556604879049e-07, "ema_decay": 0.997803, "it_s": 0.45, "data_frac": 0.908, "fm": 0.11840196907520294, "loss": 0.11840196907520294} +{"event": "train", "step": 3600, "lr": 3.18825646801314e-06, "backbone_lr": 3.18825646801314e-07, "ema_decay": 0.997849, "it_s": 0.43, "data_frac": 0.913, "fm": 0.11895469002425671, "loss": 0.11895469002425671} +{"event": "train", "step": 3700, "lr": 1.8018569652073381e-06, "backbone_lr": 1.801856965207338e-07, "ema_decay": 0.997893, "it_s": 0.41, "data_frac": 0.916, "fm": 0.11923333063721657, "loss": 0.11923333063721657} +{"event": "train", "step": 3800, "lr": 8.035205700685167e-07, "backbone_lr": 8.035205700685167e-08, "ema_decay": 0.997934, "it_s": 0.42, "data_frac": 0.913, "fm": 0.11670800402760506, "loss": 0.11670800402760506} +{"event": "train", "step": 3900, "lr": 2.012853002380466e-07, "backbone_lr": 2.012853002380466e-08, "ema_decay": 0.997974, "it_s": 0.41, "data_frac": 0.911, "fm": 0.11914860650897026, "loss": 0.11914860650897026} +{"event": "train", "step": 4000, "lr": 0.0, "backbone_lr": 0.0, "ema_decay": 0.998012, "it_s": 0.44, "data_frac": 0.91, "fm": 0.11686161890625954, "loss": 0.11686161890625954} +{"event": "save", "step": 4000, "checkpoint": "/mnt/data0/zuo/Humanoid_benchmark/results/baselines_preflight_20261117/runs/b8-table-wipe-expert-v1/fm_dit/checkpoints/step-004000", "mse": 0.02690730057656765, "mse_row0": 0.024344196543097496, "mse_motion_token": 0.027465982362627983, "mse_inspire_left_hand_state": 0.01448493916541338, "mse_inspire_right_hand_state": 0.03337034583091736} diff --git a/b8-table-wipe/b512-cont-20261117/logs/train_log.jsonl b/b8-table-wipe/b512-cont-20261117/logs/train_log.jsonl new file mode 100644 index 0000000000000000000000000000000000000000..7baa911cb328e2999709b4118d367d91cb9da6f6 --- /dev/null +++ b/b8-table-wipe/b512-cont-20261117/logs/train_log.jsonl @@ -0,0 +1,109 @@ +{"event": "train", "step": 4025, "lr": 9.215152885383474e-05, "backbone_lr": 9.215152885383473e-06, "ema_decay": 0.998021, "it_s": 0.62, "data_frac": 0.848, "fm": 0.28632881194353105, "loss": 0.28632881194353105} +{"event": "train", "step": 4050, "lr": 9.204286963963111e-05, "backbone_lr": 9.204286963963112e-06, "ema_decay": 0.998031, "it_s": 0.98, "data_frac": 0.797, "fm": 0.19046931803226472, "loss": 0.19046931803226472} +{"event": "train", "step": 4075, "lr": 9.193352839727121e-05, "backbone_lr": 9.193352839727122e-06, "ema_decay": 0.99804, "it_s": 1.63, "data_frac": 0.659, "fm": 0.15901459693908693, "loss": 0.15901459693908693} +{"event": "train", "step": 4100, "lr": 9.182350690051133e-05, "backbone_lr": 9.182350690051134e-06, "ema_decay": 0.998049, "it_s": 1.4, "data_frac": 0.731, "fm": 0.1406265354156494, "loss": 0.1406265354156494} +{"event": "train", "step": 4125, "lr": 9.171280693414307e-05, "backbone_lr": 9.171280693414307e-06, "ema_decay": 0.998058, "it_s": 0.55, "data_frac": 0.892, "fm": 0.13139946579933168, "loss": 0.13139946579933168} +{"event": "train", "step": 4150, "lr": 9.160143029396422e-05, "backbone_lr": 9.160143029396422e-06, "ema_decay": 0.998066, "it_s": 5.01, "data_frac": 0.004, "fm": 0.12817200630903244, "loss": 0.12817200630903244} +{"event": "train", "step": 4175, "lr": 9.148937878674976e-05, "backbone_lr": 9.148937878674975e-06, "ema_decay": 0.998075, "it_s": 0.85, "data_frac": 0.824, "fm": 0.1271376582980156, "loss": 0.1271376582980156} +{"event": "train", "step": 4200, "lr": 9.13766542302225e-05, "backbone_lr": 9.13766542302225e-06, "ema_decay": 0.998084, "it_s": 1.55, "data_frac": 0.678, "fm": 0.12697737753391267, "loss": 0.12697737753391267} +{"event": "train", "step": 4225, "lr": 9.126325845302357e-05, "backbone_lr": 9.126325845302357e-06, "ema_decay": 0.998092, "it_s": 1.53, "data_frac": 0.703, "fm": 0.12229053348302842, "loss": 0.12229053348302842} +{"event": "train", "step": 4250, "lr": 9.114919329468282e-05, "backbone_lr": 9.114919329468283e-06, "ema_decay": 0.998101, "it_s": 0.76, "data_frac": 0.843, "fm": 0.12151277899742126, "loss": 0.12151277899742126} +{"event": "train", "step": 4275, "lr": 9.103446060558892e-05, "backbone_lr": 9.103446060558892e-06, "ema_decay": 0.998109, "it_s": 0.89, "data_frac": 0.811, "fm": 0.1202454110980034, "loss": 0.1202454110980034} +{"event": "train", "step": 4300, "lr": 9.091906224695935e-05, "backbone_lr": 9.091906224695935e-06, "ema_decay": 0.998117, "it_s": 3.31, "data_frac": 0.319, "fm": 0.11945601761341094, "loss": 0.11945601761341094} +{"event": "train", "step": 4325, "lr": 9.080300009081024e-05, "backbone_lr": 9.080300009081025e-06, "ema_decay": 0.998125, "it_s": 1.45, "data_frac": 0.724, "fm": 0.11844889491796494, "loss": 0.11844889491796494} +{"event": "train", "step": 4350, "lr": 9.068627601992598e-05, "backbone_lr": 9.0686276019926e-06, "ema_decay": 0.998133, "it_s": 1.62, "data_frac": 0.625, "fm": 0.11838024020195008, "loss": 0.11838024020195008} +{"event": "train", "step": 4375, "lr": 9.056889192782866e-05, "backbone_lr": 9.056889192782865e-06, "ema_decay": 0.998141, "it_s": 0.7, "data_frac": 0.854, "fm": 0.11815809279680252, "loss": 0.11815809279680252} +{"event": "train", "step": 4400, "lr": 9.045084971874738e-05, "backbone_lr": 9.045084971874738e-06, "ema_decay": 0.998149, "it_s": 0.99, "data_frac": 0.789, "fm": 0.11837879121303559, "loss": 0.11837879121303559} +{"event": "train", "step": 4425, "lr": 9.033215130758735e-05, "backbone_lr": 9.033215130758735e-06, "ema_decay": 0.998157, "it_s": 2.8, "data_frac": 0.451, "fm": 0.12120903819799424, "loss": 0.12120903819799424} +{"event": "train", "step": 4450, "lr": 9.021279861989885e-05, "backbone_lr": 9.021279861989884e-06, "ema_decay": 0.998165, "it_s": 1.42, "data_frac": 0.734, "fm": 0.11654747396707535, "loss": 0.11654747396707535} +{"event": "train", "step": 4475, "lr": 9.009279359184593e-05, "backbone_lr": 9.009279359184594e-06, "ema_decay": 0.998173, "it_s": 1.35, "data_frac": 0.737, "fm": 0.11454789787530899, "loss": 0.11454789787530899} +{"event": "train", "step": 4500, "lr": 8.997213817017507e-05, "backbone_lr": 8.997213817017508e-06, "ema_decay": 0.99818, "it_s": 0.64, "data_frac": 0.866, "fm": 0.11652687758207321, "loss": 0.11652687758207321} +{"event": "train", "step": 4525, "lr": 8.985083431218354e-05, "backbone_lr": 8.985083431218354e-06, "ema_decay": 0.998188, "it_s": 1.09, "data_frac": 0.773, "fm": 0.11339226454496383, "loss": 0.11339226454496383} +{"event": "train", "step": 4550, "lr": 8.972888398568772e-05, "backbone_lr": 8.972888398568772e-06, "ema_decay": 0.998195, "it_s": 0.99, "data_frac": 0.785, "fm": 0.11623615086078644, "loss": 0.11623615086078644} +{"event": "train", "step": 4575, "lr": 8.96062891689911e-05, "backbone_lr": 8.96062891689911e-06, "ema_decay": 0.998203, "it_s": 4.97, "data_frac": 0.002, "fm": 0.11666365176439285, "loss": 0.11666365176439285} +{"event": "train", "step": 4600, "lr": 8.948305185085225e-05, "backbone_lr": 8.948305185085226e-06, "ema_decay": 0.99821, "it_s": 1.55, "data_frac": 0.667, "fm": 0.11618060439825058, "loss": 0.11618060439825058} +{"event": "train", "step": 4625, "lr": 8.935917403045251e-05, "backbone_lr": 8.935917403045251e-06, "ema_decay": 0.998217, "it_s": 0.66, "data_frac": 0.863, "fm": 0.11370467752218247, "loss": 0.11370467752218247} +{"event": "train", "step": 4650, "lr": 8.92346577173636e-05, "backbone_lr": 8.923465771736361e-06, "ema_decay": 0.998224, "it_s": 1.7, "data_frac": 0.667, "fm": 0.11478051245212555, "loss": 0.11478051245212555} +{"event": "train", "step": 4675, "lr": 8.910950493151498e-05, "backbone_lr": 8.910950493151498e-06, "ema_decay": 0.998232, "it_s": 0.91, "data_frac": 0.803, "fm": 0.11607470244169235, "loss": 0.11607470244169235} +{"event": "train", "step": 4700, "lr": 8.898371770316111e-05, "backbone_lr": 8.898371770316113e-06, "ema_decay": 0.998239, "it_s": 1.28, "data_frac": 0.743, "fm": 0.11513609379529953, "loss": 0.11513609379529953} +{"event": "train", "step": 4725, "lr": 8.885729807284856e-05, "backbone_lr": 8.885729807284855e-06, "ema_decay": 0.998246, "it_s": 0.81, "data_frac": 0.838, "fm": 0.11020497322082519, "loss": 0.11020497322082519} +{"event": "train", "step": 4750, "lr": 8.873024809138272e-05, "backbone_lr": 8.873024809138272e-06, "ema_decay": 0.998253, "it_s": 2.64, "data_frac": 0.417, "fm": 0.11362491935491562, "loss": 0.11362491935491562} +{"event": "train", "step": 4775, "lr": 8.860256981979484e-05, "backbone_lr": 8.860256981979485e-06, "ema_decay": 0.998259, "it_s": 1.31, "data_frac": 0.75, "fm": 0.11324429392814636, "loss": 0.11324429392814636} +{"event": "train", "step": 4800, "lr": 8.847426532930831e-05, "backbone_lr": 8.84742653293083e-06, "ema_decay": 0.998266, "it_s": 1.3, "data_frac": 0.728, "fm": 0.11362754851579666, "loss": 0.11362754851579666} +{"event": "train", "step": 4825, "lr": 8.834533670130518e-05, "backbone_lr": 8.834533670130518e-06, "ema_decay": 0.998273, "it_s": 1.08, "data_frac": 0.775, "fm": 0.11129748165607452, "loss": 0.11129748165607452} +{"event": "train", "step": 4850, "lr": 8.821578602729242e-05, "backbone_lr": 8.821578602729242e-06, "ema_decay": 0.99828, "it_s": 0.83, "data_frac": 0.84, "fm": 0.1117265647649765, "loss": 0.1117265647649765} +{"event": "train", "step": 4875, "lr": 8.808561540886796e-05, "backbone_lr": 8.808561540886796e-06, "ema_decay": 0.998286, "it_s": 1.82, "data_frac": 0.59, "fm": 0.11203957498073577, "loss": 0.11203957498073577} +{"event": "train", "step": 4900, "lr": 8.795482695768658e-05, "backbone_lr": 8.795482695768658e-06, "ema_decay": 0.998293, "it_s": 1.36, "data_frac": 0.734, "fm": 0.10931766182184219, "loss": 0.10931766182184219} +{"event": "train", "step": 4925, "lr": 8.782342279542568e-05, "backbone_lr": 8.782342279542569e-06, "ema_decay": 0.998299, "it_s": 1.05, "data_frac": 0.791, "fm": 0.10985626131296158, "loss": 0.10985626131296158} +{"event": "train", "step": 4950, "lr": 8.769140505375085e-05, "backbone_lr": 8.769140505375084e-06, "ema_decay": 0.998306, "it_s": 1.07, "data_frac": 0.775, "fm": 0.1120401856303215, "loss": 0.1120401856303215} +{"event": "train", "step": 4975, "lr": 8.755877587428128e-05, "backbone_lr": 8.755877587428127e-06, "ema_decay": 0.998312, "it_s": 1.09, "data_frac": 0.791, "fm": 0.10802255183458329, "loss": 0.10802255183458329} +{"event": "train", "step": 5000, "lr": 8.742553740855506e-05, "backbone_lr": 8.742553740855507e-06, "ema_decay": 0.998318, "it_s": 1.12, "data_frac": 0.775, "fm": 0.11093634516000747, "loss": 0.11093634516000747} +{"event": "train", "step": 5025, "lr": 8.729169181799422e-05, "backbone_lr": 8.729169181799423e-06, "ema_decay": 0.998325, "it_s": 1.11, "data_frac": 0.766, "fm": 0.1084786233305931, "loss": 0.1084786233305931} +{"event": "train", "step": 5050, "lr": 8.715724127386972e-05, "backbone_lr": 8.715724127386971e-06, "ema_decay": 0.998331, "it_s": 2.0, "data_frac": 0.604, "fm": 0.10712229013442993, "loss": 0.10712229013442993} +{"event": "train", "step": 5075, "lr": 8.702218795726617e-05, "backbone_lr": 8.702218795726619e-06, "ema_decay": 0.998337, "it_s": 1.18, "data_frac": 0.735, "fm": 0.11033450186252594, "loss": 0.11033450186252594} +{"event": "train", "step": 5100, "lr": 8.688653405904652e-05, "backbone_lr": 8.688653405904653e-06, "ema_decay": 0.998343, "it_s": 1.02, "data_frac": 0.808, "fm": 0.11009035259485245, "loss": 0.11009035259485245} +{"event": "train", "step": 5125, "lr": 8.675028177981643e-05, "backbone_lr": 8.675028177981643e-06, "ema_decay": 0.998349, "it_s": 0.92, "data_frac": 0.807, "fm": 0.10828320592641831, "loss": 0.10828320592641831} +{"event": "train", "step": 5150, "lr": 8.661343332988869e-05, "backbone_lr": 8.661343332988869e-06, "ema_decay": 0.998355, "it_s": 0.88, "data_frac": 0.817, "fm": 0.10717267870903015, "loss": 0.10717267870903015} +{"event": "train", "step": 5175, "lr": 8.647599092924721e-05, "backbone_lr": 8.64759909292472e-06, "ema_decay": 0.998361, "it_s": 5.38, "data_frac": 0.001, "fm": 0.10801177144050599, "loss": 0.10801177144050599} +{"event": "train", "step": 5200, "lr": 8.633795680751116e-05, "backbone_lr": 8.633795680751116e-06, "ema_decay": 0.998367, "it_s": 1.56, "data_frac": 0.689, "fm": 0.105967099070549, "loss": 0.105967099070549} +{"event": "train", "step": 5225, "lr": 8.619933320389872e-05, "backbone_lr": 8.619933320389872e-06, "ema_decay": 0.998373, "it_s": 0.9, "data_frac": 0.828, "fm": 0.10755074709653854, "loss": 0.10755074709653854} +{"event": "train", "step": 5250, "lr": 8.606012236719073e-05, "backbone_lr": 8.606012236719073e-06, "ema_decay": 0.998379, "it_s": 0.91, "data_frac": 0.814, "fm": 0.10677639991044999, "loss": 0.10677639991044999} +{"event": "train", "step": 5275, "lr": 8.592032655569429e-05, "backbone_lr": 8.59203265556943e-06, "ema_decay": 0.998385, "it_s": 0.96, "data_frac": 0.803, "fm": 0.10488777875900268, "loss": 0.10488777875900268} +{"event": "train", "step": 5300, "lr": 8.577994803720606e-05, "backbone_lr": 8.577994803720605e-06, "ema_decay": 0.99839, "it_s": 1.74, "data_frac": 0.664, "fm": 0.10423666447401046, "loss": 0.10423666447401046} +{"event": "train", "step": 5325, "lr": 8.56389890889755e-05, "backbone_lr": 8.56389890889755e-06, "ema_decay": 0.998396, "it_s": 0.96, "data_frac": 0.819, "fm": 0.10578320920467377, "loss": 0.10578320920467377} +{"event": "train", "step": 5350, "lr": 8.549745199766792e-05, "backbone_lr": 8.549745199766792e-06, "ema_decay": 0.998402, "it_s": 3.47, "data_frac": 0.307, "fm": 0.10501939445734024, "loss": 0.10501939445734024} +{"event": "train", "step": 5375, "lr": 8.535533905932738e-05, "backbone_lr": 8.535533905932739e-06, "ema_decay": 0.998407, "it_s": 0.92, "data_frac": 0.805, "fm": 0.10691273421049118, "loss": 0.10691273421049118} +{"event": "train", "step": 5400, "lr": 8.521265257933948e-05, "backbone_lr": 8.521265257933948e-06, "ema_decay": 0.998413, "it_s": 0.96, "data_frac": 0.802, "fm": 0.1036634960770607, "loss": 0.1036634960770607} +{"event": "train", "step": 5425, "lr": 8.506939487239388e-05, "backbone_lr": 8.506939487239389e-06, "ema_decay": 0.998418, "it_s": 1.88, "data_frac": 0.638, "fm": 0.10052153080701828, "loss": 0.10052153080701828} +{"event": "train", "step": 5450, "lr": 8.492556826244687e-05, "backbone_lr": 8.492556826244687e-06, "ema_decay": 0.998424, "it_s": 0.75, "data_frac": 0.856, "fm": 0.1060214227437973, "loss": 0.1060214227437973} +{"event": "train", "step": 5475, "lr": 8.47811750826835e-05, "backbone_lr": 8.478117508268352e-06, "ema_decay": 0.998429, "it_s": 1.01, "data_frac": 0.796, "fm": 0.10572416603565216, "loss": 0.10572416603565216} +{"event": "train", "step": 5500, "lr": 8.463621767547998e-05, "backbone_lr": 8.463621767547998e-06, "ema_decay": 0.998434, "it_s": 4.74, "data_frac": 0.006, "fm": 0.10358604192733764, "loss": 0.10358604192733764} +{"event": "train", "step": 5525, "lr": 8.449069839236538e-05, "backbone_lr": 8.44906983923654e-06, "ema_decay": 0.99844, "it_s": 0.95, "data_frac": 0.808, "fm": 0.10235648989677429, "loss": 0.10235648989677429} +{"event": "train", "step": 5550, "lr": 8.434461959398376e-05, "backbone_lr": 8.434461959398377e-06, "ema_decay": 0.998445, "it_s": 1.71, "data_frac": 0.674, "fm": 0.10302803814411163, "loss": 0.10302803814411163} +{"event": "train", "step": 5575, "lr": 8.419798365005569e-05, "backbone_lr": 8.419798365005569e-06, "ema_decay": 0.99845, "it_s": 0.89, "data_frac": 0.826, "fm": 0.1070358219742775, "loss": 0.1070358219742775} +{"event": "train", "step": 5600, "lr": 8.405079293933986e-05, "backbone_lr": 8.405079293933986e-06, "ema_decay": 0.998455, "it_s": 0.81, "data_frac": 0.833, "fm": 0.10346478521823883, "loss": 0.10346478521823883} +{"event": "train", "step": 5625, "lr": 8.390304984959454e-05, "backbone_lr": 8.390304984959455e-06, "ema_decay": 0.998461, "it_s": 1.17, "data_frac": 0.741, "fm": 0.10347686499357224, "loss": 0.10347686499357224} +{"event": "train", "step": 5650, "lr": 8.375475677753881e-05, "backbone_lr": 8.375475677753882e-06, "ema_decay": 0.998466, "it_s": 4.63, "data_frac": 0.087, "fm": 0.10413286983966827, "loss": 0.10413286983966827} +{"event": "train", "step": 5675, "lr": 8.360591612881364e-05, "backbone_lr": 8.360591612881363e-06, "ema_decay": 0.998471, "it_s": 1.7, "data_frac": 0.684, "fm": 0.10362743228673935, "loss": 0.10362743228673935} +{"event": "train", "step": 5700, "lr": 8.345653031794292e-05, "backbone_lr": 8.345653031794292e-06, "ema_decay": 0.998476, "it_s": 0.95, "data_frac": 0.79, "fm": 0.10708901762962342, "loss": 0.10708901762962342} +{"event": "train", "step": 5725, "lr": 8.330660176829428e-05, "backbone_lr": 8.330660176829427e-06, "ema_decay": 0.998481, "it_s": 0.81, "data_frac": 0.813, "fm": 0.10352396816015244, "loss": 0.10352396816015244} +{"event": "train", "step": 5750, "lr": 8.315613291203976e-05, "backbone_lr": 8.315613291203977e-06, "ema_decay": 0.998486, "it_s": 1.12, "data_frac": 0.773, "fm": 0.1011051020026207, "loss": 0.1011051020026207} +{"event": "train", "step": 5775, "lr": 8.30051261901164e-05, "backbone_lr": 8.30051261901164e-06, "ema_decay": 0.998491, "it_s": 4.2, "data_frac": 0.206, "fm": 0.09954795092344285, "loss": 0.09954795092344285} +{"event": "train", "step": 5800, "lr": 8.285358405218655e-05, "backbone_lr": 8.285358405218655e-06, "ema_decay": 0.998496, "it_s": 1.92, "data_frac": 0.64, "fm": 0.10256652504205704, "loss": 0.10256652504205704} +{"event": "train", "step": 5825, "lr": 8.270150895659823e-05, "backbone_lr": 8.270150895659824e-06, "ema_decay": 0.9985, "it_s": 0.9, "data_frac": 0.82, "fm": 0.10183177888393402, "loss": 0.10183177888393402} +{"event": "train", "step": 5850, "lr": 8.25489033703452e-05, "backbone_lr": 8.25489033703452e-06, "ema_decay": 0.998505, "it_s": 1.12, "data_frac": 0.767, "fm": 0.1011198627948761, "loss": 0.1011198627948761} +{"event": "train", "step": 5875, "lr": 8.239576976902695e-05, "backbone_lr": 8.239576976902694e-06, "ema_decay": 0.99851, "it_s": 0.87, "data_frac": 0.806, "fm": 0.10307048767805099, "loss": 0.10307048767805099} +{"event": "train", "step": 5900, "lr": 8.224211063680853e-05, "backbone_lr": 8.224211063680854e-06, "ema_decay": 0.998515, "it_s": 1.49, "data_frac": 0.71, "fm": 0.10266599476337433, "loss": 0.10266599476337433} +{"event": "train", "step": 5925, "lr": 8.208792846638029e-05, "backbone_lr": 8.20879284663803e-06, "ema_decay": 0.998519, "it_s": 0.96, "data_frac": 0.818, "fm": 0.10186379879713059, "loss": 0.10186379879713059} +{"event": "train", "step": 5950, "lr": 8.19332257589174e-05, "backbone_lr": 8.19332257589174e-06, "ema_decay": 0.998524, "it_s": 1.88, "data_frac": 0.615, "fm": 0.1005827847123146, "loss": 0.1005827847123146} +{"event": "train", "step": 5975, "lr": 8.177800502403928e-05, "backbone_lr": 8.177800502403928e-06, "ema_decay": 0.998529, "it_s": 1.06, "data_frac": 0.788, "fm": 0.10216392815113068, "loss": 0.10216392815113068} +{"event": "train", "step": 6000, "lr": 8.162226877976887e-05, "backbone_lr": 8.162226877976886e-06, "ema_decay": 0.998533, "it_s": 1.14, "data_frac": 0.762, "fm": 0.10421739488840104, "loss": 0.10421739488840104} +{"event": "save", "step": 6000, "checkpoint": "/mnt/data0/zuo/Humanoid_benchmark/results/fmdit_continue_20261117_01/runs/b8/fm_dit/checkpoints/step-006000", "mse": 0.022521788254380226, "mse_row0": 0.02053411304950714, "mse_motion_token": 0.022982770577073097, "mse_inspire_left_hand_state": 0.012518338859081268, "mse_inspire_right_hand_state": 0.027608102187514305} +{"event": "train", "step": 6025, "lr": 8.146601955249188e-05, "backbone_lr": 8.146601955249187e-06, "ema_decay": 0.998538, "it_s": 1.62, "data_frac": 0.424, "fm": 0.1021818333864212, "loss": 0.1021818333864212} +{"event": "train", "step": 6050, "lr": 8.130925987691569e-05, "backbone_lr": 8.13092598769157e-06, "ema_decay": 0.998542, "it_s": 0.7, "data_frac": 0.862, "fm": 0.10002265453338623, "loss": 0.10002265453338623} +{"event": "train", "step": 6075, "lr": 8.11519922960283e-05, "backbone_lr": 8.115199229602831e-06, "ema_decay": 0.998547, "it_s": 0.97, "data_frac": 0.789, "fm": 0.09909536123275757, "loss": 0.09909536123275757} +{"event": "train", "step": 6100, "lr": 8.099421936105702e-05, "backbone_lr": 8.099421936105702e-06, "ema_decay": 0.998551, "it_s": 2.88, "data_frac": 0.438, "fm": 0.09719828993082047, "loss": 0.09719828993082047} +{"event": "train", "step": 6125, "lr": 8.083594363142717e-05, "backbone_lr": 8.083594363142717e-06, "ema_decay": 0.998556, "it_s": 1.02, "data_frac": 0.787, "fm": 0.10099037319421768, "loss": 0.10099037319421768} +{"event": "train", "step": 6150, "lr": 8.067716767472045e-05, "backbone_lr": 8.067716767472045e-06, "ema_decay": 0.99856, "it_s": 1.79, "data_frac": 0.64, "fm": 0.09931565254926682, "loss": 0.09931565254926682} +{"event": "train", "step": 6175, "lr": 8.051789406663342e-05, "backbone_lr": 8.051789406663342e-06, "ema_decay": 0.998565, "it_s": 0.86, "data_frac": 0.817, "fm": 0.10005048602819443, "loss": 0.10005048602819443} +{"event": "train", "step": 6200, "lr": 8.035812539093557e-05, "backbone_lr": 8.035812539093557e-06, "ema_decay": 0.998569, "it_s": 0.83, "data_frac": 0.825, "fm": 0.10188953727483749, "loss": 0.10188953727483749} +{"event": "train", "step": 6225, "lr": 8.019786423942752e-05, "backbone_lr": 8.019786423942752e-06, "ema_decay": 0.998573, "it_s": 3.74, "data_frac": 0.261, "fm": 0.10008735865354539, "loss": 0.10008735865354539} +{"event": "train", "step": 6250, "lr": 8.003711321189895e-05, "backbone_lr": 8.003711321189895e-06, "ema_decay": 0.998578, "it_s": 1.24, "data_frac": 0.72, "fm": 0.09857472032308578, "loss": 0.09857472032308578} +{"event": "train", "step": 6275, "lr": 7.987587491608637e-05, "backbone_lr": 7.987587491608636e-06, "ema_decay": 0.998582, "it_s": 1.69, "data_frac": 0.671, "fm": 0.09919695109128952, "loss": 0.09919695109128952} +{"event": "train", "step": 6300, "lr": 7.971415196763088e-05, "backbone_lr": 7.971415196763088e-06, "ema_decay": 0.998586, "it_s": 0.74, "data_frac": 0.854, "fm": 0.09775012910366059, "loss": 0.09775012910366059} +{"event": "train", "step": 6325, "lr": 7.95519469900357e-05, "backbone_lr": 7.95519469900357e-06, "ema_decay": 0.99859, "it_s": 0.81, "data_frac": 0.836, "fm": 0.09813733667135238, "loss": 0.09813733667135238} +{"event": "train", "step": 6350, "lr": 7.938926261462366e-05, "backbone_lr": 7.938926261462366e-06, "ema_decay": 0.998594, "it_s": 1.39, "data_frac": 0.723, "fm": 0.10025485634803771, "loss": 0.10025485634803771} +{"event": "train", "step": 6375, "lr": 7.922610148049445e-05, "backbone_lr": 7.922610148049445e-06, "ema_decay": 0.998599, "it_s": 3.94, "data_frac": 0.272, "fm": 0.09809325218200683, "loss": 0.09809325218200683} +{"event": "train", "step": 6400, "lr": 7.906246623448183e-05, "backbone_lr": 7.906246623448184e-06, "ema_decay": 0.998603, "it_s": 1.85, "data_frac": 0.659, "fm": 0.09990756928920747, "loss": 0.09990756928920747} +{"event": "train", "step": 6425, "lr": 7.889835953111075e-05, "backbone_lr": 7.889835953111075e-06, "ema_decay": 0.998607, "it_s": 0.71, "data_frac": 0.861, "fm": 0.09899431318044663, "loss": 0.09899431318044663} +{"event": "train", "step": 6450, "lr": 7.873378403255419e-05, "backbone_lr": 7.87337840325542e-06, "ema_decay": 0.998611, "it_s": 0.82, "data_frac": 0.823, "fm": 0.09775156676769256, "loss": 0.09775156676769256} +{"event": "train", "step": 6475, "lr": 7.856874240859005e-05, "backbone_lr": 7.856874240859006e-06, "ema_decay": 0.998615, "it_s": 1.1, "data_frac": 0.775, "fm": 0.10037847697734832, "loss": 0.10037847697734832} +{"event": "train", "step": 6500, "lr": 7.840323733655778e-05, "backbone_lr": 7.84032373365578e-06, "ema_decay": 0.998619, "it_s": 2.59, "data_frac": 0.528, "fm": 0.09699027627706527, "loss": 0.09699027627706527} +{"event": "train", "step": 6525, "lr": 7.823727150131506e-05, "backbone_lr": 7.823727150131506e-06, "ema_decay": 0.998623, "it_s": 0.62, "data_frac": 0.876, "fm": 0.09724076330661774, "loss": 0.09724076330661774} +{"event": "train", "step": 6550, "lr": 7.807084759519405e-05, "backbone_lr": 7.807084759519405e-06, "ema_decay": 0.998627, "it_s": 4.35, "data_frac": 0.005, "fm": 0.0968179738521576, "loss": 0.0968179738521576} +{"event": "train", "step": 6575, "lr": 7.790396831795792e-05, "backbone_lr": 7.790396831795792e-06, "ema_decay": 0.998631, "it_s": 1.04, "data_frac": 0.793, "fm": 0.09870140582323074, "loss": 0.09870140582323074} +{"event": "train", "step": 6600, "lr": 7.773663637675694e-05, "backbone_lr": 7.773663637675695e-06, "ema_decay": 0.998634, "it_s": 0.98, "data_frac": 0.81, "fm": 0.09852319538593292, "loss": 0.09852319538593292} +{"event": "train", "step": 6625, "lr": 7.756885448608459e-05, "backbone_lr": 7.75688544860846e-06, "ema_decay": 0.998638, "it_s": 2.27, "data_frac": 0.572, "fm": 0.0948163315653801, "loss": 0.0948163315653801} +{"event": "train", "step": 6650, "lr": 7.740062536773352e-05, "backbone_lr": 7.740062536773352e-06, "ema_decay": 0.998642, "it_s": 0.64, "data_frac": 0.876, "fm": 0.0981231078505516, "loss": 0.0981231078505516} +{"event": "train", "step": 6675, "lr": 7.723195175075136e-05, "backbone_lr": 7.723195175075136e-06, "ema_decay": 0.998646, "it_s": 1.13, "data_frac": 0.767, "fm": 0.10084572613239288, "loss": 0.10084572613239288} +{"event": "train", "step": 6700, "lr": 7.706283637139658e-05, "backbone_lr": 7.706283637139658e-06, "ema_decay": 0.99865, "it_s": 3.77, "data_frac": 0.286, "fm": 0.09731092363595963, "loss": 0.09731092363595963} diff --git a/b8-table-wipe/b512-cont-20261117/logs/train_stdout.log b/b8-table-wipe/b512-cont-20261117/logs/train_stdout.log new file mode 100644 index 0000000000000000000000000000000000000000..f9cc64fd7e290877eae76f846fe49c5c82975a10 --- /dev/null +++ b/b8-table-wipe/b512-cont-20261117/logs/train_stdout.log @@ -0,0 +1,112 @@ +{"event": "preflight", "augmentation": "hb-aug-v2-20261109", "samples": 64, "state_dim": 44, "body_dims": 29, "images_changed": 64, "body_values_moved": 1.0, "max_body_offset_deg": 2.5819, "actions_bit_identical": true, "non_body_state_bit_identical": true, "eval_items_deterministic": 8} +{"event": "resumed", "step": 4000} +{"event": "start", "policy": "fm_dit", "params": 22540684, "backbone_params": 11166912, "backbone_weights": "IMAGENET1K_V1", "backbone_norm": "frozen_bn", "train_frames": 61498, "steps": 20000, "from_step": 4000, "cameras": ["ego_view", "chest"], "hw": [168, 224]} +{"event": "train", "step": 4025, "lr": 9.215152885383474e-05, "backbone_lr": 9.215152885383473e-06, "ema_decay": 0.998021, "it_s": 0.62, "data_frac": 0.848, "fm": 0.28632881194353105, "loss": 0.28632881194353105} +{"event": "train", "step": 4050, "lr": 9.204286963963111e-05, "backbone_lr": 9.204286963963112e-06, "ema_decay": 0.998031, "it_s": 0.98, "data_frac": 0.797, "fm": 0.19046931803226472, "loss": 0.19046931803226472} +{"event": "train", "step": 4075, "lr": 9.193352839727121e-05, "backbone_lr": 9.193352839727122e-06, "ema_decay": 0.99804, "it_s": 1.63, "data_frac": 0.659, "fm": 0.15901459693908693, "loss": 0.15901459693908693} +{"event": "train", "step": 4100, "lr": 9.182350690051133e-05, "backbone_lr": 9.182350690051134e-06, "ema_decay": 0.998049, "it_s": 1.4, "data_frac": 0.731, "fm": 0.1406265354156494, "loss": 0.1406265354156494} +{"event": "train", "step": 4125, "lr": 9.171280693414307e-05, "backbone_lr": 9.171280693414307e-06, "ema_decay": 0.998058, "it_s": 0.55, "data_frac": 0.892, "fm": 0.13139946579933168, "loss": 0.13139946579933168} +{"event": "train", "step": 4150, "lr": 9.160143029396422e-05, "backbone_lr": 9.160143029396422e-06, "ema_decay": 0.998066, "it_s": 5.01, "data_frac": 0.004, "fm": 0.12817200630903244, "loss": 0.12817200630903244} +{"event": "train", "step": 4175, "lr": 9.148937878674976e-05, "backbone_lr": 9.148937878674975e-06, "ema_decay": 0.998075, "it_s": 0.85, "data_frac": 0.824, "fm": 0.1271376582980156, "loss": 0.1271376582980156} +{"event": "train", "step": 4200, "lr": 9.13766542302225e-05, "backbone_lr": 9.13766542302225e-06, "ema_decay": 0.998084, "it_s": 1.55, "data_frac": 0.678, "fm": 0.12697737753391267, "loss": 0.12697737753391267} +{"event": "train", "step": 4225, "lr": 9.126325845302357e-05, "backbone_lr": 9.126325845302357e-06, "ema_decay": 0.998092, "it_s": 1.53, "data_frac": 0.703, "fm": 0.12229053348302842, "loss": 0.12229053348302842} +{"event": "train", "step": 4250, "lr": 9.114919329468282e-05, "backbone_lr": 9.114919329468283e-06, "ema_decay": 0.998101, "it_s": 0.76, "data_frac": 0.843, "fm": 0.12151277899742126, "loss": 0.12151277899742126} +{"event": "train", "step": 4275, "lr": 9.103446060558892e-05, "backbone_lr": 9.103446060558892e-06, "ema_decay": 0.998109, "it_s": 0.89, "data_frac": 0.811, "fm": 0.1202454110980034, "loss": 0.1202454110980034} +{"event": "train", "step": 4300, "lr": 9.091906224695935e-05, "backbone_lr": 9.091906224695935e-06, "ema_decay": 0.998117, "it_s": 3.31, "data_frac": 0.319, "fm": 0.11945601761341094, "loss": 0.11945601761341094} +{"event": "train", "step": 4325, "lr": 9.080300009081024e-05, "backbone_lr": 9.080300009081025e-06, "ema_decay": 0.998125, "it_s": 1.45, "data_frac": 0.724, "fm": 0.11844889491796494, "loss": 0.11844889491796494} +{"event": "train", "step": 4350, "lr": 9.068627601992598e-05, "backbone_lr": 9.0686276019926e-06, "ema_decay": 0.998133, "it_s": 1.62, "data_frac": 0.625, "fm": 0.11838024020195008, "loss": 0.11838024020195008} +{"event": "train", "step": 4375, "lr": 9.056889192782866e-05, "backbone_lr": 9.056889192782865e-06, "ema_decay": 0.998141, "it_s": 0.7, "data_frac": 0.854, "fm": 0.11815809279680252, "loss": 0.11815809279680252} +{"event": "train", "step": 4400, "lr": 9.045084971874738e-05, "backbone_lr": 9.045084971874738e-06, "ema_decay": 0.998149, "it_s": 0.99, "data_frac": 0.789, "fm": 0.11837879121303559, "loss": 0.11837879121303559} +{"event": "train", "step": 4425, "lr": 9.033215130758735e-05, "backbone_lr": 9.033215130758735e-06, "ema_decay": 0.998157, "it_s": 2.8, "data_frac": 0.451, "fm": 0.12120903819799424, "loss": 0.12120903819799424} +{"event": "train", "step": 4450, "lr": 9.021279861989885e-05, "backbone_lr": 9.021279861989884e-06, "ema_decay": 0.998165, "it_s": 1.42, "data_frac": 0.734, "fm": 0.11654747396707535, "loss": 0.11654747396707535} +{"event": "train", "step": 4475, "lr": 9.009279359184593e-05, "backbone_lr": 9.009279359184594e-06, "ema_decay": 0.998173, "it_s": 1.35, "data_frac": 0.737, "fm": 0.11454789787530899, "loss": 0.11454789787530899} +{"event": "train", "step": 4500, "lr": 8.997213817017507e-05, "backbone_lr": 8.997213817017508e-06, "ema_decay": 0.99818, "it_s": 0.64, "data_frac": 0.866, "fm": 0.11652687758207321, "loss": 0.11652687758207321} +{"event": "train", "step": 4525, "lr": 8.985083431218354e-05, "backbone_lr": 8.985083431218354e-06, "ema_decay": 0.998188, "it_s": 1.09, "data_frac": 0.773, "fm": 0.11339226454496383, "loss": 0.11339226454496383} +{"event": "train", "step": 4550, "lr": 8.972888398568772e-05, "backbone_lr": 8.972888398568772e-06, "ema_decay": 0.998195, "it_s": 0.99, "data_frac": 0.785, "fm": 0.11623615086078644, "loss": 0.11623615086078644} +{"event": "train", "step": 4575, "lr": 8.96062891689911e-05, "backbone_lr": 8.96062891689911e-06, "ema_decay": 0.998203, "it_s": 4.97, "data_frac": 0.002, "fm": 0.11666365176439285, "loss": 0.11666365176439285} +{"event": "train", "step": 4600, "lr": 8.948305185085225e-05, "backbone_lr": 8.948305185085226e-06, "ema_decay": 0.99821, "it_s": 1.55, "data_frac": 0.667, "fm": 0.11618060439825058, "loss": 0.11618060439825058} +{"event": "train", "step": 4625, "lr": 8.935917403045251e-05, "backbone_lr": 8.935917403045251e-06, "ema_decay": 0.998217, "it_s": 0.66, "data_frac": 0.863, "fm": 0.11370467752218247, "loss": 0.11370467752218247} +{"event": "train", "step": 4650, "lr": 8.92346577173636e-05, "backbone_lr": 8.923465771736361e-06, "ema_decay": 0.998224, "it_s": 1.7, "data_frac": 0.667, "fm": 0.11478051245212555, "loss": 0.11478051245212555} +{"event": "train", "step": 4675, "lr": 8.910950493151498e-05, "backbone_lr": 8.910950493151498e-06, "ema_decay": 0.998232, "it_s": 0.91, "data_frac": 0.803, "fm": 0.11607470244169235, "loss": 0.11607470244169235} +{"event": "train", "step": 4700, "lr": 8.898371770316111e-05, "backbone_lr": 8.898371770316113e-06, "ema_decay": 0.998239, "it_s": 1.28, "data_frac": 0.743, "fm": 0.11513609379529953, "loss": 0.11513609379529953} +{"event": "train", "step": 4725, "lr": 8.885729807284856e-05, "backbone_lr": 8.885729807284855e-06, "ema_decay": 0.998246, "it_s": 0.81, "data_frac": 0.838, "fm": 0.11020497322082519, "loss": 0.11020497322082519} +{"event": "train", "step": 4750, "lr": 8.873024809138272e-05, "backbone_lr": 8.873024809138272e-06, "ema_decay": 0.998253, "it_s": 2.64, "data_frac": 0.417, "fm": 0.11362491935491562, "loss": 0.11362491935491562} +{"event": "train", "step": 4775, "lr": 8.860256981979484e-05, "backbone_lr": 8.860256981979485e-06, "ema_decay": 0.998259, "it_s": 1.31, "data_frac": 0.75, "fm": 0.11324429392814636, "loss": 0.11324429392814636} +{"event": "train", "step": 4800, "lr": 8.847426532930831e-05, "backbone_lr": 8.84742653293083e-06, "ema_decay": 0.998266, "it_s": 1.3, "data_frac": 0.728, "fm": 0.11362754851579666, "loss": 0.11362754851579666} +{"event": "train", "step": 4825, "lr": 8.834533670130518e-05, "backbone_lr": 8.834533670130518e-06, "ema_decay": 0.998273, "it_s": 1.08, "data_frac": 0.775, "fm": 0.11129748165607452, "loss": 0.11129748165607452} +{"event": "train", "step": 4850, "lr": 8.821578602729242e-05, "backbone_lr": 8.821578602729242e-06, "ema_decay": 0.99828, "it_s": 0.83, "data_frac": 0.84, "fm": 0.1117265647649765, "loss": 0.1117265647649765} +{"event": "train", "step": 4875, "lr": 8.808561540886796e-05, "backbone_lr": 8.808561540886796e-06, "ema_decay": 0.998286, "it_s": 1.82, "data_frac": 0.59, "fm": 0.11203957498073577, "loss": 0.11203957498073577} +{"event": "train", "step": 4900, "lr": 8.795482695768658e-05, "backbone_lr": 8.795482695768658e-06, "ema_decay": 0.998293, "it_s": 1.36, "data_frac": 0.734, "fm": 0.10931766182184219, "loss": 0.10931766182184219} +{"event": "train", "step": 4925, "lr": 8.782342279542568e-05, "backbone_lr": 8.782342279542569e-06, "ema_decay": 0.998299, "it_s": 1.05, "data_frac": 0.791, "fm": 0.10985626131296158, "loss": 0.10985626131296158} +{"event": "train", "step": 4950, "lr": 8.769140505375085e-05, "backbone_lr": 8.769140505375084e-06, "ema_decay": 0.998306, "it_s": 1.07, "data_frac": 0.775, "fm": 0.1120401856303215, "loss": 0.1120401856303215} +{"event": "train", "step": 4975, "lr": 8.755877587428128e-05, "backbone_lr": 8.755877587428127e-06, "ema_decay": 0.998312, "it_s": 1.09, "data_frac": 0.791, "fm": 0.10802255183458329, "loss": 0.10802255183458329} +{"event": "train", "step": 5000, "lr": 8.742553740855506e-05, "backbone_lr": 8.742553740855507e-06, "ema_decay": 0.998318, "it_s": 1.12, "data_frac": 0.775, "fm": 0.11093634516000747, "loss": 0.11093634516000747} +{"event": "train", "step": 5025, "lr": 8.729169181799422e-05, "backbone_lr": 8.729169181799423e-06, "ema_decay": 0.998325, "it_s": 1.11, "data_frac": 0.766, "fm": 0.1084786233305931, "loss": 0.1084786233305931} +{"event": "train", "step": 5050, "lr": 8.715724127386972e-05, "backbone_lr": 8.715724127386971e-06, "ema_decay": 0.998331, "it_s": 2.0, "data_frac": 0.604, "fm": 0.10712229013442993, "loss": 0.10712229013442993} +{"event": "train", "step": 5075, "lr": 8.702218795726617e-05, "backbone_lr": 8.702218795726619e-06, "ema_decay": 0.998337, "it_s": 1.18, "data_frac": 0.735, "fm": 0.11033450186252594, "loss": 0.11033450186252594} +{"event": "train", "step": 5100, "lr": 8.688653405904652e-05, "backbone_lr": 8.688653405904653e-06, "ema_decay": 0.998343, "it_s": 1.02, "data_frac": 0.808, "fm": 0.11009035259485245, "loss": 0.11009035259485245} +{"event": "train", "step": 5125, "lr": 8.675028177981643e-05, "backbone_lr": 8.675028177981643e-06, "ema_decay": 0.998349, "it_s": 0.92, "data_frac": 0.807, "fm": 0.10828320592641831, "loss": 0.10828320592641831} +{"event": "train", "step": 5150, "lr": 8.661343332988869e-05, "backbone_lr": 8.661343332988869e-06, "ema_decay": 0.998355, "it_s": 0.88, "data_frac": 0.817, "fm": 0.10717267870903015, "loss": 0.10717267870903015} +{"event": "train", "step": 5175, "lr": 8.647599092924721e-05, "backbone_lr": 8.64759909292472e-06, "ema_decay": 0.998361, "it_s": 5.38, "data_frac": 0.001, "fm": 0.10801177144050599, "loss": 0.10801177144050599} +{"event": "train", "step": 5200, "lr": 8.633795680751116e-05, "backbone_lr": 8.633795680751116e-06, "ema_decay": 0.998367, "it_s": 1.56, "data_frac": 0.689, "fm": 0.105967099070549, "loss": 0.105967099070549} +{"event": "train", "step": 5225, "lr": 8.619933320389872e-05, "backbone_lr": 8.619933320389872e-06, "ema_decay": 0.998373, "it_s": 0.9, "data_frac": 0.828, "fm": 0.10755074709653854, "loss": 0.10755074709653854} +{"event": "train", "step": 5250, "lr": 8.606012236719073e-05, "backbone_lr": 8.606012236719073e-06, "ema_decay": 0.998379, "it_s": 0.91, "data_frac": 0.814, "fm": 0.10677639991044999, "loss": 0.10677639991044999} +{"event": "train", "step": 5275, "lr": 8.592032655569429e-05, "backbone_lr": 8.59203265556943e-06, "ema_decay": 0.998385, "it_s": 0.96, "data_frac": 0.803, "fm": 0.10488777875900268, "loss": 0.10488777875900268} +{"event": "train", "step": 5300, "lr": 8.577994803720606e-05, "backbone_lr": 8.577994803720605e-06, "ema_decay": 0.99839, "it_s": 1.74, "data_frac": 0.664, "fm": 0.10423666447401046, "loss": 0.10423666447401046} +{"event": "train", "step": 5325, "lr": 8.56389890889755e-05, "backbone_lr": 8.56389890889755e-06, "ema_decay": 0.998396, "it_s": 0.96, "data_frac": 0.819, "fm": 0.10578320920467377, "loss": 0.10578320920467377} +{"event": "train", "step": 5350, "lr": 8.549745199766792e-05, "backbone_lr": 8.549745199766792e-06, "ema_decay": 0.998402, "it_s": 3.47, "data_frac": 0.307, "fm": 0.10501939445734024, "loss": 0.10501939445734024} +{"event": "train", "step": 5375, "lr": 8.535533905932738e-05, "backbone_lr": 8.535533905932739e-06, "ema_decay": 0.998407, "it_s": 0.92, "data_frac": 0.805, "fm": 0.10691273421049118, "loss": 0.10691273421049118} +{"event": "train", "step": 5400, "lr": 8.521265257933948e-05, "backbone_lr": 8.521265257933948e-06, "ema_decay": 0.998413, "it_s": 0.96, "data_frac": 0.802, "fm": 0.1036634960770607, "loss": 0.1036634960770607} +{"event": "train", "step": 5425, "lr": 8.506939487239388e-05, "backbone_lr": 8.506939487239389e-06, "ema_decay": 0.998418, "it_s": 1.88, "data_frac": 0.638, "fm": 0.10052153080701828, "loss": 0.10052153080701828} +{"event": "train", "step": 5450, "lr": 8.492556826244687e-05, "backbone_lr": 8.492556826244687e-06, "ema_decay": 0.998424, "it_s": 0.75, "data_frac": 0.856, "fm": 0.1060214227437973, "loss": 0.1060214227437973} +{"event": "train", "step": 5475, "lr": 8.47811750826835e-05, "backbone_lr": 8.478117508268352e-06, "ema_decay": 0.998429, "it_s": 1.01, "data_frac": 0.796, "fm": 0.10572416603565216, "loss": 0.10572416603565216} +{"event": "train", "step": 5500, "lr": 8.463621767547998e-05, "backbone_lr": 8.463621767547998e-06, "ema_decay": 0.998434, "it_s": 4.74, "data_frac": 0.006, "fm": 0.10358604192733764, "loss": 0.10358604192733764} +{"event": "train", "step": 5525, "lr": 8.449069839236538e-05, "backbone_lr": 8.44906983923654e-06, "ema_decay": 0.99844, "it_s": 0.95, "data_frac": 0.808, "fm": 0.10235648989677429, "loss": 0.10235648989677429} +{"event": "train", "step": 5550, "lr": 8.434461959398376e-05, "backbone_lr": 8.434461959398377e-06, "ema_decay": 0.998445, "it_s": 1.71, "data_frac": 0.674, "fm": 0.10302803814411163, "loss": 0.10302803814411163} +{"event": "train", "step": 5575, "lr": 8.419798365005569e-05, "backbone_lr": 8.419798365005569e-06, "ema_decay": 0.99845, "it_s": 0.89, "data_frac": 0.826, "fm": 0.1070358219742775, "loss": 0.1070358219742775} +{"event": "train", "step": 5600, "lr": 8.405079293933986e-05, "backbone_lr": 8.405079293933986e-06, "ema_decay": 0.998455, "it_s": 0.81, "data_frac": 0.833, "fm": 0.10346478521823883, "loss": 0.10346478521823883} +{"event": "train", "step": 5625, "lr": 8.390304984959454e-05, "backbone_lr": 8.390304984959455e-06, "ema_decay": 0.998461, "it_s": 1.17, "data_frac": 0.741, "fm": 0.10347686499357224, "loss": 0.10347686499357224} +{"event": "train", "step": 5650, "lr": 8.375475677753881e-05, "backbone_lr": 8.375475677753882e-06, "ema_decay": 0.998466, "it_s": 4.63, "data_frac": 0.087, "fm": 0.10413286983966827, "loss": 0.10413286983966827} +{"event": "train", "step": 5675, "lr": 8.360591612881364e-05, "backbone_lr": 8.360591612881363e-06, "ema_decay": 0.998471, "it_s": 1.7, "data_frac": 0.684, "fm": 0.10362743228673935, "loss": 0.10362743228673935} +{"event": "train", "step": 5700, "lr": 8.345653031794292e-05, "backbone_lr": 8.345653031794292e-06, "ema_decay": 0.998476, "it_s": 0.95, "data_frac": 0.79, "fm": 0.10708901762962342, "loss": 0.10708901762962342} +{"event": "train", "step": 5725, "lr": 8.330660176829428e-05, "backbone_lr": 8.330660176829427e-06, "ema_decay": 0.998481, "it_s": 0.81, "data_frac": 0.813, "fm": 0.10352396816015244, "loss": 0.10352396816015244} +{"event": "train", "step": 5750, "lr": 8.315613291203976e-05, "backbone_lr": 8.315613291203977e-06, "ema_decay": 0.998486, "it_s": 1.12, "data_frac": 0.773, "fm": 0.1011051020026207, "loss": 0.1011051020026207} +{"event": "train", "step": 5775, "lr": 8.30051261901164e-05, "backbone_lr": 8.30051261901164e-06, "ema_decay": 0.998491, "it_s": 4.2, "data_frac": 0.206, "fm": 0.09954795092344285, "loss": 0.09954795092344285} +{"event": "train", "step": 5800, "lr": 8.285358405218655e-05, "backbone_lr": 8.285358405218655e-06, "ema_decay": 0.998496, "it_s": 1.92, "data_frac": 0.64, "fm": 0.10256652504205704, "loss": 0.10256652504205704} +{"event": "train", "step": 5825, "lr": 8.270150895659823e-05, "backbone_lr": 8.270150895659824e-06, "ema_decay": 0.9985, "it_s": 0.9, "data_frac": 0.82, "fm": 0.10183177888393402, "loss": 0.10183177888393402} +{"event": "train", "step": 5850, "lr": 8.25489033703452e-05, "backbone_lr": 8.25489033703452e-06, "ema_decay": 0.998505, "it_s": 1.12, "data_frac": 0.767, "fm": 0.1011198627948761, "loss": 0.1011198627948761} +{"event": "train", "step": 5875, "lr": 8.239576976902695e-05, "backbone_lr": 8.239576976902694e-06, "ema_decay": 0.99851, "it_s": 0.87, "data_frac": 0.806, "fm": 0.10307048767805099, "loss": 0.10307048767805099} +{"event": "train", "step": 5900, "lr": 8.224211063680853e-05, "backbone_lr": 8.224211063680854e-06, "ema_decay": 0.998515, "it_s": 1.49, "data_frac": 0.71, "fm": 0.10266599476337433, "loss": 0.10266599476337433} +{"event": "train", "step": 5925, "lr": 8.208792846638029e-05, "backbone_lr": 8.20879284663803e-06, "ema_decay": 0.998519, "it_s": 0.96, "data_frac": 0.818, "fm": 0.10186379879713059, "loss": 0.10186379879713059} +{"event": "train", "step": 5950, "lr": 8.19332257589174e-05, "backbone_lr": 8.19332257589174e-06, "ema_decay": 0.998524, "it_s": 1.88, "data_frac": 0.615, "fm": 0.1005827847123146, "loss": 0.1005827847123146} +{"event": "train", "step": 5975, "lr": 8.177800502403928e-05, "backbone_lr": 8.177800502403928e-06, "ema_decay": 0.998529, "it_s": 1.06, "data_frac": 0.788, "fm": 0.10216392815113068, "loss": 0.10216392815113068} +{"event": "train", "step": 6000, "lr": 8.162226877976887e-05, "backbone_lr": 8.162226877976886e-06, "ema_decay": 0.998533, "it_s": 1.14, "data_frac": 0.762, "fm": 0.10421739488840104, "loss": 0.10421739488840104} +{"event": "save", "step": 6000, "checkpoint": "/mnt/data0/zuo/Humanoid_benchmark/results/fmdit_continue_20261117_01/runs/b8/fm_dit/checkpoints/step-006000", "mse": 0.022521788254380226, "mse_row0": 0.02053411304950714, "mse_motion_token": 0.022982770577073097, "mse_inspire_left_hand_state": 0.012518338859081268, "mse_inspire_right_hand_state": 0.027608102187514305} +{"event": "train", "step": 6025, "lr": 8.146601955249188e-05, "backbone_lr": 8.146601955249187e-06, "ema_decay": 0.998538, "it_s": 1.62, "data_frac": 0.424, "fm": 0.1021818333864212, "loss": 0.1021818333864212} +{"event": "train", "step": 6050, "lr": 8.130925987691569e-05, "backbone_lr": 8.13092598769157e-06, "ema_decay": 0.998542, "it_s": 0.7, "data_frac": 0.862, "fm": 0.10002265453338623, "loss": 0.10002265453338623} +{"event": "train", "step": 6075, "lr": 8.11519922960283e-05, "backbone_lr": 8.115199229602831e-06, "ema_decay": 0.998547, "it_s": 0.97, "data_frac": 0.789, "fm": 0.09909536123275757, "loss": 0.09909536123275757} +{"event": "train", "step": 6100, "lr": 8.099421936105702e-05, "backbone_lr": 8.099421936105702e-06, "ema_decay": 0.998551, "it_s": 2.88, "data_frac": 0.438, "fm": 0.09719828993082047, "loss": 0.09719828993082047} +{"event": "train", "step": 6125, "lr": 8.083594363142717e-05, "backbone_lr": 8.083594363142717e-06, "ema_decay": 0.998556, "it_s": 1.02, "data_frac": 0.787, "fm": 0.10099037319421768, "loss": 0.10099037319421768} +{"event": "train", "step": 6150, "lr": 8.067716767472045e-05, "backbone_lr": 8.067716767472045e-06, "ema_decay": 0.99856, "it_s": 1.79, "data_frac": 0.64, "fm": 0.09931565254926682, "loss": 0.09931565254926682} +{"event": "train", "step": 6175, "lr": 8.051789406663342e-05, "backbone_lr": 8.051789406663342e-06, "ema_decay": 0.998565, "it_s": 0.86, "data_frac": 0.817, "fm": 0.10005048602819443, "loss": 0.10005048602819443} +{"event": "train", "step": 6200, "lr": 8.035812539093557e-05, "backbone_lr": 8.035812539093557e-06, "ema_decay": 0.998569, "it_s": 0.83, "data_frac": 0.825, "fm": 0.10188953727483749, "loss": 0.10188953727483749} +{"event": "train", "step": 6225, "lr": 8.019786423942752e-05, "backbone_lr": 8.019786423942752e-06, "ema_decay": 0.998573, "it_s": 3.74, "data_frac": 0.261, "fm": 0.10008735865354539, "loss": 0.10008735865354539} +{"event": "train", "step": 6250, "lr": 8.003711321189895e-05, "backbone_lr": 8.003711321189895e-06, "ema_decay": 0.998578, "it_s": 1.24, "data_frac": 0.72, "fm": 0.09857472032308578, "loss": 0.09857472032308578} +{"event": "train", "step": 6275, "lr": 7.987587491608637e-05, "backbone_lr": 7.987587491608636e-06, "ema_decay": 0.998582, "it_s": 1.69, "data_frac": 0.671, "fm": 0.09919695109128952, "loss": 0.09919695109128952} +{"event": "train", "step": 6300, "lr": 7.971415196763088e-05, "backbone_lr": 7.971415196763088e-06, "ema_decay": 0.998586, "it_s": 0.74, "data_frac": 0.854, "fm": 0.09775012910366059, "loss": 0.09775012910366059} +{"event": "train", "step": 6325, "lr": 7.95519469900357e-05, "backbone_lr": 7.95519469900357e-06, "ema_decay": 0.99859, "it_s": 0.81, "data_frac": 0.836, "fm": 0.09813733667135238, "loss": 0.09813733667135238} +{"event": "train", "step": 6350, "lr": 7.938926261462366e-05, "backbone_lr": 7.938926261462366e-06, "ema_decay": 0.998594, "it_s": 1.39, "data_frac": 0.723, "fm": 0.10025485634803771, "loss": 0.10025485634803771} +{"event": "train", "step": 6375, "lr": 7.922610148049445e-05, "backbone_lr": 7.922610148049445e-06, "ema_decay": 0.998599, "it_s": 3.94, "data_frac": 0.272, "fm": 0.09809325218200683, "loss": 0.09809325218200683} +{"event": "train", "step": 6400, "lr": 7.906246623448183e-05, "backbone_lr": 7.906246623448184e-06, "ema_decay": 0.998603, "it_s": 1.85, "data_frac": 0.659, "fm": 0.09990756928920747, "loss": 0.09990756928920747} +{"event": "train", "step": 6425, "lr": 7.889835953111075e-05, "backbone_lr": 7.889835953111075e-06, "ema_decay": 0.998607, "it_s": 0.71, "data_frac": 0.861, "fm": 0.09899431318044663, "loss": 0.09899431318044663} +{"event": "train", "step": 6450, "lr": 7.873378403255419e-05, "backbone_lr": 7.87337840325542e-06, "ema_decay": 0.998611, "it_s": 0.82, "data_frac": 0.823, "fm": 0.09775156676769256, "loss": 0.09775156676769256} +{"event": "train", "step": 6475, "lr": 7.856874240859005e-05, "backbone_lr": 7.856874240859006e-06, "ema_decay": 0.998615, "it_s": 1.1, "data_frac": 0.775, "fm": 0.10037847697734832, "loss": 0.10037847697734832} +{"event": "train", "step": 6500, "lr": 7.840323733655778e-05, "backbone_lr": 7.84032373365578e-06, "ema_decay": 0.998619, "it_s": 2.59, "data_frac": 0.528, "fm": 0.09699027627706527, "loss": 0.09699027627706527} +{"event": "train", "step": 6525, "lr": 7.823727150131506e-05, "backbone_lr": 7.823727150131506e-06, "ema_decay": 0.998623, "it_s": 0.62, "data_frac": 0.876, "fm": 0.09724076330661774, "loss": 0.09724076330661774} +{"event": "train", "step": 6550, "lr": 7.807084759519405e-05, "backbone_lr": 7.807084759519405e-06, "ema_decay": 0.998627, "it_s": 4.35, "data_frac": 0.005, "fm": 0.0968179738521576, "loss": 0.0968179738521576} +{"event": "train", "step": 6575, "lr": 7.790396831795792e-05, "backbone_lr": 7.790396831795792e-06, "ema_decay": 0.998631, "it_s": 1.04, "data_frac": 0.793, "fm": 0.09870140582323074, "loss": 0.09870140582323074} +{"event": "train", "step": 6600, "lr": 7.773663637675694e-05, "backbone_lr": 7.773663637675695e-06, "ema_decay": 0.998634, "it_s": 0.98, "data_frac": 0.81, "fm": 0.09852319538593292, "loss": 0.09852319538593292} +{"event": "train", "step": 6625, "lr": 7.756885448608459e-05, "backbone_lr": 7.75688544860846e-06, "ema_decay": 0.998638, "it_s": 2.27, "data_frac": 0.572, "fm": 0.0948163315653801, "loss": 0.0948163315653801} +{"event": "train", "step": 6650, "lr": 7.740062536773352e-05, "backbone_lr": 7.740062536773352e-06, "ema_decay": 0.998642, "it_s": 0.64, "data_frac": 0.876, "fm": 0.0981231078505516, "loss": 0.0981231078505516} +{"event": "train", "step": 6675, "lr": 7.723195175075136e-05, "backbone_lr": 7.723195175075136e-06, "ema_decay": 0.998646, "it_s": 1.13, "data_frac": 0.767, "fm": 0.10084572613239288, "loss": 0.10084572613239288} +{"event": "train", "step": 6700, "lr": 7.706283637139658e-05, "backbone_lr": 7.706283637139658e-06, "ema_decay": 0.99865, "it_s": 3.77, "data_frac": 0.286, "fm": 0.09731092363595963, "loss": 0.09731092363595963} diff --git a/b8-table-wipe/b512-cont-20261117/provenance/parent_checkpoint_config.json b/b8-table-wipe/b512-cont-20261117/provenance/parent_checkpoint_config.json new file mode 100644 index 0000000000000000000000000000000000000000..6e39151a27df09bc3d1c799ea2bcca94f71f9658 --- /dev/null +++ b/b8-table-wipe/b512-cont-20261117/provenance/parent_checkpoint_config.json @@ -0,0 +1,884 @@ +{ + "policy": { + "policy": "fm_dit", + "n_cameras": 2, + "state_dim": 44, + "action_dim": 76, + "horizon": 16, + "backbone": "resnet18", + "backbone_weights": "IMAGENET1K_V1", + "backbone_norm": "frozen_bn", + "hidden_dim": 384, + "num_layers": 4, + "num_heads": 8, + "mlp_ratio": 4.0, + "dropout": 0.1, + "time_embed_dim": 128, + "num_inference_steps": null, + "num_train_timesteps": 100, + "beta_schedule": "squaredcos_cap_v2", + "prediction_type": "epsilon", + "clip_sample": true, + "act_hidden_dim": 512, + "dim_feedforward": 3200, + "enc_layers": 4, + "dec_layers": 7, + "nheads": 8, + "latent_dim": 32, + "kl_weight": 10.0, + "extra": {} + }, + "contract": { + "name": "g1_sonic_dfx6", + "robot": "g1", + "cameras": [ + "ego_view", + "chest" + ], + "state": [ + [ + "observation.state", + 0, + 29 + ], + [ + "observation.inspire_left_hand_state", + 0, + 6 + ], + [ + "observation.inspire_right_hand_state", + 0, + 6 + ], + [ + "observation.projected_gravity", + 0, + 3 + ] + ], + "action": [ + [ + "action.motion_token", + 0, + 64 + ], + [ + "observation.inspire_left_hand_state", + 0, + 6 + ], + [ + "observation.inspire_right_hand_state", + 0, + 6 + ] + ], + "state_dim": 44, + "action_dim": 76, + "action_space": "sonic_dfx6", + "body_dims": 29 + }, + "image_hw": [ + 168, + 224 + ], + "task_text": "Approach the flat block with a naturally aligned wrist, stop before closing the hand, wipe sideways at a top-view 45-degree heading, then set the block on the tabletop and release it.", + "dataset": { + "name": "b8-table-wipe-expert-v1", + "root": "/mnt/data0/zuo/Humanoid_benchmark/data/baselines/b8-table-wipe-expert-v1", + "robot_type": null, + "fps": 30.0, + "episodes": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 36, + 37, + 38, + 39, + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55, + 56, + 57, + 58, + 59, + 60, + 61, + 62, + 63, + 64, + 65, + 66, + 67, + 68, + 69, + 70, + 71, + 72, + 73, + 74, + 75, + 76, + 77, + 78, + 79, + 80, + 81, + 82, + 83, + 84, + 85, + 86, + 87, + 88, + 89, + 90, + 91, + 92, + 93, + 94, + 95, + 96, + 97, + 98, + 99 + ], + "train_episodes": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 36, + 37, + 38, + 39, + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55, + 56, + 57, + 58, + 59, + 60, + 61, + 62, + 63, + 64, + 65, + 66, + 67, + 68, + 69, + 70, + 71, + 72, + 73, + 74, + 75, + 76, + 77, + 78, + 79, + 80, + 81, + 82, + 83, + 84, + 85, + 86, + 87, + 88, + 89, + 90, + 91, + 92, + 93, + 94, + 95, + 96, + 97, + 98, + 99 + ], + "val_episodes": [], + "train_frames": 61498, + "git_commit": null + }, + "training": { + "steps": 4000, + "batch_size": 512, + "lr": 0.0001, + "betas": [ + 0.95, + 0.999 + ], + "weight_decay": 1e-06, + "warmup": 500, + "schedule": "linear warm-up + cosine", + "backbone_lr_scale": 0.1, + "ema": 0.9999, + "shift_px": 0, + "amp": true, + "seed": 0, + "augmentation": { + "spec": "docs/augmentation_v2.md", + "version": "hb-aug-v2-20261109", + "implementation": "evaluation/augmentation/hb_aug_v2.py", + "visual": { + "effects": [ + "exposure", + "contrast", + "saturation", + "hue", + "gamma", + "temperature", + "rotation_crop", + "noise", + "gaussian_blur", + "jpeg", + "motion_blur", + "sharpen", + "defocus", + "light_shadow", + "wb_channel" + ], + "boosted_per_sample": 1, + "boost": 1.5, + "pixel_scale": "H / 224", + "future_targets": "none (one observation frame)" + }, + "proprio": { + "robot": "g1", + "body_dims": 29, + "bias_deg": 2.0, + "bias_scope": "per_sample", + "noise_deg": 0.25, + "noise_clip_sigma": 3, + "delay_probability": 0.0, + "history_drop_probability": 0.0, + "velocity": false + }, + "native_image_augmentation": "off", + "evaluation": "off" + }, + "preflight": { + "samples": 64, + "state_dim": 44, + "body_dims": 29, + "images_changed": 64, + "body_values_moved": 1.0, + "max_body_offset_deg": 2.7313, + "actions_bit_identical": true, + "non_body_state_bit_identical": true, + "eval_items_deterministic": 8 + } + }, + "stats": { + "state": { + "min": [ + -0.2515260875225067, + -0.03310006856918335, + -0.08915279805660248, + 0.14301416277885437, + -0.18305416405200958, + -0.10634835064411163, + -0.2634684145450592, + -0.05348178371787071, + -0.21431514620780945, + 0.13807232677936554, + -0.2242385447025299, + -0.09678935259580612, + -0.18892650306224823, + -0.10323800146579742, + -0.09472811967134476, + -0.2798849642276764, + 0.1722967028617859, + -1.1891459226608276, + 0.3821650445461273, + -0.5277573466300964, + -0.1239900067448616, + -0.025756916031241417, + -1.5671592950820923, + -1.708814024925232, + -0.94542396068573, + -0.8548623323440552, + -0.9579849243164062, + -0.9779664278030396, + -0.5873764157295227, + 0.9998621344566345, + 0.9998221397399902, + 0.9998023509979248, + 0.9998185634613037, + 0.9975361227989197, + 0.9988760948181152, + 0.360027939081192, + 0.4790782630443573, + 0.5461887717247009, + 0.48382189869880676, + 0.6635858416557312, + 0.0, + -0.10022255033254623, + -0.03946204483509064, + -1.0 + ], + "max": [ + 0.00482684513553977, + 0.13756881654262543, + 0.2979176640510559, + 0.3711587190628052, + 0.031152762472629547, + 0.010087746195495129, + 0.019378269091248512, + 0.1355365365743637, + 0.182325080037117, + 0.4443013668060303, + -0.05482839047908783, + 0.011374615132808685, + 0.165291890501976, + 0.05307222157716751, + 0.177412211894989, + 0.22115345299243927, + 0.6989883780479431, + -0.3335850238800049, + 1.1546826362609863, + 0.28441083431243896, + 0.08963758498430252, + 0.3550512492656708, + 1.1140739917755127, + -0.22596199810504913, + 1.7804871797561646, + 1.2184052467346191, + 1.2347023487091064, + 1.5067987442016602, + 1.0105962753295898, + 1.0, + 1.0, + 1.0, + 1.0, + 0.9982408285140991, + 0.9999225735664368, + 1.0, + 1.0, + 1.0, + 0.9999973177909851, + 1.0, + 1.0, + 0.038201265037059784, + 0.023285934701561928, + -0.9949650168418884 + ], + "mean": [ + -0.07244415528619287, + 0.016438895257606716, + -0.03188766069938736, + 0.19652274108921836, + -0.09995080381043964, + -0.0335864852112993, + -0.06201331924385435, + 0.0041316509641342114, + -0.07562538691067132, + 0.22342829523439264, + -0.13939505500881355, + -0.02568257955335654, + 0.020507808718722174, + 0.0030919114143910654, + 0.11976549024164236, + -0.04130542899628228, + 0.27413152575942573, + -0.667075950232086, + 0.9776525052369189, + -0.15594609082852004, + -0.04114803172895316, + 0.07066678088305102, + -0.44530701068620304, + -0.7956643808095055, + 0.6432070891275159, + 0.07140851169135853, + -0.22403887370387132, + 0.19658286211869352, + 0.3118955244879216, + 0.9999827171293537, + 0.999982850921418, + 0.9999830789480982, + 0.9999811406495951, + 0.9977658298817017, + 0.9997360855995177, + 0.7778945983312991, + 0.8171959691692932, + 0.8399288212288848, + 0.8394280370284749, + 0.8745686375938581, + 0.12291562838728093, + -0.023435156143715264, + -0.004820223051842413, + -0.999617456030026 + ], + "std": [ + 0.045712745420827386, + 0.013154301249487431, + 0.02405439153160751, + 0.0422870935042734, + 0.01604373013239405, + 0.010666838959405088, + 0.0447255039612766, + 0.014369854397034587, + 0.025451572022126993, + 0.05279699313574748, + 0.02213303083995789, + 0.011183898115253933, + 0.07041149453770397, + 0.015833724218855847, + 0.039560888033149964, + 0.08342761969885264, + 0.0587237913039723, + 0.22301655606787774, + 0.08125572382728762, + 0.1281115145665045, + 0.03600107397023974, + 0.04123008646611267, + 0.5831912382159029, + 0.23706341250281637, + 0.5432748450536339, + 0.504036388323598, + 0.31188020039153314, + 0.3436391052756804, + 0.26504982482523237, + 1.8214790335763182e-05, + 2.3385127857745866e-05, + 2.5684815254370892e-05, + 2.368414515012511e-05, + 7.802892144541008e-05, + 6.03931448414821e-05, + 0.2307055833669559, + 0.1900819078013402, + 0.16354430170740317, + 0.16593648793853702, + 0.1286487353890603, + 0.3083183803524622, + 0.013093107258771247, + 0.004580775936054731, + 0.0002959946385408924 + ] + }, + "action": { + "min": [ + 0.0, + -0.375, + -0.4375, + -0.125, + -0.375, + -0.3125, + -0.375, + -0.3125, + -0.25, + -0.25, + -0.5625, + -0.1875, + -0.125, + -0.3125, + -0.125, + -0.25, + -0.125, + -0.1875, + -0.5, + -0.25, + -0.1875, + -0.5625, + -0.25, + -0.375, + -0.4375, + -0.0625, + -0.375, + -0.375, + -0.3125, + -0.3125, + 0.0, + 0.125, + -0.3125, + 0.0, + 0.125, + -0.4375, + -0.1875, + -0.4375, + -0.25, + -0.1875, + -0.4375, + -0.375, + -0.1875, + -0.0625, + -0.3125, + -0.25, + -0.1875, + -0.3125, + -0.3125, + -0.5625, + -0.125, + -0.125, + -0.4375, + -0.5, + -0.1875, + -0.25, + -0.375, + -0.0625, + -0.4375, + -0.25, + -0.4375, + 0.0625, + -0.25, + 0.0, + 0.9998621344566345, + 0.9998221397399902, + 0.9998023509979248, + 0.9998185634613037, + 0.9975361227989197, + 0.9988760948181152, + 0.360027939081192, + 0.4790782630443573, + 0.5461887717247009, + 0.48382189869880676, + 0.6635858416557312, + 0.0 + ], + "max": [ + 0.375, + 0.25, + 0.25, + 0.1875, + 0.1875, + 0.1875, + 0.25, + 0.25, + 0.1875, + 0.125, + 0.0625, + 0.1875, + 0.5, + 0.1875, + 0.25, + 0.0625, + 0.3125, + 0.5, + 0.1875, + 0.25, + 0.25, + 0.0625, + 0.4375, + 0.25, + 0.0, + 0.4375, + 0.0, + 0.1875, + 0.1875, + 0.1875, + 0.4375, + 0.5, + 0.3125, + 0.3125, + 0.3125, + 0.0625, + 0.1875, + 0.3125, + 0.1875, + 0.25, + 0.0, + 0.25, + 0.125, + 0.5, + 0.125, + 0.5, + 0.4375, + 0.4375, + 0.4375, + 0.0, + 0.125, + 0.25, + 0.5, + -0.0625, + 0.125, + 0.4375, + 0.5, + 0.3125, + 0.125, + 0.125, + 0.125, + 0.5, + 0.125, + 0.6875, + 1.0, + 1.0, + 1.0, + 1.0, + 0.9982408285140991, + 0.9999225735664368, + 1.0, + 1.0, + 1.0, + 0.9999973177909851, + 1.0, + 1.0 + ], + "mean": [ + 0.2026803717194055, + -0.07236617450974016, + -0.02242552603336694, + 0.02358918176200852, + -0.12101816319229894, + -0.10377979771699893, + -0.04818144492503821, + -0.09367987576831767, + -0.015672257634394615, + -0.07295460827994406, + -0.28937628052944814, + 0.023045464893167257, + 0.12778362710982472, + -0.014680355458714104, + 0.047179379817229826, + -0.12413716706234348, + 0.0648974356889655, + 0.1738440680997756, + -0.1366050521968194, + 0.05253626134183226, + 0.005097726755341637, + -0.21554664379329408, + 0.09576226056131906, + -0.11485841002959446, + -0.21714120784415752, + 0.25317489999674786, + -0.23478507431135973, + -0.10450746365735471, + -0.049666249308920614, + -0.03343909558034407, + 0.2612117467234707, + 0.28021846238902076, + 0.013058351491105402, + 0.14741942827409021, + 0.23094043708738496, + -0.2599698364174445, + 0.054112532114865525, + -0.18685465380988, + -0.021627735861328823, + -0.045454730235129595, + -0.261119264041107, + -0.09780399362580897, + -0.03171647858466942, + 0.30728844840482616, + -0.13527167550164232, + 0.061584319815278545, + 0.07089966340368792, + 0.06740158216527367, + 0.18057091287521546, + -0.27359832840092363, + -0.01098104816416794, + 0.03545237243487593, + 0.1730208705974178, + -0.26741316790789943, + -0.03493507918956714, + 0.1694120133988097, + 0.01502691144427461, + 0.05571421021821848, + -0.22518415233015707, + -0.05392248528407428, + -0.1961598344661615, + 0.3429247699112166, + -0.04006126215486683, + 0.40486580864418353, + 0.9999827171293537, + 0.999982850921418, + 0.9999830789480982, + 0.9999811406495951, + 0.9977658298817017, + 0.9997360855995177, + 0.7778945983312991, + 0.8171959691692932, + 0.8399288212288848, + 0.8394280370284749, + 0.8745686375938581, + 0.12291562838728093 + ], + "std": [ + 0.07423638798986239, + 0.10276070894626457, + 0.11812702387971753, + 0.05957396656117891, + 0.1218417031488596, + 0.08063836551976374, + 0.1020635000819969, + 0.10540346375069484, + 0.08296831170197061, + 0.05989282207073844, + 0.131405419274155, + 0.07688508477180626, + 0.15530657456533145, + 0.09090021877289593, + 0.07144535330531092, + 0.07721423672952077, + 0.061937753226413185, + 0.12535445915332458, + 0.1406645362754369, + 0.09610483490082063, + 0.0892112540257282, + 0.10101183108941218, + 0.14459992137856284, + 0.09259695505428482, + 0.07250089525780803, + 0.07751064616894329, + 0.09003536032570492, + 0.08768655804866919, + 0.0793513264967945, + 0.0799717022695259, + 0.08767655242632699, + 0.06686962750120226, + 0.11139553980852172, + 0.05962527954838032, + 0.03113638664810605, + 0.1104336480130166, + 0.0900293543854962, + 0.17353368370821917, + 0.06621053001132303, + 0.07044267349366652, + 0.07468062148900616, + 0.13559357690832427, + 0.051673248141484364, + 0.11970962808802042, + 0.09873016730126778, + 0.1482997857923691, + 0.14644960009292055, + 0.12362418143308783, + 0.14892813932159055, + 0.1264928491368565, + 0.04485723360367807, + 0.06556638017875786, + 0.23014907315132233, + 0.07366488770406758, + 0.05520901424631441, + 0.1518289001396863, + 0.1840720418570084, + 0.05832378814080937, + 0.11903714776197126, + 0.05677283141546173, + 0.09207849117719222, + 0.09729334507011746, + 0.0780996197326025, + 0.16351388381996734, + 1.8214790335763182e-05, + 2.3385127857745866e-05, + 2.5684815254370892e-05, + 2.368414515012511e-05, + 7.802892144541008e-05, + 6.03931448414821e-05, + 0.2307055833669559, + 0.1900819078013402, + 0.16354430170740317, + 0.16593648793853702, + 0.1286487353890603, + 0.3083183803524622 + ] + } + }, + "code_commit": "028df1bb7024fb1e3d176de7e60c84d3333cd97f", + "step": 4000, + "weights": "ema", + "offline_eval": { + "mse": 0.02690730057656765, + "mse_row0": 0.024344196543097496, + "mse_motion_token": 0.027465982362627983, + "mse_inspire_left_hand_state": 0.01448493916541338, + "mse_inspire_right_hand_state": 0.03337034583091736, + "samples": 256, + "frames": "train" + }, + "format": "hb_baselines/1" +} \ No newline at end of file diff --git a/b8-table-wipe/b512-cont-20261117/provenance/parent_checkpoint_protocol.json b/b8-table-wipe/b512-cont-20261117/provenance/parent_checkpoint_protocol.json new file mode 100644 index 0000000000000000000000000000000000000000..23be42192ad4903cca0ffc47b061dd54ddc0e63b --- /dev/null +++ b/b8-table-wipe/b512-cont-20261117/provenance/parent_checkpoint_protocol.json @@ -0,0 +1,11 @@ +{ + "format": "hb_baselines/1", + "step": 4000, + "policy": "fm_dit", + "files": { + "model.pt": { + "bytes": 90311637, + "sha256": "cc34f3f0c8ae25fbd3d607bd0e0e2f47741ccd15f13ca2ccf2ce6eedc271055b" + } + } +} \ No newline at end of file diff --git a/b8-table-wipe/b512-cont-20261117/provenance/parent_run.json b/b8-table-wipe/b512-cont-20261117/provenance/parent_run.json new file mode 100644 index 0000000000000000000000000000000000000000..e2d3979312f73f55dea20b79d98dea9430983c2a --- /dev/null +++ b/b8-table-wipe/b512-cont-20261117/provenance/parent_run.json @@ -0,0 +1,894 @@ +{ + "argv": [ + "/mnt/data0/zuo/Humanoid_benchmark/results/baselines_preflight_20261117/main/evaluation/baselines/train.py", + "--policy", + "fm_dit", + "--contract", + "g1_sonic_dfx6", + "--dataset", + "/mnt/data0/zuo/Humanoid_benchmark/data/baselines/b8-table-wipe-expert-v1", + "--cache-dir", + "/mnt/data0/zuo/Humanoid_benchmark/cache/baselines/frames", + "--out", + "/mnt/data0/zuo/Humanoid_benchmark/results/baselines_preflight_20261117/runs/b8-table-wipe-expert-v1/fm_dit", + "--batch-size", + "512", + "--steps", + "4000", + "--workers", + "12" + ], + "started": "2026-11-17T04:07:19", + "host": "xt-p-A100-004", + "torch": "2.9.0+cu128", + "policy": { + "policy": "fm_dit", + "n_cameras": 2, + "state_dim": 44, + "action_dim": 76, + "horizon": 16, + "backbone": "resnet18", + "backbone_weights": "IMAGENET1K_V1", + "backbone_norm": "frozen_bn", + "hidden_dim": 384, + "num_layers": 4, + "num_heads": 8, + "mlp_ratio": 4.0, + "dropout": 0.1, + "time_embed_dim": 128, + "num_inference_steps": null, + "num_train_timesteps": 100, + "beta_schedule": "squaredcos_cap_v2", + "prediction_type": "epsilon", + "clip_sample": true, + "act_hidden_dim": 512, + "dim_feedforward": 3200, + "enc_layers": 4, + "dec_layers": 7, + "nheads": 8, + "latent_dim": 32, + "kl_weight": 10.0, + "extra": {} + }, + "contract": { + "name": "g1_sonic_dfx6", + "robot": "g1", + "cameras": [ + "ego_view", + "chest" + ], + "state": [ + [ + "observation.state", + 0, + 29 + ], + [ + "observation.inspire_left_hand_state", + 0, + 6 + ], + [ + "observation.inspire_right_hand_state", + 0, + 6 + ], + [ + "observation.projected_gravity", + 0, + 3 + ] + ], + "action": [ + [ + "action.motion_token", + 0, + 64 + ], + [ + "observation.inspire_left_hand_state", + 0, + 6 + ], + [ + "observation.inspire_right_hand_state", + 0, + 6 + ] + ], + "state_dim": 44, + "action_dim": 76, + "action_space": "sonic_dfx6", + "body_dims": 29 + }, + "image_hw": [ + 168, + 224 + ], + "task_text": "Approach the flat block with a naturally aligned wrist, stop before closing the hand, wipe sideways at a top-view 45-degree heading, then set the block on the tabletop and release it.", + "dataset": { + "name": "b8-table-wipe-expert-v1", + "root": "/mnt/data0/zuo/Humanoid_benchmark/data/baselines/b8-table-wipe-expert-v1", + "robot_type": null, + "fps": 30.0, + "episodes": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 36, + 37, + 38, + 39, + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55, + 56, + 57, + 58, + 59, + 60, + 61, + 62, + 63, + 64, + 65, + 66, + 67, + 68, + 69, + 70, + 71, + 72, + 73, + 74, + 75, + 76, + 77, + 78, + 79, + 80, + 81, + 82, + 83, + 84, + 85, + 86, + 87, + 88, + 89, + 90, + 91, + 92, + 93, + 94, + 95, + 96, + 97, + 98, + 99 + ], + "train_episodes": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 36, + 37, + 38, + 39, + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55, + 56, + 57, + 58, + 59, + 60, + 61, + 62, + 63, + 64, + 65, + 66, + 67, + 68, + 69, + 70, + 71, + 72, + 73, + 74, + 75, + 76, + 77, + 78, + 79, + 80, + 81, + 82, + 83, + 84, + 85, + 86, + 87, + 88, + 89, + 90, + 91, + 92, + 93, + 94, + 95, + 96, + 97, + 98, + 99 + ], + "val_episodes": [], + "train_frames": 61498, + "git_commit": null + }, + "training": { + "steps": 4000, + "batch_size": 512, + "lr": 0.0001, + "betas": [ + 0.95, + 0.999 + ], + "weight_decay": 1e-06, + "warmup": 500, + "schedule": "linear warm-up + cosine", + "backbone_lr_scale": 0.1, + "ema": 0.9999, + "shift_px": 0, + "amp": true, + "seed": 0, + "augmentation": { + "spec": "docs/augmentation_v2.md", + "version": "hb-aug-v2-20261109", + "implementation": "evaluation/augmentation/hb_aug_v2.py", + "visual": { + "effects": [ + "exposure", + "contrast", + "saturation", + "hue", + "gamma", + "temperature", + "rotation_crop", + "noise", + "gaussian_blur", + "jpeg", + "motion_blur", + "sharpen", + "defocus", + "light_shadow", + "wb_channel" + ], + "boosted_per_sample": 1, + "boost": 1.5, + "pixel_scale": "H / 224", + "future_targets": "none (one observation frame)" + }, + "proprio": { + "robot": "g1", + "body_dims": 29, + "bias_deg": 2.0, + "bias_scope": "per_sample", + "noise_deg": 0.25, + "noise_clip_sigma": 3, + "delay_probability": 0.0, + "history_drop_probability": 0.0, + "velocity": false + }, + "native_image_augmentation": "off", + "evaluation": "off" + }, + "preflight": { + "samples": 64, + "state_dim": 44, + "body_dims": 29, + "images_changed": 64, + "body_values_moved": 1.0, + "max_body_offset_deg": 2.7313, + "actions_bit_identical": true, + "non_body_state_bit_identical": true, + "eval_items_deterministic": 8 + } + }, + "stats": { + "state": { + "min": [ + -0.2515260875225067, + -0.03310006856918335, + -0.08915279805660248, + 0.14301416277885437, + -0.18305416405200958, + -0.10634835064411163, + -0.2634684145450592, + -0.05348178371787071, + -0.21431514620780945, + 0.13807232677936554, + -0.2242385447025299, + -0.09678935259580612, + -0.18892650306224823, + -0.10323800146579742, + -0.09472811967134476, + -0.2798849642276764, + 0.1722967028617859, + -1.1891459226608276, + 0.3821650445461273, + -0.5277573466300964, + -0.1239900067448616, + -0.025756916031241417, + -1.5671592950820923, + -1.708814024925232, + -0.94542396068573, + -0.8548623323440552, + -0.9579849243164062, + -0.9779664278030396, + -0.5873764157295227, + 0.9998621344566345, + 0.9998221397399902, + 0.9998023509979248, + 0.9998185634613037, + 0.9975361227989197, + 0.9988760948181152, + 0.360027939081192, + 0.4790782630443573, + 0.5461887717247009, + 0.48382189869880676, + 0.6635858416557312, + 0.0, + -0.10022255033254623, + -0.03946204483509064, + -1.0 + ], + "max": [ + 0.00482684513553977, + 0.13756881654262543, + 0.2979176640510559, + 0.3711587190628052, + 0.031152762472629547, + 0.010087746195495129, + 0.019378269091248512, + 0.1355365365743637, + 0.182325080037117, + 0.4443013668060303, + -0.05482839047908783, + 0.011374615132808685, + 0.165291890501976, + 0.05307222157716751, + 0.177412211894989, + 0.22115345299243927, + 0.6989883780479431, + -0.3335850238800049, + 1.1546826362609863, + 0.28441083431243896, + 0.08963758498430252, + 0.3550512492656708, + 1.1140739917755127, + -0.22596199810504913, + 1.7804871797561646, + 1.2184052467346191, + 1.2347023487091064, + 1.5067987442016602, + 1.0105962753295898, + 1.0, + 1.0, + 1.0, + 1.0, + 0.9982408285140991, + 0.9999225735664368, + 1.0, + 1.0, + 1.0, + 0.9999973177909851, + 1.0, + 1.0, + 0.038201265037059784, + 0.023285934701561928, + -0.9949650168418884 + ], + "mean": [ + -0.07244415528619287, + 0.016438895257606716, + -0.03188766069938736, + 0.19652274108921836, + -0.09995080381043964, + -0.0335864852112993, + -0.06201331924385435, + 0.0041316509641342114, + -0.07562538691067132, + 0.22342829523439264, + -0.13939505500881355, + -0.02568257955335654, + 0.020507808718722174, + 0.0030919114143910654, + 0.11976549024164236, + -0.04130542899628228, + 0.27413152575942573, + -0.667075950232086, + 0.9776525052369189, + -0.15594609082852004, + -0.04114803172895316, + 0.07066678088305102, + -0.44530701068620304, + -0.7956643808095055, + 0.6432070891275159, + 0.07140851169135853, + -0.22403887370387132, + 0.19658286211869352, + 0.3118955244879216, + 0.9999827171293537, + 0.999982850921418, + 0.9999830789480982, + 0.9999811406495951, + 0.9977658298817017, + 0.9997360855995177, + 0.7778945983312991, + 0.8171959691692932, + 0.8399288212288848, + 0.8394280370284749, + 0.8745686375938581, + 0.12291562838728093, + -0.023435156143715264, + -0.004820223051842413, + -0.999617456030026 + ], + "std": [ + 0.045712745420827386, + 0.013154301249487431, + 0.02405439153160751, + 0.0422870935042734, + 0.01604373013239405, + 0.010666838959405088, + 0.0447255039612766, + 0.014369854397034587, + 0.025451572022126993, + 0.05279699313574748, + 0.02213303083995789, + 0.011183898115253933, + 0.07041149453770397, + 0.015833724218855847, + 0.039560888033149964, + 0.08342761969885264, + 0.0587237913039723, + 0.22301655606787774, + 0.08125572382728762, + 0.1281115145665045, + 0.03600107397023974, + 0.04123008646611267, + 0.5831912382159029, + 0.23706341250281637, + 0.5432748450536339, + 0.504036388323598, + 0.31188020039153314, + 0.3436391052756804, + 0.26504982482523237, + 1.8214790335763182e-05, + 2.3385127857745866e-05, + 2.5684815254370892e-05, + 2.368414515012511e-05, + 7.802892144541008e-05, + 6.03931448414821e-05, + 0.2307055833669559, + 0.1900819078013402, + 0.16354430170740317, + 0.16593648793853702, + 0.1286487353890603, + 0.3083183803524622, + 0.013093107258771247, + 0.004580775936054731, + 0.0002959946385408924 + ] + }, + "action": { + "min": [ + 0.0, + -0.375, + -0.4375, + -0.125, + -0.375, + -0.3125, + -0.375, + -0.3125, + -0.25, + -0.25, + -0.5625, + -0.1875, + -0.125, + -0.3125, + -0.125, + -0.25, + -0.125, + -0.1875, + -0.5, + -0.25, + -0.1875, + -0.5625, + -0.25, + -0.375, + -0.4375, + -0.0625, + -0.375, + -0.375, + -0.3125, + -0.3125, + 0.0, + 0.125, + -0.3125, + 0.0, + 0.125, + -0.4375, + -0.1875, + -0.4375, + -0.25, + -0.1875, + -0.4375, + -0.375, + -0.1875, + -0.0625, + -0.3125, + -0.25, + -0.1875, + -0.3125, + -0.3125, + -0.5625, + -0.125, + -0.125, + -0.4375, + -0.5, + -0.1875, + -0.25, + -0.375, + -0.0625, + -0.4375, + -0.25, + -0.4375, + 0.0625, + -0.25, + 0.0, + 0.9998621344566345, + 0.9998221397399902, + 0.9998023509979248, + 0.9998185634613037, + 0.9975361227989197, + 0.9988760948181152, + 0.360027939081192, + 0.4790782630443573, + 0.5461887717247009, + 0.48382189869880676, + 0.6635858416557312, + 0.0 + ], + "max": [ + 0.375, + 0.25, + 0.25, + 0.1875, + 0.1875, + 0.1875, + 0.25, + 0.25, + 0.1875, + 0.125, + 0.0625, + 0.1875, + 0.5, + 0.1875, + 0.25, + 0.0625, + 0.3125, + 0.5, + 0.1875, + 0.25, + 0.25, + 0.0625, + 0.4375, + 0.25, + 0.0, + 0.4375, + 0.0, + 0.1875, + 0.1875, + 0.1875, + 0.4375, + 0.5, + 0.3125, + 0.3125, + 0.3125, + 0.0625, + 0.1875, + 0.3125, + 0.1875, + 0.25, + 0.0, + 0.25, + 0.125, + 0.5, + 0.125, + 0.5, + 0.4375, + 0.4375, + 0.4375, + 0.0, + 0.125, + 0.25, + 0.5, + -0.0625, + 0.125, + 0.4375, + 0.5, + 0.3125, + 0.125, + 0.125, + 0.125, + 0.5, + 0.125, + 0.6875, + 1.0, + 1.0, + 1.0, + 1.0, + 0.9982408285140991, + 0.9999225735664368, + 1.0, + 1.0, + 1.0, + 0.9999973177909851, + 1.0, + 1.0 + ], + "mean": [ + 0.2026803717194055, + -0.07236617450974016, + -0.02242552603336694, + 0.02358918176200852, + -0.12101816319229894, + -0.10377979771699893, + -0.04818144492503821, + -0.09367987576831767, + -0.015672257634394615, + -0.07295460827994406, + -0.28937628052944814, + 0.023045464893167257, + 0.12778362710982472, + -0.014680355458714104, + 0.047179379817229826, + -0.12413716706234348, + 0.0648974356889655, + 0.1738440680997756, + -0.1366050521968194, + 0.05253626134183226, + 0.005097726755341637, + -0.21554664379329408, + 0.09576226056131906, + -0.11485841002959446, + -0.21714120784415752, + 0.25317489999674786, + -0.23478507431135973, + -0.10450746365735471, + -0.049666249308920614, + -0.03343909558034407, + 0.2612117467234707, + 0.28021846238902076, + 0.013058351491105402, + 0.14741942827409021, + 0.23094043708738496, + -0.2599698364174445, + 0.054112532114865525, + -0.18685465380988, + -0.021627735861328823, + -0.045454730235129595, + -0.261119264041107, + -0.09780399362580897, + -0.03171647858466942, + 0.30728844840482616, + -0.13527167550164232, + 0.061584319815278545, + 0.07089966340368792, + 0.06740158216527367, + 0.18057091287521546, + -0.27359832840092363, + -0.01098104816416794, + 0.03545237243487593, + 0.1730208705974178, + -0.26741316790789943, + -0.03493507918956714, + 0.1694120133988097, + 0.01502691144427461, + 0.05571421021821848, + -0.22518415233015707, + -0.05392248528407428, + -0.1961598344661615, + 0.3429247699112166, + -0.04006126215486683, + 0.40486580864418353, + 0.9999827171293537, + 0.999982850921418, + 0.9999830789480982, + 0.9999811406495951, + 0.9977658298817017, + 0.9997360855995177, + 0.7778945983312991, + 0.8171959691692932, + 0.8399288212288848, + 0.8394280370284749, + 0.8745686375938581, + 0.12291562838728093 + ], + "std": [ + 0.07423638798986239, + 0.10276070894626457, + 0.11812702387971753, + 0.05957396656117891, + 0.1218417031488596, + 0.08063836551976374, + 0.1020635000819969, + 0.10540346375069484, + 0.08296831170197061, + 0.05989282207073844, + 0.131405419274155, + 0.07688508477180626, + 0.15530657456533145, + 0.09090021877289593, + 0.07144535330531092, + 0.07721423672952077, + 0.061937753226413185, + 0.12535445915332458, + 0.1406645362754369, + 0.09610483490082063, + 0.0892112540257282, + 0.10101183108941218, + 0.14459992137856284, + 0.09259695505428482, + 0.07250089525780803, + 0.07751064616894329, + 0.09003536032570492, + 0.08768655804866919, + 0.0793513264967945, + 0.0799717022695259, + 0.08767655242632699, + 0.06686962750120226, + 0.11139553980852172, + 0.05962527954838032, + 0.03113638664810605, + 0.1104336480130166, + 0.0900293543854962, + 0.17353368370821917, + 0.06621053001132303, + 0.07044267349366652, + 0.07468062148900616, + 0.13559357690832427, + 0.051673248141484364, + 0.11970962808802042, + 0.09873016730126778, + 0.1482997857923691, + 0.14644960009292055, + 0.12362418143308783, + 0.14892813932159055, + 0.1264928491368565, + 0.04485723360367807, + 0.06556638017875786, + 0.23014907315132233, + 0.07366488770406758, + 0.05520901424631441, + 0.1518289001396863, + 0.1840720418570084, + 0.05832378814080937, + 0.11903714776197126, + 0.05677283141546173, + 0.09207849117719222, + 0.09729334507011746, + 0.0780996197326025, + 0.16351388381996734, + 1.8214790335763182e-05, + 2.3385127857745866e-05, + 2.5684815254370892e-05, + 2.368414515012511e-05, + 7.802892144541008e-05, + 6.03931448414821e-05, + 0.2307055833669559, + 0.1900819078013402, + 0.16354430170740317, + 0.16593648793853702, + 0.1286487353890603, + 0.3083183803524622 + ] + } + }, + "code_commit": "028df1bb7024fb1e3d176de7e60c84d3333cd97f" +} \ No newline at end of file diff --git a/b8-table-wipe/b512-cont-20261117/resume.json b/b8-table-wipe/b512-cont-20261117/resume.json new file mode 100644 index 0000000000000000000000000000000000000000..b91c1053db8c0b5ab63992f2c418eca37240e15e --- /dev/null +++ b/b8-table-wipe/b512-cont-20261117/resume.json @@ -0,0 +1,26 @@ +{ + "task": "b8", + "status": "paused_by_owner", + "saved_step": 6000, + "last_logged_step": 6700, + "paused_at": "2026-11-17T08:36:34.220194+00:00", + "planned_total_steps": 20000, + "dataset_repo": "humanoid-benchmark/b8-table-wipe-expert-v1", + "dataset_revision": "3b2e6180508f84f70fda8b33cd3e41c14225f0b5", + "dataset_train_frames": 61498, + "dataset_episodes": 100, + "code_directory": "code/fmdit_continue_20261117", + "source_commit": "028df1bb7024fb1e3d176de7e60c84d3333cd97f", + "parent_local_checkpoint": "/mnt/data0/zuo/Humanoid_benchmark/results/baselines_preflight_20261117/runs/b8-table-wipe-expert-v1/fm_dit/checkpoints/step-004000", + "parent_state_sha256": "e31fa6e541a93c084a279627f53a7a9c3494772ef9605b1665a267abf7bb9a8f", + "offline_eval": { + "mse": 0.022521788254380226, + "mse_row0": 0.02053411304950714, + "mse_motion_token": 0.022982770577073097, + "mse_inspire_left_hand_state": 0.012518338859081268, + "mse_inspire_right_hand_state": 0.027608102187514305, + "samples": 256, + "frames": "train" + }, + "limitation": "Portable resume starts at step 6000. Later logged updates are not in the saved state. RNG and DataLoader states were not saved by the original trainer; resume is not bit-exact." +} diff --git a/b8-table-wipe/b512-cont-20261117/run.json b/b8-table-wipe/b512-cont-20261117/run.json new file mode 100644 index 0000000000000000000000000000000000000000..0c9ae753f8d8c36809bb6a5a4c36247749e20886 --- /dev/null +++ b/b8-table-wipe/b512-cont-20261117/run.json @@ -0,0 +1,899 @@ +{ + "argv": [ + "/mnt/data0/zuo/Humanoid_benchmark/results/baselines_preflight_20261117/main/evaluation/baselines/train.py", + "--policy", + "fm_dit", + "--contract", + "g1_sonic_dfx6", + "--dataset", + "/mnt/data0/zuo/Humanoid_benchmark/data/baselines/b8-table-wipe-expert-v1", + "--cache-dir", + "/mnt/data0/zuo/Humanoid_benchmark/cache/baselines/frames", + "--out", + "/mnt/data0/zuo/Humanoid_benchmark/results/fmdit_continue_20261117_01/runs/b8/fm_dit", + "--batch-size", + "512", + "--steps", + "20000", + "--save-every", + "2000", + "--log-every", + "25", + "--workers", + "32", + "--resume" + ], + "started": "2026-11-17T15:57:43", + "host": "xt-p-A100-004", + "torch": "2.9.0+cu128", + "policy": { + "policy": "fm_dit", + "n_cameras": 2, + "state_dim": 44, + "action_dim": 76, + "horizon": 16, + "backbone": "resnet18", + "backbone_weights": "IMAGENET1K_V1", + "backbone_norm": "frozen_bn", + "hidden_dim": 384, + "num_layers": 4, + "num_heads": 8, + "mlp_ratio": 4.0, + "dropout": 0.1, + "time_embed_dim": 128, + "num_inference_steps": null, + "num_train_timesteps": 100, + "beta_schedule": "squaredcos_cap_v2", + "prediction_type": "epsilon", + "clip_sample": true, + "act_hidden_dim": 512, + "dim_feedforward": 3200, + "enc_layers": 4, + "dec_layers": 7, + "nheads": 8, + "latent_dim": 32, + "kl_weight": 10.0, + "extra": {} + }, + "contract": { + "name": "g1_sonic_dfx6", + "robot": "g1", + "cameras": [ + "ego_view", + "chest" + ], + "state": [ + [ + "observation.state", + 0, + 29 + ], + [ + "observation.inspire_left_hand_state", + 0, + 6 + ], + [ + "observation.inspire_right_hand_state", + 0, + 6 + ], + [ + "observation.projected_gravity", + 0, + 3 + ] + ], + "action": [ + [ + "action.motion_token", + 0, + 64 + ], + [ + "observation.inspire_left_hand_state", + 0, + 6 + ], + [ + "observation.inspire_right_hand_state", + 0, + 6 + ] + ], + "state_dim": 44, + "action_dim": 76, + "action_space": "sonic_dfx6", + "body_dims": 29 + }, + "image_hw": [ + 168, + 224 + ], + "task_text": "Approach the flat block with a naturally aligned wrist, stop before closing the hand, wipe sideways at a top-view 45-degree heading, then set the block on the tabletop and release it.", + "dataset": { + "name": "b8-table-wipe-expert-v1", + "root": "/mnt/data0/zuo/Humanoid_benchmark/data/baselines/b8-table-wipe-expert-v1", + "robot_type": null, + "fps": 30.0, + "episodes": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 36, + 37, + 38, + 39, + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55, + 56, + 57, + 58, + 59, + 60, + 61, + 62, + 63, + 64, + 65, + 66, + 67, + 68, + 69, + 70, + 71, + 72, + 73, + 74, + 75, + 76, + 77, + 78, + 79, + 80, + 81, + 82, + 83, + 84, + 85, + 86, + 87, + 88, + 89, + 90, + 91, + 92, + 93, + 94, + 95, + 96, + 97, + 98, + 99 + ], + "train_episodes": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 36, + 37, + 38, + 39, + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55, + 56, + 57, + 58, + 59, + 60, + 61, + 62, + 63, + 64, + 65, + 66, + 67, + 68, + 69, + 70, + 71, + 72, + 73, + 74, + 75, + 76, + 77, + 78, + 79, + 80, + 81, + 82, + 83, + 84, + 85, + 86, + 87, + 88, + 89, + 90, + 91, + 92, + 93, + 94, + 95, + 96, + 97, + 98, + 99 + ], + "val_episodes": [], + "train_frames": 61498, + "git_commit": null + }, + "training": { + "steps": 20000, + "batch_size": 512, + "lr": 0.0001, + "betas": [ + 0.95, + 0.999 + ], + "weight_decay": 1e-06, + "warmup": 500, + "schedule": "linear warm-up + cosine", + "backbone_lr_scale": 0.1, + "ema": 0.9999, + "shift_px": 0, + "amp": true, + "seed": 0, + "augmentation": { + "spec": "docs/augmentation_v2.md", + "version": "hb-aug-v2-20261109", + "implementation": "evaluation/augmentation/hb_aug_v2.py", + "visual": { + "effects": [ + "exposure", + "contrast", + "saturation", + "hue", + "gamma", + "temperature", + "rotation_crop", + "noise", + "gaussian_blur", + "jpeg", + "motion_blur", + "sharpen", + "defocus", + "light_shadow", + "wb_channel" + ], + "boosted_per_sample": 1, + "boost": 1.5, + "pixel_scale": "H / 224", + "future_targets": "none (one observation frame)" + }, + "proprio": { + "robot": "g1", + "body_dims": 29, + "bias_deg": 2.0, + "bias_scope": "per_sample", + "noise_deg": 0.25, + "noise_clip_sigma": 3, + "delay_probability": 0.0, + "history_drop_probability": 0.0, + "velocity": false + }, + "native_image_augmentation": "off", + "evaluation": "off" + }, + "preflight": { + "samples": 64, + "state_dim": 44, + "body_dims": 29, + "images_changed": 64, + "body_values_moved": 1.0, + "max_body_offset_deg": 2.5819, + "actions_bit_identical": true, + "non_body_state_bit_identical": true, + "eval_items_deterministic": 8 + } + }, + "stats": { + "state": { + "min": [ + -0.2515260875225067, + -0.03310006856918335, + -0.08915279805660248, + 0.14301416277885437, + -0.18305416405200958, + -0.10634835064411163, + -0.2634684145450592, + -0.05348178371787071, + -0.21431514620780945, + 0.13807232677936554, + -0.2242385447025299, + -0.09678935259580612, + -0.18892650306224823, + -0.10323800146579742, + -0.09472811967134476, + -0.2798849642276764, + 0.1722967028617859, + -1.1891459226608276, + 0.3821650445461273, + -0.5277573466300964, + -0.1239900067448616, + -0.025756916031241417, + -1.5671592950820923, + -1.708814024925232, + -0.94542396068573, + -0.8548623323440552, + -0.9579849243164062, + -0.9779664278030396, + -0.5873764157295227, + 0.9998621344566345, + 0.9998221397399902, + 0.9998023509979248, + 0.9998185634613037, + 0.9975361227989197, + 0.9988760948181152, + 0.360027939081192, + 0.4790782630443573, + 0.5461887717247009, + 0.48382189869880676, + 0.6635858416557312, + 0.0, + -0.10022255033254623, + -0.03946204483509064, + -1.0 + ], + "max": [ + 0.00482684513553977, + 0.13756881654262543, + 0.2979176640510559, + 0.3711587190628052, + 0.031152762472629547, + 0.010087746195495129, + 0.019378269091248512, + 0.1355365365743637, + 0.182325080037117, + 0.4443013668060303, + -0.05482839047908783, + 0.011374615132808685, + 0.165291890501976, + 0.05307222157716751, + 0.177412211894989, + 0.22115345299243927, + 0.6989883780479431, + -0.3335850238800049, + 1.1546826362609863, + 0.28441083431243896, + 0.08963758498430252, + 0.3550512492656708, + 1.1140739917755127, + -0.22596199810504913, + 1.7804871797561646, + 1.2184052467346191, + 1.2347023487091064, + 1.5067987442016602, + 1.0105962753295898, + 1.0, + 1.0, + 1.0, + 1.0, + 0.9982408285140991, + 0.9999225735664368, + 1.0, + 1.0, + 1.0, + 0.9999973177909851, + 1.0, + 1.0, + 0.038201265037059784, + 0.023285934701561928, + -0.9949650168418884 + ], + "mean": [ + -0.07244415528619287, + 0.016438895257606716, + -0.03188766069938736, + 0.19652274108921836, + -0.09995080381043964, + -0.0335864852112993, + -0.06201331924385435, + 0.0041316509641342114, + -0.07562538691067132, + 0.22342829523439264, + -0.13939505500881355, + -0.02568257955335654, + 0.020507808718722174, + 0.0030919114143910654, + 0.11976549024164236, + -0.04130542899628228, + 0.27413152575942573, + -0.667075950232086, + 0.9776525052369189, + -0.15594609082852004, + -0.04114803172895316, + 0.07066678088305102, + -0.44530701068620304, + -0.7956643808095055, + 0.6432070891275159, + 0.07140851169135853, + -0.22403887370387132, + 0.19658286211869352, + 0.3118955244879216, + 0.9999827171293537, + 0.999982850921418, + 0.9999830789480982, + 0.9999811406495951, + 0.9977658298817017, + 0.9997360855995177, + 0.7778945983312991, + 0.8171959691692932, + 0.8399288212288848, + 0.8394280370284749, + 0.8745686375938581, + 0.12291562838728093, + -0.023435156143715264, + -0.004820223051842413, + -0.999617456030026 + ], + "std": [ + 0.045712745420827386, + 0.013154301249487431, + 0.02405439153160751, + 0.0422870935042734, + 0.01604373013239405, + 0.010666838959405088, + 0.0447255039612766, + 0.014369854397034587, + 0.025451572022126993, + 0.05279699313574748, + 0.02213303083995789, + 0.011183898115253933, + 0.07041149453770397, + 0.015833724218855847, + 0.039560888033149964, + 0.08342761969885264, + 0.0587237913039723, + 0.22301655606787774, + 0.08125572382728762, + 0.1281115145665045, + 0.03600107397023974, + 0.04123008646611267, + 0.5831912382159029, + 0.23706341250281637, + 0.5432748450536339, + 0.504036388323598, + 0.31188020039153314, + 0.3436391052756804, + 0.26504982482523237, + 1.8214790335763182e-05, + 2.3385127857745866e-05, + 2.5684815254370892e-05, + 2.368414515012511e-05, + 7.802892144541008e-05, + 6.03931448414821e-05, + 0.2307055833669559, + 0.1900819078013402, + 0.16354430170740317, + 0.16593648793853702, + 0.1286487353890603, + 0.3083183803524622, + 0.013093107258771247, + 0.004580775936054731, + 0.0002959946385408924 + ] + }, + "action": { + "min": [ + 0.0, + -0.375, + -0.4375, + -0.125, + -0.375, + -0.3125, + -0.375, + -0.3125, + -0.25, + -0.25, + -0.5625, + -0.1875, + -0.125, + -0.3125, + -0.125, + -0.25, + -0.125, + -0.1875, + -0.5, + -0.25, + -0.1875, + -0.5625, + -0.25, + -0.375, + -0.4375, + -0.0625, + -0.375, + -0.375, + -0.3125, + -0.3125, + 0.0, + 0.125, + -0.3125, + 0.0, + 0.125, + -0.4375, + -0.1875, + -0.4375, + -0.25, + -0.1875, + -0.4375, + -0.375, + -0.1875, + -0.0625, + -0.3125, + -0.25, + -0.1875, + -0.3125, + -0.3125, + -0.5625, + -0.125, + -0.125, + -0.4375, + -0.5, + -0.1875, + -0.25, + -0.375, + -0.0625, + -0.4375, + -0.25, + -0.4375, + 0.0625, + -0.25, + 0.0, + 0.9998621344566345, + 0.9998221397399902, + 0.9998023509979248, + 0.9998185634613037, + 0.9975361227989197, + 0.9988760948181152, + 0.360027939081192, + 0.4790782630443573, + 0.5461887717247009, + 0.48382189869880676, + 0.6635858416557312, + 0.0 + ], + "max": [ + 0.375, + 0.25, + 0.25, + 0.1875, + 0.1875, + 0.1875, + 0.25, + 0.25, + 0.1875, + 0.125, + 0.0625, + 0.1875, + 0.5, + 0.1875, + 0.25, + 0.0625, + 0.3125, + 0.5, + 0.1875, + 0.25, + 0.25, + 0.0625, + 0.4375, + 0.25, + 0.0, + 0.4375, + 0.0, + 0.1875, + 0.1875, + 0.1875, + 0.4375, + 0.5, + 0.3125, + 0.3125, + 0.3125, + 0.0625, + 0.1875, + 0.3125, + 0.1875, + 0.25, + 0.0, + 0.25, + 0.125, + 0.5, + 0.125, + 0.5, + 0.4375, + 0.4375, + 0.4375, + 0.0, + 0.125, + 0.25, + 0.5, + -0.0625, + 0.125, + 0.4375, + 0.5, + 0.3125, + 0.125, + 0.125, + 0.125, + 0.5, + 0.125, + 0.6875, + 1.0, + 1.0, + 1.0, + 1.0, + 0.9982408285140991, + 0.9999225735664368, + 1.0, + 1.0, + 1.0, + 0.9999973177909851, + 1.0, + 1.0 + ], + "mean": [ + 0.2026803717194055, + -0.07236617450974016, + -0.02242552603336694, + 0.02358918176200852, + -0.12101816319229894, + -0.10377979771699893, + -0.04818144492503821, + -0.09367987576831767, + -0.015672257634394615, + -0.07295460827994406, + -0.28937628052944814, + 0.023045464893167257, + 0.12778362710982472, + -0.014680355458714104, + 0.047179379817229826, + -0.12413716706234348, + 0.0648974356889655, + 0.1738440680997756, + -0.1366050521968194, + 0.05253626134183226, + 0.005097726755341637, + -0.21554664379329408, + 0.09576226056131906, + -0.11485841002959446, + -0.21714120784415752, + 0.25317489999674786, + -0.23478507431135973, + -0.10450746365735471, + -0.049666249308920614, + -0.03343909558034407, + 0.2612117467234707, + 0.28021846238902076, + 0.013058351491105402, + 0.14741942827409021, + 0.23094043708738496, + -0.2599698364174445, + 0.054112532114865525, + -0.18685465380988, + -0.021627735861328823, + -0.045454730235129595, + -0.261119264041107, + -0.09780399362580897, + -0.03171647858466942, + 0.30728844840482616, + -0.13527167550164232, + 0.061584319815278545, + 0.07089966340368792, + 0.06740158216527367, + 0.18057091287521546, + -0.27359832840092363, + -0.01098104816416794, + 0.03545237243487593, + 0.1730208705974178, + -0.26741316790789943, + -0.03493507918956714, + 0.1694120133988097, + 0.01502691144427461, + 0.05571421021821848, + -0.22518415233015707, + -0.05392248528407428, + -0.1961598344661615, + 0.3429247699112166, + -0.04006126215486683, + 0.40486580864418353, + 0.9999827171293537, + 0.999982850921418, + 0.9999830789480982, + 0.9999811406495951, + 0.9977658298817017, + 0.9997360855995177, + 0.7778945983312991, + 0.8171959691692932, + 0.8399288212288848, + 0.8394280370284749, + 0.8745686375938581, + 0.12291562838728093 + ], + "std": [ + 0.07423638798986239, + 0.10276070894626457, + 0.11812702387971753, + 0.05957396656117891, + 0.1218417031488596, + 0.08063836551976374, + 0.1020635000819969, + 0.10540346375069484, + 0.08296831170197061, + 0.05989282207073844, + 0.131405419274155, + 0.07688508477180626, + 0.15530657456533145, + 0.09090021877289593, + 0.07144535330531092, + 0.07721423672952077, + 0.061937753226413185, + 0.12535445915332458, + 0.1406645362754369, + 0.09610483490082063, + 0.0892112540257282, + 0.10101183108941218, + 0.14459992137856284, + 0.09259695505428482, + 0.07250089525780803, + 0.07751064616894329, + 0.09003536032570492, + 0.08768655804866919, + 0.0793513264967945, + 0.0799717022695259, + 0.08767655242632699, + 0.06686962750120226, + 0.11139553980852172, + 0.05962527954838032, + 0.03113638664810605, + 0.1104336480130166, + 0.0900293543854962, + 0.17353368370821917, + 0.06621053001132303, + 0.07044267349366652, + 0.07468062148900616, + 0.13559357690832427, + 0.051673248141484364, + 0.11970962808802042, + 0.09873016730126778, + 0.1482997857923691, + 0.14644960009292055, + 0.12362418143308783, + 0.14892813932159055, + 0.1264928491368565, + 0.04485723360367807, + 0.06556638017875786, + 0.23014907315132233, + 0.07366488770406758, + 0.05520901424631441, + 0.1518289001396863, + 0.1840720418570084, + 0.05832378814080937, + 0.11903714776197126, + 0.05677283141546173, + 0.09207849117719222, + 0.09729334507011746, + 0.0780996197326025, + 0.16351388381996734, + 1.8214790335763182e-05, + 2.3385127857745866e-05, + 2.5684815254370892e-05, + 2.368414515012511e-05, + 7.802892144541008e-05, + 6.03931448414821e-05, + 0.2307055833669559, + 0.1900819078013402, + 0.16354430170740317, + 0.16593648793853702, + 0.1286487353890603, + 0.3083183803524622 + ] + } + }, + "code_commit": "028df1bb7024fb1e3d176de7e60c84d3333cd97f" +} \ No newline at end of file diff --git a/b8-table-wipe/b512-cont-20261117/step-006000/config.json b/b8-table-wipe/b512-cont-20261117/step-006000/config.json new file mode 100644 index 0000000000000000000000000000000000000000..f10f1d53eded07070465ea13d81d8c05f159e577 --- /dev/null +++ b/b8-table-wipe/b512-cont-20261117/step-006000/config.json @@ -0,0 +1,884 @@ +{ + "policy": { + "policy": "fm_dit", + "n_cameras": 2, + "state_dim": 44, + "action_dim": 76, + "horizon": 16, + "backbone": "resnet18", + "backbone_weights": "IMAGENET1K_V1", + "backbone_norm": "frozen_bn", + "hidden_dim": 384, + "num_layers": 4, + "num_heads": 8, + "mlp_ratio": 4.0, + "dropout": 0.1, + "time_embed_dim": 128, + "num_inference_steps": null, + "num_train_timesteps": 100, + "beta_schedule": "squaredcos_cap_v2", + "prediction_type": "epsilon", + "clip_sample": true, + "act_hidden_dim": 512, + "dim_feedforward": 3200, + "enc_layers": 4, + "dec_layers": 7, + "nheads": 8, + "latent_dim": 32, + "kl_weight": 10.0, + "extra": {} + }, + "contract": { + "name": "g1_sonic_dfx6", + "robot": "g1", + "cameras": [ + "ego_view", + "chest" + ], + "state": [ + [ + "observation.state", + 0, + 29 + ], + [ + "observation.inspire_left_hand_state", + 0, + 6 + ], + [ + "observation.inspire_right_hand_state", + 0, + 6 + ], + [ + "observation.projected_gravity", + 0, + 3 + ] + ], + "action": [ + [ + "action.motion_token", + 0, + 64 + ], + [ + "observation.inspire_left_hand_state", + 0, + 6 + ], + [ + "observation.inspire_right_hand_state", + 0, + 6 + ] + ], + "state_dim": 44, + "action_dim": 76, + "action_space": "sonic_dfx6", + "body_dims": 29 + }, + "image_hw": [ + 168, + 224 + ], + "task_text": "Approach the flat block with a naturally aligned wrist, stop before closing the hand, wipe sideways at a top-view 45-degree heading, then set the block on the tabletop and release it.", + "dataset": { + "name": "b8-table-wipe-expert-v1", + "root": "/mnt/data0/zuo/Humanoid_benchmark/data/baselines/b8-table-wipe-expert-v1", + "robot_type": null, + "fps": 30.0, + "episodes": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 36, + 37, + 38, + 39, + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55, + 56, + 57, + 58, + 59, + 60, + 61, + 62, + 63, + 64, + 65, + 66, + 67, + 68, + 69, + 70, + 71, + 72, + 73, + 74, + 75, + 76, + 77, + 78, + 79, + 80, + 81, + 82, + 83, + 84, + 85, + 86, + 87, + 88, + 89, + 90, + 91, + 92, + 93, + 94, + 95, + 96, + 97, + 98, + 99 + ], + "train_episodes": [ + 0, + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 10, + 11, + 12, + 13, + 14, + 15, + 16, + 17, + 18, + 19, + 20, + 21, + 22, + 23, + 24, + 25, + 26, + 27, + 28, + 29, + 30, + 31, + 32, + 33, + 34, + 35, + 36, + 37, + 38, + 39, + 40, + 41, + 42, + 43, + 44, + 45, + 46, + 47, + 48, + 49, + 50, + 51, + 52, + 53, + 54, + 55, + 56, + 57, + 58, + 59, + 60, + 61, + 62, + 63, + 64, + 65, + 66, + 67, + 68, + 69, + 70, + 71, + 72, + 73, + 74, + 75, + 76, + 77, + 78, + 79, + 80, + 81, + 82, + 83, + 84, + 85, + 86, + 87, + 88, + 89, + 90, + 91, + 92, + 93, + 94, + 95, + 96, + 97, + 98, + 99 + ], + "val_episodes": [], + "train_frames": 61498, + "git_commit": null + }, + "training": { + "steps": 20000, + "batch_size": 512, + "lr": 0.0001, + "betas": [ + 0.95, + 0.999 + ], + "weight_decay": 1e-06, + "warmup": 500, + "schedule": "linear warm-up + cosine", + "backbone_lr_scale": 0.1, + "ema": 0.9999, + "shift_px": 0, + "amp": true, + "seed": 0, + "augmentation": { + "spec": "docs/augmentation_v2.md", + "version": "hb-aug-v2-20261109", + "implementation": "evaluation/augmentation/hb_aug_v2.py", + "visual": { + "effects": [ + "exposure", + "contrast", + "saturation", + "hue", + "gamma", + "temperature", + "rotation_crop", + "noise", + "gaussian_blur", + "jpeg", + "motion_blur", + "sharpen", + "defocus", + "light_shadow", + "wb_channel" + ], + "boosted_per_sample": 1, + "boost": 1.5, + "pixel_scale": "H / 224", + "future_targets": "none (one observation frame)" + }, + "proprio": { + "robot": "g1", + "body_dims": 29, + "bias_deg": 2.0, + "bias_scope": "per_sample", + "noise_deg": 0.25, + "noise_clip_sigma": 3, + "delay_probability": 0.0, + "history_drop_probability": 0.0, + "velocity": false + }, + "native_image_augmentation": "off", + "evaluation": "off" + }, + "preflight": { + "samples": 64, + "state_dim": 44, + "body_dims": 29, + "images_changed": 64, + "body_values_moved": 1.0, + "max_body_offset_deg": 2.5819, + "actions_bit_identical": true, + "non_body_state_bit_identical": true, + "eval_items_deterministic": 8 + } + }, + "stats": { + "state": { + "min": [ + -0.2515260875225067, + -0.03310006856918335, + -0.08915279805660248, + 0.14301416277885437, + -0.18305416405200958, + -0.10634835064411163, + -0.2634684145450592, + -0.05348178371787071, + -0.21431514620780945, + 0.13807232677936554, + -0.2242385447025299, + -0.09678935259580612, + -0.18892650306224823, + -0.10323800146579742, + -0.09472811967134476, + -0.2798849642276764, + 0.1722967028617859, + -1.1891459226608276, + 0.3821650445461273, + -0.5277573466300964, + -0.1239900067448616, + -0.025756916031241417, + -1.5671592950820923, + -1.708814024925232, + -0.94542396068573, + -0.8548623323440552, + -0.9579849243164062, + -0.9779664278030396, + -0.5873764157295227, + 0.9998621344566345, + 0.9998221397399902, + 0.9998023509979248, + 0.9998185634613037, + 0.9975361227989197, + 0.9988760948181152, + 0.360027939081192, + 0.4790782630443573, + 0.5461887717247009, + 0.48382189869880676, + 0.6635858416557312, + 0.0, + -0.10022255033254623, + -0.03946204483509064, + -1.0 + ], + "max": [ + 0.00482684513553977, + 0.13756881654262543, + 0.2979176640510559, + 0.3711587190628052, + 0.031152762472629547, + 0.010087746195495129, + 0.019378269091248512, + 0.1355365365743637, + 0.182325080037117, + 0.4443013668060303, + -0.05482839047908783, + 0.011374615132808685, + 0.165291890501976, + 0.05307222157716751, + 0.177412211894989, + 0.22115345299243927, + 0.6989883780479431, + -0.3335850238800049, + 1.1546826362609863, + 0.28441083431243896, + 0.08963758498430252, + 0.3550512492656708, + 1.1140739917755127, + -0.22596199810504913, + 1.7804871797561646, + 1.2184052467346191, + 1.2347023487091064, + 1.5067987442016602, + 1.0105962753295898, + 1.0, + 1.0, + 1.0, + 1.0, + 0.9982408285140991, + 0.9999225735664368, + 1.0, + 1.0, + 1.0, + 0.9999973177909851, + 1.0, + 1.0, + 0.038201265037059784, + 0.023285934701561928, + -0.9949650168418884 + ], + "mean": [ + -0.07244415528619287, + 0.016438895257606716, + -0.03188766069938736, + 0.19652274108921836, + -0.09995080381043964, + -0.0335864852112993, + -0.06201331924385435, + 0.0041316509641342114, + -0.07562538691067132, + 0.22342829523439264, + -0.13939505500881355, + -0.02568257955335654, + 0.020507808718722174, + 0.0030919114143910654, + 0.11976549024164236, + -0.04130542899628228, + 0.27413152575942573, + -0.667075950232086, + 0.9776525052369189, + -0.15594609082852004, + -0.04114803172895316, + 0.07066678088305102, + -0.44530701068620304, + -0.7956643808095055, + 0.6432070891275159, + 0.07140851169135853, + -0.22403887370387132, + 0.19658286211869352, + 0.3118955244879216, + 0.9999827171293537, + 0.999982850921418, + 0.9999830789480982, + 0.9999811406495951, + 0.9977658298817017, + 0.9997360855995177, + 0.7778945983312991, + 0.8171959691692932, + 0.8399288212288848, + 0.8394280370284749, + 0.8745686375938581, + 0.12291562838728093, + -0.023435156143715264, + -0.004820223051842413, + -0.999617456030026 + ], + "std": [ + 0.045712745420827386, + 0.013154301249487431, + 0.02405439153160751, + 0.0422870935042734, + 0.01604373013239405, + 0.010666838959405088, + 0.0447255039612766, + 0.014369854397034587, + 0.025451572022126993, + 0.05279699313574748, + 0.02213303083995789, + 0.011183898115253933, + 0.07041149453770397, + 0.015833724218855847, + 0.039560888033149964, + 0.08342761969885264, + 0.0587237913039723, + 0.22301655606787774, + 0.08125572382728762, + 0.1281115145665045, + 0.03600107397023974, + 0.04123008646611267, + 0.5831912382159029, + 0.23706341250281637, + 0.5432748450536339, + 0.504036388323598, + 0.31188020039153314, + 0.3436391052756804, + 0.26504982482523237, + 1.8214790335763182e-05, + 2.3385127857745866e-05, + 2.5684815254370892e-05, + 2.368414515012511e-05, + 7.802892144541008e-05, + 6.03931448414821e-05, + 0.2307055833669559, + 0.1900819078013402, + 0.16354430170740317, + 0.16593648793853702, + 0.1286487353890603, + 0.3083183803524622, + 0.013093107258771247, + 0.004580775936054731, + 0.0002959946385408924 + ] + }, + "action": { + "min": [ + 0.0, + -0.375, + -0.4375, + -0.125, + -0.375, + -0.3125, + -0.375, + -0.3125, + -0.25, + -0.25, + -0.5625, + -0.1875, + -0.125, + -0.3125, + -0.125, + -0.25, + -0.125, + -0.1875, + -0.5, + -0.25, + -0.1875, + -0.5625, + -0.25, + -0.375, + -0.4375, + -0.0625, + -0.375, + -0.375, + -0.3125, + -0.3125, + 0.0, + 0.125, + -0.3125, + 0.0, + 0.125, + -0.4375, + -0.1875, + -0.4375, + -0.25, + -0.1875, + -0.4375, + -0.375, + -0.1875, + -0.0625, + -0.3125, + -0.25, + -0.1875, + -0.3125, + -0.3125, + -0.5625, + -0.125, + -0.125, + -0.4375, + -0.5, + -0.1875, + -0.25, + -0.375, + -0.0625, + -0.4375, + -0.25, + -0.4375, + 0.0625, + -0.25, + 0.0, + 0.9998621344566345, + 0.9998221397399902, + 0.9998023509979248, + 0.9998185634613037, + 0.9975361227989197, + 0.9988760948181152, + 0.360027939081192, + 0.4790782630443573, + 0.5461887717247009, + 0.48382189869880676, + 0.6635858416557312, + 0.0 + ], + "max": [ + 0.375, + 0.25, + 0.25, + 0.1875, + 0.1875, + 0.1875, + 0.25, + 0.25, + 0.1875, + 0.125, + 0.0625, + 0.1875, + 0.5, + 0.1875, + 0.25, + 0.0625, + 0.3125, + 0.5, + 0.1875, + 0.25, + 0.25, + 0.0625, + 0.4375, + 0.25, + 0.0, + 0.4375, + 0.0, + 0.1875, + 0.1875, + 0.1875, + 0.4375, + 0.5, + 0.3125, + 0.3125, + 0.3125, + 0.0625, + 0.1875, + 0.3125, + 0.1875, + 0.25, + 0.0, + 0.25, + 0.125, + 0.5, + 0.125, + 0.5, + 0.4375, + 0.4375, + 0.4375, + 0.0, + 0.125, + 0.25, + 0.5, + -0.0625, + 0.125, + 0.4375, + 0.5, + 0.3125, + 0.125, + 0.125, + 0.125, + 0.5, + 0.125, + 0.6875, + 1.0, + 1.0, + 1.0, + 1.0, + 0.9982408285140991, + 0.9999225735664368, + 1.0, + 1.0, + 1.0, + 0.9999973177909851, + 1.0, + 1.0 + ], + "mean": [ + 0.2026803717194055, + -0.07236617450974016, + -0.02242552603336694, + 0.02358918176200852, + -0.12101816319229894, + -0.10377979771699893, + -0.04818144492503821, + -0.09367987576831767, + -0.015672257634394615, + -0.07295460827994406, + -0.28937628052944814, + 0.023045464893167257, + 0.12778362710982472, + -0.014680355458714104, + 0.047179379817229826, + -0.12413716706234348, + 0.0648974356889655, + 0.1738440680997756, + -0.1366050521968194, + 0.05253626134183226, + 0.005097726755341637, + -0.21554664379329408, + 0.09576226056131906, + -0.11485841002959446, + -0.21714120784415752, + 0.25317489999674786, + -0.23478507431135973, + -0.10450746365735471, + -0.049666249308920614, + -0.03343909558034407, + 0.2612117467234707, + 0.28021846238902076, + 0.013058351491105402, + 0.14741942827409021, + 0.23094043708738496, + -0.2599698364174445, + 0.054112532114865525, + -0.18685465380988, + -0.021627735861328823, + -0.045454730235129595, + -0.261119264041107, + -0.09780399362580897, + -0.03171647858466942, + 0.30728844840482616, + -0.13527167550164232, + 0.061584319815278545, + 0.07089966340368792, + 0.06740158216527367, + 0.18057091287521546, + -0.27359832840092363, + -0.01098104816416794, + 0.03545237243487593, + 0.1730208705974178, + -0.26741316790789943, + -0.03493507918956714, + 0.1694120133988097, + 0.01502691144427461, + 0.05571421021821848, + -0.22518415233015707, + -0.05392248528407428, + -0.1961598344661615, + 0.3429247699112166, + -0.04006126215486683, + 0.40486580864418353, + 0.9999827171293537, + 0.999982850921418, + 0.9999830789480982, + 0.9999811406495951, + 0.9977658298817017, + 0.9997360855995177, + 0.7778945983312991, + 0.8171959691692932, + 0.8399288212288848, + 0.8394280370284749, + 0.8745686375938581, + 0.12291562838728093 + ], + "std": [ + 0.07423638798986239, + 0.10276070894626457, + 0.11812702387971753, + 0.05957396656117891, + 0.1218417031488596, + 0.08063836551976374, + 0.1020635000819969, + 0.10540346375069484, + 0.08296831170197061, + 0.05989282207073844, + 0.131405419274155, + 0.07688508477180626, + 0.15530657456533145, + 0.09090021877289593, + 0.07144535330531092, + 0.07721423672952077, + 0.061937753226413185, + 0.12535445915332458, + 0.1406645362754369, + 0.09610483490082063, + 0.0892112540257282, + 0.10101183108941218, + 0.14459992137856284, + 0.09259695505428482, + 0.07250089525780803, + 0.07751064616894329, + 0.09003536032570492, + 0.08768655804866919, + 0.0793513264967945, + 0.0799717022695259, + 0.08767655242632699, + 0.06686962750120226, + 0.11139553980852172, + 0.05962527954838032, + 0.03113638664810605, + 0.1104336480130166, + 0.0900293543854962, + 0.17353368370821917, + 0.06621053001132303, + 0.07044267349366652, + 0.07468062148900616, + 0.13559357690832427, + 0.051673248141484364, + 0.11970962808802042, + 0.09873016730126778, + 0.1482997857923691, + 0.14644960009292055, + 0.12362418143308783, + 0.14892813932159055, + 0.1264928491368565, + 0.04485723360367807, + 0.06556638017875786, + 0.23014907315132233, + 0.07366488770406758, + 0.05520901424631441, + 0.1518289001396863, + 0.1840720418570084, + 0.05832378814080937, + 0.11903714776197126, + 0.05677283141546173, + 0.09207849117719222, + 0.09729334507011746, + 0.0780996197326025, + 0.16351388381996734, + 1.8214790335763182e-05, + 2.3385127857745866e-05, + 2.5684815254370892e-05, + 2.368414515012511e-05, + 7.802892144541008e-05, + 6.03931448414821e-05, + 0.2307055833669559, + 0.1900819078013402, + 0.16354430170740317, + 0.16593648793853702, + 0.1286487353890603, + 0.3083183803524622 + ] + } + }, + "code_commit": "028df1bb7024fb1e3d176de7e60c84d3333cd97f", + "step": 6000, + "weights": "ema", + "offline_eval": { + "mse": 0.022521788254380226, + "mse_row0": 0.02053411304950714, + "mse_motion_token": 0.022982770577073097, + "mse_inspire_left_hand_state": 0.012518338859081268, + "mse_inspire_right_hand_state": 0.027608102187514305, + "samples": 256, + "frames": "train" + }, + "format": "hb_baselines/1" +} \ No newline at end of file diff --git a/b8-table-wipe/b512-cont-20261117/step-006000/model.pt b/b8-table-wipe/b512-cont-20261117/step-006000/model.pt new file mode 100644 index 0000000000000000000000000000000000000000..a0a0570347ff1d0add1a28fcdcefdf1c8af5574a --- /dev/null +++ b/b8-table-wipe/b512-cont-20261117/step-006000/model.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:49f877e3a020061ca3fd91b94f178596a9348d02548e116ea261eeecb5675ed8 +size 90311637 diff --git a/b8-table-wipe/b512-cont-20261117/step-006000/protocol.json b/b8-table-wipe/b512-cont-20261117/step-006000/protocol.json new file mode 100644 index 0000000000000000000000000000000000000000..ab417502bf0a63ac570084bb189a6486289bdb8c --- /dev/null +++ b/b8-table-wipe/b512-cont-20261117/step-006000/protocol.json @@ -0,0 +1,11 @@ +{ + "format": "hb_baselines/1", + "step": 6000, + "policy": "fm_dit", + "files": { + "model.pt": { + "bytes": 90311637, + "sha256": "49f877e3a020061ca3fd91b94f178596a9348d02548e116ea261eeecb5675ed8" + } + } +} \ No newline at end of file diff --git a/b8-table-wipe/b512-cont-20261117/train_state.pt b/b8-table-wipe/b512-cont-20261117/train_state.pt new file mode 100644 index 0000000000000000000000000000000000000000..c4a12c4e978f174f7929509f2b5867e8a27a1423 --- /dev/null +++ b/b8-table-wipe/b512-cont-20261117/train_state.pt @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:defddc0d62576449aad58f234a7023e3e21eb0289831e03dac2c19ffd888f097 +size 361027889 diff --git a/code/fmdit_continue_20261117/augmentation/__init__.py b/code/fmdit_continue_20261117/augmentation/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..7737a758f4f1311b1f658e8a64f626371eae52d3 --- /dev/null +++ b/code/fmdit_continue_20261117/augmentation/__init__.py @@ -0,0 +1,110 @@ +"""Humanoid Benchmark data augmentation spec v2 (docs/augmentation_v2.md): the training-time entry every model uses. + +`hb_aug_v2.py` is an unmodified copy of HumanWAM `hb_training/hb_groot/hb_aug_v2.py` (branch bai, commit af94cde, +git blob 75542085), the implementation GR00T, pi0.5, Psi-Zero, ACT, Delta Flow and DiT4DiT train with. Do not edit +it here; a new version of the spec is a new file with a new VERSION. + +How a training entry uses it (the FM-DiT / DP / ACT baselines in evaluation/baselines do exactly this): + +- Visual, once per training sample: after the model's native deterministic resize / center crop and before its + normalization, stack every camera and every frame of the sample into x float [T, V, 3, H, W] in [0, 1] and call + `augment_images(x, obs_mask)` (obs_mask[t] False marks a future image target: it gets the colour, geometry, light + patch and white-balance effects of its camera but none of the sensor/optics effects). `augment_frames_uint8` does + the same for uint8 (V, H, W, 3) frames. Pixel-size effects scale with H / 224 inside hb_aug_v2. +- Proprio, once per training sample: `perturb_body(state, body_dims)` adds the spec's per-joint bias (U +-2 deg, + drawn per sample, shared by every frame of the sample) and N(0, 0.25 deg) noise clipped at 3 sigma to the leading + `body_dims` raw-radian joint angles (G1 29, H2 31 incl. the head), before normalization. Hands, gravity and every + action label stay bit-identical. No proprio delay, no history drop. +- The model's own image augmentation (random crop / shift, colour jitter, rotation, ...) is off. +- Evaluation, offline checks and rollouts call none of this: same resize, same input layout, same normalization. +- Randomness comes from `worker_rng()`, one numpy Generator per process (DataLoader worker) seeded from its pid and + os.urandom, never from the global np.random (which DataLoader seeds identically on every rank). + +The interface state is raw and velocity-free: G1 44 = body 29 + measured hands 12 + projected gravity 3; H2 78 = +75 joints (body 31 incl. head, Sharpa hands 22 + 22) + projected gravity 3 (`STATE_DIMS`, `check_state_layout`). +""" +from __future__ import annotations + +import math +import os +import re + +import numpy as np + +from . import hb_aug_v2 + +VERSION = 'hb-aug-v2-20261109' +if hb_aug_v2.VERSION != VERSION: + raise ImportError(f'evaluation/augmentation/hb_aug_v2.py is {hb_aug_v2.VERSION}, expected {VERSION}') + +BODY_JOINTS = {'g1': 29, 'h2_sharpa': 31} # leading raw-radian body joint angles of the state (H2: incl. head) +STATE_DIMS = {'g1': 44, 'h2_sharpa': 78} # the benchmark state of each robot, no joint velocity +BIAS_DEG, NOISE_DEG = 2.0, 0.25 # = hb_aug_v2.perturb_joints + +_RNG = {'pid': None, 'rng': None} + + +def worker_rng() -> np.random.Generator: + """This process's generator, re-created after a fork (DataLoader workers) from its pid and os.urandom.""" + pid = os.getpid() + if _RNG['pid'] != pid: + _RNG['rng'] = np.random.default_rng([pid, int.from_bytes(os.urandom(8), 'little')]) + _RNG['pid'] = pid + return _RNG['rng'] + + +def augment_images(x, obs_mask=None, rng=None): + """hb_aug_v2.augment: x float [T, V, 3, H, W] in [0, 1], one call per training sample -> (augmented, params).""" + if x.ndim != 5 or x.shape[2] != 3: + raise ValueError(f'expected [T, V, 3, H, W], got {tuple(x.shape)}') + return hb_aug_v2.augment(x, rng if rng is not None else worker_rng(), obs_mask=obs_mask) + + +def augment_frames_uint8(frames, rng=None): + """One sample's uint8 frames (V, H, W, 3) (one time step) -> augmented uint8 frames of the same shape (rounded to + the nearest level, as the GR00T and Psi-Zero integrations do), and the parameters.""" + import torch + frames = torch.as_tensor(frames) + if frames.dtype != torch.uint8 or frames.ndim != 4 or frames.shape[-1] != 3: + raise ValueError(f'expected uint8 (V, H, W, 3) frames, got {frames.dtype} {tuple(frames.shape)}') + x = frames.permute(0, 3, 1, 2).float().div(255.0)[None] # [1, V, 3, H, W] + y, params = augment_images(x, rng=rng) + out = (y[0] * 255.0 + 0.5).clamp(0, 255).to(torch.uint8).permute(0, 2, 3, 1).contiguous() + return out, params + + +def perturb_body(state, body_dims: int, rng=None) -> np.ndarray: + """A float32 copy of `state` (..., D) whose first `body_dims` values got hb_aug_v2.perturb_joints (one bias per + joint for the whole sample, noise per value); the other values are copied bit for bit.""" + state = np.array(state, dtype=np.float32, copy=True) + if not 0 < body_dims <= state.shape[-1]: + raise ValueError(f'body_dims {body_dims} outside the state width {state.shape[-1]}') + state[..., :body_dims] = hb_aug_v2.perturb_joints(state[..., :body_dims], rng if rng is not None else worker_rng()) + return state + + +def check_state_layout(robot: str, state_dim: int, columns=()) -> None: + """The benchmark interface: G1 44 / H2 78 raw state values, no velocity column.""" + if robot not in STATE_DIMS: + raise KeyError(f'no augmentation layout for robot {robot!r}; known: {sorted(STATE_DIMS)}') + if state_dim != STATE_DIMS[robot]: + raise ValueError(f'{robot} state must be {STATE_DIMS[robot]}-D (no joint velocity), got {state_dim}') + velocity = [c for c in columns if re.search(r'(velocit|(^|[._])(vel|qvel|dq)([._]|$))', str(c).lower())] + if velocity: + raise ValueError(f'joint velocity is not allowed in the model input: {velocity}') + + +def max_body_offset_rad() -> float: + """The largest |perturbation| perturb_body can add to one joint: bias 2 deg + clipped noise 3 x 0.25 deg.""" + return math.radians(BIAS_DEG + 3 * NOISE_DEG) + + +def describe(robot: str, body_dims: int) -> dict: + """The record a training run stores (run.json / checkpoint config).""" + return dict(spec='docs/augmentation_v2.md', version=VERSION, implementation='evaluation/augmentation/hb_aug_v2.py', + visual=dict(effects=list(hb_aug_v2.EFFECTS), boosted_per_sample=1, boost=1.5, + pixel_scale='H / 224', future_targets='none (one observation frame)'), + proprio=dict(robot=robot, body_dims=int(body_dims), bias_deg=BIAS_DEG, bias_scope='per_sample', + noise_deg=NOISE_DEG, noise_clip_sigma=3, delay_probability=0.0, + history_drop_probability=0.0, velocity=False), + native_image_augmentation='off', evaluation='off') diff --git a/code/fmdit_continue_20261117/augmentation/check_spec.py b/code/fmdit_continue_20261117/augmentation/check_spec.py new file mode 100644 index 0000000000000000000000000000000000000000..2bba9d4f6a2b6ad752d62099795dd79fc4f96f2e --- /dev/null +++ b/code/fmdit_continue_20261117/augmentation/check_spec.py @@ -0,0 +1,174 @@ +"""Audit hb_aug_v2.sample() against docs/augmentation_v2.md by drawing many parameter sets. + + python -I evaluation/augmentation/check_spec.py [--draws 300000] [--height 168] [--out report.json] + +Checks, for the boosted (1.5 x) and the unboosted (1 x) draws of every effect separately: the drawn values lie in the +spec's range and reach both ends of it, the firing probability matches the spec (within 5 binomial sigma), and the +boosted effect is uniform over the 15 effects (chi-square, 14 degrees of freedom, p > 0.001). Pixel-size effects +(motion blur, defocus) are checked at --height / 224. Prints one JSON report; exits 1 if any check fails. +""" +from __future__ import annotations + +import argparse +import json +import math +from pathlib import Path +import sys + +import numpy as np + +sys.path.insert(0, str(Path(__file__).resolve().parents[1])) +import augmentation # noqa: E402 (asserts VERSION) +from augmentation import hb_aug_v2 as A # noqa: E402 + +CHI2_14_P001 = 36.123 # chi-square quantile, 14 degrees of freedom, p = 0.001 + +# (probability, low, high) at 1 x; the 1.5 x range is centred on identity (the spec's bracketed values) +SPEC = { + 'exposure': (1.0, -0.22, 0.22, 0.0), 'contrast': (1.0, 0.88, 1.12, 1.0), 'saturation': (1.0, 0.76, 1.24, 1.0), + 'hue': (1.0, -0.015, 0.015, 0.0), 'gamma': (1.0, 0.89, 1.11, 1.0), 'temperature': (1.0, -0.066, 0.066, 0.0), + 'rotation_crop': (1.0, -1.5, 1.5, 0.0), # degrees; extra crop zoom U(0, 0.02) on top of |angle| x 0.015 + 'noise': (0.5, 4 / 255, 4 / 255, None), + 'gaussian_blur': (0.15, 0.25, 0.65, None), # sigma (px) + 'jpeg': (0.35, 70, 95, None), # quality; 1.5 x: q -> 100 - 1.5 (100 - q) + 'motion_blur': (0.15, 3.0, 7.0, None), # length at 224 px; angle U(0, 180) + 'sharpen': (0.15, 0.2, 0.6, None), + 'defocus': (0.10, 1.0, 2.0, None), # disk radius at 224 px + 'light_shadow': (0.30, 0.70, 1.30, 1.0), # gain; 1-2 blobs per camera, size 0.15-0.35, centre 0.1-0.9 + 'wb_channel': (0.30, 0.90, 1.10, 1.0), # gain of one channel per camera +} + + +def expected_range(name, boost, px): + prob, lo, hi, centre = SPEC[name] + k = 1.5 if boost else 1.0 + if name == 'jpeg': + return int(round(100 - k * (100 - lo))), int(round(100 - k * (100 - hi))) + if centre is not None: + return centre + k * (lo - centre), centre + k * (hi - centre) + scale = k * (px if name in ('motion_blur', 'defocus') else 1.0) + return lo * scale, hi * scale + + +def values(p, name, views): + """The drawn magnitudes of `name` in one parameter set (empty when it did not fire), plus side values.""" + v = p.get(name) # rotation_crop lives in p['geometry'] + if name in ('exposure', 'contrast', 'saturation', 'hue', 'gamma', 'temperature'): + return [v], {} + if name == 'rotation_crop': + angles = [a for a, _ in p['geometry']] + extra = [s - 1.0 - abs(a) * 0.015 for a, s in p['geometry']] + return angles, dict(crop_extra=extra) + if name == 'noise' or name in ('gaussian_blur', 'sharpen', 'defocus'): + return ([v] if v else []), {} + if name == 'jpeg': + return ([v] if v is not None else []), {} + if name == 'motion_blur': + return ([v[0]] if v is not None else []), dict(angle=[v[1]] if v is not None else []) + if name == 'light_shadow': + if v is None: + return [], {} + blobs = [b for cam in v for b in cam] + return [b[3] for b in blobs], dict(size=[b[2] for b in blobs], centre=[c for b in blobs for c in b[:2]], + blobs_per_camera=[len(cam) for cam in v], cameras=[len(v)]) + if name == 'wb_channel': + if v is None: + return [], {} + changed = [[i for i, g in enumerate(row) if g != 1.0] for row in v] + return [row[i] for row in v for i in range(3) if row[i] != 1.0], dict( + channels_changed=[len(c) for c in changed], cameras=[len(v)]) + raise KeyError(name) + + +def audit(draws=300000, height=168, views=2, seed=0): + rng = np.random.default_rng(seed) + px = height / 224.0 + boosts = [] + acc = {(n, b): dict(fired=0, total=0, vals=[], side={}) for n in A.EFFECTS for b in (False, True)} + for _ in range(draws): + p = A.sample(rng, views, height) + boosts.append(p['boost']) + for name in A.EFFECTS: + entry = acc[(name, p['boost'] == name)] + vals, side = values(p, name, views) + entry['total'] += 1 + entry['fired'] += int(bool(vals)) + entry['vals'].extend(vals) + for key, items in side.items(): + entry['side'].setdefault(key, []).extend(items) + counts = {n: boosts.count(n) for n in A.EFFECTS} + expect = draws / len(A.EFFECTS) + chi2 = sum((c - expect) ** 2 / expect for c in counts.values()) + report = dict(version=A.VERSION, draws=draws, height=height, pixel_scale=px, views=views, + boost=dict(counts=counts, chi2=round(chi2, 2), chi2_limit_p001=CHI2_14_P001, + max_rel_dev=round(max(abs(c - expect) / expect for c in counts.values()), 4), + ok=bool(chi2 < CHI2_14_P001)), + effects={}, failures=[]) + for name in A.EFFECTS: + rows = {} + for boost in (False, True): + e = acc[(name, boost)] + prob = SPEC[name][0] + lo, hi = expected_range(name, boost, px) + vals = np.asarray(e['vals'], dtype=np.float64) + fired = e['fired'] / max(1, e['total']) + sigma = math.sqrt(prob * (1 - prob) / max(1, e['total'])) if 0 < prob < 1 else 0.0 + span = max(hi - lo, 1e-12) + row = dict(samples=e['total'], fired=round(fired, 4), spec_probability=prob, + spec_range=[round(lo, 6), round(hi, 6)], + observed=[round(float(vals.min()), 6), round(float(vals.max()), 6)] if len(vals) else None) + ok = abs(fired - prob) <= 5 * sigma + 1e-12 and len(vals) > 0 + ok = ok and vals.min() >= lo - 1e-9 and vals.max() <= hi + 1e-9 + if hi > lo: # continuous / integer range: both ends reached (2 % of the span) + reach = 0.02 * span if name != 'jpeg' else 0.0 + ok = ok and vals.min() <= lo + reach and vals.max() >= hi - reach + side = e['side'] + k = 1.5 if boost else 1.0 + checks = [] + if name == 'rotation_crop': + x = np.asarray(side['crop_extra']) + checks.append(('crop_extra', x.min() >= -1e-9 and x.max() <= 0.02 * k + 1e-9 and x.max() > 0.019 * k)) + row['crop_extra'] = [round(float(x.min()), 5), round(float(x.max()), 5)] + if name == 'motion_blur': + x = np.asarray(side['angle']) + checks.append(('angle_0_180', x.min() >= 0 and x.max() <= 180 and x.max() > 176 and x.min() < 4)) + if name == 'light_shadow': + checks.append(('size', min(side['size']) >= 0.15 and max(side['size']) <= 0.35)) + checks.append(('centre', min(side['centre']) >= 0.1 and max(side['centre']) <= 0.9)) + checks.append(('blobs_1_2', set(side['blobs_per_camera']) == {1, 2})) + checks.append(('per_camera', set(side['cameras']) == {views})) + if name == 'wb_channel': + checks.append(('one_channel', set(side['channels_changed']) <= {0, 1})) + checks.append(('per_camera', set(side['cameras']) == {views})) + if checks: + row['side_checks'] = {n: bool(c) for n, c in checks} + ok = ok and all(c for _, c in checks) + row['ok'] = bool(ok) + if not ok: + report['failures'].append(f'{name} {"1.5x" if boost else "1x"}') + rows['1.5x' if boost else '1x'] = row + report['effects'][name] = rows + if not report['boost']['ok']: + report['failures'].append('boost not uniform') + report['ok'] = not report['failures'] + return report + + +def main(): + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument('--draws', type=int, default=300000) + ap.add_argument('--height', type=int, default=168, help='model input height (pixel effects scale by H / 224)') + ap.add_argument('--views', type=int, default=2) + ap.add_argument('--seed', type=int, default=0) + ap.add_argument('--out', type=Path) + args = ap.parse_args() + report = audit(args.draws, args.height, args.views, args.seed) + text = json.dumps(report, indent=1) + if args.out: + args.out.write_text(text) + print(text) + sys.exit(0 if report['ok'] else 1) + + +if __name__ == '__main__': + main() diff --git a/code/fmdit_continue_20261117/augmentation/hb_aug_v2.py b/code/fmdit_continue_20261117/augmentation/hb_aug_v2.py new file mode 100644 index 0000000000000000000000000000000000000000..75542085518d31cc73146060a3c34099eee6ae8d --- /dev/null +++ b/code/fmdit_continue_20261117/augmentation/hb_aug_v2.py @@ -0,0 +1,163 @@ +"""Humanoid Benchmark data augmentation v2 (project CLAUDE.md), model-agnostic torch implementation. + +augment(x, rng, obs_mask=None) takes x: float [T, V, 3, H, W] in [0, 1] (T frames, V cameras) and returns the +augmented tensor. Each call is one training sample: one of the 15 effects is chosen uniformly and its magnitude +is scaled by 1.5 around identity; the rest stay at x1. Colour is shared across cameras and time; geometry, light +patches and white balance are drawn per camera and fixed over time; sensor/optics effects (noise, gaussian blur, +JPEG, motion blur, sharpen, defocus) apply only to observation frames (obs_mask True). +Pixel-size parameters are defined at 224 px and scaled by H / 224. +perturb_joints(q, rng) applies the proprio spec: per-joint bias +-2 deg plus N(0, 0.25 deg) clipped at 3 sigma. +""" +import io +import math + +import numpy as np +import torch +import torch.nn.functional as F +from PIL import Image +from torchvision.transforms import InterpolationMode +from torchvision.transforms import functional as TF + +EFFECTS = ("exposure", "contrast", "saturation", "hue", "gamma", "temperature", "rotation_crop", "noise", + "gaussian_blur", "jpeg", "motion_blur", "sharpen", "defocus", "light_shadow", "wb_channel") +CENTER = {"exposure": 0.0, "contrast": 1.0, "saturation": 1.0, "hue": 0.0, "gamma": 1.0, "temperature": 0.0} +RANGE = {"exposure": (-0.22, 0.22), "contrast": (0.88, 1.12), "saturation": (0.76, 1.24), "hue": (-0.015, 0.015), + "gamma": (0.89, 1.11), "temperature": (-0.066, 0.066)} +VERSION = "hb-aug-v2-20261109" + + +def sample(rng, views, height): + boost = EFFECTS[int(rng.integers(len(EFFECTS)))] + + def k(name): + return 1.5 if boost == name else 1.0 + + px = height / 224.0 + p = {"boost": boost} + for name, (a, b) in RANGE.items(): + c = CENTER[name] + p[name] = float(rng.uniform(c + k(name) * (a - c), c + k(name) * (b - c))) + p["geometry"] = [] + for _ in range(views): + angle = float(rng.uniform(-1.5 * k("rotation_crop"), 1.5 * k("rotation_crop"))) + p["geometry"].append((angle, 1.0 + abs(angle) * 0.015 + float(rng.uniform(0, 0.02 * k("rotation_crop"))))) + p["noise"] = 4.0 / 255.0 * k("noise") if rng.random() < 0.5 else 0.0 + p["gaussian_blur"] = float(rng.uniform(0.25, 0.65)) * k("gaussian_blur") if rng.random() < 0.15 else 0.0 + if rng.random() < 0.35: + lo = int(round(100 - k("jpeg") * 30)) + hi = int(round(100 - k("jpeg") * 5)) + p["jpeg"] = int(rng.integers(lo, hi + 1)) + else: + p["jpeg"] = None + p["motion_blur"] = ((float(rng.uniform(3, 7)) * k("motion_blur") * px, float(rng.uniform(0, 180))) + if rng.random() < 0.15 else None) + p["sharpen"] = float(rng.uniform(0.2, 0.6)) * k("sharpen") if rng.random() < 0.15 else 0.0 + p["defocus"] = float(rng.uniform(1, 2)) * k("defocus") * px if rng.random() < 0.10 else 0.0 + if rng.random() < 0.30: + g = 0.30 * k("light_shadow") + p["light_shadow"] = [[(float(rng.uniform(0.1, 0.9)), float(rng.uniform(0.1, 0.9)), + float(rng.uniform(0.15, 0.35)), float(rng.uniform(1 - g, 1 + g))) + for _ in range(int(rng.integers(1, 3)))] for _ in range(views)] + else: + p["light_shadow"] = None + if rng.random() < 0.30: + g = 0.10 * k("wb_channel") + gains = [] + for _ in range(views): + row = [1.0, 1.0, 1.0] + row[int(rng.integers(0, 3))] = float(rng.uniform(1 - g, 1 + g)) + gains.append(row) + p["wb_channel"] = gains + else: + p["wb_channel"] = None + return p + + +def _conv(x, kernel): + n, c, h, w = x.shape + k = kernel.to(x)[None, None].repeat(c, 1, 1, 1) + pad = kernel.shape[-1] // 2 + return F.conv2d(F.pad(x, (pad, pad, pad, pad), mode="reflect"), k, groups=c) + + +def motion_kernel(length, angle_deg): + size = max(3, int(round(length)) | 1) + k = torch.zeros(size, size) + c = size // 2 + th = math.radians(angle_deg) + for s in np.linspace(-(length - 1) / 2, (length - 1) / 2, 8 * size): + k[int(round(c - s * math.sin(th))), int(round(c + s * math.cos(th)))] = 1.0 + return k / k.sum() + + +def disk_kernel(r): + size = 2 * int(math.ceil(r)) + 1 + c = size // 2 + yy, xx = torch.meshgrid(torch.arange(size), torch.arange(size), indexing="ij") + k = (((yy - c) ** 2 + (xx - c) ** 2) <= (r + 0.5) ** 2).float() + return k / k.sum() + + +def apply(x, p, obs_mask=None, noise_seed=0): + t, v, c, h, w = x.shape + x = x.clone().clamp(0, 1) + for vi, (angle, scale) in enumerate(p["geometry"]): + x[:, vi] = TF.affine(x[:, vi], angle=angle, translate=[0, 0], scale=scale, shear=[0, 0], + interpolation=InterpolationMode.BILINEAR) + if p["light_shadow"] is not None: + yy, xx = torch.meshgrid(torch.linspace(0, 1, h), torch.linspace(0, 1, w), indexing="ij") + for vi, blobs in enumerate(p["light_shadow"]): + m = torch.ones(h, w) + for cx, cy, s, g in blobs: + m = m + (g - 1.0) * torch.exp(-((xx - cx) ** 2 + (yy - cy) ** 2) / (2 * s * s)) + x[:, vi] = (x[:, vi] * m.clamp(min=0).to(x)).clamp(0, 1) + if p["wb_channel"] is not None: + for vi, g in enumerate(p["wb_channel"]): + x[:, vi] = (x[:, vi] * torch.tensor(g, dtype=x.dtype).view(1, 3, 1, 1)).clamp(0, 1) + y = x.reshape(t * v, c, h, w) + y = (y * float(2.0 ** p["exposure"])).clamp(0, 1) + d = p["temperature"] + y = (y * y.new_tensor([1.0 + d, 1.0, 1.0 - d]).view(1, 3, 1, 1)).clamp(0, 1) + y = ((y - 0.5) * p["contrast"] + 0.5).clamp(0, 1) + y = TF.adjust_saturation(y, p["saturation"]) + y = TF.adjust_hue(y, p["hue"]) + y = y.clamp(0, 1).pow(p["gamma"]) + x = y.reshape(t, v, c, h, w) + mask = torch.ones(t, dtype=torch.bool) if obs_mask is None else torch.as_tensor(obs_mask, dtype=torch.bool) + if bool(mask.any()): + o = x[mask].reshape(-1, c, h, w) + if p["gaussian_blur"] > 0: + o = TF.gaussian_blur(o, [3, 3], [p["gaussian_blur"]] * 2) + if p["motion_blur"] is not None: + o = _conv(o, motion_kernel(*p["motion_blur"])) + if p["defocus"] > 0: + o = _conv(o, disk_kernel(p["defocus"])) + if p["sharpen"] > 0: + o = o + p["sharpen"] * (o - TF.gaussian_blur(o, [5, 5], [1.0, 1.0])) + o = o.clamp(0, 1) + if p["noise"] > 0: + gen = torch.Generator().manual_seed(int(noise_seed)) + o = (o + torch.randn(o.shape, generator=gen, dtype=o.dtype) * p["noise"]).clamp(0, 1) + if p["jpeg"] is not None: + for i in range(o.shape[0]): + buf = io.BytesIO() + TF.to_pil_image(o[i]).save(buf, format="JPEG", quality=int(p["jpeg"])) + buf.seek(0) + with Image.open(buf) as im: + o[i] = TF.pil_to_tensor(im.convert("RGB")).to(o.dtype).div(255.0) + x[mask] = o.reshape(-1, v, c, h, w) + return x.clamp(0, 1) + + +def augment(x, rng, obs_mask=None): + p = sample(rng, x.shape[1], x.shape[-2]) + return apply(x, p, obs_mask, noise_seed=int(rng.integers(0, 2 ** 62))), p + + +def perturb_joints(q, rng): + q = np.asarray(q, dtype=np.float64) + d = q.shape[-1] + bias = rng.uniform(-math.radians(2.0), math.radians(2.0), size=d) + sigma = math.radians(0.25) + noise = np.clip(rng.normal(0.0, sigma, size=q.shape), -3 * sigma, 3 * sigma) + return (q + bias + noise).astype(np.float32) diff --git a/code/fmdit_continue_20261117/augmentation/preview.py b/code/fmdit_continue_20261117/augmentation/preview.py new file mode 100644 index 0000000000000000000000000000000000000000..04e283526f682853496f9c185ce4d5bc6ed7333e --- /dev/null +++ b/code/fmdit_continue_20261117/augmentation/preview.py @@ -0,0 +1,162 @@ +"""Render a preview of the augmentation spec v2 as a model sees it: the cached training frames of a LeRobot dataset +(evaluation/baselines frame cache, e.g. 168x224) next to independent augmentations of the same frames, made with +`augmentation.augment_frames_uint8`, the call the baselines' training items use. + + python -I evaluation/augmentation/preview.py --dataset --contract g1_sonic_dfx6 \\ + --episodes 0:100 --samples 24 --out + +Writes /preview_desktop.mp4 (per sample: one row per camera; original, then --draws augmentations, each +labelled with the boosted effect and the effects that fired), /preview_phone.mp4 (the same samples stacked +vertically, cameras side by side), /contact_sheet.jpg (the first samples) and /preview.json (samples and +parameters). Frames are magnified --zoom times (nearest neighbour); each sample is held --seconds. +""" +from __future__ import annotations + +import argparse +import json +from pathlib import Path +import sys + +import numpy as np + +EVAL = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(EVAL)) +import augmentation # noqa: E402 +from baselines.data import CONTRACTS, ChunkDataset, LeRobotDataset, frame_cache, load_lowdim, parse_episodes # noqa: E402 + + +def fired(p) -> list: + """Short labels of the effects active in a parameter set beyond the always-on colour and geometry.""" + out = [] + if p['noise']: + out.append('noise') + if p['gaussian_blur']: + out.append(f"blur{p['gaussian_blur']:.2f}") + if p['jpeg'] is not None: + out.append(f"jpeg{p['jpeg']}") + if p['motion_blur'] is not None: + out.append(f"motion{p['motion_blur'][0]:.1f}px") + if p['sharpen']: + out.append('sharpen') + if p['defocus']: + out.append(f"defocus{p['defocus']:.1f}") + if p['light_shadow'] is not None: + out.append('light') + if p['wb_channel'] is not None: + out.append('wb') + return out + + +def font(size): + from PIL import ImageFont + for path in ('/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf', 'C:/Windows/Fonts/arial.ttf'): + try: + return ImageFont.truetype(path, size) + except OSError: + pass + return ImageFont.load_default(size=size) + + +def tile(img, zoom, text, fnt, bar): + from PIL import Image, ImageDraw + im = Image.fromarray(np.asarray(img)).resize((img.shape[1] * zoom, img.shape[0] * zoom), Image.NEAREST) + canvas = Image.new('RGB', (im.width, im.height + bar), (20, 20, 20)) + canvas.paste(im, (0, bar)) + ImageDraw.Draw(canvas).text((4, 2), text, fill=(240, 240, 240), font=fnt) + return np.asarray(canvas) + + +def grid(rows, gap=4): + """rows: list of lists of HxWx3 tiles (equal sizes) -> one image with `gap` px separators.""" + h, w = rows[0][0].shape[:2] + out = np.full((len(rows) * h + (len(rows) - 1) * gap, len(rows[0]) * w + (len(rows[0]) - 1) * gap, 3), 60, + np.uint8) + for r, row in enumerate(rows): + for c, t in enumerate(row): + out[r * (h + gap):r * (h + gap) + h, c * (w + gap):c * (w + gap) + w] = t + return out + + +def even(img): + h, w = img.shape[:2] + return np.ascontiguousarray(img[:h - h % 2, :w - w % 2]) + + +def write_video(path, frames, fps, hold): + import av + with av.open(str(path), 'w') as out: + stream = out.add_stream('libx264', rate=fps) + stream.height, stream.width = frames[0].shape[:2] + stream.pix_fmt = 'yuv420p' + stream.options = dict(crf='18', preset='medium') + for img in frames: + frame = av.VideoFrame.from_ndarray(img, format='rgb24') + for _ in range(hold): + for packet in stream.encode(frame): + out.mux(packet) + for packet in stream.encode(): + out.mux(packet) + + +def main(): + ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument('--dataset', type=Path, required=True) + ap.add_argument('--contract', choices=sorted(CONTRACTS), required=True) + ap.add_argument('--episodes') + ap.add_argument('--image-size', default='168x224') + ap.add_argument('--cache-dir', type=Path, default=Path('~/hb_models/baselines/cache')) + ap.add_argument('--samples', type=int, default=24) + ap.add_argument('--draws', type=int, default=3, help='augmentations shown per sample') + ap.add_argument('--zoom', type=int, default=2) + ap.add_argument('--seconds', type=float, default=2.5) + ap.add_argument('--seed', type=int, default=0) + ap.add_argument('--out', type=Path, required=True) + args = ap.parse_args() + + contract = CONTRACTS[args.contract] + hw = tuple(int(v) for v in args.image_size.lower().split('x')) + dataset = LeRobotDataset(args.dataset) + dataset.check_contract(contract, contract.cameras) + episodes = parse_episodes(args.episodes, dataset.lengths) + lowdim = load_lowdim(dataset, contract, episodes) + frames = frame_cache(dataset, lowdim, contract.cameras, hw, args.cache_dir) + plain = ChunkDataset(lowdim, frames, contract.cameras, hw, 16, list(range(len(episodes)))) + rng = np.random.default_rng(args.seed) + picks = np.sort(rng.choice(len(plain), size=min(args.samples, len(plain)), replace=False)) + args.out.mkdir(parents=True, exist_ok=True) + zoom, bar = args.zoom, 22 * args.zoom // 2 + 6 + fnt = font(11 * zoom // 2 + 3) + desktop, phone, records = [], [], [] + for n, k in enumerate(picks): + images, _, _ = plain[int(k)] + i = int(plain.index[k]) + pos = int(np.searchsorted(lowdim['starts'], i, side='right') - 1) + episode, frame = lowdim['episodes'][pos], i - int(lowdim['starts'][pos]) + draws = [augmentation.augment_frames_uint8(images, rng=rng) for _ in range(args.draws)] + cams = list(contract.cameras) + head = f'#{n + 1} ep{episode} f{frame}' + cols = [[tile(images[v].numpy(), zoom, f'{head} {cams[v]} original', fnt, bar) for v in range(len(cams))]] + for d, (img, p) in enumerate(draws): + label = f"aug{d + 1} 1.5x: {p['boost']} | on: " + (' '.join(fired(p)) or 'colour+geometry only') + cols.append([tile(img[v].numpy(), zoom, label if v == 0 else f'aug{d + 1} {cams[v]}', fnt, bar) + for v in range(len(cams))]) + desktop.append(even(grid([[col[v] for col in cols] for v in range(len(cams))]))) + phone.append(even(grid([list(col) for col in cols]))) + records.append(dict(sample=n + 1, episode=episode, frame=frame, draws=[ + dict(boost=p['boost'], fired=fired(p), **{key: p[key] for key in augmentation.hb_aug_v2.RANGE}) + for _, p in draws])) + fps = 10 + hold = max(1, int(round(args.seconds * fps))) + write_video(args.out / 'preview_desktop.mp4', desktop, fps, hold) + write_video(args.out / 'preview_phone.mp4', phone, fps, hold) + from PIL import Image + Image.fromarray(grid([[d] for d in desktop[:4]], gap=12)).save(args.out / 'contact_sheet.jpg', quality=90) + (args.out / 'preview.json').write_text(json.dumps(dict( + version=augmentation.VERSION, dataset=str(dataset.root), contract=contract.name, cameras=list(contract.cameras), + image_hw=list(hw), zoom=zoom, seconds_per_sample=args.seconds, samples=records), indent=1)) + print(json.dumps(dict(event='preview', out=str(args.out), samples=len(records), + desktop=list(desktop[0].shape), phone=list(phone[0].shape)))) + + +if __name__ == '__main__': + main() diff --git a/code/fmdit_continue_20261117/baselines/__init__.py b/code/fmdit_continue_20261117/baselines/__init__.py new file mode 100644 index 0000000000000000000000000000000000000000..a10810f9b2e3754911a648dcc14c5eafc0171b6c --- /dev/null +++ b/code/fmdit_continue_20261117/baselines/__init__.py @@ -0,0 +1,8 @@ +"""FM-DiT, DP (Diffusion Policy, DDPM-DiT) and ACT visuomotor baselines of the Humanoid Benchmark. + +Ported from DextraVerse's imitation-learning core (`dextraverse/algorithms/il/models/{fm_dit,ddpm_dit,act}.py` and +their networks), torch only. `train.py` trains one on a task's LeRobot v2.1 expert dataset through a robot contract +(`data.CONTRACTS`: G1 state44/action76 or H2 state78/action75); `runtime.BaselinePolicy` serves a checkpoint to the +G1 evaluation adapters (`evaluation/policy/{FMDiT,DP,ACT}/model.py`, via `adapters.py`) and to the H2 adapter +(`debug/newton/h2_policy_adapters/baseline.py`). Usage and the environment: evaluation/README.md#baselines. +""" diff --git a/code/fmdit_continue_20261117/baselines/act_nets.py b/code/fmdit_continue_20261117/baselines/act_nets.py new file mode 100644 index 0000000000000000000000000000000000000000..d36b8233cbb2c87b6a603e4100a3c88c8cd407a0 --- /dev/null +++ b/code/fmdit_continue_20261117/baselines/act_nets.py @@ -0,0 +1,615 @@ +"""ACT (Zhao et al. 2023, "Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware") network core: the +ResNet backbone with frozen BatchNorm and sine position embedding, the DETR encoder-decoder transformer and the CVAE +action-chunk decoder. + +Ported unchanged from DextraVerse (`dextraverse/algorithms/il/models/nets/act/`), which vendors the ACT repository's +`detr/models/{backbone,position_encoding,transformer,detr_vae}.py` (Apache-2.0, (c) Facebook, Inc. and its +affiliates; see THIRD_PARTY_NOTICES.md) with the NestedTensor/argparse plumbing removed, explicit `qpos_dim` / +`action_dim` instead of the ALOHA 14-DoF constants, and one backbone shared by every camera (upstream behaviour). +The policy uses `return_intermediate_dec=False`: `Transformer.forward(...)[0]` is the final decoder layer. +""" +from __future__ import annotations + +import copy +import math +from typing import Optional + +import numpy as np +import torch +import torch.nn.functional as F +import torchvision +from torch import Tensor, nn +from torchvision.models._utils import IntermediateLayerGetter + + +class PositionEmbeddingSine(nn.Module): + """Standard DETR sine embedding over a ``(B, C, H, W)`` feature map.""" + + def __init__( + self, num_pos_feats: int = 64, temperature: int = 10000, normalize: bool = False + ) -> None: + super().__init__() + self.num_pos_feats = num_pos_feats + self.temperature = temperature + self.normalize = normalize + self.scale = 2 * math.pi + + def forward(self, x: torch.Tensor) -> torch.Tensor: + not_mask = torch.ones((1, x.shape[-2], x.shape[-1]), device=x.device) + y_embed = not_mask.cumsum(1, dtype=torch.float32) + x_embed = not_mask.cumsum(2, dtype=torch.float32) + if self.normalize: + eps = 1e-6 + y_embed = y_embed / (y_embed[:, -1:, :] + eps) * self.scale + x_embed = x_embed / (x_embed[:, :, -1:] + eps) * self.scale + + dim_t = torch.arange(self.num_pos_feats, dtype=torch.float32, device=x.device) + dim_t = self.temperature ** (2 * (dim_t // 2) / self.num_pos_feats) + + pos_x = x_embed[:, :, :, None] / dim_t + pos_y = y_embed[:, :, :, None] / dim_t + pos_x = torch.stack( + (pos_x[:, :, :, 0::2].sin(), pos_x[:, :, :, 1::2].cos()), dim=4 + ).flatten(3) + pos_y = torch.stack( + (pos_y[:, :, :, 0::2].sin(), pos_y[:, :, :, 1::2].cos()), dim=4 + ).flatten(3) + return torch.cat((pos_y, pos_x), dim=3).permute(0, 3, 1, 2) + + +class FrozenBatchNorm2d(nn.Module): + """BatchNorm2d with fixed statistics and affine parameters (the DETR/ACT backbone norm).""" + + def __init__(self, n: int) -> None: + super().__init__() + self.register_buffer("weight", torch.ones(n)) + self.register_buffer("bias", torch.zeros(n)) + self.register_buffer("running_mean", torch.zeros(n)) + self.register_buffer("running_var", torch.ones(n)) + + def _load_from_state_dict(self, state_dict, prefix, local_metadata, strict, *args): + state_dict.pop(prefix + "num_batches_tracked", None) + super()._load_from_state_dict(state_dict, prefix, local_metadata, strict, *args) + + def forward(self, x: torch.Tensor) -> torch.Tensor: + w = self.weight.reshape(1, -1, 1, 1) + b = self.bias.reshape(1, -1, 1, 1) + rv = self.running_var.reshape(1, -1, 1, 1) + rm = self.running_mean.reshape(1, -1, 1, 1) + scale = w * (rv + 1e-5).rsqrt() + return x * scale + (b - rm * scale) + + +class Backbone(nn.Module): + """ResNet trunk returning the layer4 feature map.""" + + def __init__(self, name: str = "resnet18", weights: str | None = None) -> None: + super().__init__() + trunk = getattr(torchvision.models, name)( + replace_stride_with_dilation=[False, False, False], + weights=weights, + norm_layer=FrozenBatchNorm2d, + ) + self.body = IntermediateLayerGetter(trunk, return_layers={"layer4": "0"}) + self.num_channels = 512 if name in ("resnet18", "resnet34") else 2048 + + def forward(self, x: torch.Tensor) -> dict[str, torch.Tensor]: + return self.body(x) + + +class Joiner(nn.Sequential): + """Backbone + position embedding, returning matched feature/position lists.""" + + def __init__(self, backbone: Backbone, position_embedding: PositionEmbeddingSine) -> None: + super().__init__(backbone, position_embedding) + self.num_channels = backbone.num_channels + + def forward(self, x: torch.Tensor) -> tuple[list[torch.Tensor], list[torch.Tensor]]: + features = list(self[0](x).values()) + pos = [self[1](f).to(f.dtype) for f in features] + return features, pos + + +def build_backbone( + name: str = "resnet18", hidden_dim: int = 512, weights: str | None = None +) -> Joiner: + """The ACT camera backbone: ResNet + sine embedding sized to the transformer width.""" + return Joiner(Backbone(name, weights=weights), PositionEmbeddingSine(hidden_dim // 2)) + + +class Transformer(nn.Module): + def __init__( + self, + d_model=512, + nhead=8, + num_encoder_layers=6, + num_decoder_layers=6, + dim_feedforward=2048, + dropout=0.1, + activation="relu", + normalize_before=False, + return_intermediate_dec=False, + ): + super().__init__() + + encoder_layer = TransformerEncoderLayer( + d_model, nhead, dim_feedforward, dropout, activation, normalize_before + ) + encoder_norm = nn.LayerNorm(d_model) if normalize_before else None + self.encoder = TransformerEncoder(encoder_layer, num_encoder_layers, encoder_norm) + + decoder_layer = TransformerDecoderLayer( + d_model, nhead, dim_feedforward, dropout, activation, normalize_before + ) + decoder_norm = nn.LayerNorm(d_model) + self.decoder = TransformerDecoder( + decoder_layer, + num_decoder_layers, + decoder_norm, + return_intermediate=return_intermediate_dec, + ) + + self._reset_parameters() + + self.d_model = d_model + self.nhead = nhead + + def _reset_parameters(self): + for p in self.parameters(): + if p.dim() > 1: + nn.init.xavier_uniform_(p) + + def forward( + self, + src, + mask, + query_embed, + pos_embed, + latent_input=None, + proprio_input=None, + additional_pos_embed=None, + ): + # TODO flatten only when input has H and W + if len(src.shape) == 4: # has H and W + # flatten NxCxHxW to HWxNxC + bs, c, h, w = src.shape + src = src.flatten(2).permute(2, 0, 1) + pos_embed = pos_embed.flatten(2).permute(2, 0, 1).repeat(1, bs, 1) + query_embed = query_embed.unsqueeze(1).repeat(1, bs, 1) + # mask = mask.flatten(1) + + additional_pos_embed = additional_pos_embed.unsqueeze(1).repeat( + 1, bs, 1 + ) # seq, bs, dim + pos_embed = torch.cat([additional_pos_embed, pos_embed], axis=0) + + addition_input = torch.stack([latent_input, proprio_input], axis=0) + src = torch.cat([addition_input, src], axis=0) + else: + assert len(src.shape) == 3 + # flatten NxHWxC to HWxNxC + bs, hw, c = src.shape + src = src.permute(1, 0, 2) + pos_embed = pos_embed.unsqueeze(1).repeat(1, bs, 1) + query_embed = query_embed.unsqueeze(1).repeat(1, bs, 1) + + tgt = torch.zeros_like(query_embed) + memory = self.encoder(src, src_key_padding_mask=mask, pos=pos_embed) + hs = self.decoder( + tgt, memory, memory_key_padding_mask=mask, pos=pos_embed, query_pos=query_embed + ) + hs = hs.transpose(1, 2) + return hs + + +class TransformerEncoder(nn.Module): + def __init__(self, encoder_layer, num_layers, norm=None): + super().__init__() + self.layers = _get_clones(encoder_layer, num_layers) + self.num_layers = num_layers + self.norm = norm + + def forward( + self, + src, + mask: Optional[Tensor] = None, + src_key_padding_mask: Optional[Tensor] = None, + pos: Optional[Tensor] = None, + ): + output = src + + for layer in self.layers: + output = layer( + output, src_mask=mask, src_key_padding_mask=src_key_padding_mask, pos=pos + ) + + if self.norm is not None: + output = self.norm(output) + + return output + + +class TransformerDecoder(nn.Module): + def __init__(self, decoder_layer, num_layers, norm=None, return_intermediate=False): + super().__init__() + self.layers = _get_clones(decoder_layer, num_layers) + self.num_layers = num_layers + self.norm = norm + self.return_intermediate = return_intermediate + + def forward( + self, + tgt, + memory, + tgt_mask: Optional[Tensor] = None, + memory_mask: Optional[Tensor] = None, + tgt_key_padding_mask: Optional[Tensor] = None, + memory_key_padding_mask: Optional[Tensor] = None, + pos: Optional[Tensor] = None, + query_pos: Optional[Tensor] = None, + ): + output = tgt + + intermediate = [] + + for layer in self.layers: + output = layer( + output, + memory, + tgt_mask=tgt_mask, + memory_mask=memory_mask, + tgt_key_padding_mask=tgt_key_padding_mask, + memory_key_padding_mask=memory_key_padding_mask, + pos=pos, + query_pos=query_pos, + ) + if self.return_intermediate: + intermediate.append(self.norm(output)) + + if self.norm is not None: + output = self.norm(output) + if self.return_intermediate: + intermediate.pop() + intermediate.append(output) + + if self.return_intermediate: + return torch.stack(intermediate) + + return output.unsqueeze(0) + + +class TransformerEncoderLayer(nn.Module): + def __init__( + self, + d_model, + nhead, + dim_feedforward=2048, + dropout=0.1, + activation="relu", + normalize_before=False, + ): + super().__init__() + self.self_attn = nn.MultiheadAttention(d_model, nhead, dropout=dropout) + # Implementation of Feedforward model + self.linear1 = nn.Linear(d_model, dim_feedforward) + self.dropout = nn.Dropout(dropout) + self.linear2 = nn.Linear(dim_feedforward, d_model) + + self.norm1 = nn.LayerNorm(d_model) + self.norm2 = nn.LayerNorm(d_model) + self.dropout1 = nn.Dropout(dropout) + self.dropout2 = nn.Dropout(dropout) + + self.activation = _get_activation_fn(activation) + self.normalize_before = normalize_before + + def with_pos_embed(self, tensor, pos: Optional[Tensor]): + return tensor if pos is None else tensor + pos + + def forward_post( + self, + src, + src_mask: Optional[Tensor] = None, + src_key_padding_mask: Optional[Tensor] = None, + pos: Optional[Tensor] = None, + ): + q = k = self.with_pos_embed(src, pos) + src2 = self.self_attn( + q, k, value=src, attn_mask=src_mask, key_padding_mask=src_key_padding_mask + )[0] + src = src + self.dropout1(src2) + src = self.norm1(src) + src2 = self.linear2(self.dropout(self.activation(self.linear1(src)))) + src = src + self.dropout2(src2) + src = self.norm2(src) + return src + + def forward_pre( + self, + src, + src_mask: Optional[Tensor] = None, + src_key_padding_mask: Optional[Tensor] = None, + pos: Optional[Tensor] = None, + ): + src2 = self.norm1(src) + q = k = self.with_pos_embed(src2, pos) + src2 = self.self_attn( + q, k, value=src2, attn_mask=src_mask, key_padding_mask=src_key_padding_mask + )[0] + src = src + self.dropout1(src2) + src2 = self.norm2(src) + src2 = self.linear2(self.dropout(self.activation(self.linear1(src2)))) + src = src + self.dropout2(src2) + return src + + def forward( + self, + src, + src_mask: Optional[Tensor] = None, + src_key_padding_mask: Optional[Tensor] = None, + pos: Optional[Tensor] = None, + ): + if self.normalize_before: + return self.forward_pre(src, src_mask, src_key_padding_mask, pos) + return self.forward_post(src, src_mask, src_key_padding_mask, pos) + + +class TransformerDecoderLayer(nn.Module): + def __init__( + self, + d_model, + nhead, + dim_feedforward=2048, + dropout=0.1, + activation="relu", + normalize_before=False, + ): + super().__init__() + self.self_attn = nn.MultiheadAttention(d_model, nhead, dropout=dropout) + self.multihead_attn = nn.MultiheadAttention(d_model, nhead, dropout=dropout) + # Implementation of Feedforward model + self.linear1 = nn.Linear(d_model, dim_feedforward) + self.dropout = nn.Dropout(dropout) + self.linear2 = nn.Linear(dim_feedforward, d_model) + + self.norm1 = nn.LayerNorm(d_model) + self.norm2 = nn.LayerNorm(d_model) + self.norm3 = nn.LayerNorm(d_model) + self.dropout1 = nn.Dropout(dropout) + self.dropout2 = nn.Dropout(dropout) + self.dropout3 = nn.Dropout(dropout) + + self.activation = _get_activation_fn(activation) + self.normalize_before = normalize_before + + def with_pos_embed(self, tensor, pos: Optional[Tensor]): + return tensor if pos is None else tensor + pos + + def forward_post( + self, + tgt, + memory, + tgt_mask: Optional[Tensor] = None, + memory_mask: Optional[Tensor] = None, + tgt_key_padding_mask: Optional[Tensor] = None, + memory_key_padding_mask: Optional[Tensor] = None, + pos: Optional[Tensor] = None, + query_pos: Optional[Tensor] = None, + ): + q = k = self.with_pos_embed(tgt, query_pos) + tgt2 = self.self_attn( + q, k, value=tgt, attn_mask=tgt_mask, key_padding_mask=tgt_key_padding_mask + )[0] + tgt = tgt + self.dropout1(tgt2) + tgt = self.norm1(tgt) + tgt2 = self.multihead_attn( + query=self.with_pos_embed(tgt, query_pos), + key=self.with_pos_embed(memory, pos), + value=memory, + attn_mask=memory_mask, + key_padding_mask=memory_key_padding_mask, + )[0] + tgt = tgt + self.dropout2(tgt2) + tgt = self.norm2(tgt) + tgt2 = self.linear2(self.dropout(self.activation(self.linear1(tgt)))) + tgt = tgt + self.dropout3(tgt2) + tgt = self.norm3(tgt) + return tgt + + def forward_pre( + self, + tgt, + memory, + tgt_mask: Optional[Tensor] = None, + memory_mask: Optional[Tensor] = None, + tgt_key_padding_mask: Optional[Tensor] = None, + memory_key_padding_mask: Optional[Tensor] = None, + pos: Optional[Tensor] = None, + query_pos: Optional[Tensor] = None, + ): + tgt2 = self.norm1(tgt) + q = k = self.with_pos_embed(tgt2, query_pos) + tgt2 = self.self_attn( + q, k, value=tgt2, attn_mask=tgt_mask, key_padding_mask=tgt_key_padding_mask + )[0] + tgt = tgt + self.dropout1(tgt2) + tgt2 = self.norm2(tgt) + tgt2 = self.multihead_attn( + query=self.with_pos_embed(tgt2, query_pos), + key=self.with_pos_embed(memory, pos), + value=memory, + attn_mask=memory_mask, + key_padding_mask=memory_key_padding_mask, + )[0] + tgt = tgt + self.dropout2(tgt2) + tgt2 = self.norm3(tgt) + tgt2 = self.linear2(self.dropout(self.activation(self.linear1(tgt2)))) + tgt = tgt + self.dropout3(tgt2) + return tgt + + def forward( + self, + tgt, + memory, + tgt_mask: Optional[Tensor] = None, + memory_mask: Optional[Tensor] = None, + tgt_key_padding_mask: Optional[Tensor] = None, + memory_key_padding_mask: Optional[Tensor] = None, + pos: Optional[Tensor] = None, + query_pos: Optional[Tensor] = None, + ): + if self.normalize_before: + return self.forward_pre( + tgt, + memory, + tgt_mask, + memory_mask, + tgt_key_padding_mask, + memory_key_padding_mask, + pos, + query_pos, + ) + return self.forward_post( + tgt, + memory, + tgt_mask, + memory_mask, + tgt_key_padding_mask, + memory_key_padding_mask, + pos, + query_pos, + ) + + +def _get_clones(module, N): + return nn.ModuleList([copy.deepcopy(module) for i in range(N)]) + + +def _get_activation_fn(activation): + """Return an activation function given a string.""" + if activation == "relu": + return F.relu + if activation == "gelu": + return F.gelu + raise RuntimeError(f"activation should be relu/gelu, not {activation}.") + + + +def reparametrize(mu: torch.Tensor, logvar: torch.Tensor) -> torch.Tensor: + std = logvar.div(2).exp() + return mu + std * torch.randn_like(std) + + +def get_sinusoid_encoding_table(n_position: int, d_hid: int) -> torch.Tensor: + def angle_vec(position: int) -> list[float]: + return [position / np.power(10000, 2 * (j // 2) / d_hid) for j in range(d_hid)] + + table = np.array([angle_vec(i) for i in range(n_position)]) + table[:, 0::2] = np.sin(table[:, 0::2]) + table[:, 1::2] = np.cos(table[:, 1::2]) + return torch.FloatTensor(table).unsqueeze(0) + + +def build_style_encoder( + hidden_dim: int, nheads: int, dim_feedforward: int, enc_layers: int, dropout: float +) -> TransformerEncoder: + """The CVAE style encoder (a plain transformer encoder over [CLS, qpos, action seq]).""" + layer = TransformerEncoderLayer(hidden_dim, nheads, dim_feedforward, dropout, "relu", False) + return TransformerEncoder(layer, enc_layers, norm=None) + + +class DETRVAE(nn.Module): + """CVAE decoder: (qpos, images[, action seq]) → ``num_queries`` future actions.""" + + def __init__( + self, + backbone: Joiner, + transformer: Transformer, + encoder: TransformerEncoder, + *, + qpos_dim: int, + action_dim: int, + num_queries: int, + latent_dim: int = 32, + ) -> None: + super().__init__() + self.num_queries = num_queries + self.transformer = transformer + self.encoder = encoder + hidden_dim = transformer.d_model + self.action_head = nn.Linear(hidden_dim, action_dim) + self.query_embed = nn.Embedding(num_queries, hidden_dim) + self.backbone = backbone # ONE backbone shared by every camera (upstream behavior) + self.input_proj = nn.Conv2d(backbone.num_channels, hidden_dim, kernel_size=1) + self.input_proj_robot_state = nn.Linear(qpos_dim, hidden_dim) + + # CVAE style encoder inputs + self.latent_dim = latent_dim + self.cls_embed = nn.Embedding(1, hidden_dim) + self.encoder_action_proj = nn.Linear(action_dim, hidden_dim) + self.encoder_joint_proj = nn.Linear(qpos_dim, hidden_dim) + self.latent_proj = nn.Linear(hidden_dim, latent_dim * 2) + self.register_buffer( + "pos_table", get_sinusoid_encoding_table(1 + 1 + num_queries, hidden_dim) + ) + # decoder side + self.latent_out_proj = nn.Linear(latent_dim, hidden_dim) + self.additional_pos_embed = nn.Embedding(2, hidden_dim) # latent + proprio slots + + def forward( + self, + qpos: torch.Tensor, + images: torch.Tensor, + actions: torch.Tensor | None = None, + is_pad: torch.Tensor | None = None, + ) -> tuple[torch.Tensor, tuple[torch.Tensor | None, torch.Tensor | None]]: + """``qpos (B, Dq)``, ``images (B, ncam, C, H, W)``; train passes ``actions (B, T, Da)``.""" + is_training = actions is not None + bs = qpos.shape[0] + if is_training: + action_embed = self.encoder_action_proj(actions) + qpos_embed = self.encoder_joint_proj(qpos).unsqueeze(1) + cls_embed = self.cls_embed.weight.unsqueeze(0).repeat(bs, 1, 1) + encoder_input = torch.cat([cls_embed, qpos_embed, action_embed], dim=1) + encoder_input = encoder_input.permute(1, 0, 2) + if is_pad is None: + is_pad = torch.zeros(bs, actions.shape[1], dtype=torch.bool, device=qpos.device) + cls_joint_is_pad = torch.full((bs, 2), False, device=qpos.device) + key_pad = torch.cat([cls_joint_is_pad, is_pad], dim=1) + pos_embed = self.pos_table.clone().detach().permute(1, 0, 2) + encoder_output = self.encoder( + encoder_input, pos=pos_embed, src_key_padding_mask=key_pad + )[0] + latent_info = self.latent_proj(encoder_output) + mu = latent_info[:, : self.latent_dim] + logvar = latent_info[:, self.latent_dim :] + latent_input = self.latent_out_proj(reparametrize(mu, logvar)) + else: + mu = logvar = None + latent_sample = torch.zeros(bs, self.latent_dim, device=qpos.device) + latent_input = self.latent_out_proj(latent_sample) + + all_cam_features, all_cam_pos = [], [] + for cam_id in range(images.shape[1]): + features, pos = self.backbone(images[:, cam_id]) + all_cam_features.append(self.input_proj(features[0])) + all_cam_pos.append(pos[0]) + proprio_input = self.input_proj_robot_state(qpos) + src = torch.cat(all_cam_features, dim=3) # cameras fold into the width axis + pos = torch.cat(all_cam_pos, dim=3) + hs = self.transformer( + src, + None, + self.query_embed.weight, + pos, + latent_input, + proprio_input, + self.additional_pos_embed.weight, + )[0] + return self.action_head(hs), (mu, logvar) + + +def kl_divergence(mu: torch.Tensor, logvar: torch.Tensor) -> torch.Tensor: + """Total KL(q(z|x) || N(0, I)), summed over latent dims, meaned over the batch.""" + klds = -0.5 * (1 + logvar - mu.pow(2) - logvar.exp()) + return klds.sum(1).mean() diff --git a/code/fmdit_continue_20261117/baselines/adapters.py b/code/fmdit_continue_20261117/baselines/adapters.py new file mode 100644 index 0000000000000000000000000000000000000000..6da8ae02058a8559126021d160d9602e43f8297f --- /dev/null +++ b/code/fmdit_continue_20261117/baselines/adapters.py @@ -0,0 +1,48 @@ +"""The G1 evaluation adapters of the baselines (evaluation/README.md#the-model-interface): `make_model('fm_dit')` +returns the `Model` class that `evaluation/policy/FMDiT/model.py` exports (likewise DP, ACT). + +Observation: the current ego_view and chest frames (480x640 RGB, resized as the training cache) and state44; no past +actions or earlier frames (the baselines read one frame). Output: the checkpoint's whole chunk (16 rows of action76, +row 0 executed at the request's frame), `execution_steps` rows of it executed before the next query (deploy.yml +`config: execution_steps`, default the whole chunk), the request's seed for the sampling noise (FM-DiT, DP). +""" +from __future__ import annotations + +from .runtime import BaselinePolicy + +WIDTH = {'sonic_dfx6': 76} + + +def require(condition, message): + if not condition: + raise RuntimeError(message) + + +def make_model(policy_name: str): + class Model: + history_length = 1 # the current head and chest frames only + video_frames = 0 + + def __init__(self, cfg): + self.policy = BaselinePolicy(cfg['checkpoint_path'], device=cfg.get('device', 'cuda:0'), + policy=policy_name, robot='g1') + contract = self.policy.contract + require(cfg['action_space'] in WIDTH and contract['action_space'] == cfg['action_space'] + and self.policy.action_dim == WIDTH[cfg['action_space']] and self.policy.state_dim == 44 + and list(self.policy.cameras) == ['ego_view', 'chest'], + f"checkpoint contract {contract['name']} does not serve {cfg['action_space']}") + extra = cfg.get('config') or {} + if extra.get('check_task', True): + require(cfg['task'].strip() == (self.policy.task_text or '').strip(), + f'task text {cfg["task"]!r} is not the training task {self.policy.task_text!r}') + self.execution_steps = int(extra.get('execution_steps') or self.policy.horizon) + require(1 <= self.execution_steps <= self.policy.horizon, f'execution_steps {self.execution_steps}') + self.info = dict(self.policy.info(), execution_steps=self.execution_steps) + + def get_action(self, obs): + action, _ = self.policy.predict(obs['images'], obs['state'], seed=obs['seed']) + return dict(action=action, execution_steps=self.execution_steps, + sampling_steps=self.policy.sampling_steps) + + Model.__name__ = Model.__qualname__ = f'Model_{policy_name}' + return Model diff --git a/code/fmdit_continue_20261117/baselines/data.py b/code/fmdit_continue_20261117/baselines/data.py new file mode 100644 index 0000000000000000000000000000000000000000..d0fd56bcc55af00a09be9a5ed86137b91273d487 --- /dev/null +++ b/code/fmdit_continue_20261117/baselines/data.py @@ -0,0 +1,288 @@ +"""Training data of the baselines: a task's LeRobot v2.1 expert dataset read through a robot contract. + +A contract names the cameras and which dataset columns form the policy state and the action rows, the same layouts +the registered GR00T / Psi0 checkpoints were trained on: + +- `g1_sonic_dfx6` (G1 + INSPIRE, the evaluation/README.md interface; ego_view, chest): state44 = + `observation.state[0:29]` body, measured `observation.inspire_{left,right}_hand_state`, + `observation.projected_gravity`; action76 (`sonic_dfx6`) = `action.motion_token` 64 and the measured DFX6 of the + same row (row 0 = the current opening); +- `h2_sharpa` (H2 + Sharpa, debug/newton/h2_sharpa_policy_eval.py; head, chest): state78 = `observation.state` 75, + `observation.projected_gravity` 3; action75 = `action.body_target` 31, `action.hands` 44 (absolute targets); +- `h2_sharpa_planner` (H2 + Sharpa a14 v2, h2_policy_planner_eval.py; head, chest): state78; action82 = action75 + + `action.planner` 7. + +Frames: every camera's video is decoded once (PyAV), each frame resized with `resize_frame` (OpenCV INTER_AREA, the +same function the policy runtime applies to live frames) and stored as a uint8 memory map in a cache directory keyed by +the dataset, episodes, camera and size. A training sample is the frame t of an episode: the cameras and the state at t, +and the action rows t..t+horizon-1 (rows past the episode's end repeat its last row, as DextraVerse's windows). +""" +from __future__ import annotations + +from dataclasses import dataclass +import hashlib +import json +import os +from pathlib import Path +import sys + +import numpy as np + +CACHE_VERSION = 1 + + +@dataclass(frozen=True) +class Contract: + name: str + robot: str + cameras: tuple + state: tuple # ((column, start, end), ...) concatenated in order + action: tuple + action_space: str | None = None # the G1 evaluation interface's action space, if any + body_dims: int = 0 # leading raw-radian body joint angles of the state (augmentation spec v2 perturbs these) + + @property + def state_dim(self) -> int: + return sum(end - start for _, start, end in self.state) + + @property + def action_dim(self) -> int: + return sum(end - start for _, start, end in self.action) + + def to_dict(self, cameras=None) -> dict: + return dict(name=self.name, robot=self.robot, cameras=list(cameras or self.cameras), + state=[list(s) for s in self.state], action=[list(a) for a in self.action], + state_dim=self.state_dim, action_dim=self.action_dim, action_space=self.action_space, + body_dims=self.body_dims) + + +_G1_HANDS = (('observation.inspire_left_hand_state', 0, 6), ('observation.inspire_right_hand_state', 0, 6)) +_H2_STATE = (('observation.state', 0, 75), ('observation.projected_gravity', 0, 3)) +_H2_ACTION = (('action.body_target', 0, 31), ('action.hands', 0, 44)) +# body_dims: G1 observation.state[0:29] are the 29 body joints; H2 observation.state[0:31] are legs 6+6, waist 3, +# head 2, arms 7+7 (docs/unitree_h2_sharpa.md), then the Sharpa hands. +CONTRACTS = {c.name: c for c in ( + Contract('g1_sonic_dfx6', 'g1', ('ego_view', 'chest'), + (('observation.state', 0, 29), *_G1_HANDS, ('observation.projected_gravity', 0, 3)), + (('action.motion_token', 0, 64), *_G1_HANDS), action_space='sonic_dfx6', body_dims=29), + Contract('h2_sharpa', 'h2_sharpa', ('head', 'chest'), _H2_STATE, _H2_ACTION, body_dims=31), + Contract('h2_sharpa_planner', 'h2_sharpa', ('head', 'chest'), _H2_STATE, + (*_H2_ACTION, ('action.planner', 0, 7)), body_dims=31), +)} + + +def augmentation_module(): + """evaluation/augmentation: the augmentation spec v2 shared by every model's training (imported on demand).""" + evaluation = str(Path(__file__).resolve().parents[1]) + if evaluation not in sys.path: + sys.path.insert(0, evaluation) + import augmentation + return augmentation + + +def resize_frame(frame: np.ndarray, hw) -> np.ndarray: + """(H, W, 3) uint8 RGB -> (h, w, 3) uint8, OpenCV INTER_AREA (training cache and live frames alike).""" + import cv2 + frame = np.ascontiguousarray(frame, dtype=np.uint8) + if frame.ndim != 3 or frame.shape[2] != 3: + raise ValueError(f'expected an (H, W, 3) RGB frame, got {frame.shape}') + h, w = int(hw[0]), int(hw[1]) + if frame.shape[:2] == (h, w): + return frame + return cv2.resize(frame, (w, h), interpolation=cv2.INTER_AREA) + + +class LeRobotDataset: + """A LeRobot v2.1 dataset directory (meta/info.json, meta/episodes.jsonl, data/, videos/).""" + + def __init__(self, root): + self.root = Path(root).expanduser().resolve() + self.info = json.loads((self.root / 'meta/info.json').read_text()) + if self.info.get('codebase_version') != 'v2.1': + raise ValueError(f"{self.root}: LeRobot {self.info.get('codebase_version')}, expected v2.1") + rows = [json.loads(line) for line in (self.root / 'meta/episodes.jsonl').read_text().splitlines() if line] + self.lengths = {int(r['episode_index']): int(r['length']) for r in rows} + tasks = [json.loads(line) for line in (self.root / 'meta/tasks.jsonl').read_text().splitlines() if line] + self.tasks = {int(t['task_index']): t['task'] for t in tasks} + self.fps = float(self.info['fps']) + + @property + def name(self) -> str: + """The dataset's folder name (its parent's for an export folder named `lerobot`).""" + return self.root.parent.name if self.root.name == 'lerobot' else self.root.name + + def chunk(self, episode: int) -> int: + return episode // int(self.info['chunks_size']) + + def parquet(self, episode: int) -> Path: + return self.root / self.info['data_path'].format(episode_chunk=self.chunk(episode), episode_index=episode) + + def video(self, episode: int, camera: str) -> Path: + return self.root / self.info['video_path'].format(episode_chunk=self.chunk(episode), episode_index=episode, + video_key=f'observation.images.{camera}') + + def check_contract(self, contract: Contract, cameras): + features = self.info['features'] + for column, start, end in (*contract.state, *contract.action): + if column not in features: + raise KeyError(f'{self.root.name}: no column {column!r} (contract {contract.name})') + width = int(np.prod(features[column]['shape'])) + if end > width: + raise ValueError(f'{column} has {width} values; the contract reads {start}:{end}') + for camera in cameras: + key = f'observation.images.{camera}' + if features.get(key, {}).get('dtype') != 'video': + raise KeyError(f'{self.root.name}: no video {key!r}') + + +def parse_episodes(spec: str | None, available) -> list: + """'0:100', '0,3,7' or None (all), restricted to the dataset's episodes, sorted.""" + available = sorted(available) + if not spec: + return available + chosen = set() + for part in spec.split(','): + if ':' in part: + lo, hi = part.split(':') + chosen.update(range(int(lo or 0), int(hi) if hi else available[-1] + 1)) + else: + chosen.add(int(part)) + missing = sorted(chosen - set(available)) + if missing: + raise ValueError(f'episodes not in the dataset: {missing[:10]}') + return sorted(chosen) + + +def load_lowdim(dataset: LeRobotDataset, contract: Contract, episodes) -> dict: + """State (N, state_dim) and action (N, action_dim) float32 of the episodes in order, plus each episode's start + row, length and task text.""" + import pyarrow.parquet as pq + columns = sorted({c for c, _, _ in (*contract.state, *contract.action)} | {'frame_index', 'task_index'}) + states, actions, starts, lengths, tasks, offset = [], [], [], [], set(), 0 + for episode in episodes: + table = pq.read_table(dataset.parquet(episode), columns=columns) + data = {c: np.asarray(table.column(c).to_pylist(), dtype=np.float64) for c in columns} + n = len(data['frame_index']) + if n != dataset.lengths[episode] or not np.array_equal(data['frame_index'], np.arange(n)): + raise ValueError(f'episode {episode}: {n} rows / frame_index differ from episodes.jsonl') + + def pick(spec): + return np.concatenate([data[c].reshape(n, -1)[:, s:e] for c, s, e in spec], axis=1).astype(np.float32) + + states.append(pick(contract.state)) + actions.append(pick(contract.action)) + tasks.update(dataset.tasks[int(i)] for i in np.unique(data['task_index'])) + starts.append(offset) + lengths.append(n) + offset += n + state, action = np.concatenate(states), np.concatenate(actions) + if not (np.isfinite(state).all() and np.isfinite(action).all()): + raise ValueError('non-finite state or action values') + return dict(state=state, action=action, starts=np.asarray(starts), lengths=np.asarray(lengths), + episodes=list(episodes), tasks=sorted(tasks)) + + +def statistics(lowdim: dict, rows: np.ndarray) -> dict: + out = {} + for key in ('state', 'action'): + x = lowdim[key][rows].astype(np.float64) + out[key] = dict(min=x.min(0).tolist(), max=x.max(0).tolist(), mean=x.mean(0).tolist(), std=x.std(0).tolist()) + return out + + +# -- frame cache -------------------------------------------------------------------------------------------------- +def _decode_into(job): + path, out_path, total, start, length, hw = job + import av + frames = np.memmap(out_path, dtype=np.uint8, mode='r+', shape=(total, hw[0], hw[1], 3)) + count = 0 + with av.open(str(path)) as container: + stream = container.streams.video[0] + stream.thread_type = 'AUTO' + for frame in container.decode(stream): + if count < length: + frames[start + count] = resize_frame(frame.to_ndarray(format='rgb24'), hw) + count += 1 + frames.flush() + del frames + if count != length: + raise ValueError(f'{path}: {count} video frames, {length} data rows') + return length + + +def frame_cache(dataset: LeRobotDataset, lowdim: dict, cameras, hw, cache_root, workers=8) -> dict: + """{camera: path of the (N, h, w, 3) uint8 memory map} for the episodes of `lowdim`, decoded once and reused.""" + from multiprocessing import get_context + total = int(lowdim['lengths'].sum()) + paths = {} + for camera in cameras: + key = json.dumps(dict(version=CACHE_VERSION, root=str(dataset.root), episodes=lowdim['episodes'], + lengths=lowdim['lengths'].tolist(), camera=camera, hw=list(hw)), sort_keys=True) + digest = hashlib.sha256(key.encode()).hexdigest()[:16] + folder = Path(cache_root).expanduser() / f'{dataset.name}_{camera}_{hw[0]}x{hw[1]}_{digest}' + folder.mkdir(parents=True, exist_ok=True) + out, meta = folder / 'frames.u8', folder / 'meta.json' + if meta.exists() and json.loads(meta.read_text()).get('complete'): + paths[camera] = out + continue + np.memmap(out, dtype=np.uint8, mode='w+', shape=(total, hw[0], hw[1], 3)).flush() + jobs = [(dataset.video(e, camera), out, total, int(s), int(n), tuple(hw)) + for e, s, n in zip(lowdim['episodes'], lowdim['starts'], lowdim['lengths'])] + with get_context('spawn').Pool(max(1, int(workers))) as pool: # the caller may hold torch threads + for done, _ in enumerate(pool.imap_unordered(_decode_into, jobs), 1): + if done % 20 == 0 or done == len(jobs): + print(json.dumps(dict(event='frame_cache', camera=camera, episodes=done, of=len(jobs))), + flush=True) + meta.write_text(json.dumps(dict(json.loads(key), frames=total, complete=True), indent=1)) + paths[camera] = out + return paths + + +class ChunkDataset: + """torch-style dataset of (images (n_cameras, h, w, 3) uint8, state (state_dim,), action (horizon, action_dim)) + over the frames of the selected episodes (positions into lowdim['episodes']). Memory maps open lazily per + process (DataLoader workers). + + `body_dims` > 0 turns on the training augmentation (docs/augmentation_v2.md, evaluation/augmentation): per item, + the cached frames of all cameras get one hb_aug_v2.augment call (one time step, no future targets) and the first + `body_dims` state values the per-sample joint bias + noise; the action rows are never touched. 0 (evaluation, + offline checks) returns the cached frames and the recorded state unchanged, deterministically.""" + + def __init__(self, lowdim: dict, frames: dict, cameras, hw, horizon: int, positions, body_dims: int = 0): + self.lowdim, self.paths, self.cameras, self.hw, self.horizon = lowdim, frames, list(cameras), tuple(hw), horizon + self.positions, self.body_dims = list(positions), int(body_dims) + if self.body_dims: + augmentation_module() # asserts the spec version before any worker starts + self.total = int(lowdim['lengths'].sum()) + index, end = [], np.zeros(self.total, np.int64) + for p in positions: + s, n = int(lowdim['starts'][p]), int(lowdim['lengths'][p]) + index.append(np.arange(s, s + n)) + end[s:s + n] = s + n + self.index, self.end = np.concatenate(index), end + self._maps, self._pid = None, None + + def __len__(self): + return len(self.index) + + def _frames(self): + if self._maps is None or self._pid != os.getpid(): + self._maps = {c: np.memmap(self.paths[c], dtype=np.uint8, mode='r', shape=(self.total, *self.hw, 3)) + for c in self.cameras} + self._pid = os.getpid() + return self._maps + + def rows(self, i: int) -> np.ndarray: + return np.minimum(np.arange(i, i + self.horizon), self.end[i] - 1) + + def __getitem__(self, k): + import torch + i = int(self.index[k]) + maps = self._frames() + images = torch.from_numpy(np.stack([maps[c][i] for c in self.cameras])) + state = self.lowdim['state'][i] + if self.body_dims: + augmentation = augmentation_module() + images, _ = augmentation.augment_frames_uint8(images) + state = augmentation.perturb_body(state, self.body_dims) + return images, torch.from_numpy(state), torch.from_numpy(self.lowdim['action'][self.rows(i)]) diff --git a/code/fmdit_continue_20261117/baselines/nets.py b/code/fmdit_continue_20261117/baselines/nets.py new file mode 100644 index 0000000000000000000000000000000000000000..a104b1579d2689c463e76d00c8b4b1ef05f43910 --- /dev/null +++ b/code/fmdit_continue_20261117/baselines/nets.py @@ -0,0 +1,207 @@ +"""Networks of the FM-DiT and DP (DDPM-DiT) baselines: the AdaLN DiT action backbone and the camera + state encoder. + +Ported from DextraVerse's imitation-learning core (`dextraverse/algorithms/il/models/nets/dit.py`, `layers.py`, +`positional_embedding.py`, `vision/backbones.py`, `vision/encoder.py`), unchanged in architecture and initialization. +Left out: the ROVE improvement embedding, the spatial-softmax / R3M / tactile options and the inpainting path (the +baselines always condition through the global observation feature). Added: the ResNet's normalization choice +(`get_resnet`): DextraVerse's GroupNorm for a backbone trained from scratch, or BatchNorm frozen at the pretrained +statistics (the DETR / ACT practice) for an ImageNet-initialized backbone that is fine-tuned with the policy. +""" +from __future__ import annotations + +import math + +import torch +import torch.nn as nn +import torch.nn.functional as F +import torchvision + +IMAGENET_MEAN = (0.485, 0.456, 0.406) +IMAGENET_STD = (0.229, 0.224, 0.225) + + +class SinusoidalPosEmb(nn.Module): + def __init__(self, dim): + super().__init__() + self.dim = dim + + def forward(self, x): + half_dim = self.dim // 2 + emb = math.log(10000) / (half_dim - 1) + emb = torch.exp(torch.arange(half_dim, device=x.device) * -emb) + emb = x[:, None] * emb[None, :] + return torch.cat((emb.sin(), emb.cos()), dim=-1) + + +class RotaryPosEmb(nn.Module): + """Rotary position embedding over each attention head's (S, head_dim) slice.""" + + def __init__(self, dim): + super().__init__() + self.dim = dim + self.register_buffer('inv_freq', 1.0 / (10000 ** (torch.arange(0, dim, 2).float() / dim))) + + def forward(self, x): # x: (B, heads, S, head_dim) + t = torch.arange(x.shape[-2], device=x.device, dtype=self.inv_freq.dtype) + freqs = torch.outer(t, self.inv_freq.to(x.device)) + emb = torch.cat([freqs, freqs], dim=-1) + cos, sin = emb.cos()[None, None], emb.sin()[None, None] + x1, x2 = x[..., : x.shape[-1] // 2], x[..., x.shape[-1] // 2:] + return x * cos + torch.cat([-x2, x1], dim=-1) * sin + + +class Mlp(nn.Module): + """The ViT MLP (timm.layers.Mlp) as DextraVerse's nets/layers.py.""" + + def __init__(self, in_features, hidden_features=None, out_features=None, act_layer=nn.GELU, drop=0.0): + super().__init__() + out_features = out_features or in_features + hidden_features = hidden_features or in_features + self.fc1 = nn.Linear(in_features, hidden_features) + self.act = act_layer() + self.drop1 = nn.Dropout(drop) + self.norm = nn.Identity() + self.fc2 = nn.Linear(hidden_features, out_features) + self.drop2 = nn.Dropout(drop) + + def forward(self, x): + return self.drop2(self.fc2(self.norm(self.drop1(self.act(self.fc1(x)))))) + + +class Attention(nn.Module): + def __init__(self, dim, num_heads=8, qkv_bias=True, attn_drop=0.0, proj_drop=0.0, qk_norm=True): + super().__init__() + assert dim % num_heads == 0, 'dim should be divisible by num_heads' + self.num_heads = num_heads + self.head_dim = dim // num_heads + self.attn_drop = attn_drop + self.proj_drop = nn.Dropout(proj_drop) + self.qkv = nn.Linear(dim, dim * 3, bias=qkv_bias) + self.q_norm = nn.LayerNorm(self.head_dim) if qk_norm else nn.Identity() + self.k_norm = nn.LayerNorm(self.head_dim) if qk_norm else nn.Identity() + self.proj = nn.Linear(dim, dim) + self.rope = RotaryPosEmb(self.head_dim) + + def forward(self, x): + B, S, C = x.shape + qkv = self.qkv(x).reshape(B, S, 3, self.num_heads, self.head_dim).permute(2, 0, 3, 1, 4) + q, k, v = qkv.unbind(0) + q, k = self.rope(self.q_norm(q)), self.rope(self.k_norm(k)) + x = F.scaled_dot_product_attention(q, k, v, dropout_p=self.attn_drop if self.training else 0.0) + return self.proj_drop(self.proj(x.transpose(1, 2).reshape(B, S, C))) + + +class AdaLNBlock(nn.Module): + """Adaptive-LayerNorm transformer block as in DiT (Peebles et al. 2022).""" + + def __init__(self, dim, num_heads, mlp_ratio=4.0, dropout=0.0): + super().__init__() + self.norm1 = nn.LayerNorm(dim, elementwise_affine=False, eps=1e-6) + self.attn = Attention(dim, num_heads=num_heads, attn_drop=dropout, proj_drop=dropout) + self.norm2 = nn.LayerNorm(dim, elementwise_affine=False, eps=1e-6) + self.mlp = Mlp(in_features=dim, hidden_features=int(dim * mlp_ratio), + act_layer=lambda: nn.GELU(approximate='tanh'), drop=0) + self.ada_ln = nn.Sequential(nn.SiLU(), nn.Linear(dim, 6 * dim)) + self.dim = dim + nn.init.constant_(self.ada_ln[-1].weight, 0) + nn.init.constant_(self.ada_ln[-1].bias, 0) + + def forward(self, x, t, c): + B = x.shape[0] + gamma1, gamma2, scale1, scale2, shift1, shift2 = self.ada_ln(F.silu(t + c)).view(B, 6, 1, self.dim).unbind(1) + x = x + self.attn(self.norm1(x).mul(scale1.add(1)).add_(shift1)).mul_(gamma1) + return x + self.mlp(self.norm2(x).mul(scale2.add(1)).add_(shift2)).mul_(gamma2) + + +class FlowTransformer(nn.Module): + """DiT action backbone: AdaLN blocks over the H action tokens, conditioned on time_embed(t) + cond_embed(obs). + + As DextraVerse, the generic Xavier initialization runs after the blocks' zero init of their AdaLN projection, so + it is not the full AdaLN-Zero recipe (kept: it is the reference's training behaviour).""" + + def __init__(self, input_dim, condition_dim, hidden_dim, output_dim, num_layers, num_heads, mlp_ratio=4.0, + dropout=0.1, time_embed_dim=256): + super().__init__() + self.input_proj = nn.Linear(input_dim, hidden_dim) + self.time_embed = nn.Sequential(SinusoidalPosEmb(time_embed_dim), nn.Linear(time_embed_dim, time_embed_dim * 4), + nn.Mish(), nn.Linear(time_embed_dim * 4, hidden_dim)) + self.cond_embed = nn.Linear(condition_dim, hidden_dim) + self.transformer_blocks = nn.ModuleList( + [AdaLNBlock(hidden_dim, num_heads, mlp_ratio=mlp_ratio, dropout=dropout) for _ in range(num_layers)]) + self.norm = nn.LayerNorm(hidden_dim) + self.out_proj = nn.Linear(hidden_dim, output_dim) + + def basic_init(module): + if isinstance(module, nn.Linear): + nn.init.xavier_uniform_(module.weight) + if module.bias is not None: + nn.init.constant_(module.bias, 0) + + self.apply(basic_init) + nn.init.normal_(self.time_embed[1].weight, std=0.02) + nn.init.normal_(self.time_embed[3].weight, std=0.02) + + def forward(self, x, t, global_cond): + if not torch.is_tensor(t): + t = torch.tensor(t, device=x.device) + if t.ndim == 0: + t = t.expand(x.shape[0]) + x = self.input_proj(x) + t = self.time_embed(t) + c = self.cond_embed(global_cond) + for block in self.transformer_blocks: + x = block(x, t, c) + return self.out_proj(self.norm(x)) + + +def replace_batchnorm_with_groupnorm(module: nn.Module) -> nn.Module: + """DextraVerse's use_group_norm: every BatchNorm2d becomes GroupNorm(num_features // 16 groups).""" + for name, child in module.named_children(): + if isinstance(child, nn.BatchNorm2d): + setattr(module, name, nn.GroupNorm(num_groups=child.num_features // 16, num_channels=child.num_features)) + else: + replace_batchnorm_with_groupnorm(child) + return module + + +NORMS = ('group', 'frozen_bn') + + +def get_resnet(name='resnet18', weights=None, norm='group') -> nn.Module: + """A torchvision ResNet with its classifier removed: (N, 3, H, W) -> (N, 512) globally pooled. + + weights: torchvision weights name (e.g. 'IMAGENET1K_V1') to initialize from, or None (random initialization; + a checkpoint load supplies every weight). norm: 'group' replaces every BatchNorm with GroupNorm (DextraVerse's + use_group_norm, for training from scratch: pretrained BatchNorm statistics and affine are dropped); 'frozen_bn' + keeps BatchNorm frozen at its (pretrained) running statistics and affine, the convolutions fine-tune.""" + if norm == 'frozen_bn': + from .act_nets import FrozenBatchNorm2d + resnet = getattr(torchvision.models, name)(weights=weights, norm_layer=FrozenBatchNorm2d) + elif norm == 'group': + resnet = replace_batchnorm_with_groupnorm(getattr(torchvision.models, name)(weights=weights)) + else: + raise ValueError(f'norm must be one of {NORMS}, got {norm!r}') + resnet.fc = nn.Identity() + return resnet + + +class ImageStateEncoder(nn.Module): + """DextraVerse MultiImageObsEncoder as configured for FM-DiT / DDPM-DiT: one ResNet shared by every camera + (share_rgb_model; `get_resnet` weights and norm), ImageNet normalization of the [0, 1] images, the pooled features + of each camera in the contract's camera order, then the normalized state. + Output (B, n_cameras * 512 + state_dim).""" + + def __init__(self, n_cameras, state_dim, backbone='resnet18', weights=None, norm='group'): + super().__init__() + self.rgb = get_resnet(backbone, weights=weights, norm=norm) + self.n_cameras, self.state_dim = int(n_cameras), int(state_dim) + self.register_buffer('mean', torch.tensor(IMAGENET_MEAN).view(1, 3, 1, 1), persistent=False) + self.register_buffer('std', torch.tensor(IMAGENET_STD).view(1, 3, 1, 1), persistent=False) + self.feature_dim = self.n_cameras * (512 if backbone in ('resnet18', 'resnet34') else 2048) + self.state_dim + + def forward(self, images, state): + """images (B, n_cameras, 3, H, W) in [0, 1]; state (B, state_dim) normalized.""" + B = images.shape[0] + x = (images.flatten(0, 1) - self.mean) / self.std + features = self.rgb(x).reshape(B, -1) # camera features side by side, in camera order + return torch.cat([features, state], dim=-1) diff --git a/code/fmdit_continue_20261117/baselines/policies.py b/code/fmdit_continue_20261117/baselines/policies.py new file mode 100644 index 0000000000000000000000000000000000000000..41aabebb1109ef2a5bd7222ffc1b8cd6e96077f1 --- /dev/null +++ b/code/fmdit_continue_20261117/baselines/policies.py @@ -0,0 +1,281 @@ +"""The three baseline policies behind one contract, ported from DextraVerse's imitation-learning core: + +- `fm_dit`, `FMDiTPolicy` (reference: dextraverse/algorithms/il/models/fm_dit.py + fm_base.py): flow matching, MSE on + the velocity `a - noise`; sampled with 10 midpoint steps from noise (20 DiT calls); +- `dp`, `DPPolicy` (ddpm_dit.py + ddpm_base.py: Diffusion Policy with a DiT backbone): DDPM epsilon MSE over 100 + steps; sampled with the 100 DDPM steps; +- `act`, `ACTPolicy` (act.py + nets/act/): L1 + `kl_weight` x KL of the CVAE; decoded from the zero latent in one pass. + +Contract (every policy): `compute_loss(images, state, action) -> (loss, metrics)` and +`predict(images, state, generator) -> (action, normalized action)`; images (B, n_cameras, 3, H, W) float in [0, 1], +state (B, state_dim) and action (B, horizon, action_dim) in physical units. Each policy owns its `Normalizer` +(DextraVerse's LinearNormalizer 'limits' mode: state and action to [-1, 1] per dimension, constant dimensions shifted +to 0), saved in its state dict. One observation frame (n_obs_steps 1: the current images and state); the chunk's row 0 +is the action of the observation's own frame (the LeRobot rows t..t+horizon-1). The configuration that builds a +policy is a plain dict (`PolicyConfig` fields), stored in the checkpoint. + +The ResNet18 starts from torchvision's ImageNet-1k weights (`backbone_weights` 'IMAGENET1K_V1', the default since +2026-11-16) and is fine-tuned with the policy, its BatchNorm frozen at the pretrained statistics (`backbone_norm` +'frozen_bn', as ACT always does); `backbone_weights` None trains from scratch with DextraVerse's GroupNorm encoder +(FM-DiT, DP) or ACT's frozen-BatchNorm one. A checkpoint load builds the structure without fetching any weights. +""" +from __future__ import annotations + +from dataclasses import asdict, dataclass, field, replace + +import torch +import torch.nn as nn +import torch.nn.functional as F + +from .act_nets import DETRVAE, Transformer, build_backbone, build_style_encoder, kl_divergence +from .nets import IMAGENET_MEAN, IMAGENET_STD, FlowTransformer, ImageStateEncoder + + +@dataclass +class PolicyConfig: + """Everything that builds a policy (stored in the checkpoint's config.json). Model defaults are DextraVerse's + configs/algorithm/il/model/{fm_dit,ddpm_dit,act}.yaml.""" + policy: str + n_cameras: int + state_dim: int + action_dim: int + horizon: int = 16 + backbone: str = 'resnet18' + backbone_weights: str | None = 'IMAGENET1K_V1' # torchvision weights to initialize from; None = from scratch + backbone_norm: str | None = None # 'frozen_bn' or 'group'; None: frozen_bn when pretrained or ACT, else group + # FM-DiT / DP (DiT backbone) + hidden_dim: int = 384 + num_layers: int = 4 + num_heads: int = 8 + mlp_ratio: float = 4.0 + dropout: float = 0.1 + time_embed_dim: int = 128 + num_inference_steps: int | None = None # None: FM-DiT 10 midpoint steps, DP every training timestep (100) + # DP noise schedule (diffusers DDPMScheduler) + num_train_timesteps: int = 100 + beta_schedule: str = 'squaredcos_cap_v2' + prediction_type: str = 'epsilon' + clip_sample: bool = True + # ACT + act_hidden_dim: int = 512 + dim_feedforward: int = 3200 + enc_layers: int = 4 + dec_layers: int = 7 + nheads: int = 8 + latent_dim: int = 32 + kl_weight: float = 10.0 + extra: dict = field(default_factory=dict) + + def __post_init__(self): + if self.backbone_norm is None: + self.backbone_norm = 'frozen_bn' if self.backbone_weights or self.policy == 'act' else 'group' + if self.policy == 'act' and self.backbone_norm != 'frozen_bn': + raise ValueError("ACT's backbone keeps frozen BatchNorm (backbone_norm 'frozen_bn')") + + @classmethod + def from_dict(cls, data: dict) -> PolicyConfig: + known = {k: v for k, v in data.items() if k in cls.__dataclass_fields__} + return cls(**known) + + def to_dict(self) -> dict: + return asdict(self) + + +class Normalizer(nn.Module): + """Per-dimension affine map to [-1, 1] from the training data's min/max (DextraVerse LinearNormalizer, mode + 'limits', fit_offset, range_eps 1e-4): x_n = x * scale + offset; a dimension whose range is below range_eps gets + scale 1 and maps its value to 0.""" + + def __init__(self, dim: int): + super().__init__() + self.register_buffer('scale', torch.ones(dim)) + self.register_buffer('offset', torch.zeros(dim)) + + @torch.no_grad() + def fit(self, minimum, maximum, range_eps=1e-4): + lo = torch.as_tensor(minimum, dtype=torch.float32) + hi = torch.as_tensor(maximum, dtype=torch.float32) + span = hi - lo + ignore = span < range_eps + span[ignore] = 2.0 + scale = 2.0 / span + offset = -1.0 - scale * lo + offset[ignore] = -lo[ignore] + self.scale.copy_(scale) + self.offset.copy_(offset) + + def normalize(self, x): + return x * self.scale + self.offset + + def unnormalize(self, x): + return (x - self.offset) / self.scale + + +class BasePolicy(nn.Module): + name = '' + backbone_prefix = '' # state-dict prefix of the image backbone (its own learning rate in training) + + def __init__(self, cfg: PolicyConfig, init_weights=None): + super().__init__() + self.cfg = cfg + self.state_norm = Normalizer(cfg.state_dim) + self.action_norm = Normalizer(cfg.action_dim) + + def fit_normalizers(self, stats: dict): + self.state_norm.fit(stats['state']['min'], stats['state']['max']) + self.action_norm.fit(stats['action']['min'], stats['action']['max']) + + @property + def sampling_steps(self) -> int: + return 1 + + def compute_loss(self, images, state, action): + raise NotImplementedError + + def predict(self, images, state, generator=None): + raise NotImplementedError + + +class DiTPolicy(BasePolicy): + """Shared by FM-DiT and DP: the shared-ResNet image + state encoder and the AdaLN DiT over the action chunk + (global observation conditioning, DextraVerse obs_as_global_cond).""" + backbone_prefix = 'obs_encoder.rgb.' + + def __init__(self, cfg: PolicyConfig, init_weights=None): + super().__init__(cfg) + self.obs_encoder = ImageStateEncoder(cfg.n_cameras, cfg.state_dim, backbone=cfg.backbone, + weights=init_weights, norm=cfg.backbone_norm) + self.model = FlowTransformer(input_dim=cfg.action_dim, condition_dim=self.obs_encoder.feature_dim, + hidden_dim=cfg.hidden_dim, output_dim=cfg.action_dim, num_layers=cfg.num_layers, + num_heads=cfg.num_heads, mlp_ratio=cfg.mlp_ratio, dropout=cfg.dropout, + time_embed_dim=cfg.time_embed_dim) + + def condition(self, images, state): + return self.obs_encoder(images, self.state_norm.normalize(state)) + + +class FMDiTPolicy(DiTPolicy): + """Flow-matching DiT (DextraVerse FlowMatchingDiTImagePolicy): x_t = (1 - t) noise + t a, t ~ U(0, 1), the DiT + regresses the velocity a - noise; sampling integrates t = 0 -> 1 with the midpoint rule.""" + name = 'fm_dit' + + @property + def sampling_steps(self) -> int: + return int(self.cfg.num_inference_steps or 10) + + def compute_loss(self, images, state, action): + cond = self.condition(images, state) + target = self.action_norm.normalize(action) + noise = torch.randn_like(target) + t = torch.rand(target.shape[0], device=target.device) + te = t.view(-1, 1, 1) + x_t = (1 - te) * noise + te * target + loss = F.mse_loss(self.model(x_t, t, cond), target - noise) + return loss, dict(fm=float(loss.detach())) + + @torch.no_grad() + def predict(self, images, state, generator=None): + cond = self.condition(images, state) + B, steps = images.shape[0], self.sampling_steps + x = torch.randn((B, self.cfg.horizon, self.cfg.action_dim), device=cond.device, dtype=cond.dtype, + generator=generator) + times = torch.linspace(0, 1.0, steps + 1, device=cond.device) + for i in range(steps): + t0 = times[i].view(1).expand(B) + t1 = times[i + 1].view(1).expand(B) + dt = (t1 - t0).view(-1, 1, 1) + x_mid = x + self.model(x, t0, cond) * dt / 2 + x = x + dt * self.model(x_mid, t0 + (t1 - t0) / 2, cond) + return self.action_norm.unnormalize(x), x + + +class DPPolicy(DiTPolicy): + """Diffusion Policy with the DiT backbone (DextraVerse DiffusionDiTImagePolicy, ddpm_dit.yaml): DDPM with 100 + training steps, squaredcos_cap_v2 betas, fixed_small variance, clipped samples, epsilon prediction; sampled with + every training step (100 DiT calls).""" + name = 'dp' + + def __init__(self, cfg: PolicyConfig, init_weights=None): + super().__init__(cfg, init_weights) + from diffusers.schedulers.scheduling_ddpm import DDPMScheduler + self.noise_scheduler = DDPMScheduler(num_train_timesteps=cfg.num_train_timesteps, beta_start=0.0001, + beta_end=0.02, beta_schedule=cfg.beta_schedule, + variance_type='fixed_small', clip_sample=cfg.clip_sample, + prediction_type=cfg.prediction_type) + + @property + def sampling_steps(self) -> int: + return int(self.cfg.num_inference_steps or self.cfg.num_train_timesteps) + + def compute_loss(self, images, state, action): + cond = self.condition(images, state) + target = self.action_norm.normalize(action) + noise = torch.randn_like(target) + t = torch.randint(0, self.noise_scheduler.config.num_train_timesteps, (target.shape[0],), + device=target.device).long() + noisy = self.noise_scheduler.add_noise(target, noise, t) + pred = self.model(noisy, t, cond) + goal = noise if self.cfg.prediction_type == 'epsilon' else target + loss = F.mse_loss(pred, goal) + return loss, dict(ddpm=float(loss.detach())) + + @torch.no_grad() + def predict(self, images, state, generator=None): + cond = self.condition(images, state) + scheduler = self.noise_scheduler + x = torch.randn((images.shape[0], self.cfg.horizon, self.cfg.action_dim), device=cond.device, + dtype=cond.dtype, generator=generator) + scheduler.set_timesteps(self.sampling_steps, device=cond.device) + for t in scheduler.timesteps: + x = scheduler.step(self.model(x, t.to(cond.device), cond), t, x, generator=generator).prev_sample + return self.action_norm.unnormalize(x), x + + +class ACTPolicy(BasePolicy): + """ACT (DextraVerse ACTImagePolicy, act.yaml): ImageNet-normalized cameras through one ResNet with frozen + BatchNorm (spatial tokens, no pooling), the normalized state as qpos, a CVAE style encoder over the action chunk in + training, the DETR decoder with `horizon` queries; loss L1 + kl_weight x KL; inference decodes from the zero + latent (deterministic).""" + name = 'act' + backbone_prefix = 'model.backbone.' + + def __init__(self, cfg: PolicyConfig, init_weights=None): + super().__init__(cfg) + transformer = Transformer(d_model=cfg.act_hidden_dim, nhead=cfg.nheads, num_encoder_layers=cfg.enc_layers, + num_decoder_layers=cfg.dec_layers, dim_feedforward=cfg.dim_feedforward, + dropout=cfg.dropout, return_intermediate_dec=False) + self.model = DETRVAE(build_backbone(cfg.backbone, hidden_dim=cfg.act_hidden_dim, weights=init_weights), + transformer, + build_style_encoder(cfg.act_hidden_dim, cfg.nheads, cfg.dim_feedforward, cfg.enc_layers, + cfg.dropout), + qpos_dim=cfg.state_dim, action_dim=cfg.action_dim, num_queries=cfg.horizon, + latent_dim=cfg.latent_dim) + self.register_buffer('mean', torch.tensor(IMAGENET_MEAN).view(1, 1, 3, 1, 1), persistent=False) + self.register_buffer('std', torch.tensor(IMAGENET_STD).view(1, 1, 3, 1, 1), persistent=False) + + def compute_loss(self, images, state, action): + target = self.action_norm.normalize(action) + a_hat, (mu, logvar) = self.model(self.state_norm.normalize(state), (images - self.mean) / self.std, + actions=target) + l1 = F.l1_loss(a_hat, target) + kl = kl_divergence(mu, logvar) + return l1 + self.cfg.kl_weight * kl, dict(l1=float(l1.detach()), kl=float(kl.detach())) + + @torch.no_grad() + def predict(self, images, state, generator=None): # noqa: ARG002 - deterministic (zero latent) + a_hat, _ = self.model(self.state_norm.normalize(state), (images - self.mean) / self.std) + return self.action_norm.unnormalize(a_hat), a_hat + + +POLICIES = {cls.name: cls for cls in (FMDiTPolicy, DPPolicy, ACTPolicy)} +LABELS = {'fm_dit': 'FM-DiT', 'dp': 'DP (DDPM-DiT)', 'act': 'ACT'} + + +def build_policy(cfg: PolicyConfig | dict, pretrained: bool = True) -> BasePolicy: + """The policy of `cfg`; pretrained=True initializes the backbone from cfg.backbone_weights (training), + False builds the same structure with random weights (a checkpoint load, which must not download anything).""" + cfg = cfg if isinstance(cfg, PolicyConfig) else PolicyConfig.from_dict(cfg) + if cfg.policy not in POLICIES: + raise KeyError(f'unknown baseline {cfg.policy!r}; one of {sorted(POLICIES)}') + cfg = replace(cfg) # a copy (resolved backbone_norm) + return POLICIES[cfg.policy](cfg, init_weights=cfg.backbone_weights if pretrained else None) diff --git a/code/fmdit_continue_20261117/baselines/requirements.txt b/code/fmdit_continue_20261117/baselines/requirements.txt new file mode 100644 index 0000000000000000000000000000000000000000..9f97db4822c6da22bb4f8026920879262a675aa9 --- /dev/null +++ b/code/fmdit_continue_20261117/baselines/requirements.txt @@ -0,0 +1,12 @@ +# Environment of the FM-DiT / DP / ACT baselines (training and the policy server), Python 3.12, CUDA 12.8 wheels: +# uv venv --python 3.12 +# uv pip install --python /bin/python -r evaluation/baselines/requirements.txt \ +# --extra-index-url https://download.pytorch.org/whl/cu128 --index-strategy unsafe-best-match +torch==2.9.0+cu128 +torchvision==0.24.0+cu128 +diffusers==0.35.1 +av==15.1.0 +pyarrow==23.0.1 +numpy==2.2.6 +opencv-python-headless==4.11.0.86 +pytest==8.4.2 diff --git a/code/fmdit_continue_20261117/baselines/runtime.py b/code/fmdit_continue_20261117/baselines/runtime.py new file mode 100644 index 0000000000000000000000000000000000000000..14e9825510f200578bdeb3c01ead3e54256a70c1 --- /dev/null +++ b/code/fmdit_continue_20261117/baselines/runtime.py @@ -0,0 +1,123 @@ +"""Checkpoints and inference of the baselines. + +A checkpoint is a directory written by train.py (`/checkpoints/step-NNNNNN/`, `final` links the last one): + +- `model.pt`: torch.save of {format, config (the config.json text), state_dict}: the EMA weights (or the raw ones when + EMA is off) with the policy's state and action normalizers, loaded strictly with `weights_only=True`; +- `config.json`: the policy configuration, the contract (robot, cameras, state and action columns), the image size, + the training task text, dataset, episodes, recipe and statistics; +- `protocol.json`: model.pt's size and SHA-256. + +`BaselinePolicy.predict` takes the live frames of the checkpoint's cameras (any size; resized with data.resize_frame, +as the training cache), the physical state of the contract and a seed, and returns the physical (horizon, +action_dim) chunk with its normalized form; row 0 is the action of the observation's frame. +""" +from __future__ import annotations + +import hashlib +import json +from pathlib import Path + +import numpy as np + +FORMAT = 'hb_baselines/1' + + +def sha256(path) -> str: + digest = hashlib.sha256() + with Path(path).open('rb') as stream: + for block in iter(lambda: stream.read(16 << 20), b''): + digest.update(block) + return digest.hexdigest() + + +def save_checkpoint(folder, policy, config: dict): + """Write model.pt, config.json and protocol.json of `policy` (an nn.Module) into `folder`.""" + import torch + folder = Path(folder) + folder.mkdir(parents=True, exist_ok=True) + text = json.dumps(dict(config, format=FORMAT), indent=1) + state = {k: v.detach().cpu().contiguous() for k, v in policy.state_dict().items()} + torch.save(dict(format=FORMAT, config=text, state_dict=state), folder / 'model.pt.tmp') + (folder / 'model.pt.tmp').replace(folder / 'model.pt') + (folder / 'config.json').write_text(text) + (folder / 'protocol.json').write_text(json.dumps(dict( + format=FORMAT, step=config.get('step'), policy=config['policy']['policy'], + files={'model.pt': dict(bytes=(folder / 'model.pt').stat().st_size, sha256=sha256(folder / 'model.pt'))}), + indent=1)) + + +def load_checkpoint(folder, device='cpu', verify=True): + """(policy in eval mode on `device`, config dict) of a checkpoint directory.""" + import torch + from .policies import build_policy + folder = Path(folder) + if verify: + protocol = json.loads((folder / 'protocol.json').read_text()) + entry = protocol['files']['model.pt'] + if (folder / 'model.pt').stat().st_size != entry['bytes']: + raise ValueError(f'{folder}/model.pt differs from protocol.json') + payload = torch.load(folder / 'model.pt', map_location='cpu', weights_only=True) + if payload.get('format') != FORMAT: + raise ValueError(f"{folder}: format {payload.get('format')!r}, expected {FORMAT}") + config = json.loads(payload['config']) + if json.loads((folder / 'config.json').read_text()) != config: + raise ValueError(f'{folder}: config.json differs from the one inside model.pt') + policy = build_policy(config['policy'], pretrained=False) # every weight comes from model.pt + policy.load_state_dict(payload['state_dict'], strict=True) + return policy.to(device).eval(), config + + +class BaselinePolicy: + """A loaded checkpoint for closed-loop inference (one request at a time).""" + + def __init__(self, checkpoint, device='cuda:0', policy: str | None = None, robot: str | None = None): + import torch + self.torch = torch + self.device = torch.device(device) + self.model, self.config = load_checkpoint(checkpoint, device=self.device) + self.checkpoint = str(Path(checkpoint).resolve()) + name, contract = self.config['policy']['policy'], self.config['contract'] + if policy is not None and name != policy: + raise ValueError(f'{checkpoint} is a {name!r} checkpoint, not {policy!r}') + if robot is not None and contract['robot'] != robot: + raise ValueError(f"{checkpoint} was trained for {contract['robot']!r} ({contract['name']}), not {robot!r}") + self.name, self.contract = name, contract + self.cameras = tuple(contract['cameras']) + self.image_hw = tuple(self.config['image_hw']) + self.horizon = int(self.config['policy']['horizon']) + self.state_dim, self.action_dim = int(contract['state_dim']), int(contract['action_dim']) + self.sampling_steps = int(self.model.sampling_steps) + self.task_text = self.config.get('task_text') + + def info(self) -> dict: + return dict(model=self.name, checkpoint=self.checkpoint, step=self.config.get('step'), + contract=self.contract['name'], views=list(self.cameras), image_hw=list(self.image_hw), + action_chunk=self.horizon, action_dim=self.action_dim, state_dim=self.state_dim, + sampling_steps=self.sampling_steps, task_text=self.task_text, + training_data=self.config.get('dataset', {}).get('name')) + + def images_tensor(self, images: dict): + from .data import resize_frame + missing = [c for c in self.cameras if c not in images] + if missing: + raise KeyError(f'missing cameras {missing}; the checkpoint reads {list(self.cameras)}') + frames = np.stack([resize_frame(np.asarray(images[c]), self.image_hw) for c in self.cameras]) + x = self.torch.from_numpy(frames).to(self.device).permute(0, 3, 1, 2).float().div_(255.) + return x.unsqueeze(0) # (1, n_cameras, 3, h, w) + + def predict(self, images: dict, state, seed: int = 0): + """(action (horizon, action_dim) float32 physical, normalized (horizon, action_dim) float32).""" + torch = self.torch + state = np.asarray(state, np.float32).reshape(-1) + if state.shape != (self.state_dim,) or not np.isfinite(state).all(): + raise ValueError(f'expected a finite state of {self.state_dim} values, got {state.shape}') + generator = torch.Generator(device=self.device) + generator.manual_seed(int(seed) % (2 ** 63 - 1)) + with torch.inference_mode(): + action, raw = self.model.predict(self.images_tensor(images), + torch.from_numpy(state)[None].to(self.device), generator=generator) + action = action[0].float().cpu().numpy() + if action.shape != (self.horizon, self.action_dim) or not np.isfinite(action).all(): + raise ValueError(f'invalid prediction {action.shape}') + return np.ascontiguousarray(action, np.float32), raw[0].float().cpu().numpy().astype(np.float32) diff --git a/code/fmdit_continue_20261117/baselines/train.py b/code/fmdit_continue_20261117/baselines/train.py new file mode 100644 index 0000000000000000000000000000000000000000..cde41dd3d6af9fe5693f06203e40aaabd1901ca8 --- /dev/null +++ b/code/fmdit_continue_20261117/baselines/train.py @@ -0,0 +1,395 @@ +"""Train an FM-DiT, DP (DDPM-DiT) or ACT baseline on a task's LeRobot v2.1 expert dataset. + + /bin/python -I evaluation/baselines/train.py --policy fm_dit --contract h2_sharpa \\ + --dataset data/h2_pour_water/h2_pour_water_r1/lerobot --out --steps 60000 + +The recipe is DextraVerse's shared IL recipe (configs/algorithm/il/train.yaml; runners/train_dex_flow_matching.py): +AdamW (lr 1e-4, betas 0.95/0.999, weight decay 1e-6), cosine decay (here after a 500-step linear warm-up, as +Diffusion Policy), whole-policy EMA (Diffusion Policy's warm-up schedule, max decay 0.9999) whose weights are the +saved checkpoint, batch 64, one observation frame and 16-row chunks. Added for the humanoid views: frames resized +to 168x224 (the 480x640 aspect). bf16 autocast on CUDA. + +Augmentation (2026-11-16): the Humanoid Benchmark spec v2 (docs/augmentation_v2.md, evaluation/augmentation), the +same for every model of the benchmark, is on by default (--augmentation hb-v2): per training item one hb_aug_v2 call +on the cached 168x224 frames of all cameras (colour shared, geometry / light patches / white balance per camera, one +of the 15 effects boosted 1.5 x) and the per-sample joint bias (+-2 deg) + noise (0.25 deg, clipped 3 sigma) on the +body joint angles of the raw state (G1 29, H2 31), before the policy's normalizers; actions untouched. The native +random shift is off (--shift 0; a shift with hb-v2 is refused). Before the first step a preflight reads training items +through the augmented and the plain path and checks the state width (G1 44 / H2 78, no velocity), that only the body +joints move and by at most 2.75 deg, that the action rows are bit-identical, that the images change, and that the +offline-check items are deterministic; it is stored in run.json (--preflight-only stops after it). --augmentation +none is an explicit exception to the spec, recorded in run.json and the checkpoints. The ResNet18 starts from torchvision's ImageNet-1k weights +(--backbone-weights IMAGENET1K_V1, downloaded once to the torch hub cache) and is fine-tuned at 0.1 x the learning +rate (--backbone-lr-scale, the DETR practice) with its BatchNorm frozen; --backbone-weights none trains it from +scratch (DextraVerse's GroupNorm encoder for FM-DiT and DP). + +Writes into --out: run.json (arguments), train_log.jsonl, checkpoints/step-NNNNNN/ every --save-every steps and at the +end (runtime.py format), checkpoints/final -> the last one, and train_state.pt (optimizer, schedule, EMA, step) that +--resume continues from. Each save also measures the EMA policy offline on fixed frames (held-out episodes with +--val-episodes, else training frames): normalized-action MSE overall and per action group. +""" +from __future__ import annotations + +import argparse +import copy +from datetime import datetime +import json +import math +import os +from pathlib import Path +import subprocess +import sys +import time + +import numpy as np + +HERE = Path(__file__).resolve().parent +REPO = HERE.parents[1] +if str(HERE.parent) not in sys.path: + sys.path.insert(0, str(HERE.parent)) + +from baselines.data import (CONTRACTS, ChunkDataset, LeRobotDataset, augmentation_module, frame_cache, # noqa: E402 + load_lowdim, parse_episodes, statistics) +from baselines.policies import POLICIES, PolicyConfig, build_policy # noqa: E402 +from baselines.runtime import save_checkpoint # noqa: E402 + +DEFAULTS = dict(lr=1e-4, betas=(0.95, 0.999), weight_decay=1e-6, warmup=500, ema=0.9999, batch_size=64, + backbone_weights='IMAGENET1K_V1', backbone_lr_scale=0.1) + + +def git_commit(): + try: + return subprocess.run(['git', '-C', str(REPO), 'rev-parse', 'HEAD'], capture_output=True, text=True, + check=True).stdout.strip() + except Exception: # noqa: BLE001 - outside a checkout + return None + + +def parse_hw(text): + h, w = text.lower().split('x') + return int(h), int(w) + + +def random_shift(images, pad): + """DrQ random shift: replicate-pad by `pad` pixels and crop back at a random offset per sample. + images (B, N, 3, H, W) float.""" + import torch + import torch.nn.functional as F + if pad <= 0: + return images + B, N, C, H, W = images.shape + x = F.pad(images.reshape(B * N, C, H, W), (pad,) * 4, mode='replicate') + oy = torch.randint(0, 2 * pad + 1, (B * N,), device=images.device) + ox = torch.randint(0, 2 * pad + 1, (B * N,), device=images.device) + rows = (oy[:, None] + torch.arange(H, device=images.device)[None])[:, None, :, None].expand(-1, C, -1, W + 2 * pad) + x = x.gather(2, rows) + cols = (ox[:, None] + torch.arange(W, device=images.device)[None])[:, None, None, :].expand(-1, C, H, -1) + return x.gather(3, cols).reshape(B, N, C, H, W) + + +class EMA: + """Diffusion Policy's EMAModel schedule: decay = min(max_decay, 1 - (1 + step) ** -0.75), parameters averaged, + buffers (normalizers) copied.""" + + def __init__(self, model, max_decay): + self.model = copy.deepcopy(model).eval().requires_grad_(False) + self.max_decay = float(max_decay) + + def update(self, model, step): + import torch + decay = 0.0 if step <= 0 else min(self.max_decay, max(0.0, 1 - (1 + step) ** -0.75)) + with torch.no_grad(): + for e, p in zip(self.model.parameters(), model.parameters()): + e.mul_(decay).add_(p.detach(), alpha=1 - decay) + for e, b in zip(self.model.buffers(), model.buffers()): + e.copy_(b) + return decay + + +def preflight(train_set, eval_set, contract, samples=64, seed=0) -> dict: + """Training items of `train_set` (augmented unless its body_dims is 0) against the same frames read without + augmentation, and two reads of the offline-check items; raises on any violation of docs/augmentation_v2.md.""" + import torch + plain = ChunkDataset(train_set.lowdim, train_set.paths, train_set.cameras, train_set.hw, train_set.horizon, + train_set.positions) # the same frames without augmentation + picks = np.random.default_rng(seed).choice(len(plain), size=min(samples, len(plain)), replace=False) + nb, sd = train_set.body_dims, contract.state_dim + limit = augmentation_module().max_body_offset_rad() * (1 + 1e-5) + images_changed = body_moved = 0 + worst = 0.0 + for k in picks: + (im, st, ac), (im0, st0, ac0) = train_set[int(k)], plain[int(k)] + if st.shape != (sd,) or st0.shape != (sd,) or ac.shape != ac0.shape: + raise SystemExit(f'preflight: state {tuple(st.shape)} / action {tuple(ac.shape)}, expected ({sd},)') + if not torch.equal(ac, ac0): + raise SystemExit('preflight: an action label differs from the recorded data') + if not torch.equal(st[nb:], st0[nb:]): + raise SystemExit(f'preflight: a state value past the {nb} body joints changed') + delta = (st[:nb].double() - st0[:nb].double()).abs() + if nb: + worst = max(worst, float(delta.max())) + if worst > limit: + raise SystemExit(f'preflight: a body joint moved {math.degrees(worst):.3f} deg (> 2.75)') + body_moved += int((delta > 0).sum()) + images_changed += int(not torch.equal(im, im0)) + n = len(picks) + moved = body_moved / max(1, n * nb) + if nb and (images_changed < 0.9 * n or moved < 0.95): + raise SystemExit(f'preflight: augmentation inactive ({images_changed}/{n} images changed, {moved:.3f} of the ' + 'body joint values moved)') + if not nb and (images_changed or worst): + raise SystemExit('preflight: items changed with the augmentation off') + eval_picks = np.random.default_rng(seed + 1).choice(len(eval_set), size=min(8, len(eval_set)), replace=False) + for k in eval_picks: + a, b = eval_set[int(k)], eval_set[int(k)] + if eval_set.body_dims or not all(torch.equal(x, y) for x, y in zip(a, b)): + raise SystemExit('preflight: the offline-check items are not deterministic') + return dict(samples=n, state_dim=sd, body_dims=nb, images_changed=images_changed, + body_values_moved=round(moved, 4), + max_body_offset_deg=round(math.degrees(worst), 4), actions_bit_identical=True, + non_body_state_bit_identical=True, eval_items_deterministic=len(eval_picks)) + + +def offline_eval(policy, dataset, samples, device, groups): + """Normalized-action MSE of policy.predict on fixed dataset items (seeded), overall and per action group.""" + import torch + policy.eval() + errs = [] + generator = torch.Generator(device=device) + generator.manual_seed(0) + for start in range(0, len(samples), 32): + batch = [dataset[int(k)] for k in samples[start:start + 32]] + images = torch.stack([b[0] for b in batch]).to(device).permute(0, 1, 4, 2, 3).float().div_(255.) + state = torch.stack([b[1] for b in batch]).to(device) + target = torch.stack([b[2] for b in batch]).to(device) + _, raw = policy.predict(images, state, generator=generator) + errs.append((raw.float() - policy.action_norm.normalize(target)).pow(2).cpu().numpy()) + err = np.concatenate(errs) # (S, horizon, action_dim) + out = dict(mse=float(err.mean()), mse_row0=float(err[:, 0].mean())) + offset = 0 + for column, start, end in groups: + out[f'mse_{column.split(".")[-1]}'] = float(err[..., offset:offset + end - start].mean()) + offset += end - start + return out + + +def main(): + p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + p.add_argument('--policy', choices=sorted(POLICIES), required=True) + p.add_argument('--dataset', type=Path, required=True, help='LeRobot v2.1 dataset directory') + p.add_argument('--contract', choices=sorted(CONTRACTS), required=True) + p.add_argument('--cameras', help='comma-separated cameras (default: the contract\'s)') + p.add_argument('--episodes', help="episodes to use, e.g. '0:100' (default: all)") + p.add_argument('--val-episodes', type=int, default=0, help='hold out the last N of them for the offline check') + p.add_argument('--out', type=Path, required=True) + p.add_argument('--steps', type=int, default=60000) + p.add_argument('--batch-size', type=int, default=DEFAULTS['batch_size']) + p.add_argument('--lr', type=float, default=DEFAULTS['lr']) + p.add_argument('--weight-decay', type=float, default=DEFAULTS['weight_decay']) + p.add_argument('--warmup', type=int, default=DEFAULTS['warmup']) + p.add_argument('--ema', type=float, default=DEFAULTS['ema'], help='max EMA decay; 0 saves the raw weights') + p.add_argument('--horizon', type=int, default=16) + p.add_argument('--image-size', default='168x224', help='HxW of the policy images') + p.add_argument('--augmentation', choices=['hb-v2', 'none'], default='hb-v2', + help='training augmentation: the benchmark spec v2 (default; docs/augmentation_v2.md) or none ' + '(an explicit exception, recorded in run.json)') + p.add_argument('--shift', type=int, default=0, + help='native random-shift augmentation in pixels (default 0: off; only with --augmentation none)') + p.add_argument('--preflight-only', action='store_true', help='run the data preflight, write run.json and stop') + p.add_argument('--backbone-weights', default=DEFAULTS['backbone_weights'], + help="torchvision weights the ResNet18 starts from (default %(default)s); 'none': from scratch") + p.add_argument('--backbone-lr-scale', type=float, default=DEFAULTS['backbone_lr_scale'], + help='learning rate of the image backbone relative to --lr (default %(default)s)') + p.add_argument('--model', action='append', default=[], metavar='KEY=VALUE', + help='override a PolicyConfig field (JSON value), e.g. num_layers=6; repeatable') + p.add_argument('--cache-dir', type=Path, default=Path('~/hb_models/baselines/cache')) + p.add_argument('--workers', type=int, default=32, + help='DataLoader workers (and frame-cache decoders); the v2 augmentation costs ~17 ms of one CPU ' + 'core per item, so 8 workers feed only ~6 steps/s at batch 64') + p.add_argument('--prefetch', type=int, default=2, help='batches prefetched per worker') + p.add_argument('--save-every', type=int, default=10000) + p.add_argument('--log-every', type=int, default=100) + p.add_argument('--eval-samples', type=int, default=256) + p.add_argument('--seed', type=int, default=0) + p.add_argument('--device', default='cuda:0') + p.add_argument('--no-amp', action='store_true') + p.add_argument('--resume', action='store_true', help='continue from /train_state.pt') + args = p.parse_args() + + out = args.out.expanduser().resolve() + out.mkdir(parents=True, exist_ok=True) + contract = CONTRACTS[args.contract] + aug = augmentation_module() + aug.check_state_layout(contract.robot, contract.state_dim, [column for column, _, _ in contract.state]) + hb_aug = args.augmentation == 'hb-v2' + if hb_aug and args.shift: + raise SystemExit('--shift is the native random shift, which the augmentation spec v2 turns off ' + '(docs/augmentation_v2.md); use it only with --augmentation none') + if contract.body_dims != aug.BODY_JOINTS[contract.robot]: + raise SystemExit(f'contract {contract.name}: body_dims {contract.body_dims}, the spec perturbs ' + f'{aug.BODY_JOINTS[contract.robot]} body joints') + augmentation = (aug.describe(contract.robot, contract.body_dims) if hb_aug else + dict(version=None, exception='--augmentation none (not the benchmark default)')) + cameras = tuple(args.cameras.split(',')) if args.cameras else contract.cameras + hw = parse_hw(args.image_size) + dataset = LeRobotDataset(args.dataset) + dataset.check_contract(contract, cameras) + episodes = parse_episodes(args.episodes, dataset.lengths) + if not 0 <= args.val_episodes < len(episodes): + raise SystemExit('--val-episodes must leave training episodes') + lowdim = load_lowdim(dataset, contract, episodes) + if len(lowdim['tasks']) != 1: + raise SystemExit(f"the baselines take no language: one task per dataset, found {lowdim['tasks']}") + n_train = len(episodes) - args.val_episodes + train_pos, val_pos = list(range(n_train)), list(range(n_train, len(episodes))) + train_rows = np.concatenate([np.arange(s, s + n) for s, n in + zip(lowdim['starts'][:n_train], lowdim['lengths'][:n_train])]) + stats = statistics(lowdim, train_rows) + frames = frame_cache(dataset, lowdim, cameras, hw, args.cache_dir, workers=args.workers) + train_set = ChunkDataset(lowdim, frames, cameras, hw, args.horizon, train_pos, + body_dims=contract.body_dims if hb_aug else 0) + eval_set = ChunkDataset(lowdim, frames, cameras, hw, args.horizon, val_pos or train_pos) # never augmented + checks = preflight(train_set, eval_set, contract, seed=args.seed) + print(json.dumps(dict(event='preflight', augmentation=augmentation.get('version'), **checks)), flush=True) + if args.preflight_only: + (out / 'run.json').write_text(json.dumps(dict( + argv=sys.argv, started=datetime.now().isoformat(timespec='seconds'), host=os.uname().nodename, + contract=contract.to_dict(cameras), image_hw=list(hw), augmentation=augmentation, preflight=checks, + shift_px=args.shift, code_commit=git_commit()), indent=1, default=str)) + return + + import torch + torch.manual_seed(args.seed) + np.random.seed(args.seed) + device = torch.device(args.device) + overrides = {k: json.loads(v) for k, v in (item.split('=', 1) for item in args.model)} + weights = None if str(args.backbone_weights).lower() in ('none', '') else args.backbone_weights + pcfg = PolicyConfig.from_dict({**dict(policy=args.policy, n_cameras=len(cameras), state_dim=contract.state_dim, + action_dim=contract.action_dim, horizon=args.horizon, + backbone_weights=weights), **overrides}) + policy = build_policy(pcfg) + policy.fit_normalizers(stats) + policy.to(device) + ema = EMA(policy, args.ema) if args.ema > 0 else None + named = list(policy.named_parameters()) + backbone = [p for n, p in named if n.startswith(policy.backbone_prefix)] + rest = [p for n, p in named if not n.startswith(policy.backbone_prefix)] + if not backbone: + raise SystemExit(f'no backbone parameters under {policy.backbone_prefix!r}') + optimizer = torch.optim.AdamW([dict(params=rest), dict(params=backbone, lr=args.lr * args.backbone_lr_scale)], + lr=args.lr, betas=DEFAULTS['betas'], weight_decay=args.weight_decay) + + def lr_lambda(step): + if step < args.warmup: + return (step + 1) / max(1, args.warmup) + return 0.5 * (1 + math.cos(math.pi * min(1.0, (step - args.warmup) / max(1, args.steps - args.warmup)))) + + scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda) + step = 0 + state_path = out / 'train_state.pt' + if args.resume and state_path.exists(): + saved = torch.load(state_path, map_location='cpu', weights_only=False) + if saved['config']['policy'] != pcfg.to_dict(): + raise SystemExit('train_state.pt was written for another policy configuration') + before = saved['config'].get('training', {}) + if (before.get('augmentation') or {}).get('version') != augmentation.get('version') \ + or before.get('shift_px') != args.shift: + raise SystemExit('train_state.pt was trained with another augmentation (version / --shift); resuming ' + 'would change it mid-run') + policy.load_state_dict(saved['policy']) + optimizer.load_state_dict(saved['optimizer']) + scheduler.load_state_dict(saved['scheduler']) + if ema is not None: + ema.model.load_state_dict(saved['ema']) + step = int(saved['step']) + print(json.dumps(dict(event='resumed', step=step)), flush=True) + + samples = np.random.default_rng(1).choice(len(eval_set), size=min(args.eval_samples, len(eval_set)), replace=False) + config = dict( + policy=pcfg.to_dict(), contract=contract.to_dict(cameras), image_hw=list(hw), task_text=lowdim['tasks'][0], + dataset=dict(name=dataset.name, + root=str(dataset.root), robot_type=dataset.info.get('robot_type'), fps=dataset.fps, + episodes=episodes, train_episodes=episodes[:n_train], val_episodes=episodes[n_train:], + train_frames=int(len(train_set)), + git_commit=dataset.info.get('script_config', {}).get('git_commit')), + training=dict(steps=args.steps, batch_size=args.batch_size, lr=args.lr, betas=list(DEFAULTS['betas']), + weight_decay=args.weight_decay, warmup=args.warmup, schedule='linear warm-up + cosine', + backbone_lr_scale=args.backbone_lr_scale, + ema=args.ema, shift_px=args.shift, amp=not args.no_amp, seed=args.seed, + augmentation=augmentation, preflight=checks), + stats=stats, code_commit=git_commit()) + run = dict(argv=sys.argv, started=datetime.now().isoformat(timespec='seconds'), host=os.uname().nodename, + torch=torch.__version__, **config) + (out / 'run.json').write_text(json.dumps(run, indent=1, default=str)) + loader = torch.utils.data.DataLoader(train_set, batch_size=args.batch_size, shuffle=True, drop_last=True, + num_workers=args.workers, pin_memory=True, persistent_workers=True, + prefetch_factor=args.prefetch) + params = sum(x.numel() for x in policy.parameters()) + print(json.dumps(dict(event='start', policy=args.policy, params=params, + backbone_params=sum(x.numel() for x in backbone), backbone_weights=pcfg.backbone_weights, + backbone_norm=pcfg.backbone_norm, train_frames=len(train_set), steps=args.steps, + from_step=step, cameras=list(cameras), hw=list(hw))), flush=True) + log = (out / 'train_log.jsonl').open('a') + amp = (not args.no_amp) and device.type == 'cuda' + + def save(step): + folder = out / f'checkpoints/step-{step:06d}' + model = ema.model if ema is not None else policy + metrics = offline_eval(model, eval_set, samples, device, contract.action) + save_checkpoint(folder, model, dict(config, step=step, weights='ema' if ema is not None else 'raw', + offline_eval=dict(metrics, samples=len(samples), + frames='val' if val_pos else 'train'))) + final = out / 'checkpoints/final' + if final.is_symlink() or final.exists(): + final.unlink() + final.symlink_to(folder.name) + torch.save(dict(policy=policy.state_dict(), optimizer=optimizer.state_dict(), scheduler=scheduler.state_dict(), + ema=None if ema is None else ema.model.state_dict(), step=step, config=config), + out / 'train_state.pt.tmp') + (out / 'train_state.pt.tmp').replace(state_path) + record = dict(event='save', step=step, checkpoint=str(folder), **metrics) + print(json.dumps(record), flush=True) + log.write(json.dumps(record) + '\n') + log.flush() + policy.train() + + policy.train() + t0, data_s, acc = time.monotonic(), 0.0, {} + while step < args.steps: + tick = time.monotonic() + for images, state, action in loader: + data_s += time.monotonic() - tick + images = images.to(device, non_blocking=True).permute(0, 1, 4, 2, 3).float().div_(255.) + images = random_shift(images, args.shift) + state, action = state.to(device, non_blocking=True), action.to(device, non_blocking=True) + with torch.autocast(device.type, dtype=torch.bfloat16, enabled=amp): + loss, metrics = policy.compute_loss(images, state, action) + optimizer.zero_grad(set_to_none=True) + loss.backward() + optimizer.step() + scheduler.step() + step += 1 + decay = ema.update(policy, step) if ema is not None else 0.0 + for k, v in dict(metrics, loss=float(loss.detach())).items(): + acc[k] = acc.get(k, 0.0) + v + if step % args.log_every == 0: + elapsed = time.monotonic() - t0 + record = dict(event='train', step=step, lr=scheduler.get_last_lr()[0], + backbone_lr=scheduler.get_last_lr()[1], ema_decay=round(decay, 6), + it_s=round(args.log_every / elapsed, 2), data_frac=round(data_s / elapsed, 3), + **{k: v / args.log_every for k, v in acc.items()}) + print(json.dumps(record), flush=True) + log.write(json.dumps(record) + '\n') + log.flush() + t0, data_s, acc = time.monotonic(), 0.0, {} + if step % args.save_every == 0 or step == args.steps: + save(step) + if step >= args.steps: + break + tick = time.monotonic() + log.close() + print(json.dumps(dict(event='done', step=step, checkpoint=str(out / 'checkpoints/final'))), flush=True) + + +if __name__ == '__main__': + main() diff --git a/code/fmdit_continue_20261117/docs/augmentation_v2.md b/code/fmdit_continue_20261117/docs/augmentation_v2.md new file mode 100644 index 0000000000000000000000000000000000000000..14d5c7bcbbea70b299c7a5b090b3f7722d21a57d --- /dev/null +++ b/code/fmdit_continue_20261117/docs/augmentation_v2.md @@ -0,0 +1,105 @@ +# Humanoid Benchmark 数据增强默认规范 v2 + +> 本仓库副本(2026-11-16):从 HumanWAM `hb_training/docs/augmentation_v2.md`(bai 分支,提交 af94cde)原样复制,正文未改;下文「代码对照」表里的路径指 HumanWAM 的 `hb_training/`。本仓库的接入方式和代码位置见文末「本仓库的接入」一节。 + +> 来源:项目规则(CLAUDE.md)里的「数据增强默认规范 v2」。2026-11-09 用户确认定稿;2026-11-10(关节速度)和 2026-11-12(本体感知)两次修订。本文件按原文整理(格式改为 Markdown,删去了与训练排程相关的句子),末尾附代码对照。用户说「数据增强」即指本规范。 + +## 开训前必做 + +任何训练(含短测、续训、微调,任何模型:ImageWAM Δ-flow、GR00T、DiT4DiT、π0.5、Psi-Zero、ACT 等)在启动前,必须对照本规范逐项核对本次运行的配置、数据管线和增强代码。只有用户专门声明的例外才可偏离,例外要写进该实验目录的记录里并在汇报中说明。渲染增强预览视频时同样按此规范。 + +本版取代 2026-11-09 第一版(A3/A7 与 B6 v5t7 Δ-flow 首次 60 轮训练所用版本)。用户看过 1.5 倍单项预览和新增五项增强的预览后确认:从今往后所有训练默认使用本版,除非用户专门声明例外。 + +## 一、执行方式 + +- 在线增强:加载训练样本时动态组合,每次采样独立抽取,同一样本内两路相机与时间维度按下文规则共享或独立。 +- 保留原始图像、轨迹和动作标签,不改动监督值。训练增强从第 1 轮开始。 +- 评估和 rollout 关闭所有随机扰动,保留相同的输入排列及归一化。 +- 每个训练样本先从第二节的 15 项视觉增强中均匀随机选出 1 项,把该项的幅度放大到 1.5 倍,其余 14 项保持 1 倍。「1.5 倍」指扰动量本身变为原来的 1.5 倍(以「不变」为中心放大),不是在原值上再加 1.5 倍;各项的触发概率不变。 +- 各模型原生自带的图像增强(随机裁切、颜色抖动等)一律关闭,统一使用本规范,避免叠加。 + +## 二、视觉(1 倍基准值;括号内为被选中放大时的 1.5 倍值) + +**颜色**——每个样本在两路相机和时间维度共享同一组参数,观测图与未来图像目标都应用: + +1. 曝光 ±0.22 EV(±0.33 EV) +2. 对比度 0.88–1.12(0.82–1.18) +3. 饱和度 0.76–1.24(0.64–1.36) +4. 色相 ±0.015(±0.0225) +5. gamma 0.89–1.11(0.835–1.165) +6. 归一化色温偏移 ±0.066(±0.099) + +**几何**——每路相机独立抽取,观测图与未来图像目标使用对应的同一变换: + +7. 旋转 ±1.5°,并做轻微裁切缩放(额外 0.02)(旋转 ±2.25°,裁切 0.03) + +**传感器与光学**——只用于观测图,不用于未来图像目标: + +8. 高斯噪声:概率 50%,σ = 4/255(6/255) +9. 高斯模糊:概率 15%,3×3 核,σ 0.25–0.65(0.375–0.975) +10. JPEG 压缩:概率 35%,质量 70–95(55–92,按 q → 100 − 1.5 × (100 − q) 换算) +11. 运动模糊:概率 15%,线长 3–7 px(按 224 px 模型输入计;1.5 倍为 4.5–10.5 px),方向在 0–180° 随机,同一样本内固定 +12. 锐化:概率 15%,unsharp mask 强度 0.2–0.6,σ = 1 px(0.3–0.9) +13. 离焦:概率 10%,圆盘模糊半径 1–2 px(1.5–3 px) + +**场景光照与白平衡**——观测图与未来图像目标都应用,同一样本内时间上固定,两路相机各自独立抽取: + +14. 局部光照/阴影:概率 30%,每路相机 1–2 个柔和光斑,增益 0.70–1.30(0.55–1.45),光斑尺寸为画面 15%–35% +15. 白平衡单通道偏移:概率 30%,每路相机随机选一个 RGB 通道,增益 0.90–1.10(0.85–1.15) + +不同模型输入分辨率不是 224 px 时,像素类参数(运动模糊线长、离焦半径)按输入边长与 224 的比例换算。 + +## 三、本体感知(2026-11-12 用户修订:所有模型以 GR00T 在 A20 basket transfer 上的训练为准对齐) + +- 29 个身体关节逐关节采样 ±2° 常偏,再叠加 σ = 0.25° 的零均值高斯噪声,噪声裁剪在 ±3σ。做法和 GR00T 的 hb_launch.py 一致:调用 `hb_aug_v2.perturb_joints`,按它的语义采样常偏。监督动作和实测手部标签保留原值。 +- 2026-11-12 起,所有新训练都去掉第一版里的「15% 概率本体延迟 1–2 个控制周期」和「15% 概率历史丢帧」,因为 GR00T 从来没有做过这两项,所以控制频率(50 Hz / 30 Hz)也不再影响训练。 +- 已训完的 π0.5、Psi-Zero、ACT 训练用了延迟和丢帧,结果保留,差异写进实验记录和论文附录。 + +## 四、关节速度:一律不用(2026-11-10 用户更正,取代第一版和 v2 原文中的速度条款) + +任何模型、任何任务的训练数据和模型输入都绝不能加入关节速度维度,也不计算、不保存速度统计。本体输入只有关节角、实测手部和投影重力: + +- G1(B6、A3、A7 等):身体关节角 29、实测双手 12、投影重力 3,共 44 维。 +- H2(A18 等):75 个关节(身体 31 含头,Sharpa 双手 22+22)和投影重力 3,共 78 维。 + +参照基准是 HF `humanoid-benchmark/deltaflow` 里的 `legacy/b6-pedal-bin/real30-from-sim60-20261108`:`proprio_dim 44`、`action_state_dim 44`、`velocity_channels: []`、数据类 `WholeBodyRobustLeRobotDataset`。 + +ImageWAM Δ-flow 的动作监督仍是 64 维 motion token 加 12 维同帧实测双手(G1 共 76 维;H2 为 75 维)。动作块和历史长度均为 16,未来图像使用 t+16。 + +两层原则(2026-11-10 用户确认): + +- benchmark 接口层所有模型完全一致:每个模型拿到的都是上述 44 维(G1)或 78 维(H2)原始 state,不含速度;相机、动作空间、训练数据、增强、训练长度也一致。 +- 模型内部预处理按各模型官方配方:归一化、离散化、编码都按原生做法。例如 DiT4DiT 官方 G1 配方对关节角做 sin/cos:G1 是 29 个关节做 sin/cos 再加 15 维 min_max,共 73 维;H2 是 75 个关节做 sin/cos 再加重力 3 维,共 153 维。sin/cos 是同样 44/78 个数的确定性变换,不引入新信息,与速度不同。 +- 偏离原生配方之处写进实验记录和论文附录。batch size 不同不要求重训,接口维度不对齐则必须改正。 +- 开训前必须核对:模型实际收到的原始本体是 44/78 维且不含速度(要实际构造模型或取一个 batch 检查,不能只看配置);编码后的维度符合该模型的原生配方。 + +## 五、适用范围 + +本版在现有数据上进行,跳过起步姿势库和仿真渲染层的灯光、相机安装角与镜头内参随机化;二维旋转、裁切、颜色、光照斑块和白平衡属于上述在线视觉增强。适用于所有模型和任务。 + +## 代码对照(路径相对 `hb_training/`) + +| 规范条目 | 实现 | +|---|---| +| 二、15 项视觉增强,每样本随机一项放大 1.5 倍 | `hb_groot/hb_aug_v2.py`:`sample()` 抽参数,`apply()` 执行,入口 `augment(x, rng, obs_mask=None)`,`x` 为 `[T, V, 3, H, W]`、取值 0–1;版本号 `VERSION = "hb-aug-v2-20261109"`。`hb_pi05/hb_aug_v2.py` 是同一文件的副本(π0.5 环境用)。 | +| 颜色共享;几何、光斑、白平衡按相机独立;传感器与光学类只用于观测图 | 同上:颜色参数每样本一组;几何、光斑、白平衡按相机逐路抽取、时间上固定;`obs_mask` 为 False 的帧(未来图像目标)不加传感器与光学类效果。 | +| 像素类参数按输入边长换算 | 同上:以 224 px 为基准,按 `H / 224` 缩放。 | +| 三、本体常偏 ±2° + σ 0.25° 噪声,无延迟、无丢帧 | `hb_aug_v2.perturb_joints`(GR00T 由 `hb_groot/hb_launch.py` 调用)。Psi-Zero 和 ACT 在 `hb_psi0/hb_psi0_config.py` 里按相同常数和语义实现(`BIAS_DEG 2.0`、`NOISE_DEG 0.25`),并在启动时核对 `hb_aug_v2` 的版本号;延迟概率参数 `hb-delay-prob`,ACT 固定为 0。 | +| 四、本体不含速度(G1 44 / H2 78) | Psi-Zero / ACT 的数据视图由 `hb_psi0/prepare_data.py` 生成(`observation.hb44` / `observation.hb78`);`hb_act/hb_act_config.py` 断言 state 只能是 44 或 78 维。 | +| 一、关闭原生图像增强 | ACT:`hb_act/hb_act_config.py` 断言原生 ColorJitter(`img_aug`)关闭,在原生 resize 与中心裁剪到 224×224 之后,对所有相机一起调用 `hb_aug_v2.augment`。 | +| 一、评估和 rollout 关闭随机扰动 | 推理路径不调用增强;开环评估适配器见 `eval-action-mse/adapters/`,协议见 `eval-action-mse/README.md`。 | +| 各模型接入点 | GR00T `hb_groot/hb_launch.py`;π0.5 `hb_pi05/hb_g1_data.py`;Psi-Zero `hb_psi0/hb_psi0_config.py`;ACT `hb_act/hb_act_config.py`;Delta Flow `hb_deltaflow/wholebody_hbaug.py`、`hb_deltaflow/h2_hbaug.py`;DiT4DiT `hb_dit4dit/hb_d4d_data.py`。 | + +## 本仓库的接入(Humanoid_benchmark,2026-11-16) + +| 内容 | 位置 | +|---|---| +| 参考实现(原样副本,git blob 75542085,与 HumanWAM `hb_groot/hb_aug_v2.py` 逐字节相同) | `evaluation/augmentation/hb_aug_v2.py` | +| 通用训练入口:导入时断言 `VERSION == "hb-aug-v2-20261109"`;`worker_rng()`(每个进程一个 `np.random.Generator`,pid + os.urandom 播种);`augment_images(x[T,V,3,H,W], obs_mask)`;`augment_frames_uint8(frames[V,H,W,3])`;`perturb_body(state, body_dims)`;`check_state_layout(robot, state_dim, columns)`(G1 44 / H2 78、无速度列);`describe()`(写进 run.json 的记录) | `evaluation/augmentation/__init__.py` | +| `sample()` 统计审计(被放大项是否均匀、各项 1 倍/1.5 倍范围、触发概率) | `evaluation/augmentation/check_spec.py` | +| 增强预览(缓存帧与增强帧并排,电脑版 + 手机版视频) | `evaluation/augmentation/preview.py` | +| FM-DiT、DP、ACT:训练样本在 168x224 缓存帧(INTER_AREA 确定性缩放)之后、模型归一化之前增强;本体只扰动 G1 前 29 维 / H2 前 31 维(含头);原生 random shift 关闭;开训前 preflight | `evaluation/baselines/data.py`(`ChunkDataset(body_dims=...)`)、`evaluation/baselines/train.py`(`--augmentation hb-v2` 默认、`preflight()`) | +| CPU 测试 | `tests/framework/test_augmentation.py` | +| 验收记录与偏离说明 | `debug/reports/augmentation_v2_baselines/README.md` | + +以后新加模型的训练入口按 `evaluation/augmentation/__init__.py` 文档字符串里的步骤接入:放在模型原生的确定性 resize/中心裁剪之后、归一化之前;一个样本的所有相机、所有帧一起调用一次 `augment_images`(有未来图像目标时用 `obs_mask` 标出观测帧);本体用 `perturb_body` 只扰动身体关节、在原始弧度上、归一化之前;关闭模型原生图像增强;评估和推理路径不调用增强;开训前实际取 batch 核对(可参照 `evaluation/baselines/train.py` 的 `preflight()`)。 diff --git a/code/fmdit_continue_20261117/resume.py b/code/fmdit_continue_20261117/resume.py new file mode 100644 index 0000000000000000000000000000000000000000..ee20c6cf643ddc0089f477580d595be7bc8e2748 --- /dev/null +++ b/code/fmdit_continue_20261117/resume.py @@ -0,0 +1,58 @@ +"""Explicitly resume one packaged run in a fresh output directory.""" +import argparse +import json +from pathlib import Path +import shlex +import shutil +import subprocess +import sys + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--task', choices=['b5', 'b8', 'a20'], required=True) + parser.add_argument('--dataset', type=Path, required=True) + parser.add_argument('--out', type=Path, required=True) + parser.add_argument('--cache-dir', type=Path, required=True) + parser.add_argument('--steps', type=int, default=20000) + parser.add_argument('--workers', type=int, default=32) + parser.add_argument('--device', default='cuda:0') + parser.add_argument('--dry-run', action='store_true') + args = parser.parse_args() + code = Path(__file__).resolve().parent + repo = code.parents[1] + paths = { + 'b5': 'b5-chair-push-standing-start/b512-cont-20261117', + 'b8': 'b8-table-wipe/b512-cont-20261117', + 'a20': 'a20-basket-transfer/b512-trimv2-cont-20261117', + } + package = repo / paths[args.task] + metadata = json.loads((package / 'resume.json').read_text()) + if args.steps <= metadata['saved_step']: + parser.error('--steps must exceed saved step 6000') + dataset, out = args.dataset.expanduser().resolve(), args.out.expanduser().resolve() + state = package / 'train_state.pt' + if not state.is_file(): + parser.error(f'Missing resume state: {state}') + if not (dataset / 'meta/info.json').is_file(): + parser.error(f'Not a LeRobot dataset: {dataset}') + if out.exists(): + parser.error('Choose a fresh output directory; existing runs are preserved') + argv = [sys.executable, '-I', str(code / 'baselines/train.py'), + '--policy', 'fm_dit', '--contract', 'g1_sonic_dfx6', + '--dataset', str(dataset), '--out', str(out), '--resume', + '--cache-dir', str(args.cache_dir.expanduser().resolve()), + '--steps', str(args.steps), '--batch-size', '512', '--warmup', '500', + '--workers', str(args.workers), '--save-every', '2000', + '--log-every', '25', '--device', args.device] + print(shlex.join(argv), flush=True) + if args.dry_run: + return + out.mkdir(parents=True, exist_ok=False) + shutil.copy2(state, out / 'train_state.pt') + shutil.copy2(package / 'resume.json', out / 'resume_source.json') + raise SystemExit(subprocess.call(argv)) + + +if __name__ == '__main__': + main()