Index three completed fresh-data PI0.5 peaks with all eval50 reports and zero-success tie provenance

README.md

@@ -6,13 +6,15 @@ license: mit
 
 ## single_task
 
-One PI0.5 LoRA checkpoint per ManiLadder task (103 tasks), each finetuned on 20 demos.
+One PI0.5 LoRA checkpoint per ManiLadder task (106 tasks), each finetuned on 20 demos.
 Checkpoints are selected by the highest **50-episode** success rate; ties use the earliest step.
 
 - The 60 previously published tasks use fixed evaluation seeds 1000–1049.
 - The 22 completed retraining runs published through 2026-10-30 use seeds 800000–800049. All reached 40,000 training steps. 15 have all eight 5k–40k evaluations; 7 have seven evaluations because one intermediate checkpoint was not retained. Their selected weights are the best among the available evaluated checkpoints.
 - The 21 tasks from the HF-data retraining campaign `pi05_hf21_20261025` (added 2026-10-29; raw demos freshly downloaded from Linslab/ManiLadder_Dataset) use seeds 1000–1049. All reached 40,000 training steps and have all eight 5k–40k evaluations. See [its manifest](single_task/pi05_hf21_20261025_manifest.json).
 
+- The 3 fresh-data runs in `pi05_newdata_3tasks_20261028` use pinned dataset revision `d120457e637b01679fbb40fd4de1d2394f0d55c9`, 20 demos per task, 40,000 training steps, and all eight 5k–40k eval50 checkpoints with seeds 800000–800049. **All three scored 0/50 at every checkpoint**; step 5,000 is the earliest tied checkpoint. See [their manifest](single_task/pi05_newdata_3tasks_20261028_manifest.json).
+
 See [retraining provenance and missing steps](single_task/pi05_retrain_eval50_20261022_manifest.json). These scores use different seed sets across the two groups and are not a controlled comparison.
 
 Each `single_task/<task>/` folder contains:
@@ -22,7 +24,7 @@ Each `single_task/<task>/` folder contains:
 - `eval_result.json` — 50-episode evaluation of this checkpoint
 
 `single_task/results.csv` lists the selected checkpoint per task; `single_task/all_checkpoints_eval.csv` has the
-50-episode result of every evaluated checkpoint (814 total).
+50-episode result of every evaluated checkpoint (838 total).
 
 Evaluation: `exec_steps=16`, `sample_steps=10`. Tasks whose checkpoints all scored 0/50 are still included;
 for them the step choice is arbitrary.
@@ -68,6 +70,7 @@ for them the step choice is arbitrary.
 | DualArmOpenLaptop-v1 | 2 | 25000 | 0.42 |
 | DualArmPegInHole-v1 | 2 | 5000 | 0.18 |
 | DualArmPlaceRopeOverRod-v1 | 2 | 40000 | 0.46 |
+| DualArmPouringTwoCups-v1 | 2 | 5000 | 0.00 |
 | DualArmPushObject-v1 | 2 | 25000 | 0.06 |
 | DualArmStraightenRope-v1 | 2 | 35000 | 0.48 |
 | DualArmStuffCloth-v1 | 2 | 15000 | 0.40 |
@@ -85,7 +88,9 @@ for them the step choice is arbitrary.
 | ScoopWater-v1 | 2 | 15000 | 0.44 |
 | StraightenTwoRopesSequential-v1 | 2 | 15000 | 0.36 |
 | ClutterGrasp-v1 | 3 | 15000 | 0.16 |
+| DualArmAssembleBracket-v1 | 3 | 5000 | 0.00 |
 | DualArmFoldCarton-v1 | 3 | 25000 | 0.14 |
+| DualArmLayerClayInDish-v1 | 3 | 5000 | 0.00 |
 | DualArmMakeLine-v1 | 3 | 5000 | 0.00 |
 | DualArmOpenFridge-v1 | 3 | 5000 | 0.00 |
 | DualArmOrganizeCabinet-v1 | 3 | 5000 | 0.00 |

single_task/all_checkpoints_eval.csv

@@ -307,6 +307,14 @@ DualArmPlaceRopeOverRod-v1,level_2,25000,0.28,14,324.9540537391882
 DualArmPlaceRopeOverRod-v1,level_2,30000,0.38,19,296.7787297666073
 DualArmPlaceRopeOverRod-v1,level_2,35000,0.08,4,298.22417598265224
 DualArmPlaceRopeOverRod-v1,level_2,40000,0.46,23,289.2976510642795
+DualArmPouringTwoCups-v1,level_2,5000,0.0,0,2313.0995259738716
+DualArmPouringTwoCups-v1,level_2,10000,0.0,0,2159.899790851027
+DualArmPouringTwoCups-v1,level_2,15000,0.0,0,1917.7337250904739
+DualArmPouringTwoCups-v1,level_2,20000,0.0,0,2533.5854001205757
+DualArmPouringTwoCups-v1,level_2,25000,0.0,0,2307.9573923527146
+DualArmPouringTwoCups-v1,level_2,30000,0.0,0,2610.355446477852
+DualArmPouringTwoCups-v1,level_2,35000,0.0,0,2287.075251869607
+DualArmPouringTwoCups-v1,level_2,40000,0.0,0,2507.2310234421384
 DualArmPushObject-v1,level_2,5000,0.0,0,0.7291
 DualArmPushObject-v1,level_2,10000,0.0,0,1.45198
 DualArmPushObject-v1,level_2,15000,0.0,0,3.1949400000000003
@@ -442,6 +450,14 @@ ClutterGrasp-v1,level_3,25000,0.12,6,nan
 ClutterGrasp-v1,level_3,30000,0.16,8,nan
 ClutterGrasp-v1,level_3,35000,0.14,7,nan
 ClutterGrasp-v1,level_3,40000,0.16,8,nan
+DualArmAssembleBracket-v1,level_3,5000,0.0,0,404.7625858747959
+DualArmAssembleBracket-v1,level_3,10000,0.0,0,610.9061906814575
+DualArmAssembleBracket-v1,level_3,15000,0.0,0,863.3327129042149
+DualArmAssembleBracket-v1,level_3,20000,0.0,0,582.8615937566757
+DualArmAssembleBracket-v1,level_3,25000,0.0,0,1010.4638057649136
+DualArmAssembleBracket-v1,level_3,30000,0.0,0,978.0950652432442
+DualArmAssembleBracket-v1,level_3,35000,0.0,0,771.3933880138397
+DualArmAssembleBracket-v1,level_3,40000,0.0,0,945.0748059856892
 DualArmFoldCarton-v1,level_3,5000,0.0,0,6345.062264726162
 DualArmFoldCarton-v1,level_3,10000,0.02,1,8489.919916040897
 DualArmFoldCarton-v1,level_3,15000,0.04,2,9307.387320964337
@@ -450,6 +466,14 @@ DualArmFoldCarton-v1,level_3,25000,0.14,7,10516.428281509876
 DualArmFoldCarton-v1,level_3,30000,0.06,3,9551.720534439088
 DualArmFoldCarton-v1,level_3,35000,0.1,5,10555.199788849353
 DualArmFoldCarton-v1,level_3,40000,0.04,2,9328.819932291508
+DualArmLayerClayInDish-v1,level_3,5000,0.0,0,421.427312732935
+DualArmLayerClayInDish-v1,level_3,10000,0.0,0,419.9999928474426
+DualArmLayerClayInDish-v1,level_3,15000,0.0,0,436.26779628276824
+DualArmLayerClayInDish-v1,level_3,20000,0.0,0,419.9999928474426
+DualArmLayerClayInDish-v1,level_3,25000,0.0,0,436.08672299027444
+DualArmLayerClayInDish-v1,level_3,30000,0.0,0,426.5973575603962
+DualArmLayerClayInDish-v1,level_3,35000,0.0,0,493.774410200119
+DualArmLayerClayInDish-v1,level_3,40000,0.0,0,477.8500285220146
 DualArmMakeLine-v1,level_3,5000,0.0,0,11880.993419999999
 DualArmMakeLine-v1,level_3,10000,0.0,0,12168.136680000001
 DualArmMakeLine-v1,level_3,15000,0.0,0,12659.39848

single_task/pi05_newdata_3tasks_20261028_manifest.json

@@ -0,0 +1,57 @@
+{
+  "campaign": "pi05_newdata_3tasks_20261028",
+  "episodes_per_eval": 50,
+  "episode_seed_base": 800000,
+  "tasks": [
+    {
+      "task": "DualArmPouringTwoCups-v1",
+      "level": "level_2",
+      "suite": "deformable/dual_arm_dex",
+      "step": 5000,
+      "success_rate": 0.0,
+      "successes": 0,
+      "episodes": 50,
+      "episode_seed_base": 800000,
+      "ckpts_evaluated": 8,
+      "missing_eval_steps": [],
+      "campaign": "pi05_newdata_3tasks_20261028",
+      "checkpoint": "single_task/DualArmPouringTwoCups-v1/lora_step_0005000.npz",
+      "checkpoint_sha256": "e6f0c091d86fa5b989128fc20867d33a488c2fc3893c7adc4cba979676bec741",
+      "dataset_revision": "d120457e637b01679fbb40fd4de1d2394f0d55c9"
+    },
+    {
+      "task": "DualArmLayerClayInDish-v1",
+      "level": "level_3",
+      "suite": "deformable/dual_arm_dex",
+      "step": 5000,
+      "success_rate": 0.0,
+      "successes": 0,
+      "episodes": 50,
+      "episode_seed_base": 800000,
+      "ckpts_evaluated": 8,
+      "missing_eval_steps": [],
+      "campaign": "pi05_newdata_3tasks_20261028",
+      "checkpoint": "single_task/DualArmLayerClayInDish-v1/lora_step_0005000.npz",
+      "checkpoint_sha256": "60b9ea56c7eb9eb51ea8623541f5f4e70f6435d2ecde5fa9ce99ae58d0b611a9",
+      "dataset_revision": "d120457e637b01679fbb40fd4de1d2394f0d55c9"
+    },
+    {
+      "task": "DualArmAssembleBracket-v1",
+      "level": "level_3",
+      "suite": "rigid_body/dual_arm_gripper",
+      "step": 5000,
+      "success_rate": 0.0,
+      "successes": 0,
+      "episodes": 50,
+      "episode_seed_base": 800000,
+      "ckpts_evaluated": 8,
+      "missing_eval_steps": [],
+      "campaign": "pi05_newdata_3tasks_20261028",
+      "checkpoint": "single_task/DualArmAssembleBracket-v1/lora_step_0005000.npz",
+      "checkpoint_sha256": "162f72d171b4fa6d9511056d713dd0724de51816955f256465cb4e3a83a3761a",
+      "dataset_revision": "d120457e637b01679fbb40fd4de1d2394f0d55c9"
+    }
+  ],
+  "selection": "Highest eval50 success; earliest checkpoint breaks ties. All three tasks scored zero at all eight checkpoints.",
+  "evaluation_note": "TwoCups 20k\u201340k reports combine serial prefixes and validated five-episode blocks. Original prefix reward totals have three-decimal precision; successes and seeds are exact."
+}

single_task/results.csv

@@ -38,6 +38,7 @@ DualArmOpenBottle-v1,level_2,30000,0.06,3,8,single_task/DualArmOpenBottle-v1/lor
 DualArmOpenLaptop-v1,level_2,25000,0.42,21,8,single_task/DualArmOpenLaptop-v1/lora_step_0025000.npz
 DualArmPegInHole-v1,level_2,5000,0.18,9,8,single_task/DualArmPegInHole-v1/lora_step_0005000.npz
 DualArmPlaceRopeOverRod-v1,level_2,40000,0.46,23,8,single_task/DualArmPlaceRopeOverRod-v1/lora_step_0040000.npz
+DualArmPouringTwoCups-v1,level_2,5000,0.0,0,8,single_task/DualArmPouringTwoCups-v1/lora_step_0005000.npz
 DualArmPushObject-v1,level_2,25000,0.06,3,8,single_task/DualArmPushObject-v1/lora_step_0025000.npz
 DualArmStraightenRope-v1,level_2,35000,0.48,24,7,single_task/DualArmStraightenRope-v1/lora_step_0035000.npz
 DualArmStuffCloth-v1,level_2,15000,0.4,20,8,single_task/DualArmStuffCloth-v1/lora_step_0015000.npz
@@ -55,7 +56,9 @@ RopeBridgeDex-v1,level_2,30000,0.78,39,8,single_task/RopeBridgeDex-v1/lora_step_
 ScoopWater-v1,level_2,15000,0.44,22,8,single_task/ScoopWater-v1/lora_step_0015000.npz
 StraightenTwoRopesSequential-v1,level_2,15000,0.36,18,8,single_task/StraightenTwoRopesSequential-v1/lora_step_0015000.npz
 ClutterGrasp-v1,level_3,15000,0.16,8,8,single_task/ClutterGrasp-v1/lora_step_0015000.npz
+DualArmAssembleBracket-v1,level_3,5000,0.0,0,8,single_task/DualArmAssembleBracket-v1/lora_step_0005000.npz
 DualArmFoldCarton-v1,level_3,25000,0.14,7,8,single_task/DualArmFoldCarton-v1/lora_step_0025000.npz
+DualArmLayerClayInDish-v1,level_3,5000,0.0,0,8,single_task/DualArmLayerClayInDish-v1/lora_step_0005000.npz
 DualArmMakeLine-v1,level_3,5000,0.0,0,8,single_task/DualArmMakeLine-v1/lora_step_0005000.npz
 DualArmOpenFridge-v1,level_3,5000,0.0,0,8,single_task/DualArmOpenFridge-v1/lora_step_0005000.npz
 DualArmOrganizeCabinet-v1,level_3,5000,0.0,0,8,single_task/DualArmOrganizeCabinet-v1/lora_step_0005000.npz