Auto-sync: 2026-06-27 10:19:52 (part 2)

results/dovla_cil_run_report_2026-06-27.md

@@ -0,0 +1,57 @@
+# DoVLA-CIL Run Report - 2026-06-27
+
+## Executive Result
+
+The deployed h=16 behavior-cloning policy remains weak, but the interventional field becomes strong when used on the action lattice it was trained to score.
+
+| Method | Mean success | Gain vs h=16 policy | Notes |
+|---|---:|---:|---|
+| h=16 policy, rank checkpoint | 29.74% | -- | Previous online rollout baseline |
+| h=16 policy, best-policy checkpoint | 27.01% | -2.72 pp | Lower val BC did not improve rollout |
+| Gaussian field search | 29.10% | -0.64 pp | Off-manifold candidates hurt |
+| Lattice field selection, no expert candidates | 56.99% | +27.25 pp | Conservative, reviewer-safe result |
+| Lattice field selection, full lattice | 69.33% | +39.59 pp | Includes expert proposal in candidate set |
+
+Canonical summaries:
+
+- `results/h16_field_sweep_summary.md`
+- `results/h16_lattice_summary.md`
+- `results/h16_lattice_no_expert_summary.md`
+- `results/h16_policy_ckpt_summary.md`
+
+## Jobs Submitted
+
+| Job | Purpose | Status |
+|---|---|---|
+| 14827325 | Gaussian field-guided h=16 rollout sweep | Completed |
+| 14827529 | Summarize Gaussian field sweep | Completed |
+| 14827496 | Full lattice-selected rollout | Completed |
+| 14829910 | Regenerate full lattice summary | Completed |
+| 14828516 | Lattice-selected rollout excluding expert candidates | Completed |
+| 14829912 | Regenerate no-expert lattice summary | Completed |
+| 14827420 | Retrain h=16 with `best_policy.pt` checkpoint saving | Completed |
+| 14827421 | Evaluate `best_policy.pt` | Completed |
+| 14827422 | Summarize `best_policy.pt` rollout | Completed |
+
+## Interpretation
+
+The central publishable finding is not "longer horizon fixes BC." It does not.
+
+The better story is:
+
+> Counterfactual intervention lattices expose a learnable local utility field. Direct action-generation BC cannot exploit the field, and Gaussian test-time search is off-manifold. But when the field is queried on the same kind of counterfactual action lattice it was trained on, it becomes an effective decision rule, reaching 56.99% success without expert candidates and 69.33% with the full lattice, against a 29.74% policy baseline and an 86.78% oracle ceiling.
+
+This supports a clean novelty claim: DoVLA-CIL is not just more demonstrations or a larger VLA. It is a counterfactual data engine plus a path-independent action-utility field, and the field must be deployed on intervention-lattice proposals rather than arbitrary action noise.
+
+## Reviewer-Safe Claims
+
+- Strong conservative result: no-expert lattice selection improves success from 29.74% to 56.99%.
+- Strong upper result: full lattice selection improves success to 69.33%, but must be labeled as including expert proposals.
+- Negative but useful ablation: Gaussian field search fails, showing the field is not a generic black-box action optimizer off the data manifold.
+- Negative checkpoint ablation: selecting checkpoint by lower BC validation loss does not improve online rollout.
+
+## Next Best Experiments
+
+1. Generate an on-the-fly non-expert proposal lattice that does not depend on stored validation candidates, then evaluate with `selection_mode=lattice`.
+2. Train a proposal model to imitate successful non-expert lattice candidates so deployment does not require an explicit lattice store.
+3. Add final paper tables separating `policy`, `field_gaussian`, `field_lattice_no_expert`, `field_lattice_full`, and `oracle`.

results/h16_lattice_no_expert_summary.json

@@ -0,0 +1,227 @@
+{
+  "run_root": "/scratch/knguy52/dovla/experiments/dovla_h16_rollout_runs",
+  "out_name": "lattice_no_expert_rollout.json",
+  "num_completed": 3,
+  "baseline_h4_policy_success": 0.2967,
+  "baseline_h16_policy_success": 0.29739130434782607,
+  "mean_success": 0.5698550724637681,
+  "std_success": 0.046188021535170105,
+  "mean_progress": 0.7501174603224449,
+  "mean_oracle_success": 0.8678260869565217,
+  "mean_action_mse_to_best": 0.4590102900127523,
+  "gain_vs_h4": 0.2731550724637681,
+  "gain_vs_h16_policy": 0.27246376811594203,
+  "selected_candidate_type_counts": {
+    "lattice_near_miss": 1263,
+    "lattice_no_op": 222,
+    "lattice_random_negative": 144,
+    "lattice_wrong_direction": 34,
+    "lattice_wrong_gripper": 62
+  },
+  "rows": [
+    {
+      "seed": 0,
+      "path": "/scratch/knguy52/dovla/experiments/dovla_h16_rollout_runs/seed_0/lattice_no_expert_rollout.json",
+      "selection_mode": "lattice",
+      "num_candidates": 16,
+      "num_groups": 575,
+      "policy_rollout_success_rate": 0.5165217391304348,
+      "policy_rollout_progress": 0.7308024795159049,
+      "oracle_success_rate": 0.8573913043478261,
+      "action_mse_to_best": 0.4291479373977864,
+      "per_task": {
+        "LiftPegUpright-v1": {
+          "action_mse_to_best": 0.2888141949958115,
+          "expert_success_rate": 0.8865979381443299,
+          "num_groups": 97,
+          "oracle_success_rate": 0.9278350515463918,
+          "policy_expert_regret": 0.45705366580142187,
+          "policy_oracle_regret": 0.47029784183526774,
+          "policy_rollout_progress": 0.8553990655645882,
+          "policy_rollout_success_rate": 0.5773195876288659,
+          "restore_max_error": 4.76837158203125e-07
+        },
+        "PickCube-v1": {
+          "action_mse_to_best": 0.4340394471700375,
+          "expert_success_rate": 0.9375,
+          "num_groups": 208,
+          "oracle_success_rate": 0.9471153846153846,
+          "policy_expert_regret": 0.6624748194769311,
+          "policy_oracle_regret": 0.6691413935309706,
+          "policy_rollout_progress": 0.7513381671697761,
+          "policy_rollout_success_rate": 0.5048076923076923,
+          "restore_max_error": 4.76837158203125e-07
+        },
+        "PullCube-v1": {
+          "action_mse_to_best": 0.5183976920587676,
+          "expert_success_rate": 0.19480519480519481,
+          "num_groups": 77,
+          "oracle_success_rate": 0.36363636363636365,
+          "policy_expert_regret": 0.1787171581058533,
+          "policy_oracle_regret": 0.37732240485100005,
+          "policy_rollout_progress": 0.3172257935942768,
+          "policy_rollout_success_rate": 0.16883116883116883,
+          "restore_max_error": 4.76837158203125e-07
+        },
+        "PushCube-v1": {
+          "action_mse_to_best": 0.39197504613300826,
+          "expert_success_rate": 0.8617021276595744,
+          "num_groups": 94,
+          "oracle_success_rate": 0.9893617021276596,
+          "policy_expert_regret": 0.2812512713543912,
+          "policy_oracle_regret": 0.3391457902307206,
+          "policy_rollout_progress": 0.8417052735990667,
+          "policy_rollout_success_rate": 0.8191489361702128,
+          "restore_max_error": 4.76837158203125e-07
+        },
+        "StackCube-v1": {
+          "action_mse_to_best": 0.5222485399266441,
+          "expert_success_rate": 0.6767676767676768,
+          "num_groups": 99,
+          "oracle_success_rate": 0.8585858585858586,
+          "policy_expert_regret": 0.5384464679342328,
+          "policy_oracle_regret": 0.574341721606977,
+          "policy_rollout_progress": 0.7819464218736899,
+          "policy_rollout_success_rate": 0.46464646464646464,
+          "restore_max_error": 3.948807716369629e-07
+        }
+      }
+    },
+    {
+      "seed": 1,
+      "path": "/scratch/knguy52/dovla/experiments/dovla_h16_rollout_runs/seed_1/lattice_no_expert_rollout.json",
+      "selection_mode": "lattice",
+      "num_candidates": 16,
+      "num_groups": 575,
+      "policy_rollout_success_rate": 0.5965217391304348,
+      "policy_rollout_progress": 0.756582982868928,
+      "oracle_success_rate": 0.8695652173913043,
+      "action_mse_to_best": 0.44271026729079693,
+      "per_task": {
+        "LiftPegUpright-v1": {
+          "action_mse_to_best": 0.3169037180301626,
+          "expert_success_rate": 0.8584070796460177,
+          "num_groups": 113,
+          "oracle_success_rate": 0.9380530973451328,
+          "policy_expert_regret": 0.39746501211044005,
+          "policy_oracle_regret": 0.43380374942733124,
+          "policy_rollout_progress": 0.858787138245802,
+          "policy_rollout_success_rate": 0.6283185840707964,
+          "restore_max_error": 4.76837158203125e-07
+        },
+        "PickCube-v1": {
+          "action_mse_to_best": 0.4862241648218554,
+          "expert_success_rate": 0.9402173913043478,
+          "num_groups": 184,
+          "oracle_success_rate": 0.9456521739130435,
+          "policy_expert_regret": 0.4408608768540232,
+          "policy_oracle_regret": 0.44088488132895337,
+          "policy_rollout_progress": 0.8249527744381971,
+          "policy_rollout_success_rate": 0.657608695652174,
+          "restore_max_error": 4.76837158203125e-07
+        },
+        "PullCube-v1": {
+          "action_mse_to_best": 0.5220103718977618,
+          "expert_success_rate": 0.25,
+          "num_groups": 76,
+          "oracle_success_rate": 0.40789473684210525,
+          "policy_expert_regret": 0.2234261535524734,
+          "policy_oracle_regret": 0.44941299718706623,
+          "policy_rollout_progress": 0.32774039636071967,
+          "policy_rollout_success_rate": 0.17105263157894737,
+          "restore_max_error": 4.76837158203125e-07
+        },
+        "PushCube-v1": {
+          "action_mse_to_best": 0.35074781935225735,
+          "expert_success_rate": 0.8198198198198198,
+          "num_groups": 111,
+          "oracle_success_rate": 1.0,
+          "policy_expert_regret": 0.24994636481409674,
+          "policy_oracle_regret": 0.3667086133280316,
+          "policy_rollout_progress": 0.8314895848701666,
+          "policy_rollout_success_rate": 0.8018018018018018,
+          "restore_max_error": 4.76837158203125e-07
+        },
+        "StackCube-v1": {
+          "action_mse_to_best": 0.5568927584093187,
+          "expert_success_rate": 0.6923076923076923,
+          "num_groups": 91,
+          "oracle_success_rate": 0.8571428571428571,
+          "policy_expert_regret": 0.47163032846791403,
+          "policy_oracle_regret": 0.5352666910205569,
+          "policy_rollout_progress": 0.7582125712882032,
+          "policy_rollout_success_rate": 0.5384615384615384,
+          "restore_max_error": 3.948807716369629e-07
+        }
+      }
+    },
+    {
+      "seed": 2,
+      "path": "/scratch/knguy52/dovla/experiments/dovla_h16_rollout_runs/seed_2/lattice_no_expert_rollout.json",
+      "selection_mode": "lattice",
+      "num_candidates": 16,
+      "num_groups": 575,
+      "policy_rollout_success_rate": 0.5965217391304348,
+      "policy_rollout_progress": 0.7629669185825017,
+      "oracle_success_rate": 0.8765217391304347,
+      "action_mse_to_best": 0.5051726653496735,
+      "per_task": {
+        "LiftPegUpright-v1": {
+          "action_mse_to_best": 0.3387383111627666,
+          "expert_success_rate": 0.8229166666666666,
+          "num_groups": 96,
+          "oracle_success_rate": 0.9270833333333334,
+          "policy_expert_regret": 0.349092789615194,
+          "policy_oracle_regret": 0.38970588706433773,
+          "policy_rollout_progress": 0.8847468371192614,
+          "policy_rollout_success_rate": 0.6458333333333334,
+          "restore_max_error": 3.5762786865234375e-07
+        },
+        "PickCube-v1": {
+          "action_mse_to_best": 0.5211711752068515,
+          "expert_success_rate": 0.9444444444444444,
+          "num_groups": 198,
+          "oracle_success_rate": 0.9595959595959596,
+          "policy_expert_regret": 0.38791667779136185,
+          "policy_oracle_regret": 0.38796894105546403,
+          "policy_rollout_progress": 0.8505876985086938,
+          "policy_rollout_success_rate": 0.702020202020202,
+          "restore_max_error": 4.76837158203125e-07
+        },
+        "PullCube-v1": {
+          "action_mse_to_best": 0.682747141416702,
+          "expert_success_rate": 0.24444444444444444,
+          "num_groups": 90,
+          "oracle_success_rate": 0.4666666666666667,
+          "policy_expert_regret": 0.2159867504404651,
+          "policy_oracle_regret": 0.4278881938921081,
+          "policy_rollout_progress": 0.38586775238315263,
+          "policy_rollout_success_rate": 0.25555555555555554,
+          "restore_max_error": 4.0978193283081055e-07
+        },
+        "PushCube-v1": {
+          "action_mse_to_best": 0.3426748497024326,
+          "expert_success_rate": 0.8514851485148515,
+          "num_groups": 101,
+          "oracle_success_rate": 1.0,
+          "policy_expert_regret": 0.33127955694009764,
+          "policy_oracle_regret": 0.40476624254543003,
+          "policy_rollout_progress": 0.8130555396327878,
+          "policy_rollout_success_rate": 0.7821782178217822,
+          "restore_max_error": 4.76837158203125e-07
+        },
+        "StackCube-v1": {
+          "action_mse_to_best": 0.6522901051781244,
+          "expert_success_rate": 0.7666666666666667,
+          "num_groups": 90,
+          "oracle_success_rate": 0.9111111111111111,
+          "policy_expert_regret": 0.6501921941836675,
+          "policy_oracle_regret": 0.6890770451890098,
+          "policy_rollout_progress": 0.7611912255485852,
+          "policy_rollout_success_rate": 0.4444444444444444,
+          "restore_max_error": 4.76837158203125e-07
+        }
+      }
+    }
+  ]
+}

results/h16_lattice_no_expert_summary.md

@@ -0,0 +1,24 @@
+# h=16 Lattice-Selected Rollout
+
+Run root: `/scratch/knguy52/dovla/experiments/dovla_h16_rollout_runs`
+Completed seeds: 3
+Baseline h=4 policy success: 29.67%
+Baseline h=16 policy success: 29.74%
+
+Mean success: 56.99% +/- 4.62%
+Gain vs h=16 policy: +27.25%
+Mean oracle success: 86.78%
+Mean progress: 75.01%
+
+| seed | success | progress | oracle | candidates | action MSE |
+|---:|---:|---:|---:|---:|---:|
+| 0 | 51.65% | 73.08% | 85.74% | 16 | 0.429 |
+| 1 | 59.65% | 75.66% | 86.96% | 16 | 0.443 |
+| 2 | 59.65% | 76.30% | 87.65% | 16 | 0.505 |
+
+Selected candidate types:
+- lattice_near_miss: 1263
+- lattice_no_op: 222
+- lattice_random_negative: 144
+- lattice_wrong_direction: 34
+- lattice_wrong_gripper: 62

results/h16_lattice_summary.json

@@ -0,0 +1,228 @@
+{
+  "run_root": "/scratch/knguy52/dovla/experiments/dovla_h16_rollout_runs",
+  "out_name": "lattice_rollout.json",
+  "num_completed": 3,
+  "baseline_h4_policy_success": 0.2967,
+  "baseline_h16_policy_success": 0.29739130434782607,
+  "mean_success": 0.6933333333333334,
+  "std_success": 0.035655708555783816,
+  "mean_progress": 0.8108624100247803,
+  "mean_oracle_success": 0.8678260869565217,
+  "mean_action_mse_to_best": 0.43754452051773696,
+  "gain_vs_h4": 0.39663333333333334,
+  "gain_vs_h16_policy": 0.3959420289855073,
+  "selected_candidate_type_counts": {
+    "lattice_expert": 977,
+    "lattice_near_miss": 348,
+    "lattice_no_op": 177,
+    "lattice_random_negative": 138,
+    "lattice_wrong_direction": 30,
+    "lattice_wrong_gripper": 55
+  },
+  "rows": [
+    {
+      "seed": 0,
+      "path": "/scratch/knguy52/dovla/experiments/dovla_h16_rollout_runs/seed_0/lattice_rollout.json",
+      "selection_mode": "lattice",
+      "num_candidates": 16,
+      "num_groups": 575,
+      "policy_rollout_success_rate": 0.6521739130434783,
+      "policy_rollout_progress": 0.792444871255885,
+      "oracle_success_rate": 0.8573913043478261,
+      "action_mse_to_best": 0.414993303792207,
+      "per_task": {
+        "LiftPegUpright-v1": {
+          "action_mse_to_best": 0.28245431445983055,
+          "expert_success_rate": 0.8865979381443299,
+          "num_groups": 97,
+          "oracle_success_rate": 0.9278350515463918,
+          "policy_expert_regret": 0.2610538084175169,
+          "policy_oracle_regret": 0.27425835227843415,
+          "policy_rollout_progress": 0.9071177396270418,
+          "policy_rollout_success_rate": 0.7216494845360825,
+          "restore_max_error": 4.76837158203125e-07
+        },
+        "PickCube-v1": {
+          "action_mse_to_best": 0.41056775828250325,
+          "expert_success_rate": 0.9375,
+          "num_groups": 208,
+          "oracle_success_rate": 0.9471153846153846,
+          "policy_expert_regret": 0.26980068733414203,
+          "policy_oracle_regret": 0.2764672613881815,
+          "policy_rollout_progress": 0.8699935431269785,
+          "policy_rollout_success_rate": 0.7788461538461539,
+          "restore_max_error": 4.76837158203125e-07
+        },
+        "PullCube-v1": {
+          "action_mse_to_best": 0.5239523940465667,
+          "expert_success_rate": 0.19480519480519481,
+          "num_groups": 77,
+          "oracle_success_rate": 0.36363636363636365,
+          "policy_expert_regret": 0.15346811585999154,
+          "policy_oracle_regret": 0.37490944806244464,
+          "policy_rollout_progress": 0.3196387503828321,
+          "policy_rollout_success_rate": 0.16883116883116883,
+          "restore_max_error": 4.76837158203125e-07
+        },
+        "PushCube-v1": {
+          "action_mse_to_best": 0.38503194246143896,
+          "expert_success_rate": 0.8617021276595744,
+          "num_groups": 94,
+          "oracle_success_rate": 0.9893617021276596,
+          "policy_expert_regret": 0.28232911355952,
+          "policy_oracle_regret": 0.37923398661486646,
+          "policy_rollout_progress": 0.8140388716091501,
+          "policy_rollout_success_rate": 0.7872340425531915,
+          "restore_max_error": 4.76837158203125e-07
+        },
+        "StackCube-v1": {
+          "action_mse_to_best": 0.4978550557797361,
+          "expert_success_rate": 0.6767676767676768,
+          "num_groups": 99,
+          "oracle_success_rate": 0.8585858585858586,
+          "policy_expert_regret": 0.2738224623179195,
+          "policy_oracle_regret": 0.4027370100069528,
+          "policy_rollout_progress": 0.8643927833046576,
+          "policy_rollout_success_rate": 0.5656565656565656,
+          "restore_max_error": 3.948807716369629e-07
+        }
+      }
+    },
+    {
+      "seed": 1,
+      "path": "/scratch/knguy52/dovla/experiments/dovla_h16_rollout_runs/seed_1/lattice_rollout.json",
+      "selection_mode": "lattice",
+      "num_candidates": 16,
+      "num_groups": 575,
+      "policy_rollout_success_rate": 0.7147826086956521,
+      "policy_rollout_progress": 0.8205086112848442,
+      "oracle_success_rate": 0.8695652173913043,
+      "action_mse_to_best": 0.41892785357189893,
+      "per_task": {
+        "LiftPegUpright-v1": {
+          "action_mse_to_best": 0.31486138409473396,
+          "expert_success_rate": 0.8584070796460177,
+          "num_groups": 113,
+          "oracle_success_rate": 0.9380530973451328,
+          "policy_expert_regret": 0.2672024345503444,
+          "policy_oracle_regret": 0.2922932648025783,
+          "policy_rollout_progress": 0.8941378234762006,
+          "policy_rollout_success_rate": 0.7345132743362832,
+          "restore_max_error": 4.76837158203125e-07
+        },
+        "PickCube-v1": {
+          "action_mse_to_best": 0.44310908647173125,
+          "expert_success_rate": 0.9402173913043478,
+          "num_groups": 184,
+          "oracle_success_rate": 0.9456521739130435,
+          "policy_expert_regret": 0.17421010318819596,
+          "policy_oracle_regret": 0.1742341076631261,
+          "policy_rollout_progress": 0.9089716651436427,
+          "policy_rollout_success_rate": 0.8478260869565217,
+          "restore_max_error": 4.76837158203125e-07
+        },
+        "PullCube-v1": {
+          "action_mse_to_best": 0.5217870954473159,
+          "expert_success_rate": 0.25,
+          "num_groups": 76,
+          "oracle_success_rate": 0.40789473684210525,
+          "policy_expert_regret": 0.12901650541952173,
+          "policy_oracle_regret": 0.3323121542037514,
+          "policy_rollout_progress": 0.379051765659824,
+          "policy_rollout_success_rate": 0.23684210526315788,
+          "restore_max_error": 4.76837158203125e-07
+        },
+        "PushCube-v1": {
+          "action_mse_to_best": 0.34412760804365344,
+          "expert_success_rate": 0.8198198198198198,
+          "num_groups": 111,
+          "oracle_success_rate": 1.0,
+          "policy_expert_regret": 0.1992848778630162,
+          "policy_oracle_regret": 0.3170433482071301,
+          "policy_rollout_progress": 0.8541278229640411,
+          "policy_rollout_success_rate": 0.8288288288288288,
+          "restore_max_error": 4.76837158203125e-07
+        },
+        "StackCube-v1": {
+          "action_mse_to_best": 0.5045947664120531,
+          "expert_success_rate": 0.6923076923076923,
+          "num_groups": 91,
+          "oracle_success_rate": 0.8571428571428571,
+          "policy_expert_regret": 0.1715098601448667,
+          "policy_oracle_regret": 0.2727490429367338,
+          "policy_rollout_progress": 0.8778897638504322,
+          "policy_rollout_success_rate": 0.6813186813186813,
+          "restore_max_error": 3.948807716369629e-07
+        }
+      }
+    },
+    {
+      "seed": 2,
+      "path": "/scratch/knguy52/dovla/experiments/dovla_h16_rollout_runs/seed_2/lattice_rollout.json",
+      "selection_mode": "lattice",
+      "num_candidates": 16,
+      "num_groups": 575,
+      "policy_rollout_success_rate": 0.7130434782608696,
+      "policy_rollout_progress": 0.8196337475336116,
+      "oracle_success_rate": 0.8765217391304347,
+      "action_mse_to_best": 0.47871240418910493,
+      "per_task": {
+        "LiftPegUpright-v1": {
+          "action_mse_to_best": 0.33342877044075675,
+          "expert_success_rate": 0.8229166666666666,
+          "num_groups": 96,
+          "oracle_success_rate": 0.9270833333333334,
+          "policy_expert_regret": 0.22194465808570385,
+          "policy_oracle_regret": 0.28883908068140346,
+          "policy_rollout_progress": 0.9184654237081608,
+          "policy_rollout_success_rate": 0.7395833333333334,
+          "restore_max_error": 3.5762786865234375e-07
+        },
+        "PickCube-v1": {
+          "action_mse_to_best": 0.4627937324570887,
+          "expert_success_rate": 0.9444444444444444,
+          "num_groups": 198,
+          "oracle_success_rate": 0.9595959595959596,
+          "policy_expert_regret": 0.10305700688199564,
+          "policy_oracle_regret": 0.11014045741070401,
+          "policy_rollout_progress": 0.9385366825518584,
+          "policy_rollout_success_rate": 0.898989898989899,
+          "restore_max_error": 4.76837158203125e-07
+        },
+        "PullCube-v1": {
+          "action_mse_to_best": 0.6827966811549332,
+          "expert_success_rate": 0.24444444444444444,
+          "num_groups": 90,
+          "oracle_success_rate": 0.4666666666666667,
+          "policy_expert_regret": 0.15796189763479762,
+          "policy_oracle_regret": 0.40946051337652734,
+          "policy_rollout_progress": 0.39318432178762225,
+          "policy_rollout_success_rate": 0.26666666666666666,
+          "restore_max_error": 4.0978193283081055e-07
+        },
+        "PushCube-v1": {
+          "action_mse_to_best": 0.34167051138264115,
+          "expert_success_rate": 0.8514851485148515,
+          "num_groups": 101,
+          "oracle_success_rate": 1.0,
+          "policy_expert_regret": 0.24242213811024582,
+          "policy_oracle_regret": 0.29742342971339086,
+          "policy_rollout_progress": 0.8609924118707676,
+          "policy_rollout_success_rate": 0.8415841584158416,
+          "restore_max_error": 4.76837158203125e-07
+        },
+        "StackCube-v1": {
+          "action_mse_to_best": 0.6184098718480932,
+          "expert_success_rate": 0.7666666666666667,
+          "num_groups": 90,
+          "oracle_success_rate": 0.9111111111111111,
+          "policy_expert_regret": 0.3941795117325253,
+          "policy_oracle_regret": 0.4843028916252984,
+          "policy_rollout_progress": 0.8326626494526863,
+          "policy_rollout_success_rate": 0.5777777777777777,
+          "restore_max_error": 4.76837158203125e-07
+        }
+      }
+    }
+  ]
+}

results/h16_lattice_summary.md

@@ -0,0 +1,25 @@
+# h=16 Lattice-Selected Rollout
+
+Run root: `/scratch/knguy52/dovla/experiments/dovla_h16_rollout_runs`
+Completed seeds: 3
+Baseline h=4 policy success: 29.67%
+Baseline h=16 policy success: 29.74%
+
+Mean success: 69.33% +/- 3.57%
+Gain vs h=16 policy: +39.59%
+Mean oracle success: 86.78%
+Mean progress: 81.09%
+
+| seed | success | progress | oracle | candidates | action MSE |
+|---:|---:|---:|---:|---:|---:|
+| 0 | 65.22% | 79.24% | 85.74% | 16 | 0.415 |
+| 1 | 71.48% | 82.05% | 86.96% | 16 | 0.419 |
+| 2 | 71.30% | 81.96% | 87.65% | 16 | 0.479 |
+
+Selected candidate types:
+- lattice_expert: 977
+- lattice_near_miss: 348
+- lattice_no_op: 177
+- lattice_random_negative: 138
+- lattice_wrong_direction: 30
+- lattice_wrong_gripper: 55

results/h16_policy_ckpt_summary.json

@@ -0,0 +1,252 @@
+{
+  "run_root": "/scratch/knguy52/dovla/experiments/dovla_h16_policy_ckpt_runs",
+  "objective": "base",
+  "num_completed": 3,
+  "baseline_h4_policy_success": 0.2967,
+  "baseline_h16_rank_checkpoint_success": 0.29739130434782607,
+  "mean_success": 0.27014492753623187,
+  "std_success": 0.035013724917197675,
+  "mean_progress": 0.5127486971950728,
+  "mean_action_mse_to_best": 0.41113467054252606,
+  "gain_vs_h4": -0.026555072463768148,
+  "gain_vs_h16_rank_checkpoint": -0.027246376811594197,
+  "rows": [
+    {
+      "seed": 0,
+      "path": "/scratch/knguy52/dovla/experiments/dovla_h16_policy_ckpt_runs/base/seed_0/policy_rollout.json",
+      "num_groups": 575,
+      "policy_rollout_success_rate": 0.2747826086956522,
+      "policy_rollout_progress": 0.530730156995022,
+      "oracle_success_rate": 0.8573913043478261,
+      "action_mse_to_best": 0.38363921120114947,
+      "best_policy_val": {
+        "bc_loss": 0.278208808766471,
+        "field_effect_loss": 0.009297492267150018,
+        "field_potential_loss": 0.26523639841212165,
+        "field_preference_loss": 0.5116484082407422,
+        "lattice_edges": 3833.3333333333335,
+        "progress_mae": 0.19217014147175682,
+        "rank_acc": 0.8298350108994378,
+        "rank_loss": 0.5116421315405104,
+        "regret_mae": 0.40311697456571793,
+        "success_accuracy": 0.8784792224566141,
+        "total_loss": 1.144673228263855
+      },
+      "per_task": {
+        "LiftPegUpright-v1": {
+          "action_mse_to_best": 0.3370508882649166,
+          "expert_success_rate": 0.8865979381443299,
+          "num_groups": 97,
+          "oracle_success_rate": 0.9278350515463918,
+          "policy_expert_regret": 1.1605938944927197,
+          "policy_oracle_regret": 1.2001252182058453,
+          "policy_rollout_progress": 0.5482521015651447,
+          "policy_rollout_success_rate": 0.15463917525773196,
+          "restore_max_error": 4.76837158203125e-07
+        },
+        "PickCube-v1": {
+          "action_mse_to_best": 0.31313961764009524,
+          "expert_success_rate": 0.9375,
+          "num_groups": 208,
+          "oracle_success_rate": 0.9471153846153846,
+          "policy_expert_regret": 1.147200564157174,
+          "policy_oracle_regret": 1.1606021968558566,
+          "policy_rollout_progress": 0.564308604850549,
+          "policy_rollout_success_rate": 0.22115384615384615,
+          "restore_max_error": 4.76837158203125e-07
+        },
+        "PullCube-v1": {
+          "action_mse_to_best": 0.5131220405364966,
+          "expert_success_rate": 0.19480519480519481,
+          "num_groups": 77,
+          "oracle_success_rate": 0.36363636363636365,
+          "policy_expert_regret": 0.2314507949661899,
+          "policy_oracle_regret": 0.32168451832099393,
+          "policy_rollout_progress": 0.3784269967249462,
+          "policy_rollout_success_rate": 0.2597402597402597,
+          "restore_max_error": 4.76837158203125e-07
+        },
+        "PushCube-v1": {
+          "action_mse_to_best": 0.402122884602385,
+          "expert_success_rate": 0.8617021276595744,
+          "num_groups": 94,
+          "oracle_success_rate": 0.9893617021276596,
+          "policy_expert_regret": 0.5956213959354035,
+          "policy_oracle_regret": 0.6544046972660308,
+          "policy_rollout_progress": 0.6860208346488628,
+          "policy_rollout_success_rate": 0.6595744680851063,
+          "restore_max_error": 4.76837158203125e-07
+        },
+        "StackCube-v1": {
+          "action_mse_to_best": 0.4591476922817152,
+          "expert_success_rate": 0.6767676767676768,
+          "num_groups": 99,
+          "oracle_success_rate": 0.8585858585858586,
+          "policy_expert_regret": 1.0776312859973523,
+          "policy_oracle_regret": 1.26736486349443,
+          "policy_rollout_progress": 0.41402383238980267,
+          "policy_rollout_success_rate": 0.15151515151515152,
+          "restore_max_error": 3.948807716369629e-07
+        }
+      }
+    },
+    {
+      "seed": 1,
+      "path": "/scratch/knguy52/dovla/experiments/dovla_h16_policy_ckpt_runs/base/seed_1/policy_rollout.json",
+      "num_groups": 575,
+      "policy_rollout_success_rate": 0.23304347826086957,
+      "policy_rollout_progress": 0.4766378657271464,
+      "oracle_success_rate": 0.8695652173913043,
+      "action_mse_to_best": 0.42292791193756074,
+      "best_policy_val": {
+        "bc_loss": 0.2710145554608769,
+        "field_effect_loss": 0.010103965022911629,
+        "field_potential_loss": 0.28313595718807644,
+        "field_preference_loss": 0.523477166891098,
+        "lattice_edges": 3833.3333333333335,
+        "progress_mae": 0.19897940589321983,
+        "rank_acc": 0.8108105460802714,
+        "rank_loss": 0.5234645042154524,
+        "regret_mae": 0.373554312520557,
+        "success_accuracy": 0.8641913069619073,
+        "total_loss": 1.1782064769003127
+      },
+      "per_task": {
+        "LiftPegUpright-v1": {
+          "action_mse_to_best": 0.35198108434874925,
+          "expert_success_rate": 0.8584070796460177,
+          "num_groups": 113,
+          "oracle_success_rate": 0.9380530973451328,
+          "policy_expert_regret": 1.1627261715125194,
+          "policy_oracle_regret": 1.2533149083631228,
+          "policy_rollout_progress": 0.5348512005489484,
+          "policy_rollout_success_rate": 0.13274336283185842,
+          "restore_max_error": 4.76837158203125e-07
+        },
+        "PickCube-v1": {
+          "action_mse_to_best": 0.33714081593991624,
+          "expert_success_rate": 0.9402173913043478,
+          "num_groups": 184,
+          "oracle_success_rate": 0.9456521739130435,
+          "policy_expert_regret": 1.303022225893548,
+          "policy_oracle_regret": 1.3105591308608975,
+          "policy_rollout_progress": 0.4878872205584269,
+          "policy_rollout_success_rate": 0.125,
+          "restore_max_error": 4.76837158203125e-07
+        },
+        "PullCube-v1": {
+          "action_mse_to_best": 0.5864368125453199,
+          "expert_success_rate": 0.25,
+          "num_groups": 76,
+          "oracle_success_rate": 0.40789473684210525,
+          "policy_expert_regret": 0.3705535115298341,
+          "policy_oracle_regret": 0.6008050318137464,
+          "policy_rollout_progress": 0.247043040638234,
+          "policy_rollout_success_rate": 0.10526315789473684,
+          "restore_max_error": 4.76837158203125e-07
+        },
+        "PushCube-v1": {
+          "action_mse_to_best": 0.4040020576149628,
+          "expert_success_rate": 0.8198198198198198,
+          "num_groups": 111,
+          "oracle_success_rate": 1.0,
+          "policy_expert_regret": 0.6428923311534228,
+          "policy_oracle_regret": 0.7433026666576797,
+          "policy_rollout_progress": 0.6440847207297076,
+          "policy_rollout_success_rate": 0.6126126126126126,
+          "restore_max_error": 4.76837158203125e-07
+        },
+        "StackCube-v1": {
+          "action_mse_to_best": 0.5710148412202086,
+          "expert_success_rate": 0.6923076923076923,
+          "num_groups": 91,
+          "oracle_success_rate": 0.8571428571428571,
+          "policy_expert_regret": 1.0961015530667462,
+          "policy_oracle_regret": 1.230444488780839,
+          "policy_rollout_progress": 0.36910619185521054,
+          "policy_rollout_success_rate": 0.21978021978021978,
+          "restore_max_error": 3.948807716369629e-07
+        }
+      }
+    },
+    {
+      "seed": 2,
+      "path": "/scratch/knguy52/dovla/experiments/dovla_h16_policy_ckpt_runs/base/seed_2/policy_rollout.json",
+      "num_groups": 575,
+      "policy_rollout_success_rate": 0.3026086956521739,
+      "policy_rollout_progress": 0.5308780688630498,
+      "oracle_success_rate": 0.8765217391304347,
+      "action_mse_to_best": 0.426836888488868,
+      "best_policy_val": {
+        "bc_loss": 0.2535104958547486,
+        "field_effect_loss": 0.00966053934664362,
+        "field_potential_loss": 0.26335224095318055,
+        "field_preference_loss": 0.5147431525919173,
+        "lattice_edges": 3833.3333333333335,
+        "progress_mae": 0.19942427343792385,
+        "rank_acc": 0.8290975723001692,
+        "rank_loss": 0.5147051960229874,
+        "regret_mae": 0.3818853911426332,
+        "success_accuracy": 0.8690461185243394,
+        "total_loss": 1.1259862383206685
+      },
+      "per_task": {
+        "LiftPegUpright-v1": {
+          "action_mse_to_best": 0.3665522066488241,
+          "expert_success_rate": 0.8229166666666666,
+          "num_groups": 96,
+          "oracle_success_rate": 0.9270833333333334,
+          "policy_expert_regret": 1.1764638209715486,
+          "policy_oracle_regret": 1.2963196545218427,
+          "policy_rollout_progress": 0.47813306966175634,
+          "policy_rollout_success_rate": 0.14583333333333334,
+          "restore_max_error": 3.5762786865234375e-07
+        },
+        "PickCube-v1": {
+          "action_mse_to_best": 0.34438839633807994,
+          "expert_success_rate": 0.9444444444444444,
+          "num_groups": 198,
+          "oracle_success_rate": 0.9595959595959596,
+          "policy_expert_regret": 1.1031859448612338,
+          "policy_oracle_regret": 1.1244608974366477,
+          "policy_rollout_progress": 0.5753103712140911,
+          "policy_rollout_success_rate": 0.24242424242424243,
+          "restore_max_error": 4.76837158203125e-07
+        },
+        "PullCube-v1": {
+          "action_mse_to_best": 0.5974176011358698,
+          "expert_success_rate": 0.24444444444444444,
+          "num_groups": 90,
+          "oracle_success_rate": 0.4666666666666667,
+          "policy_expert_regret": 0.3594287911636962,
+          "policy_oracle_regret": 0.5890606641355488,
+          "policy_rollout_progress": 0.30660248436033727,
+          "policy_rollout_success_rate": 0.16666666666666666,
+          "restore_max_error": 4.0978193283081055e-07
+        },
+        "PushCube-v1": {
+          "action_mse_to_best": 0.39548876992251614,
+          "expert_success_rate": 0.8514851485148515,
+          "num_groups": 101,
+          "oracle_success_rate": 1.0,
+          "policy_expert_regret": 0.40027354908461615,
+          "policy_oracle_regret": 0.40027354908461615,
+          "policy_rollout_progress": 0.8076472429945918,
+          "policy_rollout_success_rate": 0.7920792079207921,
+          "restore_max_error": 4.76837158203125e-07
+        },
+        "StackCube-v1": {
+          "action_mse_to_best": 0.5371260744829972,
+          "expert_success_rate": 0.7666666666666667,
+          "num_groups": 90,
+          "oracle_success_rate": 0.9111111111111111,
+          "policy_expert_regret": 1.1670463437835374,
+          "policy_oracle_regret": 1.303001377152072,
+          "policy_rollout_progress": 0.4030674030383428,
+          "policy_rollout_success_rate": 0.18888888888888888,
+          "restore_max_error": 4.76837158203125e-07
+        }
+      }
+    }
+  ]
+}

results/h16_policy_ckpt_summary.md

@@ -0,0 +1,17 @@
+# h=16 Best-Policy Checkpoint Rollout
+
+Run root: `/scratch/knguy52/dovla/experiments/dovla_h16_policy_ckpt_runs`
+Completed seeds: 3
+Baseline h=4 policy success: 29.67%
+Baseline h=16 rank-checkpoint success: 29.74%
+
+Mean success: 27.01% +/- 3.50%
+Gain vs h=16 rank checkpoint: -2.72%
+Mean progress: 51.27%
+Mean action MSE to best: 0.411
+
+| seed | success | progress | oracle | action MSE |
+|---:|---:|---:|---:|---:|
+| 0 | 27.48% | 53.07% | 85.74% | 0.384 |
+| 1 | 23.30% | 47.66% | 86.96% | 0.423 |
+| 2 | 30.26% | 53.09% | 87.65% | 0.427 |