Hugging Face's logo

anhtld
/
vla

Model card Files
xet
 Community
anhtld commited on
Commit
c97fa00
·
verified ·
1 Parent(s): c9f6598

Auto-sync: 2026-06-30 10:31:48

Browse files
Files changed (5) hide show
  1. results/README.md +10 -4
  2. results/h16_bestpolicy_nooponly_prepend_margin0p05_summary.json +436 -0
  3. results/h16_bestpolicy_nooponly_prepend_margin0p05_summary.md +19 -0
  4. results/h16_bestpolicy_types2sparse_prepend_margin0p05_summary.json +439 -0
  5. results/h16_bestpolicy_types2sparse_prepend_margin0p05_summary.md +19 -0
results/README.md CHANGED
@@ -160,8 +160,11 @@ at margin `0.05`. Anti-goal family masking improves the row but does not beat
160
  compatible tangent transport: safe4 reaches 33.45%, sparse no-op/wrong-gripper
161
  34.26%, and no-op-only 34.43%. The result is a useful negative diagnostic:
162
  direct typed proposal generation adds clean support, but the generated support
163
- is still not local enough. Best-policy checkpoint ablations are pending as eval
164
- arrays `14963143` and `14963147`, with summaries `14963150` and `14963149`.
 
 
 
165
 
166
  - Code supports `--retrieval-residual-scale`, defaulting to `1.0` so the
167
  completed 32.12% result is unchanged.
@@ -422,8 +425,11 @@ arrays `14963143` and `14963147`, with summaries `14963150` and `14963149`.
422
  - Jobs `14963030` -> `14963044`, `14963031` -> `14963042`, and `14963032` ->
423
  `14963043` completed typed proposal family masking at 33.45%, 34.26%, and
424
  34.43%.
425
- - Jobs `14963143` -> `14963150` and `14963147` -> `14963149` are pending
426
- best-policy checkpoint ablations for sparse/no-op typed proposals.
 
 
 
427
 
428
  Paper table automation:
429
 
 
160
  compatible tangent transport: safe4 reaches 33.45%, sparse no-op/wrong-gripper
161
  34.26%, and no-op-only 34.43%. The result is a useful negative diagnostic:
162
  direct typed proposal generation adds clean support, but the generated support
163
+ is still not local enough. Best-policy checkpoint ablations are also negative:
164
+ sparse no-op/wrong-gripper drops to 32.29%, and no-op-only drops to 33.16%.
165
+ A no-op-only proposal-head retrain is pending as train/eval/summary
166
+ `14963500` -> `14963505` -> `14963507` to test whether the failure is due to
167
+ multi-family proposal-head interference.
168
 
169
  - Code supports `--retrieval-residual-scale`, defaulting to `1.0` so the
170
  completed 32.12% result is unchanged.
 
425
  - Jobs `14963030` -> `14963044`, `14963031` -> `14963042`, and `14963032` ->
426
  `14963043` completed typed proposal family masking at 33.45%, 34.26%, and
427
  34.43%.
428
+ - Jobs `14963143` -> `14963150` and `14963147` -> `14963149` completed
429
+ best-policy checkpoint ablations for sparse/no-op typed proposals at 32.29%
430
+ and 33.16%.
431
+ - Jobs `14963500` -> `14963505` -> `14963507` are pending for the no-op-only
432
+ proposal-head retrain.
433
 
434
  Paper table automation:
435
 
results/h16_bestpolicy_nooponly_prepend_margin0p05_summary.json ADDED
@@ -0,0 +1,436 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "run_root": "/scratch/knguy52/dovla/experiments/dovla_h16_policy_ckpt_runs",
3
+ "objective": "near_miss_policy_bc5_typedprop_p2",
4
+ "out_name": "policy_rollout_proposal_lattice_nooponly_bestpolicy_prepend_margin0p05.json",
5
+ "num_completed": 3,
6
+ "baseline_h4_policy_success": 0.2967,
7
+ "baseline_h16_rank_checkpoint_success": 0.29739130434782607,
8
+ "mean_success": 0.33159420289855074,
9
+ "std_success": 0.03719191868369956,
10
+ "mean_progress": 0.5653533038507769,
11
+ "mean_action_mse_to_best": 0.4323545880660506,
12
+ "gain_vs_h4": 0.03489420289855072,
13
+ "gain_vs_h16_rank_checkpoint": 0.03420289855072467,
14
+ "rows": [
15
+ {
16
+ "seed": 0,
17
+ "path": "/scratch/knguy52/dovla/experiments/dovla_h16_policy_ckpt_runs/near_miss_policy_bc5_typedprop_p2/seed_0/policy_rollout_proposal_lattice_nooponly_bestpolicy_prepend_margin0p05.json",
18
+ "num_groups": 575,
19
+ "selection_mode": "proposal_lattice",
20
+ "num_candidates": 2,
21
+ "candidate_sigma": 0.0,
22
+ "proposal_types": [
23
+ "expert",
24
+ "near_miss",
25
+ "no_op",
26
+ "wrong_gripper",
27
+ "wrong_direction",
28
+ "random_negative"
29
+ ],
30
+ "proposal_lattice_types": [
31
+ "no_op"
32
+ ],
33
+ "selection_margin": 0.05,
34
+ "prepend_policy_candidate": true,
35
+ "field_optim_steps": 0,
36
+ "field_optim_step_size": 0.0,
37
+ "field_optim_trust_radius": 0.0,
38
+ "field_optim_l2_penalty": 0.0,
39
+ "retrieval_neighbors": 0,
40
+ "retrieval_metric": "none",
41
+ "retrieval_type_min_success": 0.0,
42
+ "retrieval_type_success_bonus_scale": 0.0,
43
+ "retrieval_residual_consensus_penalty_scale": 0.0,
44
+ "retrieval_residual_min_source_progress": 0.0,
45
+ "retrieval_residual_min_source_advantage": -1000000000.0,
46
+ "retrieval_residual_source_progress_bonus_scale": 0.0,
47
+ "retrieval_residual_source_score_bonus_scale": 0.0,
48
+ "retrieval_residual_source_advantage_bonus_scale": 0.0,
49
+ "retrieval_residual_composite_l2_penalty_scale": 0.0,
50
+ "retrieval_residual_scale": 0.0,
51
+ "retrieval_residual_scales": [],
52
+ "retrieval_residual_anchor": "none",
53
+ "retrieval_residual_direction": "none",
54
+ "retrieval_residual_reduce": "none",
55
+ "retrieval_residual_challenger_types": [],
56
+ "retrieval_residual_challenger_scales": [],
57
+ "retrieval_residual_challenger_margin": 0.0,
58
+ "lattice_exclude_types": [],
59
+ "candidate_type_bonuses": {},
60
+ "candidate_type_bonus_components": false,
61
+ "candidate_oracle_rollouts": 0,
62
+ "candidate_oracle_unique_tolerance": 1e-06,
63
+ "candidate_oracle_success_rate": null,
64
+ "candidate_oracle_progress": null,
65
+ "candidate_oracle_score": null,
66
+ "candidate_oracle_regret": null,
67
+ "candidate_oracle_score_gain_over_selected": null,
68
+ "candidate_oracle_unique_count": null,
69
+ "candidate_oracle_improvement_rate": null,
70
+ "candidate_oracle_selected_branch_success_rate": null,
71
+ "candidate_oracle_selected_branch_progress": null,
72
+ "candidate_oracle_type_counts": {},
73
+ "candidate_oracle_best_branch_rank": null,
74
+ "candidate_oracle_best_branch_rank_counts": {},
75
+ "candidate_oracle_branch_success_rates": [],
76
+ "candidate_oracle_branch_progress": [],
77
+ "candidate_oracle_branch_score_gains_over_selected": [],
78
+ "selected_residual_scale_counts": {},
79
+ "policy_rollout_success_rate": 0.2991304347826087,
80
+ "policy_rollout_progress": 0.5511523539054652,
81
+ "oracle_success_rate": 0.8573913043478261,
82
+ "action_mse_to_best": 0.40204733323971437,
83
+ "best_policy_val": {
84
+ "bc_loss": 0.1345199979841709,
85
+ "field_effect_loss": 0.009223711728635762,
86
+ "field_potential_loss": 0.2614195512400733,
87
+ "field_preference_loss": 0.5129645930396186,
88
+ "lattice_edges": 3833.3333333333335,
89
+ "progress_mae": 0.18681737035512924,
90
+ "proposal_loss": 0.15201550225416818,
91
+ "rank_acc": 0.8316339155038198,
92
+ "rank_loss": 0.5129594521390067,
93
+ "regret_mae": 0.3556148376729753,
94
+ "success_accuracy": 0.8784897228082021,
95
+ "total_loss": 1.8417018850644429
96
+ },
97
+ "per_task": {
98
+ "LiftPegUpright-v1": {
99
+ "action_mse_to_best": 0.3164978881184127,
100
+ "expert_success_rate": 0.8865979381443299,
101
+ "num_groups": 97,
102
+ "oracle_success_rate": 0.9278350515463918,
103
+ "policy_expert_regret": 1.0569167591861843,
104
+ "policy_oracle_regret": 1.096554077777666,
105
+ "policy_rollout_progress": 0.6340098494721442,
106
+ "policy_rollout_success_rate": 0.18556701030927836,
107
+ "restore_max_error": 2.384185791015625e-07
108
+ },
109
+ "PickCube-v1": {
110
+ "action_mse_to_best": 0.34170422394464156,
111
+ "expert_success_rate": 0.9375,
112
+ "num_groups": 208,
113
+ "oracle_success_rate": 0.9471153846153846,
114
+ "policy_expert_regret": 1.1250538387192557,
115
+ "policy_oracle_regret": 1.1317204127732952,
116
+ "policy_rollout_progress": 0.5715542031953541,
117
+ "policy_rollout_success_rate": 0.23557692307692307,
118
+ "restore_max_error": 2.384185791015625e-07
119
+ },
120
+ "PullCube-v1": {
121
+ "action_mse_to_best": 0.6159144512607486,
122
+ "expert_success_rate": 0.19480519480519481,
123
+ "num_groups": 77,
124
+ "oracle_success_rate": 0.36363636363636365,
125
+ "policy_expert_regret": 0.2849003163879955,
126
+ "policy_oracle_regret": 0.5363171282768636,
127
+ "policy_rollout_progress": 0.27096225638184457,
128
+ "policy_rollout_success_rate": 0.12987012987012986,
129
+ "restore_max_error": 2.3096799850463867e-07
130
+ },
131
+ "PushCube-v1": {
132
+ "action_mse_to_best": 0.3711102392643373,
133
+ "expert_success_rate": 0.8617021276595744,
134
+ "num_groups": 94,
135
+ "oracle_success_rate": 0.9893617021276596,
136
+ "policy_expert_regret": 0.552364246642336,
137
+ "policy_oracle_regret": 0.6303656067936978,
138
+ "policy_rollout_progress": 0.6994216272488554,
139
+ "policy_rollout_success_rate": 0.6702127659574468,
140
+ "restore_max_error": 2.384185791015625e-07
141
+ },
142
+ "StackCube-v1": {
143
+ "action_mse_to_best": 0.47568351158524175,
144
+ "expert_success_rate": 0.6767676767676768,
145
+ "num_groups": 99,
146
+ "oracle_success_rate": 0.8585858585858586,
147
+ "policy_expert_regret": 0.8507854416213855,
148
+ "policy_oracle_regret": 1.016750856631934,
149
+ "policy_rollout_progress": 0.5042489609031966,
150
+ "policy_rollout_success_rate": 0.32323232323232326,
151
+ "restore_max_error": 2.384185791015625e-07
152
+ }
153
+ }
154
+ },
155
+ {
156
+ "seed": 1,
157
+ "path": "/scratch/knguy52/dovla/experiments/dovla_h16_policy_ckpt_runs/near_miss_policy_bc5_typedprop_p2/seed_1/policy_rollout_proposal_lattice_nooponly_bestpolicy_prepend_margin0p05.json",
158
+ "num_groups": 575,
159
+ "selection_mode": "proposal_lattice",
160
+ "num_candidates": 2,
161
+ "candidate_sigma": 0.0,
162
+ "proposal_types": [
163
+ "expert",
164
+ "near_miss",
165
+ "no_op",
166
+ "wrong_gripper",
167
+ "wrong_direction",
168
+ "random_negative"
169
+ ],
170
+ "proposal_lattice_types": [
171
+ "no_op"
172
+ ],
173
+ "selection_margin": 0.05,
174
+ "prepend_policy_candidate": true,
175
+ "field_optim_steps": 0,
176
+ "field_optim_step_size": 0.0,
177
+ "field_optim_trust_radius": 0.0,
178
+ "field_optim_l2_penalty": 0.0,
179
+ "retrieval_neighbors": 0,
180
+ "retrieval_metric": "none",
181
+ "retrieval_type_min_success": 0.0,
182
+ "retrieval_type_success_bonus_scale": 0.0,
183
+ "retrieval_residual_consensus_penalty_scale": 0.0,
184
+ "retrieval_residual_min_source_progress": 0.0,
185
+ "retrieval_residual_min_source_advantage": -1000000000.0,
186
+ "retrieval_residual_source_progress_bonus_scale": 0.0,
187
+ "retrieval_residual_source_score_bonus_scale": 0.0,
188
+ "retrieval_residual_source_advantage_bonus_scale": 0.0,
189
+ "retrieval_residual_composite_l2_penalty_scale": 0.0,
190
+ "retrieval_residual_scale": 0.0,
191
+ "retrieval_residual_scales": [],
192
+ "retrieval_residual_anchor": "none",
193
+ "retrieval_residual_direction": "none",
194
+ "retrieval_residual_reduce": "none",
195
+ "retrieval_residual_challenger_types": [],
196
+ "retrieval_residual_challenger_scales": [],
197
+ "retrieval_residual_challenger_margin": 0.0,
198
+ "lattice_exclude_types": [],
199
+ "candidate_type_bonuses": {},
200
+ "candidate_type_bonus_components": false,
201
+ "candidate_oracle_rollouts": 0,
202
+ "candidate_oracle_unique_tolerance": 1e-06,
203
+ "candidate_oracle_success_rate": null,
204
+ "candidate_oracle_progress": null,
205
+ "candidate_oracle_score": null,
206
+ "candidate_oracle_regret": null,
207
+ "candidate_oracle_score_gain_over_selected": null,
208
+ "candidate_oracle_unique_count": null,
209
+ "candidate_oracle_improvement_rate": null,
210
+ "candidate_oracle_selected_branch_success_rate": null,
211
+ "candidate_oracle_selected_branch_progress": null,
212
+ "candidate_oracle_type_counts": {},
213
+ "candidate_oracle_best_branch_rank": null,
214
+ "candidate_oracle_best_branch_rank_counts": {},
215
+ "candidate_oracle_branch_success_rates": [],
216
+ "candidate_oracle_branch_progress": [],
217
+ "candidate_oracle_branch_score_gains_over_selected": [],
218
+ "selected_residual_scale_counts": {},
219
+ "policy_rollout_success_rate": 0.37217391304347824,
220
+ "policy_rollout_progress": 0.5908677209994715,
221
+ "oracle_success_rate": 0.8695652173913043,
222
+ "action_mse_to_best": 0.4281022290192018,
223
+ "best_policy_val": {
224
+ "bc_loss": 0.12785478805502257,
225
+ "field_effect_loss": 0.010064840265032318,
226
+ "field_potential_loss": 0.29153423342439866,
227
+ "field_preference_loss": 0.5177145335409377,
228
+ "lattice_edges": 3833.3333333333335,
229
+ "progress_mae": 0.19837388147910437,
230
+ "proposal_loss": 0.14544310627712143,
231
+ "rank_acc": 0.8285107778178321,
232
+ "rank_loss": 0.5177104257875018,
233
+ "regret_mae": 0.37083542181385887,
234
+ "success_accuracy": 0.8705372114976248,
235
+ "total_loss": 1.8337699174880981
236
+ },
237
+ "per_task": {
238
+ "LiftPegUpright-v1": {
239
+ "action_mse_to_best": 0.3211825935054669,
240
+ "expert_success_rate": 0.8584070796460177,
241
+ "num_groups": 113,
242
+ "oracle_success_rate": 0.9380530973451328,
243
+ "policy_expert_regret": 0.9411458524718749,
244
+ "policy_oracle_regret": 0.999543223892693,
245
+ "policy_rollout_progress": 0.6494711567606546,
246
+ "policy_rollout_success_rate": 0.2831858407079646,
247
+ "restore_max_error": 2.384185791015625e-07
248
+ },
249
+ "PickCube-v1": {
250
+ "action_mse_to_best": 0.4430281473655866,
251
+ "expert_success_rate": 0.9402173913043478,
252
+ "num_groups": 184,
253
+ "oracle_success_rate": 0.9456521739130435,
254
+ "policy_expert_regret": 0.9318571271445684,
255
+ "policy_oracle_regret": 0.9318811316194985,
256
+ "policy_rollout_progress": 0.6274347850172173,
257
+ "policy_rollout_success_rate": 0.3641304347826087,
258
+ "restore_max_error": 2.384185791015625e-07
259
+ },
260
+ "PullCube-v1": {
261
+ "action_mse_to_best": 0.5424103456944546,
262
+ "expert_success_rate": 0.25,
263
+ "num_groups": 76,
264
+ "oracle_success_rate": 0.40789473684210525,
265
+ "policy_expert_regret": 0.27426378164244325,
266
+ "policy_oracle_regret": 0.48424072754814435,
267
+ "policy_rollout_progress": 0.3173962665329638,
268
+ "policy_rollout_success_rate": 0.19736842105263158,
269
+ "restore_max_error": 2.384185791015625e-07
270
+ },
271
+ "PushCube-v1": {
272
+ "action_mse_to_best": 0.35307818720838774,
273
+ "expert_success_rate": 0.8198198198198198,
274
+ "num_groups": 111,
275
+ "oracle_success_rate": 1.0,
276
+ "policy_expert_regret": 0.3658739776761682,
277
+ "policy_oracle_regret": 0.46758256812353394,
278
+ "policy_rollout_progress": 0.7756606751197094,
279
+ "policy_rollout_success_rate": 0.7567567567567568,
280
+ "restore_max_error": 2.3795291781425476e-07
281
+ },
282
+ "StackCube-v1": {
283
+ "action_mse_to_best": 0.5267374115576456,
284
+ "expert_success_rate": 0.6923076923076923,
285
+ "num_groups": 91,
286
+ "oracle_success_rate": 0.8571428571428571,
287
+ "policy_expert_regret": 1.0439347614953807,
288
+ "policy_oracle_regret": 1.2033467446709727,
289
+ "policy_rollout_progress": 0.4471455738767163,
290
+ "policy_rollout_success_rate": 0.17582417582417584,
291
+ "restore_max_error": 1.9371509552001953e-07
292
+ }
293
+ }
294
+ },
295
+ {
296
+ "seed": 2,
297
+ "path": "/scratch/knguy52/dovla/experiments/dovla_h16_policy_ckpt_runs/near_miss_policy_bc5_typedprop_p2/seed_2/policy_rollout_proposal_lattice_nooponly_bestpolicy_prepend_margin0p05.json",
298
+ "num_groups": 575,
299
+ "selection_mode": "proposal_lattice",
300
+ "num_candidates": 2,
301
+ "candidate_sigma": 0.0,
302
+ "proposal_types": [
303
+ "expert",
304
+ "near_miss",
305
+ "no_op",
306
+ "wrong_gripper",
307
+ "wrong_direction",
308
+ "random_negative"
309
+ ],
310
+ "proposal_lattice_types": [
311
+ "no_op"
312
+ ],
313
+ "selection_margin": 0.05,
314
+ "prepend_policy_candidate": true,
315
+ "field_optim_steps": 0,
316
+ "field_optim_step_size": 0.0,
317
+ "field_optim_trust_radius": 0.0,
318
+ "field_optim_l2_penalty": 0.0,
319
+ "retrieval_neighbors": 0,
320
+ "retrieval_metric": "none",
321
+ "retrieval_type_min_success": 0.0,
322
+ "retrieval_type_success_bonus_scale": 0.0,
323
+ "retrieval_residual_consensus_penalty_scale": 0.0,
324
+ "retrieval_residual_min_source_progress": 0.0,
325
+ "retrieval_residual_min_source_advantage": -1000000000.0,
326
+ "retrieval_residual_source_progress_bonus_scale": 0.0,
327
+ "retrieval_residual_source_score_bonus_scale": 0.0,
328
+ "retrieval_residual_source_advantage_bonus_scale": 0.0,
329
+ "retrieval_residual_composite_l2_penalty_scale": 0.0,
330
+ "retrieval_residual_scale": 0.0,
331
+ "retrieval_residual_scales": [],
332
+ "retrieval_residual_anchor": "none",
333
+ "retrieval_residual_direction": "none",
334
+ "retrieval_residual_reduce": "none",
335
+ "retrieval_residual_challenger_types": [],
336
+ "retrieval_residual_challenger_scales": [],
337
+ "retrieval_residual_challenger_margin": 0.0,
338
+ "lattice_exclude_types": [],
339
+ "candidate_type_bonuses": {},
340
+ "candidate_type_bonus_components": false,
341
+ "candidate_oracle_rollouts": 0,
342
+ "candidate_oracle_unique_tolerance": 1e-06,
343
+ "candidate_oracle_success_rate": null,
344
+ "candidate_oracle_progress": null,
345
+ "candidate_oracle_score": null,
346
+ "candidate_oracle_regret": null,
347
+ "candidate_oracle_score_gain_over_selected": null,
348
+ "candidate_oracle_unique_count": null,
349
+ "candidate_oracle_improvement_rate": null,
350
+ "candidate_oracle_selected_branch_success_rate": null,
351
+ "candidate_oracle_selected_branch_progress": null,
352
+ "candidate_oracle_type_counts": {},
353
+ "candidate_oracle_best_branch_rank": null,
354
+ "candidate_oracle_best_branch_rank_counts": {},
355
+ "candidate_oracle_branch_success_rates": [],
356
+ "candidate_oracle_branch_progress": [],
357
+ "candidate_oracle_branch_score_gains_over_selected": [],
358
+ "selected_residual_scale_counts": {},
359
+ "policy_rollout_success_rate": 0.3234782608695652,
360
+ "policy_rollout_progress": 0.5540398366473939,
361
+ "oracle_success_rate": 0.8765217391304347,
362
+ "action_mse_to_best": 0.46691420193923555,
363
+ "best_policy_val": {
364
+ "bc_loss": 0.11429175941480531,
365
+ "field_effect_loss": 0.009631710747877756,
366
+ "field_potential_loss": 0.2724720388650894,
367
+ "field_preference_loss": 0.5087620665629705,
368
+ "lattice_edges": 3833.3333333333335,
369
+ "progress_mae": 0.19653971824381086,
370
+ "proposal_loss": 0.13279944616887304,
371
+ "rank_acc": 0.8339865836832259,
372
+ "rank_loss": 0.508702975180414,
373
+ "regret_mae": 0.3901382436354955,
374
+ "success_accuracy": 0.870691223276986,
375
+ "total_loss": 1.710942678981357
376
+ },
377
+ "per_task": {
378
+ "LiftPegUpright-v1": {
379
+ "action_mse_to_best": 0.34519209470211837,
380
+ "expert_success_rate": 0.8229166666666666,
381
+ "num_groups": 96,
382
+ "oracle_success_rate": 0.9270833333333334,
383
+ "policy_expert_regret": 0.80674880479152,
384
+ "policy_oracle_regret": 0.9008693016755084,
385
+ "policy_rollout_progress": 0.6861427738331258,
386
+ "policy_rollout_success_rate": 0.3333333333333333,
387
+ "restore_max_error": 1.955777406692505e-07
388
+ },
389
+ "PickCube-v1": {
390
+ "action_mse_to_best": 0.41532564817603224,
391
+ "expert_success_rate": 0.9444444444444444,
392
+ "num_groups": 198,
393
+ "oracle_success_rate": 0.9595959595959596,
394
+ "policy_expert_regret": 0.9896244406841244,
395
+ "policy_oracle_regret": 0.9967078912128328,
396
+ "policy_rollout_progress": 0.6206366271392038,
397
+ "policy_rollout_success_rate": 0.32323232323232326,
398
+ "restore_max_error": 2.384185791015625e-07
399
+ },
400
+ "PullCube-v1": {
401
+ "action_mse_to_best": 0.6821927662188602,
402
+ "expert_success_rate": 0.24444444444444444,
403
+ "num_groups": 90,
404
+ "oracle_success_rate": 0.4666666666666667,
405
+ "policy_expert_regret": 0.3515557604324486,
406
+ "policy_oracle_regret": 0.6623216705396772,
407
+ "policy_rollout_progress": 0.28645538735306925,
408
+ "policy_rollout_success_rate": 0.14444444444444443,
409
+ "restore_max_error": 2.384185791015625e-07
410
+ },
411
+ "PushCube-v1": {
412
+ "action_mse_to_best": 0.38364718323296837,
413
+ "expert_success_rate": 0.8514851485148515,
414
+ "num_groups": 101,
415
+ "oracle_success_rate": 1.0,
416
+ "policy_expert_regret": 0.6492541296647327,
417
+ "policy_oracle_regret": 0.6866878037995631,
418
+ "policy_rollout_progress": 0.6697478397647934,
419
+ "policy_rollout_success_rate": 0.6435643564356436,
420
+ "restore_max_error": 2.384185791015625e-07
421
+ },
422
+ "StackCube-v1": {
423
+ "action_mse_to_best": 0.5884114690952831,
424
+ "expert_success_rate": 0.7666666666666667,
425
+ "num_groups": 90,
426
+ "oracle_success_rate": 0.9111111111111111,
427
+ "policy_expert_regret": 1.2079655384023984,
428
+ "policy_oracle_regret": 1.344330143597391,
429
+ "policy_rollout_progress": 0.4043514548076524,
430
+ "policy_rollout_success_rate": 0.13333333333333333,
431
+ "restore_max_error": 2.2351741790771484e-07
432
+ }
433
+ }
434
+ }
435
+ ]
436
+ }
results/h16_bestpolicy_nooponly_prepend_margin0p05_summary.md ADDED
@@ -0,0 +1,19 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # h=16 Best-Policy Checkpoint Rollout
2
+
3
+ Run root: `/scratch/knguy52/dovla/experiments/dovla_h16_policy_ckpt_runs`
4
+ Objective: `near_miss_policy_bc5_typedprop_p2`
5
+ Result file: `policy_rollout_proposal_lattice_nooponly_bestpolicy_prepend_margin0p05.json`
6
+ Completed seeds: 3
7
+ Baseline h=4 policy success: 29.67%
8
+ Baseline h=16 rank-checkpoint success: 29.74%
9
+
10
+ Mean success: 33.16% +/- 3.72%
11
+ Gain vs h=16 rank checkpoint: +3.42%
12
+ Mean progress: 56.54%
13
+ Mean action MSE to best: 0.432
14
+
15
+ | seed | mode | k | policy cand | retrieval K | retrieval metric | residual anchor | residual direction | residual reduce | min type success | type success bonus | consensus penalty | min source progress | source progress bonus | source score bonus | residual scale | residual scales | margin | sigma | opt steps | trust | success | progress | oracle | candidate oracle | oracle gain | action MSE |
16
+ |---:|---|---:|---|---:|---|---|---|---|---:|---:|---:|---:|---:|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
17
+ | 0 | proposal_lattice | 2 | yes | 0 | none | none | none | none | 0.00 | 0.000 | 0.000 | 0.00 | 0.000 | 0.000 | 0.00 | none | 0.050 | 0.00 | 0 | 0.00 | 29.91% | 55.12% | 85.74% | n/a | n/a | 0.402 |
18
+ | 1 | proposal_lattice | 2 | yes | 0 | none | none | none | none | 0.00 | 0.000 | 0.000 | 0.00 | 0.000 | 0.000 | 0.00 | none | 0.050 | 0.00 | 0 | 0.00 | 37.22% | 59.09% | 86.96% | n/a | n/a | 0.428 |
19
+ | 2 | proposal_lattice | 2 | yes | 0 | none | none | none | none | 0.00 | 0.000 | 0.000 | 0.00 | 0.000 | 0.000 | 0.00 | none | 0.050 | 0.00 | 0 | 0.00 | 32.35% | 55.40% | 87.65% | n/a | n/a | 0.467 |
results/h16_bestpolicy_types2sparse_prepend_margin0p05_summary.json ADDED
@@ -0,0 +1,439 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "run_root": "/scratch/knguy52/dovla/experiments/dovla_h16_policy_ckpt_runs",
3
+ "objective": "near_miss_policy_bc5_typedprop_p2",
4
+ "out_name": "policy_rollout_proposal_lattice_types2sparse_bestpolicy_prepend_margin0p05.json",
5
+ "num_completed": 3,
6
+ "baseline_h4_policy_success": 0.2967,
7
+ "baseline_h16_rank_checkpoint_success": 0.29739130434782607,
8
+ "mean_success": 0.3228985507246377,
9
+ "std_success": 0.036659505515970706,
10
+ "mean_progress": 0.5570779382899079,
11
+ "mean_action_mse_to_best": 0.45112336522551166,
12
+ "gain_vs_h4": 0.02619855072463767,
13
+ "gain_vs_h16_rank_checkpoint": 0.02550724637681162,
14
+ "rows": [
15
+ {
16
+ "seed": 0,
17
+ "path": "/scratch/knguy52/dovla/experiments/dovla_h16_policy_ckpt_runs/near_miss_policy_bc5_typedprop_p2/seed_0/policy_rollout_proposal_lattice_types2sparse_bestpolicy_prepend_margin0p05.json",
18
+ "num_groups": 575,
19
+ "selection_mode": "proposal_lattice",
20
+ "num_candidates": 3,
21
+ "candidate_sigma": 0.0,
22
+ "proposal_types": [
23
+ "expert",
24
+ "near_miss",
25
+ "no_op",
26
+ "wrong_gripper",
27
+ "wrong_direction",
28
+ "random_negative"
29
+ ],
30
+ "proposal_lattice_types": [
31
+ "no_op",
32
+ "wrong_gripper"
33
+ ],
34
+ "selection_margin": 0.05,
35
+ "prepend_policy_candidate": true,
36
+ "field_optim_steps": 0,
37
+ "field_optim_step_size": 0.0,
38
+ "field_optim_trust_radius": 0.0,
39
+ "field_optim_l2_penalty": 0.0,
40
+ "retrieval_neighbors": 0,
41
+ "retrieval_metric": "none",
42
+ "retrieval_type_min_success": 0.0,
43
+ "retrieval_type_success_bonus_scale": 0.0,
44
+ "retrieval_residual_consensus_penalty_scale": 0.0,
45
+ "retrieval_residual_min_source_progress": 0.0,
46
+ "retrieval_residual_min_source_advantage": -1000000000.0,
47
+ "retrieval_residual_source_progress_bonus_scale": 0.0,
48
+ "retrieval_residual_source_score_bonus_scale": 0.0,
49
+ "retrieval_residual_source_advantage_bonus_scale": 0.0,
50
+ "retrieval_residual_composite_l2_penalty_scale": 0.0,
51
+ "retrieval_residual_scale": 0.0,
52
+ "retrieval_residual_scales": [],
53
+ "retrieval_residual_anchor": "none",
54
+ "retrieval_residual_direction": "none",
55
+ "retrieval_residual_reduce": "none",
56
+ "retrieval_residual_challenger_types": [],
57
+ "retrieval_residual_challenger_scales": [],
58
+ "retrieval_residual_challenger_margin": 0.0,
59
+ "lattice_exclude_types": [],
60
+ "candidate_type_bonuses": {},
61
+ "candidate_type_bonus_components": false,
62
+ "candidate_oracle_rollouts": 0,
63
+ "candidate_oracle_unique_tolerance": 1e-06,
64
+ "candidate_oracle_success_rate": null,
65
+ "candidate_oracle_progress": null,
66
+ "candidate_oracle_score": null,
67
+ "candidate_oracle_regret": null,
68
+ "candidate_oracle_score_gain_over_selected": null,
69
+ "candidate_oracle_unique_count": null,
70
+ "candidate_oracle_improvement_rate": null,
71
+ "candidate_oracle_selected_branch_success_rate": null,
72
+ "candidate_oracle_selected_branch_progress": null,
73
+ "candidate_oracle_type_counts": {},
74
+ "candidate_oracle_best_branch_rank": null,
75
+ "candidate_oracle_best_branch_rank_counts": {},
76
+ "candidate_oracle_branch_success_rates": [],
77
+ "candidate_oracle_branch_progress": [],
78
+ "candidate_oracle_branch_score_gains_over_selected": [],
79
+ "selected_residual_scale_counts": {},
80
+ "policy_rollout_success_rate": 0.2921739130434783,
81
+ "policy_rollout_progress": 0.5449448479031739,
82
+ "oracle_success_rate": 0.8573913043478261,
83
+ "action_mse_to_best": 0.42280645585758103,
84
+ "best_policy_val": {
85
+ "bc_loss": 0.1345199979841709,
86
+ "field_effect_loss": 0.009223711728635762,
87
+ "field_potential_loss": 0.2614195512400733,
88
+ "field_preference_loss": 0.5129645930396186,
89
+ "lattice_edges": 3833.3333333333335,
90
+ "progress_mae": 0.18681737035512924,
91
+ "proposal_loss": 0.15201550225416818,
92
+ "rank_acc": 0.8316339155038198,
93
+ "rank_loss": 0.5129594521390067,
94
+ "regret_mae": 0.3556148376729753,
95
+ "success_accuracy": 0.8784897228082021,
96
+ "total_loss": 1.8417018850644429
97
+ },
98
+ "per_task": {
99
+ "LiftPegUpright-v1": {
100
+ "action_mse_to_best": 0.32257138143687214,
101
+ "expert_success_rate": 0.8865979381443299,
102
+ "num_groups": 97,
103
+ "oracle_success_rate": 0.9278350515463918,
104
+ "policy_expert_regret": 1.0754052938259755,
105
+ "policy_oracle_regret": 1.115042612417457,
106
+ "policy_rollout_progress": 0.6258305931828686,
107
+ "policy_rollout_success_rate": 0.17525773195876287,
108
+ "restore_max_error": 2.384185791015625e-07
109
+ },
110
+ "PickCube-v1": {
111
+ "action_mse_to_best": 0.34170422394464156,
112
+ "expert_success_rate": 0.9375,
113
+ "num_groups": 208,
114
+ "oracle_success_rate": 0.9471153846153846,
115
+ "policy_expert_regret": 1.1250538387192557,
116
+ "policy_oracle_regret": 1.1317204127732952,
117
+ "policy_rollout_progress": 0.5715542031953541,
118
+ "policy_rollout_success_rate": 0.23557692307692307,
119
+ "restore_max_error": 2.384185791015625e-07
120
+ },
121
+ "PullCube-v1": {
122
+ "action_mse_to_best": 0.6626108671332915,
123
+ "expert_success_rate": 0.19480519480519481,
124
+ "num_groups": 77,
125
+ "oracle_success_rate": 0.36363636363636365,
126
+ "policy_expert_regret": 0.2609970703866188,
127
+ "policy_oracle_regret": 0.5117309383515801,
128
+ "policy_rollout_progress": 0.28349948870380975,
129
+ "policy_rollout_success_rate": 0.14285714285714285,
130
+ "restore_max_error": 2.3096799850463867e-07
131
+ },
132
+ "PushCube-v1": {
133
+ "action_mse_to_best": 0.45357558640473067,
134
+ "expert_success_rate": 0.8617021276595744,
135
+ "num_groups": 94,
136
+ "oracle_success_rate": 0.9893617021276596,
137
+ "policy_expert_regret": 0.6155937510602018,
138
+ "policy_oracle_regret": 0.7127198087091141,
139
+ "policy_rollout_progress": 0.6596206168228007,
140
+ "policy_rollout_success_rate": 0.6276595744680851,
141
+ "restore_max_error": 2.384185791015625e-07
142
+ },
143
+ "StackCube-v1": {
144
+ "action_mse_to_best": 0.47568351158524175,
145
+ "expert_success_rate": 0.6767676767676768,
146
+ "num_groups": 99,
147
+ "oracle_success_rate": 0.8585858585858586,
148
+ "policy_expert_regret": 0.8507854416213855,
149
+ "policy_oracle_regret": 1.016750856631934,
150
+ "policy_rollout_progress": 0.5042489609031966,
151
+ "policy_rollout_success_rate": 0.32323232323232326,
152
+ "restore_max_error": 2.384185791015625e-07
153
+ }
154
+ }
155
+ },
156
+ {
157
+ "seed": 1,
158
+ "path": "/scratch/knguy52/dovla/experiments/dovla_h16_policy_ckpt_runs/near_miss_policy_bc5_typedprop_p2/seed_1/policy_rollout_proposal_lattice_types2sparse_bestpolicy_prepend_margin0p05.json",
159
+ "num_groups": 575,
160
+ "selection_mode": "proposal_lattice",
161
+ "num_candidates": 3,
162
+ "candidate_sigma": 0.0,
163
+ "proposal_types": [
164
+ "expert",
165
+ "near_miss",
166
+ "no_op",
167
+ "wrong_gripper",
168
+ "wrong_direction",
169
+ "random_negative"
170
+ ],
171
+ "proposal_lattice_types": [
172
+ "no_op",
173
+ "wrong_gripper"
174
+ ],
175
+ "selection_margin": 0.05,
176
+ "prepend_policy_candidate": true,
177
+ "field_optim_steps": 0,
178
+ "field_optim_step_size": 0.0,
179
+ "field_optim_trust_radius": 0.0,
180
+ "field_optim_l2_penalty": 0.0,
181
+ "retrieval_neighbors": 0,
182
+ "retrieval_metric": "none",
183
+ "retrieval_type_min_success": 0.0,
184
+ "retrieval_type_success_bonus_scale": 0.0,
185
+ "retrieval_residual_consensus_penalty_scale": 0.0,
186
+ "retrieval_residual_min_source_progress": 0.0,
187
+ "retrieval_residual_min_source_advantage": -1000000000.0,
188
+ "retrieval_residual_source_progress_bonus_scale": 0.0,
189
+ "retrieval_residual_source_score_bonus_scale": 0.0,
190
+ "retrieval_residual_source_advantage_bonus_scale": 0.0,
191
+ "retrieval_residual_composite_l2_penalty_scale": 0.0,
192
+ "retrieval_residual_scale": 0.0,
193
+ "retrieval_residual_scales": [],
194
+ "retrieval_residual_anchor": "none",
195
+ "retrieval_residual_direction": "none",
196
+ "retrieval_residual_reduce": "none",
197
+ "retrieval_residual_challenger_types": [],
198
+ "retrieval_residual_challenger_scales": [],
199
+ "retrieval_residual_challenger_margin": 0.0,
200
+ "lattice_exclude_types": [],
201
+ "candidate_type_bonuses": {},
202
+ "candidate_type_bonus_components": false,
203
+ "candidate_oracle_rollouts": 0,
204
+ "candidate_oracle_unique_tolerance": 1e-06,
205
+ "candidate_oracle_success_rate": null,
206
+ "candidate_oracle_progress": null,
207
+ "candidate_oracle_score": null,
208
+ "candidate_oracle_regret": null,
209
+ "candidate_oracle_score_gain_over_selected": null,
210
+ "candidate_oracle_unique_count": null,
211
+ "candidate_oracle_improvement_rate": null,
212
+ "candidate_oracle_selected_branch_success_rate": null,
213
+ "candidate_oracle_selected_branch_progress": null,
214
+ "candidate_oracle_type_counts": {},
215
+ "candidate_oracle_best_branch_rank": null,
216
+ "candidate_oracle_best_branch_rank_counts": {},
217
+ "candidate_oracle_branch_success_rates": [],
218
+ "candidate_oracle_branch_progress": [],
219
+ "candidate_oracle_branch_score_gains_over_selected": [],
220
+ "selected_residual_scale_counts": {},
221
+ "policy_rollout_success_rate": 0.3634782608695652,
222
+ "policy_rollout_progress": 0.5814847025639661,
223
+ "oracle_success_rate": 0.8695652173913043,
224
+ "action_mse_to_best": 0.44845469078160416,
225
+ "best_policy_val": {
226
+ "bc_loss": 0.12785478805502257,
227
+ "field_effect_loss": 0.010064840265032318,
228
+ "field_potential_loss": 0.29153423342439866,
229
+ "field_preference_loss": 0.5177145335409377,
230
+ "lattice_edges": 3833.3333333333335,
231
+ "progress_mae": 0.19837388147910437,
232
+ "proposal_loss": 0.14544310627712143,
233
+ "rank_acc": 0.8285107778178321,
234
+ "rank_loss": 0.5177104257875018,
235
+ "regret_mae": 0.37083542181385887,
236
+ "success_accuracy": 0.8705372114976248,
237
+ "total_loss": 1.8337699174880981
238
+ },
239
+ "per_task": {
240
+ "LiftPegUpright-v1": {
241
+ "action_mse_to_best": 0.3441940104687768,
242
+ "expert_success_rate": 0.8584070796460177,
243
+ "num_groups": 113,
244
+ "oracle_success_rate": 0.9380530973451328,
245
+ "policy_expert_regret": 0.9816829663726081,
246
+ "policy_oracle_regret": 1.0631181517269759,
247
+ "policy_rollout_progress": 0.6100037093974847,
248
+ "policy_rollout_success_rate": 0.24778761061946902,
249
+ "restore_max_error": 2.384185791015625e-07
250
+ },
251
+ "PickCube-v1": {
252
+ "action_mse_to_best": 0.4430281473655866,
253
+ "expert_success_rate": 0.9402173913043478,
254
+ "num_groups": 184,
255
+ "oracle_success_rate": 0.9456521739130435,
256
+ "policy_expert_regret": 0.9318571271445684,
257
+ "policy_oracle_regret": 0.9318811316194985,
258
+ "policy_rollout_progress": 0.6274347850172173,
259
+ "policy_rollout_success_rate": 0.3641304347826087,
260
+ "restore_max_error": 2.384185791015625e-07
261
+ },
262
+ "PullCube-v1": {
263
+ "action_mse_to_best": 0.5671229521747017,
264
+ "expert_success_rate": 0.25,
265
+ "num_groups": 76,
266
+ "oracle_success_rate": 0.40789473684210525,
267
+ "policy_expert_regret": 0.32581941247305,
268
+ "policy_oracle_regret": 0.5347183911240121,
269
+ "policy_rollout_progress": 0.29484581164893153,
270
+ "policy_rollout_success_rate": 0.17105263157894737,
271
+ "restore_max_error": 2.384185791015625e-07
272
+ },
273
+ "PushCube-v1": {
274
+ "action_mse_to_best": 0.4154663013109753,
275
+ "expert_success_rate": 0.8198198198198198,
276
+ "num_groups": 111,
277
+ "oracle_success_rate": 1.0,
278
+ "policy_expert_regret": 0.3666651953716536,
279
+ "policy_oracle_regret": 0.4502773642002999,
280
+ "policy_rollout_progress": 0.7839568700339343,
281
+ "policy_rollout_success_rate": 0.7657657657657657,
282
+ "restore_max_error": 2.3795291781425476e-07
283
+ },
284
+ "StackCube-v1": {
285
+ "action_mse_to_best": 0.5300246273668913,
286
+ "expert_success_rate": 0.6923076923076923,
287
+ "num_groups": 91,
288
+ "oracle_success_rate": 0.8571428571428571,
289
+ "policy_expert_regret": 1.0455002242719733,
290
+ "policy_oracle_regret": 1.2049122074475656,
291
+ "policy_rollout_progress": 0.44558011110012347,
292
+ "policy_rollout_success_rate": 0.17582417582417584,
293
+ "restore_max_error": 1.9371509552001953e-07
294
+ }
295
+ }
296
+ },
297
+ {
298
+ "seed": 2,
299
+ "path": "/scratch/knguy52/dovla/experiments/dovla_h16_policy_ckpt_runs/near_miss_policy_bc5_typedprop_p2/seed_2/policy_rollout_proposal_lattice_types2sparse_bestpolicy_prepend_margin0p05.json",
300
+ "num_groups": 575,
301
+ "selection_mode": "proposal_lattice",
302
+ "num_candidates": 3,
303
+ "candidate_sigma": 0.0,
304
+ "proposal_types": [
305
+ "expert",
306
+ "near_miss",
307
+ "no_op",
308
+ "wrong_gripper",
309
+ "wrong_direction",
310
+ "random_negative"
311
+ ],
312
+ "proposal_lattice_types": [
313
+ "no_op",
314
+ "wrong_gripper"
315
+ ],
316
+ "selection_margin": 0.05,
317
+ "prepend_policy_candidate": true,
318
+ "field_optim_steps": 0,
319
+ "field_optim_step_size": 0.0,
320
+ "field_optim_trust_radius": 0.0,
321
+ "field_optim_l2_penalty": 0.0,
322
+ "retrieval_neighbors": 0,
323
+ "retrieval_metric": "none",
324
+ "retrieval_type_min_success": 0.0,
325
+ "retrieval_type_success_bonus_scale": 0.0,
326
+ "retrieval_residual_consensus_penalty_scale": 0.0,
327
+ "retrieval_residual_min_source_progress": 0.0,
328
+ "retrieval_residual_min_source_advantage": -1000000000.0,
329
+ "retrieval_residual_source_progress_bonus_scale": 0.0,
330
+ "retrieval_residual_source_score_bonus_scale": 0.0,
331
+ "retrieval_residual_source_advantage_bonus_scale": 0.0,
332
+ "retrieval_residual_composite_l2_penalty_scale": 0.0,
333
+ "retrieval_residual_scale": 0.0,
334
+ "retrieval_residual_scales": [],
335
+ "retrieval_residual_anchor": "none",
336
+ "retrieval_residual_direction": "none",
337
+ "retrieval_residual_reduce": "none",
338
+ "retrieval_residual_challenger_types": [],
339
+ "retrieval_residual_challenger_scales": [],
340
+ "retrieval_residual_challenger_margin": 0.0,
341
+ "lattice_exclude_types": [],
342
+ "candidate_type_bonuses": {},
343
+ "candidate_type_bonus_components": false,
344
+ "candidate_oracle_rollouts": 0,
345
+ "candidate_oracle_unique_tolerance": 1e-06,
346
+ "candidate_oracle_success_rate": null,
347
+ "candidate_oracle_progress": null,
348
+ "candidate_oracle_score": null,
349
+ "candidate_oracle_regret": null,
350
+ "candidate_oracle_score_gain_over_selected": null,
351
+ "candidate_oracle_unique_count": null,
352
+ "candidate_oracle_improvement_rate": null,
353
+ "candidate_oracle_selected_branch_success_rate": null,
354
+ "candidate_oracle_selected_branch_progress": null,
355
+ "candidate_oracle_type_counts": {},
356
+ "candidate_oracle_best_branch_rank": null,
357
+ "candidate_oracle_best_branch_rank_counts": {},
358
+ "candidate_oracle_branch_success_rates": [],
359
+ "candidate_oracle_branch_progress": [],
360
+ "candidate_oracle_branch_score_gains_over_selected": [],
361
+ "selected_residual_scale_counts": {},
362
+ "policy_rollout_success_rate": 0.3130434782608696,
363
+ "policy_rollout_progress": 0.5448042644025839,
364
+ "oracle_success_rate": 0.8765217391304347,
365
+ "action_mse_to_best": 0.48210894903734974,
366
+ "best_policy_val": {
367
+ "bc_loss": 0.11429175941480531,
368
+ "field_effect_loss": 0.009631710747877756,
369
+ "field_potential_loss": 0.2724720388650894,
370
+ "field_preference_loss": 0.5087620665629705,
371
+ "lattice_edges": 3833.3333333333335,
372
+ "progress_mae": 0.19653971824381086,
373
+ "proposal_loss": 0.13279944616887304,
374
+ "rank_acc": 0.8339865836832259,
375
+ "rank_loss": 0.508702975180414,
376
+ "regret_mae": 0.3901382436354955,
377
+ "success_accuracy": 0.870691223276986,
378
+ "total_loss": 1.710942678981357
379
+ },
380
+ "per_task": {
381
+ "LiftPegUpright-v1": {
382
+ "action_mse_to_best": 0.34519209470211837,
383
+ "expert_success_rate": 0.8229166666666666,
384
+ "num_groups": 96,
385
+ "oracle_success_rate": 0.9270833333333334,
386
+ "policy_expert_regret": 0.80674880479152,
387
+ "policy_oracle_regret": 0.9008693016755084,
388
+ "policy_rollout_progress": 0.6861427738331258,
389
+ "policy_rollout_success_rate": 0.3333333333333333,
390
+ "restore_max_error": 1.955777406692505e-07
391
+ },
392
+ "PickCube-v1": {
393
+ "action_mse_to_best": 0.41532564817603224,
394
+ "expert_success_rate": 0.9444444444444444,
395
+ "num_groups": 198,
396
+ "oracle_success_rate": 0.9595959595959596,
397
+ "policy_expert_regret": 0.9896244406841244,
398
+ "policy_oracle_regret": 0.9967078912128328,
399
+ "policy_rollout_progress": 0.6206366271392038,
400
+ "policy_rollout_success_rate": 0.32323232323232326,
401
+ "restore_max_error": 2.384185791015625e-07
402
+ },
403
+ "PullCube-v1": {
404
+ "action_mse_to_best": 0.6752623427736884,
405
+ "expert_success_rate": 0.24444444444444444,
406
+ "num_groups": 90,
407
+ "oracle_success_rate": 0.4666666666666667,
408
+ "policy_expert_regret": 0.3316968430040611,
409
+ "policy_oracle_regret": 0.6424627531112896,
410
+ "policy_rollout_progress": 0.2952031936703457,
411
+ "policy_rollout_success_rate": 0.15555555555555556,
412
+ "restore_max_error": 2.384185791015625e-07
413
+ },
414
+ "PushCube-v1": {
415
+ "action_mse_to_best": 0.4799785439183216,
416
+ "expert_success_rate": 0.8514851485148515,
417
+ "num_groups": 101,
418
+ "oracle_success_rate": 1.0,
419
+ "policy_expert_regret": 0.7253382899383507,
420
+ "policy_oracle_regret": 0.8162076433401296,
421
+ "policy_rollout_progress": 0.609534930917296,
422
+ "policy_rollout_success_rate": 0.5742574257425742,
423
+ "restore_max_error": 2.384185791015625e-07
424
+ },
425
+ "StackCube-v1": {
426
+ "action_mse_to_best": 0.5843142497870657,
427
+ "expert_success_rate": 0.7666666666666667,
428
+ "num_groups": 90,
429
+ "oracle_success_rate": 0.9111111111111111,
430
+ "policy_expert_regret": 1.2081461252437697,
431
+ "policy_oracle_regret": 1.3445107304387622,
432
+ "policy_rollout_progress": 0.40417086796628104,
433
+ "policy_rollout_success_rate": 0.13333333333333333,
434
+ "restore_max_error": 2.2351741790771484e-07
435
+ }
436
+ }
437
+ }
438
+ ]
439
+ }
results/h16_bestpolicy_types2sparse_prepend_margin0p05_summary.md ADDED
@@ -0,0 +1,19 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # h=16 Best-Policy Checkpoint Rollout
2
+
3
+ Run root: `/scratch/knguy52/dovla/experiments/dovla_h16_policy_ckpt_runs`
4
+ Objective: `near_miss_policy_bc5_typedprop_p2`
5
+ Result file: `policy_rollout_proposal_lattice_types2sparse_bestpolicy_prepend_margin0p05.json`
6
+ Completed seeds: 3
7
+ Baseline h=4 policy success: 29.67%
8
+ Baseline h=16 rank-checkpoint success: 29.74%
9
+
10
+ Mean success: 32.29% +/- 3.67%
11
+ Gain vs h=16 rank checkpoint: +2.55%
12
+ Mean progress: 55.71%
13
+ Mean action MSE to best: 0.451
14
+
15
+ | seed | mode | k | policy cand | retrieval K | retrieval metric | residual anchor | residual direction | residual reduce | min type success | type success bonus | consensus penalty | min source progress | source progress bonus | source score bonus | residual scale | residual scales | margin | sigma | opt steps | trust | success | progress | oracle | candidate oracle | oracle gain | action MSE |
16
+ |---:|---|---:|---|---:|---|---|---|---|---:|---:|---:|---:|---:|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
17
+ | 0 | proposal_lattice | 3 | yes | 0 | none | none | none | none | 0.00 | 0.000 | 0.000 | 0.00 | 0.000 | 0.000 | 0.00 | none | 0.050 | 0.00 | 0 | 0.00 | 29.22% | 54.49% | 85.74% | n/a | n/a | 0.423 |
18
+ | 1 | proposal_lattice | 3 | yes | 0 | none | none | none | none | 0.00 | 0.000 | 0.000 | 0.00 | 0.000 | 0.000 | 0.00 | none | 0.050 | 0.00 | 0 | 0.00 | 36.35% | 58.15% | 86.96% | n/a | n/a | 0.448 |
19
+ | 2 | proposal_lattice | 3 | yes | 0 | none | none | none | none | 0.00 | 0.000 | 0.000 | 0.00 | 0.000 | 0.000 | 0.00 | none | 0.050 | 0.00 | 0 | 0.00 | 31.30% | 54.48% | 87.65% | n/a | n/a | 0.482 |