manual-selector-job-sync 2026-07-04T04:36:59Z workspace (part 6)
Browse filesThis view is limited to 50 files because it contains too many changes. See raw diff
- workspace/runs/ctt_base_context_obs_learned_dominance_context_perdim_trainmax_train_to_test/command.txt +1 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_perdim_trainmax_train_to_test/config.yaml +18 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_perdim_trainmax_train_to_test/data_hash.txt +6 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_perdim_trainmax_train_to_test/eval.log +3 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_perdim_trainmax_train_to_test/git_hash.txt +1 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_perdim_trainmax_train_to_test/metrics.json +0 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_perdim_trainmax_train_to_test/metrics_by_seed.json +0 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_perdim_trainmax_train_to_test/metrics_by_task.json +137 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_perdim_trainmax_train_to_test/split_hash.txt +14 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_perdim_trainmax_train_to_test/table.tex +8 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_perdim_trainmax_train_to_test/train.log +5 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_success_perdim_trainmax_train_to_test/command.txt +1 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_success_perdim_trainmax_train_to_test/config.yaml +18 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_success_perdim_trainmax_train_to_test/data_hash.txt +6 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_success_perdim_trainmax_train_to_test/eval.log +3 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_success_perdim_trainmax_train_to_test/git_hash.txt +1 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_success_perdim_trainmax_train_to_test/metrics.json +0 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_success_perdim_trainmax_train_to_test/metrics_by_seed.json +0 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_success_perdim_trainmax_train_to_test/metrics_by_task.json +137 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_success_perdim_trainmax_train_to_test/split_hash.txt +14 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_success_perdim_trainmax_train_to_test/table.tex +8 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_success_perdim_trainmax_train_to_test/train.log +5 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_success_tanh_train_to_test/metrics.json +0 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_success_tanh_train_to_test/metrics_by_seed.json +468 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_success_tanh_train_to_test/metrics_by_task.json +20 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_success_tanh_train_to_test/table.tex +3 -3
- workspace/runs/ctt_base_context_obs_learned_dominance_context_success_tanh_train_to_test/train.log +1 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_perdim_trainmax_train_to_test/command.txt +1 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_perdim_trainmax_train_to_test/config.yaml +18 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_perdim_trainmax_train_to_test/data_hash.txt +6 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_perdim_trainmax_train_to_test/eval.log +3 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_perdim_trainmax_train_to_test/git_hash.txt +1 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_perdim_trainmax_train_to_test/metrics.json +0 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_perdim_trainmax_train_to_test/metrics_by_seed.json +0 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_perdim_trainmax_train_to_test/metrics_by_task.json +137 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_perdim_trainmax_train_to_test/split_hash.txt +14 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_perdim_trainmax_train_to_test/table.tex +8 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_perdim_trainmax_train_to_test/train.log +5 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_success_perdim_trainmax_train_to_test/command.txt +1 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_success_perdim_trainmax_train_to_test/config.yaml +18 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_success_perdim_trainmax_train_to_test/data_hash.txt +6 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_success_perdim_trainmax_train_to_test/eval.log +3 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_success_perdim_trainmax_train_to_test/git_hash.txt +1 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_success_perdim_trainmax_train_to_test/metrics.json +0 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_success_perdim_trainmax_train_to_test/metrics_by_seed.json +0 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_success_perdim_trainmax_train_to_test/metrics_by_task.json +137 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_success_perdim_trainmax_train_to_test/split_hash.txt +14 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_success_perdim_trainmax_train_to_test/table.tex +8 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_success_perdim_trainmax_train_to_test/train.log +5 -0
- workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_success_tanh_train_to_test/command.txt +1 -1
workspace/runs/ctt_base_context_obs_learned_dominance_context_perdim_trainmax_train_to_test/command.txt
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
python scripts/eval_learned_dominance_selector.py --calibration-input runs/ctt_base_context_obs_train_cal_perdim_trainmax_rollout_comparison/combined_measured_candidates.json --calibration-target-index data/cil_charts_rgb_refs/train/index.json --eval-input runs/ctt_base_context_obs_test_perdim_trainmax_rollout_comparison/combined_measured_candidates.json --eval-target-index data/cil_charts_rgb_refs/test/index.json --checkpoint-template runs/ctt_residual_base_context_obs_seed{seed}/model.pt --out-dir runs/ctt_base_context_obs_learned_dominance_context_perdim_trainmax_train_to_test --feature-set context --target utility_margin --k 8 --bootstrap-samples 1000 --no-markdown-report
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_perdim_trainmax_train_to_test/config.yaml
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
bootstrap_samples: 1000
|
| 2 |
+
calibration_input: runs/ctt_base_context_obs_train_cal_perdim_trainmax_rollout_comparison/combined_measured_candidates.json
|
| 3 |
+
calibration_target_index: data/cil_charts_rgb_refs/train/index.json
|
| 4 |
+
checkpoint_template: runs/ctt_residual_base_context_obs_seed{seed}/model.pt
|
| 5 |
+
eval_input: runs/ctt_base_context_obs_test_perdim_trainmax_rollout_comparison/combined_measured_candidates.json
|
| 6 |
+
eval_target_index: data/cil_charts_rgb_refs/test/index.json
|
| 7 |
+
feature_set: context
|
| 8 |
+
fit_objective: pointwise
|
| 9 |
+
k: 8
|
| 10 |
+
no_markdown_report: True
|
| 11 |
+
out_dir: runs/ctt_base_context_obs_learned_dominance_context_perdim_trainmax_train_to_test
|
| 12 |
+
pairwise_weight: 1.0
|
| 13 |
+
ridge_lambdas: 0,0.01,0.1,1,10,100
|
| 14 |
+
selector_chart_feature_mode: base_context_obs_obj
|
| 15 |
+
source_index: None
|
| 16 |
+
success_bonus: 1.0
|
| 17 |
+
target: utility_margin
|
| 18 |
+
threshold_scope: global
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_perdim_trainmax_train_to_test/data_hash.txt
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"calibration_input": "130d86b8c5ba5dc901ee48835454693bf42c361803c01f6b2d80590cb2493491",
|
| 3 |
+
"calibration_target_index": "67973c50072aa87a3b97533b6292b08a8b4d88a95571dc22e91b6e8fd19e0d85",
|
| 4 |
+
"eval_input": "88d4bfe8c95f8d1579cbe59096cf665dc5f2dfdd99481a386a25e1f3cfa2b09d",
|
| 5 |
+
"eval_target_index": "dee1b5225485df6624747dd4c5074bcf514685b54017d05e698d2051638fbf36"
|
| 6 |
+
}
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_perdim_trainmax_train_to_test/eval.log
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
evaluated learned fallback rule on held-out measured rollout rows
|
| 2 |
+
eval_input=runs/ctt_base_context_obs_test_perdim_trainmax_rollout_comparison/combined_measured_candidates.json
|
| 3 |
+
num_eval_rows=144
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_perdim_trainmax_train_to_test/git_hash.txt
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
14e81292e2004925f81e0bb4810f6f144bfd87b7
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_perdim_trainmax_train_to_test/metrics.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_perdim_trainmax_train_to_test/metrics_by_seed.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_perdim_trainmax_train_to_test/metrics_by_task.json
ADDED
|
@@ -0,0 +1,137 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"LiftPegUpright-v1": {
|
| 3 |
+
"base_success": 0.0,
|
| 4 |
+
"base_utility": 0.4796089559793472,
|
| 5 |
+
"coverage": 0.8,
|
| 6 |
+
"execute_generated": 0.8,
|
| 7 |
+
"fallback_rate": 0.2,
|
| 8 |
+
"hidden_chart_oracle_success": 1.0,
|
| 9 |
+
"hidden_chart_oracle_utility": 2.0,
|
| 10 |
+
"outcome_ptr": 0.3333333333333333,
|
| 11 |
+
"pairwise_causal_calibration_accuracy": 0.6304232804232804,
|
| 12 |
+
"pairwise_causal_calibration_confidence": 0.5135220089933253,
|
| 13 |
+
"pairwise_causal_calibration_ece": 0.1381376537441502,
|
| 14 |
+
"pairwise_causal_calibration_pairs": 27.933333333333334,
|
| 15 |
+
"predicted_margin": -0.02998554890632619,
|
| 16 |
+
"proposal_oracle_success": 0.13333333333333333,
|
| 17 |
+
"proposal_oracle_utility": 0.6601470321416855,
|
| 18 |
+
"selected_candidate_index": 4.266666666666667,
|
| 19 |
+
"selected_success": 0.0,
|
| 20 |
+
"selected_success_gain_over_base": 0.0,
|
| 21 |
+
"selected_utility": 0.475364543000857,
|
| 22 |
+
"selected_utility_gain_over_base": -0.004244412978490194,
|
| 23 |
+
"selector_regret": 0.18489418029785157,
|
| 24 |
+
"success_selector_gap": 0.13333333333333333,
|
| 25 |
+
"success_support_gap": 0.8666666666666667,
|
| 26 |
+
"support_gap": 1.3398529678583144,
|
| 27 |
+
"tau": -0.07465520951292606
|
| 28 |
+
},
|
| 29 |
+
"PickCube-v1": {
|
| 30 |
+
"base_success": 1.0,
|
| 31 |
+
"base_utility": 2.0,
|
| 32 |
+
"coverage": 0.0,
|
| 33 |
+
"execute_generated": 0.0,
|
| 34 |
+
"fallback_rate": 1.0,
|
| 35 |
+
"hidden_chart_oracle_success": 1.0,
|
| 36 |
+
"hidden_chart_oracle_utility": 2.0,
|
| 37 |
+
"outcome_ptr": 0.0,
|
| 38 |
+
"pairwise_causal_calibration_accuracy": 0.44047619047619047,
|
| 39 |
+
"pairwise_causal_calibration_confidence": 0.5226564402213808,
|
| 40 |
+
"pairwise_causal_calibration_ece": 0.13232755397230822,
|
| 41 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 42 |
+
"predicted_margin": -0.3700514399915611,
|
| 43 |
+
"proposal_oracle_success": 0.3333333333333333,
|
| 44 |
+
"proposal_oracle_utility": 1.021850009759267,
|
| 45 |
+
"selected_candidate_index": 4.666666666666667,
|
| 46 |
+
"selected_success": 1.0,
|
| 47 |
+
"selected_success_gain_over_base": 0.0,
|
| 48 |
+
"selected_utility": 2.0,
|
| 49 |
+
"selected_utility_gain_over_base": 0.0,
|
| 50 |
+
"selector_regret": 0.0,
|
| 51 |
+
"success_selector_gap": 0.0,
|
| 52 |
+
"success_support_gap": 0.6666666666666666,
|
| 53 |
+
"support_gap": 0.9781499902407328,
|
| 54 |
+
"tau": -0.07465520951292609
|
| 55 |
+
},
|
| 56 |
+
"PullCube-v1": {
|
| 57 |
+
"base_success": 0.36,
|
| 58 |
+
"base_utility": 0.8032506120204925,
|
| 59 |
+
"coverage": 0.4533333333333333,
|
| 60 |
+
"execute_generated": 0.4533333333333333,
|
| 61 |
+
"fallback_rate": 0.5466666666666666,
|
| 62 |
+
"hidden_chart_oracle_success": 0.52,
|
| 63 |
+
"hidden_chart_oracle_utility": 1.1467878860235214,
|
| 64 |
+
"outcome_ptr": 0.32,
|
| 65 |
+
"pairwise_causal_calibration_accuracy": 0.5311771561771562,
|
| 66 |
+
"pairwise_causal_calibration_confidence": 0.5130368946827227,
|
| 67 |
+
"pairwise_causal_calibration_ece": 0.15360683599994118,
|
| 68 |
+
"pairwise_causal_calibration_pairs": 24.16,
|
| 69 |
+
"predicted_margin": -0.12701778822001247,
|
| 70 |
+
"proposal_oracle_success": 0.17333333333333334,
|
| 71 |
+
"proposal_oracle_utility": 0.46170220340291657,
|
| 72 |
+
"selected_candidate_index": 2.6266666666666665,
|
| 73 |
+
"selected_success": 0.36,
|
| 74 |
+
"selected_success_gain_over_base": 0.0,
|
| 75 |
+
"selected_utility": 0.7987058012435834,
|
| 76 |
+
"selected_utility_gain_over_base": -0.004544810776909192,
|
| 77 |
+
"selector_regret": 0.02149127927919229,
|
| 78 |
+
"success_selector_gap": 0.0,
|
| 79 |
+
"success_support_gap": 0.3466666666666667,
|
| 80 |
+
"support_gap": 0.6881442584097386,
|
| 81 |
+
"tau": -0.07465520951292605
|
| 82 |
+
},
|
| 83 |
+
"PushCube-v1": {
|
| 84 |
+
"base_success": 0.8333333333333334,
|
| 85 |
+
"base_utility": 1.6734557673335075,
|
| 86 |
+
"coverage": 0.0,
|
| 87 |
+
"execute_generated": 0.0,
|
| 88 |
+
"fallback_rate": 1.0,
|
| 89 |
+
"hidden_chart_oracle_success": 0.8333333333333334,
|
| 90 |
+
"hidden_chart_oracle_utility": 1.7015800600250561,
|
| 91 |
+
"outcome_ptr": 0.16666666666666666,
|
| 92 |
+
"pairwise_causal_calibration_accuracy": 0.5866721078485784,
|
| 93 |
+
"pairwise_causal_calibration_confidence": 0.5094999341420311,
|
| 94 |
+
"pairwise_causal_calibration_ece": 0.1509193203080439,
|
| 95 |
+
"pairwise_causal_calibration_pairs": 23.833333333333332,
|
| 96 |
+
"predicted_margin": -1.0899306043385468,
|
| 97 |
+
"proposal_oracle_success": 0.5555555555555556,
|
| 98 |
+
"proposal_oracle_utility": 1.1790093589160178,
|
| 99 |
+
"selected_candidate_index": 2.5555555555555554,
|
| 100 |
+
"selected_success": 0.8333333333333334,
|
| 101 |
+
"selected_success_gain_over_base": 0.0,
|
| 102 |
+
"selected_utility": 1.6734557673335075,
|
| 103 |
+
"selected_utility_gain_over_base": 0.0,
|
| 104 |
+
"selector_regret": 0.32654423266649246,
|
| 105 |
+
"success_selector_gap": 0.16666666666666666,
|
| 106 |
+
"success_support_gap": 0.3333333333333333,
|
| 107 |
+
"support_gap": 0.6220440144340197,
|
| 108 |
+
"tau": -0.07465520951292606
|
| 109 |
+
},
|
| 110 |
+
"StackCube-v1": {
|
| 111 |
+
"base_success": 0.09090909090909091,
|
| 112 |
+
"base_utility": 0.5003613992170854,
|
| 113 |
+
"coverage": 0.45454545454545453,
|
| 114 |
+
"execute_generated": 0.45454545454545453,
|
| 115 |
+
"fallback_rate": 0.5454545454545454,
|
| 116 |
+
"hidden_chart_oracle_success": 1.0,
|
| 117 |
+
"hidden_chart_oracle_utility": 2.0,
|
| 118 |
+
"outcome_ptr": 0.6060606060606061,
|
| 119 |
+
"pairwise_causal_calibration_accuracy": 0.5222077922077923,
|
| 120 |
+
"pairwise_causal_calibration_confidence": 0.5154809843682674,
|
| 121 |
+
"pairwise_causal_calibration_ece": 0.14110861032541971,
|
| 122 |
+
"pairwise_causal_calibration_pairs": 27.818181818181817,
|
| 123 |
+
"predicted_margin": -0.09559995857533575,
|
| 124 |
+
"proposal_oracle_success": 0.18181818181818182,
|
| 125 |
+
"proposal_oracle_utility": 0.634040683056369,
|
| 126 |
+
"selected_candidate_index": 3.0,
|
| 127 |
+
"selected_success": 0.09090909090909091,
|
| 128 |
+
"selected_success_gain_over_base": 0.0,
|
| 129 |
+
"selected_utility": 0.4970153953992959,
|
| 130 |
+
"selected_utility_gain_over_base": -0.003346003817789482,
|
| 131 |
+
"selector_regret": 0.18124081239555823,
|
| 132 |
+
"success_selector_gap": 0.12121212121212122,
|
| 133 |
+
"success_support_gap": 0.8181818181818182,
|
| 134 |
+
"support_gap": 1.365959316943631,
|
| 135 |
+
"tau": -0.07465520951292605
|
| 136 |
+
}
|
| 137 |
+
}
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_perdim_trainmax_train_to_test/split_hash.txt
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"calibration_target_index": {
|
| 3 |
+
"content_hash": "67f7b4a692f7c7e71da24378dc71e3399c5d35afc14c494869eba6087b765c42",
|
| 4 |
+
"retrieval_index_allowed": true,
|
| 5 |
+
"split": "train",
|
| 6 |
+
"split_hash": "867a94a1c67686c082765671a7bf8d74a9d633085e1b2619c83197c60bf58372"
|
| 7 |
+
},
|
| 8 |
+
"eval_target_index": {
|
| 9 |
+
"content_hash": "0c38d73dc9bc4ab756e12006671b5d12f1a23358b281e54b3f6a9ae3cbf6e9af",
|
| 10 |
+
"retrieval_index_allowed": false,
|
| 11 |
+
"split": "test",
|
| 12 |
+
"split_hash": "84151df9e88c219b76751677558cc5bbbc19082bf358cea7330158e9e403e25f"
|
| 13 |
+
}
|
| 14 |
+
}
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_perdim_trainmax_train_to_test/table.tex
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
% Auto-generated by scripts/eval_learned_dominance_selector.py
|
| 2 |
+
\begin{tabular}{lrrrrrrrrr}
|
| 3 |
+
\toprule
|
| 4 |
+
Rows & Coverage & Fallback & Base succ. & Selected succ. & Oracle succ. & OutcomePTR & Succ. support gap & Succ. selector gap & Cal. ECE \\
|
| 5 |
+
\midrule
|
| 6 |
+
144 & 0.4236 & 0.5764 & 0.3333 & 0.3333 & 0.2222 & 0.3611 & 0.5139 & 0.0625 & 0.0344 \\
|
| 7 |
+
\bottomrule
|
| 8 |
+
\end{tabular}
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_perdim_trainmax_train_to_test/train.log
ADDED
|
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
trained ridge dominance calibrator on calibration measured rows only
|
| 2 |
+
fit_objective=pointwise
|
| 3 |
+
selected_lambda=0.0
|
| 4 |
+
tau=-0.074655
|
| 5 |
+
calibration_input=runs/ctt_base_context_obs_train_cal_perdim_trainmax_rollout_comparison/combined_measured_candidates.json
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_success_perdim_trainmax_train_to_test/command.txt
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
python scripts/eval_learned_dominance_selector.py --calibration-input runs/ctt_base_context_obs_train_cal_perdim_trainmax_rollout_comparison/combined_measured_candidates.json --calibration-target-index data/cil_charts_rgb_refs/train/index.json --eval-input runs/ctt_base_context_obs_test_perdim_trainmax_rollout_comparison/combined_measured_candidates.json --eval-target-index data/cil_charts_rgb_refs/test/index.json --checkpoint-template runs/ctt_residual_base_context_obs_seed{seed}/model.pt --out-dir runs/ctt_base_context_obs_learned_dominance_context_success_perdim_trainmax_train_to_test --feature-set context --target success --k 8 --bootstrap-samples 1000 --no-markdown-report
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_success_perdim_trainmax_train_to_test/config.yaml
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
bootstrap_samples: 1000
|
| 2 |
+
calibration_input: runs/ctt_base_context_obs_train_cal_perdim_trainmax_rollout_comparison/combined_measured_candidates.json
|
| 3 |
+
calibration_target_index: data/cil_charts_rgb_refs/train/index.json
|
| 4 |
+
checkpoint_template: runs/ctt_residual_base_context_obs_seed{seed}/model.pt
|
| 5 |
+
eval_input: runs/ctt_base_context_obs_test_perdim_trainmax_rollout_comparison/combined_measured_candidates.json
|
| 6 |
+
eval_target_index: data/cil_charts_rgb_refs/test/index.json
|
| 7 |
+
feature_set: context
|
| 8 |
+
fit_objective: pointwise
|
| 9 |
+
k: 8
|
| 10 |
+
no_markdown_report: True
|
| 11 |
+
out_dir: runs/ctt_base_context_obs_learned_dominance_context_success_perdim_trainmax_train_to_test
|
| 12 |
+
pairwise_weight: 1.0
|
| 13 |
+
ridge_lambdas: 0,0.01,0.1,1,10,100
|
| 14 |
+
selector_chart_feature_mode: base_context_obs_obj
|
| 15 |
+
source_index: None
|
| 16 |
+
success_bonus: 1.0
|
| 17 |
+
target: success
|
| 18 |
+
threshold_scope: global
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_success_perdim_trainmax_train_to_test/data_hash.txt
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"calibration_input": "130d86b8c5ba5dc901ee48835454693bf42c361803c01f6b2d80590cb2493491",
|
| 3 |
+
"calibration_target_index": "67973c50072aa87a3b97533b6292b08a8b4d88a95571dc22e91b6e8fd19e0d85",
|
| 4 |
+
"eval_input": "88d4bfe8c95f8d1579cbe59096cf665dc5f2dfdd99481a386a25e1f3cfa2b09d",
|
| 5 |
+
"eval_target_index": "dee1b5225485df6624747dd4c5074bcf514685b54017d05e698d2051638fbf36"
|
| 6 |
+
}
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_success_perdim_trainmax_train_to_test/eval.log
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
evaluated learned fallback rule on held-out measured rollout rows
|
| 2 |
+
eval_input=runs/ctt_base_context_obs_test_perdim_trainmax_rollout_comparison/combined_measured_candidates.json
|
| 3 |
+
num_eval_rows=144
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_success_perdim_trainmax_train_to_test/git_hash.txt
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
14e81292e2004925f81e0bb4810f6f144bfd87b7
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_success_perdim_trainmax_train_to_test/metrics.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_success_perdim_trainmax_train_to_test/metrics_by_seed.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_success_perdim_trainmax_train_to_test/metrics_by_task.json
ADDED
|
@@ -0,0 +1,137 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"LiftPegUpright-v1": {
|
| 3 |
+
"base_success": 0.0,
|
| 4 |
+
"base_utility": 0.4796089559793472,
|
| 5 |
+
"coverage": 0.0,
|
| 6 |
+
"execute_generated": 0.0,
|
| 7 |
+
"fallback_rate": 1.0,
|
| 8 |
+
"hidden_chart_oracle_success": 1.0,
|
| 9 |
+
"hidden_chart_oracle_utility": 2.0,
|
| 10 |
+
"outcome_ptr": 0.3333333333333333,
|
| 11 |
+
"pairwise_causal_calibration_accuracy": 0.4343915343915344,
|
| 12 |
+
"pairwise_causal_calibration_confidence": 0.5071639936503661,
|
| 13 |
+
"pairwise_causal_calibration_ece": 0.12725711962690892,
|
| 14 |
+
"pairwise_causal_calibration_pairs": 27.933333333333334,
|
| 15 |
+
"predicted_margin": 0.13553303703087385,
|
| 16 |
+
"proposal_oracle_success": 0.13333333333333333,
|
| 17 |
+
"proposal_oracle_utility": 0.6601470321416855,
|
| 18 |
+
"selected_candidate_index": 2.8,
|
| 19 |
+
"selected_success": 0.0,
|
| 20 |
+
"selected_success_gain_over_base": 0.0,
|
| 21 |
+
"selected_utility": 0.4796089559793472,
|
| 22 |
+
"selected_utility_gain_over_base": 0.0,
|
| 23 |
+
"selector_regret": 0.18421521186828613,
|
| 24 |
+
"success_selector_gap": 0.13333333333333333,
|
| 25 |
+
"success_support_gap": 0.8666666666666667,
|
| 26 |
+
"support_gap": 1.3398529678583144,
|
| 27 |
+
"tau": 0.41382592280226843
|
| 28 |
+
},
|
| 29 |
+
"PickCube-v1": {
|
| 30 |
+
"base_success": 1.0,
|
| 31 |
+
"base_utility": 2.0,
|
| 32 |
+
"coverage": 0.3333333333333333,
|
| 33 |
+
"execute_generated": 0.3333333333333333,
|
| 34 |
+
"fallback_rate": 0.6666666666666666,
|
| 35 |
+
"hidden_chart_oracle_success": 1.0,
|
| 36 |
+
"hidden_chart_oracle_utility": 2.0,
|
| 37 |
+
"outcome_ptr": 0.0,
|
| 38 |
+
"pairwise_causal_calibration_accuracy": 0.35714285714285715,
|
| 39 |
+
"pairwise_causal_calibration_confidence": 0.5200052095470546,
|
| 40 |
+
"pairwise_causal_calibration_ece": 0.1628623524041974,
|
| 41 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 42 |
+
"predicted_margin": 0.41863597673713104,
|
| 43 |
+
"proposal_oracle_success": 0.3333333333333333,
|
| 44 |
+
"proposal_oracle_utility": 1.021850009759267,
|
| 45 |
+
"selected_candidate_index": 3.3333333333333335,
|
| 46 |
+
"selected_success": 0.6666666666666666,
|
| 47 |
+
"selected_success_gain_over_base": -0.3333333333333333,
|
| 48 |
+
"selected_utility": 1.3781749804814656,
|
| 49 |
+
"selected_utility_gain_over_base": -0.6218250195185343,
|
| 50 |
+
"selector_regret": 0.129598339398702,
|
| 51 |
+
"success_selector_gap": 0.0,
|
| 52 |
+
"success_support_gap": 0.6666666666666666,
|
| 53 |
+
"support_gap": 0.9781499902407328,
|
| 54 |
+
"tau": 0.41382592280226843
|
| 55 |
+
},
|
| 56 |
+
"PullCube-v1": {
|
| 57 |
+
"base_success": 0.36,
|
| 58 |
+
"base_utility": 0.8032506120204925,
|
| 59 |
+
"coverage": 0.0,
|
| 60 |
+
"execute_generated": 0.0,
|
| 61 |
+
"fallback_rate": 1.0,
|
| 62 |
+
"hidden_chart_oracle_success": 0.52,
|
| 63 |
+
"hidden_chart_oracle_utility": 1.1467878860235214,
|
| 64 |
+
"outcome_ptr": 0.32,
|
| 65 |
+
"pairwise_causal_calibration_accuracy": 0.48276723276723277,
|
| 66 |
+
"pairwise_causal_calibration_confidence": 0.510716401449069,
|
| 67 |
+
"pairwise_causal_calibration_ece": 0.10869460089822837,
|
| 68 |
+
"pairwise_causal_calibration_pairs": 24.16,
|
| 69 |
+
"predicted_margin": 0.13437515079559892,
|
| 70 |
+
"proposal_oracle_success": 0.17333333333333334,
|
| 71 |
+
"proposal_oracle_utility": 0.46170220340291657,
|
| 72 |
+
"selected_candidate_index": 3.28,
|
| 73 |
+
"selected_success": 0.36,
|
| 74 |
+
"selected_success_gain_over_base": 0.0,
|
| 75 |
+
"selected_utility": 0.8032506120204925,
|
| 76 |
+
"selected_utility_gain_over_base": 0.0,
|
| 77 |
+
"selector_regret": 0.027702728708585103,
|
| 78 |
+
"success_selector_gap": 0.0,
|
| 79 |
+
"success_support_gap": 0.3466666666666667,
|
| 80 |
+
"support_gap": 0.6881442584097386,
|
| 81 |
+
"tau": 0.4138259228022678
|
| 82 |
+
},
|
| 83 |
+
"PushCube-v1": {
|
| 84 |
+
"base_success": 0.8333333333333334,
|
| 85 |
+
"base_utility": 1.6734557673335075,
|
| 86 |
+
"coverage": 0.0,
|
| 87 |
+
"execute_generated": 0.0,
|
| 88 |
+
"fallback_rate": 1.0,
|
| 89 |
+
"hidden_chart_oracle_success": 0.8333333333333334,
|
| 90 |
+
"hidden_chart_oracle_utility": 1.7015800600250561,
|
| 91 |
+
"outcome_ptr": 0.16666666666666666,
|
| 92 |
+
"pairwise_causal_calibration_accuracy": 0.4821140297610885,
|
| 93 |
+
"pairwise_causal_calibration_confidence": 0.506508150386812,
|
| 94 |
+
"pairwise_causal_calibration_ece": 0.1696532797298677,
|
| 95 |
+
"pairwise_causal_calibration_pairs": 23.833333333333332,
|
| 96 |
+
"predicted_margin": 0.2537508278669194,
|
| 97 |
+
"proposal_oracle_success": 0.5555555555555556,
|
| 98 |
+
"proposal_oracle_utility": 1.1790093589160178,
|
| 99 |
+
"selected_candidate_index": 1.9444444444444444,
|
| 100 |
+
"selected_success": 0.8333333333333334,
|
| 101 |
+
"selected_success_gain_over_base": 0.0,
|
| 102 |
+
"selected_utility": 1.6734557673335075,
|
| 103 |
+
"selected_utility_gain_over_base": 0.0,
|
| 104 |
+
"selector_regret": 0.32654423266649246,
|
| 105 |
+
"success_selector_gap": 0.16666666666666666,
|
| 106 |
+
"success_support_gap": 0.3333333333333333,
|
| 107 |
+
"support_gap": 0.6220440144340197,
|
| 108 |
+
"tau": 0.41382592280226843
|
| 109 |
+
},
|
| 110 |
+
"StackCube-v1": {
|
| 111 |
+
"base_success": 0.09090909090909091,
|
| 112 |
+
"base_utility": 0.5003613992170854,
|
| 113 |
+
"coverage": 0.0,
|
| 114 |
+
"execute_generated": 0.0,
|
| 115 |
+
"fallback_rate": 1.0,
|
| 116 |
+
"hidden_chart_oracle_success": 1.0,
|
| 117 |
+
"hidden_chart_oracle_utility": 2.0,
|
| 118 |
+
"outcome_ptr": 0.6060606060606061,
|
| 119 |
+
"pairwise_causal_calibration_accuracy": 0.47779220779220777,
|
| 120 |
+
"pairwise_causal_calibration_confidence": 0.509119314577852,
|
| 121 |
+
"pairwise_causal_calibration_ece": 0.14450976381890085,
|
| 122 |
+
"pairwise_causal_calibration_pairs": 27.818181818181817,
|
| 123 |
+
"predicted_margin": 0.12169227037448728,
|
| 124 |
+
"proposal_oracle_success": 0.18181818181818182,
|
| 125 |
+
"proposal_oracle_utility": 0.634040683056369,
|
| 126 |
+
"selected_candidate_index": 3.9393939393939394,
|
| 127 |
+
"selected_success": 0.09090909090909091,
|
| 128 |
+
"selected_success_gain_over_base": 0.0,
|
| 129 |
+
"selected_utility": 0.5003613992170854,
|
| 130 |
+
"selected_utility_gain_over_base": 0.0,
|
| 131 |
+
"selector_regret": 0.18030828102068466,
|
| 132 |
+
"success_selector_gap": 0.12121212121212122,
|
| 133 |
+
"success_support_gap": 0.8181818181818182,
|
| 134 |
+
"support_gap": 1.365959316943631,
|
| 135 |
+
"tau": 0.4138259228022681
|
| 136 |
+
}
|
| 137 |
+
}
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_success_perdim_trainmax_train_to_test/split_hash.txt
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"calibration_target_index": {
|
| 3 |
+
"content_hash": "67f7b4a692f7c7e71da24378dc71e3399c5d35afc14c494869eba6087b765c42",
|
| 4 |
+
"retrieval_index_allowed": true,
|
| 5 |
+
"split": "train",
|
| 6 |
+
"split_hash": "867a94a1c67686c082765671a7bf8d74a9d633085e1b2619c83197c60bf58372"
|
| 7 |
+
},
|
| 8 |
+
"eval_target_index": {
|
| 9 |
+
"content_hash": "0c38d73dc9bc4ab756e12006671b5d12f1a23358b281e54b3f6a9ae3cbf6e9af",
|
| 10 |
+
"retrieval_index_allowed": false,
|
| 11 |
+
"split": "test",
|
| 12 |
+
"split_hash": "84151df9e88c219b76751677558cc5bbbc19082bf358cea7330158e9e403e25f"
|
| 13 |
+
}
|
| 14 |
+
}
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_success_perdim_trainmax_train_to_test/table.tex
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
% Auto-generated by scripts/eval_learned_dominance_selector.py
|
| 2 |
+
\begin{tabular}{lrrrrrrrrr}
|
| 3 |
+
\toprule
|
| 4 |
+
Rows & Coverage & Fallback & Base succ. & Selected succ. & Oracle succ. & OutcomePTR & Succ. support gap & Succ. selector gap & Cal. ECE \\
|
| 5 |
+
\midrule
|
| 6 |
+
144 & 0.0069 & 0.9931 & 0.3333 & 0.3264 & 0.2222 & 0.3611 & 0.5139 & 0.0625 & 0.0315 \\
|
| 7 |
+
\bottomrule
|
| 8 |
+
\end{tabular}
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_success_perdim_trainmax_train_to_test/train.log
ADDED
|
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
trained ridge dominance calibrator on calibration measured rows only
|
| 2 |
+
fit_objective=pointwise
|
| 3 |
+
selected_lambda=0.0
|
| 4 |
+
tau=0.413826
|
| 5 |
+
calibration_input=runs/ctt_base_context_obs_train_cal_perdim_trainmax_rollout_comparison/combined_measured_candidates.json
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_success_tanh_train_to_test/metrics.json
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_success_tanh_train_to_test/metrics_by_seed.json
CHANGED
|
@@ -8,6 +8,10 @@
|
|
| 8 |
"hidden_chart_oracle_success": 1.0,
|
| 9 |
"hidden_chart_oracle_utility": 2.0,
|
| 10 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
"predicted_margin": 0.197805317803883,
|
| 12 |
"proposal_oracle_success": 1.0,
|
| 13 |
"proposal_oracle_utility": 2.0,
|
|
@@ -31,6 +35,10 @@
|
|
| 31 |
"hidden_chart_oracle_success": 1.0,
|
| 32 |
"hidden_chart_oracle_utility": 2.0,
|
| 33 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 34 |
"predicted_margin": 0.1847762400277908,
|
| 35 |
"proposal_oracle_success": 0.0,
|
| 36 |
"proposal_oracle_utility": 0.7075161337852478,
|
|
@@ -54,6 +62,10 @@
|
|
| 54 |
"hidden_chart_oracle_success": 1.0,
|
| 55 |
"hidden_chart_oracle_utility": 2.0,
|
| 56 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 57 |
"predicted_margin": 0.14402410971307644,
|
| 58 |
"proposal_oracle_success": 1.0,
|
| 59 |
"proposal_oracle_utility": 2.0,
|
|
@@ -77,6 +89,10 @@
|
|
| 77 |
"hidden_chart_oracle_success": 1.0,
|
| 78 |
"hidden_chart_oracle_utility": 2.0,
|
| 79 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 80 |
"predicted_margin": 0.246546487439865,
|
| 81 |
"proposal_oracle_success": 0.0,
|
| 82 |
"proposal_oracle_utility": 0.09061422944068909,
|
|
@@ -100,6 +116,10 @@
|
|
| 100 |
"hidden_chart_oracle_success": 1.0,
|
| 101 |
"hidden_chart_oracle_utility": 2.0,
|
| 102 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 103 |
"predicted_margin": 0.23873008123815037,
|
| 104 |
"proposal_oracle_success": 0.0,
|
| 105 |
"proposal_oracle_utility": 0.14092671871185303,
|
|
@@ -123,6 +143,10 @@
|
|
| 123 |
"hidden_chart_oracle_success": 1.0,
|
| 124 |
"hidden_chart_oracle_utility": 2.0,
|
| 125 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 126 |
"predicted_margin": 0.2642625319749369,
|
| 127 |
"proposal_oracle_success": 0.0,
|
| 128 |
"proposal_oracle_utility": 0.1624142825603485,
|
|
@@ -146,6 +170,10 @@
|
|
| 146 |
"hidden_chart_oracle_success": 1.0,
|
| 147 |
"hidden_chart_oracle_utility": 2.0,
|
| 148 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 149 |
"predicted_margin": 0.2766048493279699,
|
| 150 |
"proposal_oracle_success": 0.0,
|
| 151 |
"proposal_oracle_utility": 0.20478898286819458,
|
|
@@ -169,6 +197,10 @@
|
|
| 169 |
"hidden_chart_oracle_success": 1.0,
|
| 170 |
"hidden_chart_oracle_utility": 2.0,
|
| 171 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 172 |
"predicted_margin": 0.23010369929396896,
|
| 173 |
"proposal_oracle_success": 0.0,
|
| 174 |
"proposal_oracle_utility": 0.2086353600025177,
|
|
@@ -192,6 +224,10 @@
|
|
| 192 |
"hidden_chart_oracle_success": 1.0,
|
| 193 |
"hidden_chart_oracle_utility": 2.0,
|
| 194 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 195 |
"predicted_margin": 0.27542788251735256,
|
| 196 |
"proposal_oracle_success": 0.0,
|
| 197 |
"proposal_oracle_utility": 0.5664863586425781,
|
|
@@ -215,6 +251,10 @@
|
|
| 215 |
"hidden_chart_oracle_success": 1.0,
|
| 216 |
"hidden_chart_oracle_utility": 2.0,
|
| 217 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 218 |
"predicted_margin": 0.6193304427041512,
|
| 219 |
"proposal_oracle_success": 0.0,
|
| 220 |
"proposal_oracle_utility": 0.06612107157707214,
|
|
@@ -238,6 +278,10 @@
|
|
| 238 |
"hidden_chart_oracle_success": 1.0,
|
| 239 |
"hidden_chart_oracle_utility": 2.0,
|
| 240 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 241 |
"predicted_margin": 0.667791884975374,
|
| 242 |
"proposal_oracle_success": 0.0,
|
| 243 |
"proposal_oracle_utility": 0.05957312881946564,
|
|
@@ -261,6 +305,10 @@
|
|
| 261 |
"hidden_chart_oracle_success": 1.0,
|
| 262 |
"hidden_chart_oracle_utility": 2.0,
|
| 263 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 264 |
"predicted_margin": 0.5738964308429505,
|
| 265 |
"proposal_oracle_success": 0.0,
|
| 266 |
"proposal_oracle_utility": 0.06987608969211578,
|
|
@@ -284,6 +332,10 @@
|
|
| 284 |
"hidden_chart_oracle_success": 1.0,
|
| 285 |
"hidden_chart_oracle_utility": 2.0,
|
| 286 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 287 |
"predicted_margin": 0.26820123426100245,
|
| 288 |
"proposal_oracle_success": 0.0,
|
| 289 |
"proposal_oracle_utility": 0.056939899921417236,
|
|
@@ -307,6 +359,10 @@
|
|
| 307 |
"hidden_chart_oracle_success": 1.0,
|
| 308 |
"hidden_chart_oracle_utility": 2.0,
|
| 309 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 310 |
"predicted_margin": 0.30466427900455206,
|
| 311 |
"proposal_oracle_success": 0.0,
|
| 312 |
"proposal_oracle_utility": 0.12235099077224731,
|
|
@@ -330,6 +386,10 @@
|
|
| 330 |
"hidden_chart_oracle_success": 1.0,
|
| 331 |
"hidden_chart_oracle_utility": 2.0,
|
| 332 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 333 |
"predicted_margin": 0.26540267342818874,
|
| 334 |
"proposal_oracle_success": 0.0,
|
| 335 |
"proposal_oracle_utility": 0.18062961101531982,
|
|
@@ -353,6 +413,10 @@
|
|
| 353 |
"hidden_chart_oracle_success": 1.0,
|
| 354 |
"hidden_chart_oracle_utility": 2.0,
|
| 355 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 356 |
"predicted_margin": 0.6712258551783686,
|
| 357 |
"proposal_oracle_success": 0.0,
|
| 358 |
"proposal_oracle_utility": 0.17992261052131653,
|
|
@@ -376,6 +440,10 @@
|
|
| 376 |
"hidden_chart_oracle_success": 1.0,
|
| 377 |
"hidden_chart_oracle_utility": 2.0,
|
| 378 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 379 |
"predicted_margin": 0.7232270300064332,
|
| 380 |
"proposal_oracle_success": 0.0,
|
| 381 |
"proposal_oracle_utility": 0.1777031123638153,
|
|
@@ -399,6 +467,10 @@
|
|
| 399 |
"hidden_chart_oracle_success": 1.0,
|
| 400 |
"hidden_chart_oracle_utility": 2.0,
|
| 401 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 402 |
"predicted_margin": 0.5588708482544975,
|
| 403 |
"proposal_oracle_success": 1.0,
|
| 404 |
"proposal_oracle_utility": 2.0,
|
|
@@ -422,6 +494,8 @@
|
|
| 422 |
"hidden_chart_oracle_success": 1.0,
|
| 423 |
"hidden_chart_oracle_utility": 2.0,
|
| 424 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
| 425 |
"predicted_margin": 0.1142441575920736,
|
| 426 |
"proposal_oracle_success": 1.0,
|
| 427 |
"proposal_oracle_utility": 2.0,
|
|
@@ -445,6 +519,10 @@
|
|
| 445 |
"hidden_chart_oracle_success": 1.0,
|
| 446 |
"hidden_chart_oracle_utility": 2.0,
|
| 447 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 448 |
"predicted_margin": 0.1658630525898114,
|
| 449 |
"proposal_oracle_success": 0.0,
|
| 450 |
"proposal_oracle_utility": 0.09690030664205551,
|
|
@@ -468,6 +546,10 @@
|
|
| 468 |
"hidden_chart_oracle_success": 1.0,
|
| 469 |
"hidden_chart_oracle_utility": 2.0,
|
| 470 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 471 |
"predicted_margin": 0.13791743075293594,
|
| 472 |
"proposal_oracle_success": 0.0,
|
| 473 |
"proposal_oracle_utility": 0.10843148082494736,
|
|
@@ -491,6 +573,10 @@
|
|
| 491 |
"hidden_chart_oracle_success": 0.0,
|
| 492 |
"hidden_chart_oracle_utility": 0.2857772707939148,
|
| 493 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 494 |
"predicted_margin": 0.20959728035122172,
|
| 495 |
"proposal_oracle_success": 0.0,
|
| 496 |
"proposal_oracle_utility": 0.008802552707493305,
|
|
@@ -514,6 +600,10 @@
|
|
| 514 |
"hidden_chart_oracle_success": 0.0,
|
| 515 |
"hidden_chart_oracle_utility": 0.2857772707939148,
|
| 516 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 517 |
"predicted_margin": 0.21183340573001233,
|
| 518 |
"proposal_oracle_success": 0.0,
|
| 519 |
"proposal_oracle_utility": 0.03755499795079231,
|
|
@@ -537,6 +627,10 @@
|
|
| 537 |
"hidden_chart_oracle_success": 0.0,
|
| 538 |
"hidden_chart_oracle_utility": 0.2857772707939148,
|
| 539 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 540 |
"predicted_margin": 0.18308305510200482,
|
| 541 |
"proposal_oracle_success": 0.0,
|
| 542 |
"proposal_oracle_utility": 0.03313249349594116,
|
|
@@ -560,6 +654,10 @@
|
|
| 560 |
"hidden_chart_oracle_success": 0.0,
|
| 561 |
"hidden_chart_oracle_utility": 0.1593424528837204,
|
| 562 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 563 |
"predicted_margin": 0.1969950204810189,
|
| 564 |
"proposal_oracle_success": 0.0,
|
| 565 |
"proposal_oracle_utility": 0.06010149046778679,
|
|
@@ -583,6 +681,10 @@
|
|
| 583 |
"hidden_chart_oracle_success": 0.0,
|
| 584 |
"hidden_chart_oracle_utility": 0.1593424528837204,
|
| 585 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 586 |
"predicted_margin": 0.20397936444755985,
|
| 587 |
"proposal_oracle_success": 0.0,
|
| 588 |
"proposal_oracle_utility": 0.030005255714058876,
|
|
@@ -606,6 +708,10 @@
|
|
| 606 |
"hidden_chart_oracle_success": 0.0,
|
| 607 |
"hidden_chart_oracle_utility": 0.1593424528837204,
|
| 608 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 609 |
"predicted_margin": 0.27422519801828676,
|
| 610 |
"proposal_oracle_success": 0.0,
|
| 611 |
"proposal_oracle_utility": 0.09976043552160263,
|
|
@@ -629,6 +735,10 @@
|
|
| 629 |
"hidden_chart_oracle_success": 0.0,
|
| 630 |
"hidden_chart_oracle_utility": 0.20948036015033722,
|
| 631 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 632 |
"predicted_margin": 0.6561771188937511,
|
| 633 |
"proposal_oracle_success": 1.0,
|
| 634 |
"proposal_oracle_utility": 2.0,
|
|
@@ -652,6 +762,10 @@
|
|
| 652 |
"hidden_chart_oracle_success": 0.0,
|
| 653 |
"hidden_chart_oracle_utility": 0.20948036015033722,
|
| 654 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 655 |
"predicted_margin": 0.7595411023961984,
|
| 656 |
"proposal_oracle_success": 0.0,
|
| 657 |
"proposal_oracle_utility": 0.19969357550144196,
|
|
@@ -675,6 +789,10 @@
|
|
| 675 |
"hidden_chart_oracle_success": 0.0,
|
| 676 |
"hidden_chart_oracle_utility": 0.20948036015033722,
|
| 677 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 678 |
"predicted_margin": 0.6374969590449568,
|
| 679 |
"proposal_oracle_success": 1.0,
|
| 680 |
"proposal_oracle_utility": 2.0,
|
|
@@ -698,6 +816,10 @@
|
|
| 698 |
"hidden_chart_oracle_success": 1.0,
|
| 699 |
"hidden_chart_oracle_utility": 2.0,
|
| 700 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 701 |
"predicted_margin": 0.2595476416603391,
|
| 702 |
"proposal_oracle_success": 1.0,
|
| 703 |
"proposal_oracle_utility": 2.0,
|
|
@@ -721,6 +843,10 @@
|
|
| 721 |
"hidden_chart_oracle_success": 1.0,
|
| 722 |
"hidden_chart_oracle_utility": 2.0,
|
| 723 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 724 |
"predicted_margin": 0.276906017761115,
|
| 725 |
"proposal_oracle_success": 1.0,
|
| 726 |
"proposal_oracle_utility": 2.0,
|
|
@@ -744,6 +870,10 @@
|
|
| 744 |
"hidden_chart_oracle_success": 1.0,
|
| 745 |
"hidden_chart_oracle_utility": 2.0,
|
| 746 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 747 |
"predicted_margin": 0.3504006480636836,
|
| 748 |
"proposal_oracle_success": 0.0,
|
| 749 |
"proposal_oracle_utility": 0.8133031129837036,
|
|
@@ -767,6 +897,10 @@
|
|
| 767 |
"hidden_chart_oracle_success": 1.0,
|
| 768 |
"hidden_chart_oracle_utility": 2.0,
|
| 769 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 770 |
"predicted_margin": 0.3284611884666808,
|
| 771 |
"proposal_oracle_success": 0.0,
|
| 772 |
"proposal_oracle_utility": 0.16821590065956116,
|
|
@@ -790,6 +924,10 @@
|
|
| 790 |
"hidden_chart_oracle_success": 1.0,
|
| 791 |
"hidden_chart_oracle_utility": 2.0,
|
| 792 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 793 |
"predicted_margin": 0.2760379294226463,
|
| 794 |
"proposal_oracle_success": 0.0,
|
| 795 |
"proposal_oracle_utility": 0.07200749218463898,
|
|
@@ -813,6 +951,10 @@
|
|
| 813 |
"hidden_chart_oracle_success": 1.0,
|
| 814 |
"hidden_chart_oracle_utility": 2.0,
|
| 815 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 816 |
"predicted_margin": 0.3311257655653472,
|
| 817 |
"proposal_oracle_success": 0.0,
|
| 818 |
"proposal_oracle_utility": 0.15220001339912415,
|
|
@@ -836,6 +978,10 @@
|
|
| 836 |
"hidden_chart_oracle_success": 0.0,
|
| 837 |
"hidden_chart_oracle_utility": 0.2974560856819153,
|
| 838 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 839 |
"predicted_margin": 0.23866768065123248,
|
| 840 |
"proposal_oracle_success": 0.0,
|
| 841 |
"proposal_oracle_utility": 0.04792773723602295,
|
|
@@ -859,6 +1005,10 @@
|
|
| 859 |
"hidden_chart_oracle_success": 0.0,
|
| 860 |
"hidden_chart_oracle_utility": 0.2974560856819153,
|
| 861 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 862 |
"predicted_margin": 0.3111189505567141,
|
| 863 |
"proposal_oracle_success": 0.0,
|
| 864 |
"proposal_oracle_utility": 0.2170039862394333,
|
|
@@ -882,6 +1032,10 @@
|
|
| 882 |
"hidden_chart_oracle_success": 0.0,
|
| 883 |
"hidden_chart_oracle_utility": 0.2974560856819153,
|
| 884 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 885 |
"predicted_margin": 0.2819397366030672,
|
| 886 |
"proposal_oracle_success": 0.0,
|
| 887 |
"proposal_oracle_utility": 0.09991621971130371,
|
|
@@ -905,6 +1059,10 @@
|
|
| 905 |
"hidden_chart_oracle_success": 1.0,
|
| 906 |
"hidden_chart_oracle_utility": 2.0,
|
| 907 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 908 |
"predicted_margin": 0.26276318940133603,
|
| 909 |
"proposal_oracle_success": 0.0,
|
| 910 |
"proposal_oracle_utility": 0.20236456394195557,
|
|
@@ -928,6 +1086,10 @@
|
|
| 928 |
"hidden_chart_oracle_success": 1.0,
|
| 929 |
"hidden_chart_oracle_utility": 2.0,
|
| 930 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 931 |
"predicted_margin": 0.2479399896937514,
|
| 932 |
"proposal_oracle_success": 0.0,
|
| 933 |
"proposal_oracle_utility": 0.20639251172542572,
|
|
@@ -951,6 +1113,10 @@
|
|
| 951 |
"hidden_chart_oracle_success": 1.0,
|
| 952 |
"hidden_chart_oracle_utility": 2.0,
|
| 953 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 954 |
"predicted_margin": 0.28149350829315467,
|
| 955 |
"proposal_oracle_success": 0.0,
|
| 956 |
"proposal_oracle_utility": 0.5673043727874756,
|
|
@@ -974,6 +1140,8 @@
|
|
| 974 |
"hidden_chart_oracle_success": 1.0,
|
| 975 |
"hidden_chart_oracle_utility": 2.0,
|
| 976 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
| 977 |
"predicted_margin": 0.29301731712998935,
|
| 978 |
"proposal_oracle_success": 1.0,
|
| 979 |
"proposal_oracle_utility": 2.0,
|
|
@@ -997,6 +1165,8 @@
|
|
| 997 |
"hidden_chart_oracle_success": 1.0,
|
| 998 |
"hidden_chart_oracle_utility": 2.0,
|
| 999 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
| 1000 |
"predicted_margin": 0.3803269800393895,
|
| 1001 |
"proposal_oracle_success": 1.0,
|
| 1002 |
"proposal_oracle_utility": 2.0,
|
|
@@ -1020,6 +1190,8 @@
|
|
| 1020 |
"hidden_chart_oracle_success": 1.0,
|
| 1021 |
"hidden_chart_oracle_utility": 2.0,
|
| 1022 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
| 1023 |
"predicted_margin": 0.3463013216157089,
|
| 1024 |
"proposal_oracle_success": 1.0,
|
| 1025 |
"proposal_oracle_utility": 2.0,
|
|
@@ -1043,6 +1215,10 @@
|
|
| 1043 |
"hidden_chart_oracle_success": 1.0,
|
| 1044 |
"hidden_chart_oracle_utility": 2.0,
|
| 1045 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1046 |
"predicted_margin": 0.6291681545836397,
|
| 1047 |
"proposal_oracle_success": 1.0,
|
| 1048 |
"proposal_oracle_utility": 2.0,
|
|
@@ -1066,6 +1242,8 @@
|
|
| 1066 |
"hidden_chart_oracle_success": 1.0,
|
| 1067 |
"hidden_chart_oracle_utility": 2.0,
|
| 1068 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
| 1069 |
"predicted_margin": 0.6121226617843838,
|
| 1070 |
"proposal_oracle_success": 1.0,
|
| 1071 |
"proposal_oracle_utility": 2.0,
|
|
@@ -1089,6 +1267,10 @@
|
|
| 1089 |
"hidden_chart_oracle_success": 1.0,
|
| 1090 |
"hidden_chart_oracle_utility": 2.0,
|
| 1091 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1092 |
"predicted_margin": 0.6378944690046493,
|
| 1093 |
"proposal_oracle_success": 1.0,
|
| 1094 |
"proposal_oracle_utility": 2.0,
|
|
@@ -1112,6 +1294,10 @@
|
|
| 1112 |
"hidden_chart_oracle_success": 1.0,
|
| 1113 |
"hidden_chart_oracle_utility": 2.0,
|
| 1114 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1115 |
"predicted_margin": 0.11067121611907713,
|
| 1116 |
"proposal_oracle_success": 1.0,
|
| 1117 |
"proposal_oracle_utility": 2.0,
|
|
@@ -1135,6 +1321,10 @@
|
|
| 1135 |
"hidden_chart_oracle_success": 1.0,
|
| 1136 |
"hidden_chart_oracle_utility": 2.0,
|
| 1137 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1138 |
"predicted_margin": 0.09669884825042896,
|
| 1139 |
"proposal_oracle_success": 0.0,
|
| 1140 |
"proposal_oracle_utility": 0.13508938252925873,
|
|
@@ -1158,6 +1348,10 @@
|
|
| 1158 |
"hidden_chart_oracle_success": 1.0,
|
| 1159 |
"hidden_chart_oracle_utility": 2.0,
|
| 1160 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1161 |
"predicted_margin": 0.0578055103471221,
|
| 1162 |
"proposal_oracle_success": 1.0,
|
| 1163 |
"proposal_oracle_utility": 2.0,
|
|
@@ -1181,6 +1375,10 @@
|
|
| 1181 |
"hidden_chart_oracle_success": 1.0,
|
| 1182 |
"hidden_chart_oracle_utility": 2.0,
|
| 1183 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1184 |
"predicted_margin": 0.14108783056103189,
|
| 1185 |
"proposal_oracle_success": 0.0,
|
| 1186 |
"proposal_oracle_utility": 0.02055191993713379,
|
|
@@ -1204,6 +1402,10 @@
|
|
| 1204 |
"hidden_chart_oracle_success": 1.0,
|
| 1205 |
"hidden_chart_oracle_utility": 2.0,
|
| 1206 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1207 |
"predicted_margin": 0.1226666983652919,
|
| 1208 |
"proposal_oracle_success": 0.0,
|
| 1209 |
"proposal_oracle_utility": 0.05916740372776985,
|
|
@@ -1227,6 +1429,10 @@
|
|
| 1227 |
"hidden_chart_oracle_success": 1.0,
|
| 1228 |
"hidden_chart_oracle_utility": 2.0,
|
| 1229 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1230 |
"predicted_margin": 0.07991407615376411,
|
| 1231 |
"proposal_oracle_success": 1.0,
|
| 1232 |
"proposal_oracle_utility": 2.0,
|
|
@@ -1250,6 +1456,10 @@
|
|
| 1250 |
"hidden_chart_oracle_success": 1.0,
|
| 1251 |
"hidden_chart_oracle_utility": 2.0,
|
| 1252 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1253 |
"predicted_margin": 0.25444049345463426,
|
| 1254 |
"proposal_oracle_success": 1.0,
|
| 1255 |
"proposal_oracle_utility": 2.0,
|
|
@@ -1273,6 +1483,10 @@
|
|
| 1273 |
"hidden_chart_oracle_success": 1.0,
|
| 1274 |
"hidden_chart_oracle_utility": 2.0,
|
| 1275 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1276 |
"predicted_margin": 0.24790697988984683,
|
| 1277 |
"proposal_oracle_success": 1.0,
|
| 1278 |
"proposal_oracle_utility": 2.0,
|
|
@@ -1296,6 +1510,10 @@
|
|
| 1296 |
"hidden_chart_oracle_success": 1.0,
|
| 1297 |
"hidden_chart_oracle_utility": 2.0,
|
| 1298 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1299 |
"predicted_margin": 0.3067677760628564,
|
| 1300 |
"proposal_oracle_success": 1.0,
|
| 1301 |
"proposal_oracle_utility": 2.0,
|
|
@@ -1319,6 +1537,10 @@
|
|
| 1319 |
"hidden_chart_oracle_success": 1.0,
|
| 1320 |
"hidden_chart_oracle_utility": 2.0,
|
| 1321 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1322 |
"predicted_margin": 0.20092600070845926,
|
| 1323 |
"proposal_oracle_success": 1.0,
|
| 1324 |
"proposal_oracle_utility": 2.0,
|
|
@@ -1342,6 +1564,10 @@
|
|
| 1342 |
"hidden_chart_oracle_success": 1.0,
|
| 1343 |
"hidden_chart_oracle_utility": 2.0,
|
| 1344 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1345 |
"predicted_margin": 0.16349292872459203,
|
| 1346 |
"proposal_oracle_success": 1.0,
|
| 1347 |
"proposal_oracle_utility": 2.0,
|
|
@@ -1365,6 +1591,10 @@
|
|
| 1365 |
"hidden_chart_oracle_success": 1.0,
|
| 1366 |
"hidden_chart_oracle_utility": 2.0,
|
| 1367 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1368 |
"predicted_margin": 0.17995610986243743,
|
| 1369 |
"proposal_oracle_success": 1.0,
|
| 1370 |
"proposal_oracle_utility": 2.0,
|
|
@@ -1388,6 +1618,10 @@
|
|
| 1388 |
"hidden_chart_oracle_success": 1.0,
|
| 1389 |
"hidden_chart_oracle_utility": 2.0,
|
| 1390 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1391 |
"predicted_margin": 0.11438425065440704,
|
| 1392 |
"proposal_oracle_success": 1.0,
|
| 1393 |
"proposal_oracle_utility": 2.0,
|
|
@@ -1411,6 +1645,10 @@
|
|
| 1411 |
"hidden_chart_oracle_success": 1.0,
|
| 1412 |
"hidden_chart_oracle_utility": 2.0,
|
| 1413 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1414 |
"predicted_margin": 0.17159672738556292,
|
| 1415 |
"proposal_oracle_success": 0.0,
|
| 1416 |
"proposal_oracle_utility": 0.15240873396396637,
|
|
@@ -1434,6 +1672,10 @@
|
|
| 1434 |
"hidden_chart_oracle_success": 1.0,
|
| 1435 |
"hidden_chart_oracle_utility": 2.0,
|
| 1436 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1437 |
"predicted_margin": 0.10074789459755354,
|
| 1438 |
"proposal_oracle_success": 0.0,
|
| 1439 |
"proposal_oracle_utility": 0.1992303878068924,
|
|
@@ -1457,6 +1699,10 @@
|
|
| 1457 |
"hidden_chart_oracle_success": 1.0,
|
| 1458 |
"hidden_chart_oracle_utility": 2.0,
|
| 1459 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1460 |
"predicted_margin": 0.275573013269513,
|
| 1461 |
"proposal_oracle_success": 0.5,
|
| 1462 |
"proposal_oracle_utility": 1.0780495330691338,
|
|
@@ -1480,6 +1726,10 @@
|
|
| 1480 |
"hidden_chart_oracle_success": 1.0,
|
| 1481 |
"hidden_chart_oracle_utility": 2.0,
|
| 1482 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1483 |
"predicted_margin": 0.2841151465083533,
|
| 1484 |
"proposal_oracle_success": 0.5,
|
| 1485 |
"proposal_oracle_utility": 1.0777441188693047,
|
|
@@ -1503,6 +1753,10 @@
|
|
| 1503 |
"hidden_chart_oracle_success": 1.0,
|
| 1504 |
"hidden_chart_oracle_utility": 2.0,
|
| 1505 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1506 |
"predicted_margin": 0.2679532638886026,
|
| 1507 |
"proposal_oracle_success": 0.5,
|
| 1508 |
"proposal_oracle_utility": 1.053164016455412,
|
|
@@ -1526,6 +1780,10 @@
|
|
| 1526 |
"hidden_chart_oracle_success": 1.0,
|
| 1527 |
"hidden_chart_oracle_utility": 2.0,
|
| 1528 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1529 |
"predicted_margin": 0.20663971147287907,
|
| 1530 |
"proposal_oracle_success": 0.0,
|
| 1531 |
"proposal_oracle_utility": 0.0642656534910202,
|
|
@@ -1549,6 +1807,10 @@
|
|
| 1549 |
"hidden_chart_oracle_success": 1.0,
|
| 1550 |
"hidden_chart_oracle_utility": 2.0,
|
| 1551 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1552 |
"predicted_margin": 0.26282861689601783,
|
| 1553 |
"proposal_oracle_success": 0.0,
|
| 1554 |
"proposal_oracle_utility": 0.07914192974567413,
|
|
@@ -1572,6 +1834,10 @@
|
|
| 1572 |
"hidden_chart_oracle_success": 1.0,
|
| 1573 |
"hidden_chart_oracle_utility": 2.0,
|
| 1574 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1575 |
"predicted_margin": 0.2871776156075902,
|
| 1576 |
"proposal_oracle_success": 0.0,
|
| 1577 |
"proposal_oracle_utility": 0.06764055788516998,
|
|
@@ -1595,6 +1861,10 @@
|
|
| 1595 |
"hidden_chart_oracle_success": 1.0,
|
| 1596 |
"hidden_chart_oracle_utility": 2.0,
|
| 1597 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1598 |
"predicted_margin": 0.2818040383848624,
|
| 1599 |
"proposal_oracle_success": 1.0,
|
| 1600 |
"proposal_oracle_utility": 2.0,
|
|
@@ -1618,6 +1888,10 @@
|
|
| 1618 |
"hidden_chart_oracle_success": 1.0,
|
| 1619 |
"hidden_chart_oracle_utility": 2.0,
|
| 1620 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1621 |
"predicted_margin": 0.27950117722622586,
|
| 1622 |
"proposal_oracle_success": 0.0,
|
| 1623 |
"proposal_oracle_utility": 0.22683429718017578,
|
|
@@ -1641,6 +1915,10 @@
|
|
| 1641 |
"hidden_chart_oracle_success": 1.0,
|
| 1642 |
"hidden_chart_oracle_utility": 2.0,
|
| 1643 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1644 |
"predicted_margin": 0.29289339637189027,
|
| 1645 |
"proposal_oracle_success": 0.0,
|
| 1646 |
"proposal_oracle_utility": 0.216936856508255,
|
|
@@ -1664,6 +1942,10 @@
|
|
| 1664 |
"hidden_chart_oracle_success": 0.5,
|
| 1665 |
"hidden_chart_oracle_utility": 1.1139086410403252,
|
| 1666 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1667 |
"predicted_margin": 0.10924904877533874,
|
| 1668 |
"proposal_oracle_success": 0.5,
|
| 1669 |
"proposal_oracle_utility": 1.0352020561695099,
|
|
@@ -1687,6 +1969,10 @@
|
|
| 1687 |
"hidden_chart_oracle_success": 0.5,
|
| 1688 |
"hidden_chart_oracle_utility": 1.1139086410403252,
|
| 1689 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1690 |
"predicted_margin": 0.12740485358567616,
|
| 1691 |
"proposal_oracle_success": 0.5,
|
| 1692 |
"proposal_oracle_utility": 1.0799707099795341,
|
|
@@ -1710,6 +1996,10 @@
|
|
| 1710 |
"hidden_chart_oracle_success": 0.5,
|
| 1711 |
"hidden_chart_oracle_utility": 1.1139086410403252,
|
| 1712 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1713 |
"predicted_margin": 0.18675708897764087,
|
| 1714 |
"proposal_oracle_success": 0.5,
|
| 1715 |
"proposal_oracle_utility": 1.1454492062330246,
|
|
@@ -1733,6 +2023,10 @@
|
|
| 1733 |
"hidden_chart_oracle_success": 1.0,
|
| 1734 |
"hidden_chart_oracle_utility": 2.0,
|
| 1735 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1736 |
"predicted_margin": 0.21756370458471622,
|
| 1737 |
"proposal_oracle_success": 0.0,
|
| 1738 |
"proposal_oracle_utility": 0.20391923189163208,
|
|
@@ -1756,6 +2050,10 @@
|
|
| 1756 |
"hidden_chart_oracle_success": 1.0,
|
| 1757 |
"hidden_chart_oracle_utility": 2.0,
|
| 1758 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1759 |
"predicted_margin": 0.1801716769818562,
|
| 1760 |
"proposal_oracle_success": 0.0,
|
| 1761 |
"proposal_oracle_utility": 0.20632563531398773,
|
|
@@ -1779,6 +2077,10 @@
|
|
| 1779 |
"hidden_chart_oracle_success": 1.0,
|
| 1780 |
"hidden_chart_oracle_utility": 2.0,
|
| 1781 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1782 |
"predicted_margin": 0.20469744001592355,
|
| 1783 |
"proposal_oracle_success": 0.0,
|
| 1784 |
"proposal_oracle_utility": 0.20428544282913208,
|
|
@@ -1802,6 +2104,10 @@
|
|
| 1802 |
"hidden_chart_oracle_success": 1.0,
|
| 1803 |
"hidden_chart_oracle_utility": 2.0,
|
| 1804 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1805 |
"predicted_margin": 0.2782043046573905,
|
| 1806 |
"proposal_oracle_success": 1.0,
|
| 1807 |
"proposal_oracle_utility": 2.0,
|
|
@@ -1825,6 +2131,10 @@
|
|
| 1825 |
"hidden_chart_oracle_success": 1.0,
|
| 1826 |
"hidden_chart_oracle_utility": 2.0,
|
| 1827 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1828 |
"predicted_margin": 0.2854092085136363,
|
| 1829 |
"proposal_oracle_success": 1.0,
|
| 1830 |
"proposal_oracle_utility": 2.0,
|
|
@@ -1848,6 +2158,10 @@
|
|
| 1848 |
"hidden_chart_oracle_success": 1.0,
|
| 1849 |
"hidden_chart_oracle_utility": 2.0,
|
| 1850 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1851 |
"predicted_margin": 0.29579179447511267,
|
| 1852 |
"proposal_oracle_success": 1.0,
|
| 1853 |
"proposal_oracle_utility": 2.0,
|
|
@@ -1871,6 +2185,10 @@
|
|
| 1871 |
"hidden_chart_oracle_success": 1.0,
|
| 1872 |
"hidden_chart_oracle_utility": 2.0,
|
| 1873 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1874 |
"predicted_margin": 0.6870668145379031,
|
| 1875 |
"proposal_oracle_success": 1.0,
|
| 1876 |
"proposal_oracle_utility": 2.0,
|
|
@@ -1894,6 +2212,10 @@
|
|
| 1894 |
"hidden_chart_oracle_success": 1.0,
|
| 1895 |
"hidden_chart_oracle_utility": 2.0,
|
| 1896 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1897 |
"predicted_margin": 0.7609842811296632,
|
| 1898 |
"proposal_oracle_success": 1.0,
|
| 1899 |
"proposal_oracle_utility": 2.0,
|
|
@@ -1917,6 +2239,8 @@
|
|
| 1917 |
"hidden_chart_oracle_success": 1.0,
|
| 1918 |
"hidden_chart_oracle_utility": 2.0,
|
| 1919 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
| 1920 |
"predicted_margin": 0.6954287552895091,
|
| 1921 |
"proposal_oracle_success": 1.0,
|
| 1922 |
"proposal_oracle_utility": 2.0,
|
|
@@ -1940,6 +2264,10 @@
|
|
| 1940 |
"hidden_chart_oracle_success": 1.0,
|
| 1941 |
"hidden_chart_oracle_utility": 2.0,
|
| 1942 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1943 |
"predicted_margin": 0.2043317342807088,
|
| 1944 |
"proposal_oracle_success": 0.0,
|
| 1945 |
"proposal_oracle_utility": 0.5837622880935669,
|
|
@@ -1963,6 +2291,10 @@
|
|
| 1963 |
"hidden_chart_oracle_success": 1.0,
|
| 1964 |
"hidden_chart_oracle_utility": 2.0,
|
| 1965 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1966 |
"predicted_margin": 0.16033886964133004,
|
| 1967 |
"proposal_oracle_success": 0.0,
|
| 1968 |
"proposal_oracle_utility": 0.7209141850471497,
|
|
@@ -1986,6 +2318,10 @@
|
|
| 1986 |
"hidden_chart_oracle_success": 1.0,
|
| 1987 |
"hidden_chart_oracle_utility": 2.0,
|
| 1988 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1989 |
"predicted_margin": 0.19688713080112988,
|
| 1990 |
"proposal_oracle_success": 0.0,
|
| 1991 |
"proposal_oracle_utility": 0.21979393064975739,
|
|
@@ -2009,6 +2345,10 @@
|
|
| 2009 |
"hidden_chart_oracle_success": 1.0,
|
| 2010 |
"hidden_chart_oracle_utility": 2.0,
|
| 2011 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2012 |
"predicted_margin": 0.6471664922001317,
|
| 2013 |
"proposal_oracle_success": 0.0,
|
| 2014 |
"proposal_oracle_utility": 0.12932512164115906,
|
|
@@ -2032,6 +2372,10 @@
|
|
| 2032 |
"hidden_chart_oracle_success": 1.0,
|
| 2033 |
"hidden_chart_oracle_utility": 2.0,
|
| 2034 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2035 |
"predicted_margin": 0.6675461655230028,
|
| 2036 |
"proposal_oracle_success": 1.0,
|
| 2037 |
"proposal_oracle_utility": 2.0,
|
|
@@ -2055,6 +2399,10 @@
|
|
| 2055 |
"hidden_chart_oracle_success": 1.0,
|
| 2056 |
"hidden_chart_oracle_utility": 2.0,
|
| 2057 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2058 |
"predicted_margin": 0.6010746259173304,
|
| 2059 |
"proposal_oracle_success": 1.0,
|
| 2060 |
"proposal_oracle_utility": 2.0,
|
|
@@ -2078,6 +2426,10 @@
|
|
| 2078 |
"hidden_chart_oracle_success": 1.0,
|
| 2079 |
"hidden_chart_oracle_utility": 2.0,
|
| 2080 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2081 |
"predicted_margin": 0.09578732958371006,
|
| 2082 |
"proposal_oracle_success": 1.0,
|
| 2083 |
"proposal_oracle_utility": 2.0,
|
|
@@ -2101,6 +2453,10 @@
|
|
| 2101 |
"hidden_chart_oracle_success": 1.0,
|
| 2102 |
"hidden_chart_oracle_utility": 2.0,
|
| 2103 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2104 |
"predicted_margin": 0.10249988539655566,
|
| 2105 |
"proposal_oracle_success": 0.0,
|
| 2106 |
"proposal_oracle_utility": 0.1290181279182434,
|
|
@@ -2124,6 +2480,10 @@
|
|
| 2124 |
"hidden_chart_oracle_success": 1.0,
|
| 2125 |
"hidden_chart_oracle_utility": 2.0,
|
| 2126 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2127 |
"predicted_margin": 0.09962893251516124,
|
| 2128 |
"proposal_oracle_success": 1.0,
|
| 2129 |
"proposal_oracle_utility": 2.0,
|
|
@@ -2147,6 +2507,10 @@
|
|
| 2147 |
"hidden_chart_oracle_success": 1.0,
|
| 2148 |
"hidden_chart_oracle_utility": 2.0,
|
| 2149 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2150 |
"predicted_margin": 0.1717547088237561,
|
| 2151 |
"proposal_oracle_success": 1.0,
|
| 2152 |
"proposal_oracle_utility": 2.0,
|
|
@@ -2170,6 +2534,10 @@
|
|
| 2170 |
"hidden_chart_oracle_success": 1.0,
|
| 2171 |
"hidden_chart_oracle_utility": 2.0,
|
| 2172 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2173 |
"predicted_margin": 0.20877294341834293,
|
| 2174 |
"proposal_oracle_success": 1.0,
|
| 2175 |
"proposal_oracle_utility": 2.0,
|
|
@@ -2193,6 +2561,10 @@
|
|
| 2193 |
"hidden_chart_oracle_success": 1.0,
|
| 2194 |
"hidden_chart_oracle_utility": 2.0,
|
| 2195 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2196 |
"predicted_margin": 0.13802477297102525,
|
| 2197 |
"proposal_oracle_success": 0.0,
|
| 2198 |
"proposal_oracle_utility": 0.5664259791374207,
|
|
@@ -2216,6 +2588,10 @@
|
|
| 2216 |
"hidden_chart_oracle_success": 0.0,
|
| 2217 |
"hidden_chart_oracle_utility": 0.16558238118886948,
|
| 2218 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2219 |
"predicted_margin": 0.15020438296667366,
|
| 2220 |
"proposal_oracle_success": 0.0,
|
| 2221 |
"proposal_oracle_utility": 0.04293955350294709,
|
|
@@ -2239,6 +2615,10 @@
|
|
| 2239 |
"hidden_chart_oracle_success": 0.0,
|
| 2240 |
"hidden_chart_oracle_utility": 0.16558238118886948,
|
| 2241 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2242 |
"predicted_margin": 0.17852089799806223,
|
| 2243 |
"proposal_oracle_success": 0.0,
|
| 2244 |
"proposal_oracle_utility": 0.02655014442279935,
|
|
@@ -2262,6 +2642,10 @@
|
|
| 2262 |
"hidden_chart_oracle_success": 0.0,
|
| 2263 |
"hidden_chart_oracle_utility": 0.16558238118886948,
|
| 2264 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2265 |
"predicted_margin": 0.14307776706871508,
|
| 2266 |
"proposal_oracle_success": 0.0,
|
| 2267 |
"proposal_oracle_utility": 0.03389316890388727,
|
|
@@ -2285,6 +2669,10 @@
|
|
| 2285 |
"hidden_chart_oracle_success": 0.0,
|
| 2286 |
"hidden_chart_oracle_utility": 0.17056432366371155,
|
| 2287 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2288 |
"predicted_margin": 0.11748877632482482,
|
| 2289 |
"proposal_oracle_success": 0.0,
|
| 2290 |
"proposal_oracle_utility": 0.09099660068750381,
|
|
@@ -2308,6 +2696,10 @@
|
|
| 2308 |
"hidden_chart_oracle_success": 0.0,
|
| 2309 |
"hidden_chart_oracle_utility": 0.17056432366371155,
|
| 2310 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2311 |
"predicted_margin": 0.1312853076823468,
|
| 2312 |
"proposal_oracle_success": 0.0,
|
| 2313 |
"proposal_oracle_utility": 0.08514270931482315,
|
|
@@ -2331,6 +2723,10 @@
|
|
| 2331 |
"hidden_chart_oracle_success": 0.0,
|
| 2332 |
"hidden_chart_oracle_utility": 0.17056432366371155,
|
| 2333 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2334 |
"predicted_margin": 0.1345855010442043,
|
| 2335 |
"proposal_oracle_success": 0.0,
|
| 2336 |
"proposal_oracle_utility": 0.08911379426717758,
|
|
@@ -2354,6 +2750,10 @@
|
|
| 2354 |
"hidden_chart_oracle_success": 1.0,
|
| 2355 |
"hidden_chart_oracle_utility": 2.0,
|
| 2356 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2357 |
"predicted_margin": 0.24752873446742565,
|
| 2358 |
"proposal_oracle_success": 0.0,
|
| 2359 |
"proposal_oracle_utility": 0.1631268858909607,
|
|
@@ -2377,6 +2777,10 @@
|
|
| 2377 |
"hidden_chart_oracle_success": 1.0,
|
| 2378 |
"hidden_chart_oracle_utility": 2.0,
|
| 2379 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2380 |
"predicted_margin": 0.23791187673632117,
|
| 2381 |
"proposal_oracle_success": 0.0,
|
| 2382 |
"proposal_oracle_utility": 0.15427637100219727,
|
|
@@ -2400,6 +2804,10 @@
|
|
| 2400 |
"hidden_chart_oracle_success": 1.0,
|
| 2401 |
"hidden_chart_oracle_utility": 2.0,
|
| 2402 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2403 |
"predicted_margin": 0.24387011799339325,
|
| 2404 |
"proposal_oracle_success": 0.0,
|
| 2405 |
"proposal_oracle_utility": 0.19178248941898346,
|
|
@@ -2423,6 +2831,10 @@
|
|
| 2423 |
"hidden_chart_oracle_success": 1.0,
|
| 2424 |
"hidden_chart_oracle_utility": 2.0,
|
| 2425 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2426 |
"predicted_margin": 0.0886735921808986,
|
| 2427 |
"proposal_oracle_success": 1.0,
|
| 2428 |
"proposal_oracle_utility": 2.0,
|
|
@@ -2446,6 +2858,10 @@
|
|
| 2446 |
"hidden_chart_oracle_success": 1.0,
|
| 2447 |
"hidden_chart_oracle_utility": 2.0,
|
| 2448 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2449 |
"predicted_margin": 0.18825713114760673,
|
| 2450 |
"proposal_oracle_success": 1.0,
|
| 2451 |
"proposal_oracle_utility": 2.0,
|
|
@@ -2469,6 +2885,10 @@
|
|
| 2469 |
"hidden_chart_oracle_success": 1.0,
|
| 2470 |
"hidden_chart_oracle_utility": 2.0,
|
| 2471 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2472 |
"predicted_margin": 0.08486096062823595,
|
| 2473 |
"proposal_oracle_success": 1.0,
|
| 2474 |
"proposal_oracle_utility": 2.0,
|
|
@@ -2492,6 +2912,10 @@
|
|
| 2492 |
"hidden_chart_oracle_success": 1.0,
|
| 2493 |
"hidden_chart_oracle_utility": 2.0,
|
| 2494 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2495 |
"predicted_margin": 0.2713698658138815,
|
| 2496 |
"proposal_oracle_success": 1.0,
|
| 2497 |
"proposal_oracle_utility": 2.0,
|
|
@@ -2515,6 +2939,10 @@
|
|
| 2515 |
"hidden_chart_oracle_success": 1.0,
|
| 2516 |
"hidden_chart_oracle_utility": 2.0,
|
| 2517 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2518 |
"predicted_margin": 0.30277388356719176,
|
| 2519 |
"proposal_oracle_success": 1.0,
|
| 2520 |
"proposal_oracle_utility": 2.0,
|
|
@@ -2538,6 +2966,10 @@
|
|
| 2538 |
"hidden_chart_oracle_success": 1.0,
|
| 2539 |
"hidden_chart_oracle_utility": 2.0,
|
| 2540 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2541 |
"predicted_margin": 0.3320023318656702,
|
| 2542 |
"proposal_oracle_success": 1.0,
|
| 2543 |
"proposal_oracle_utility": 2.0,
|
|
@@ -2561,6 +2993,10 @@
|
|
| 2561 |
"hidden_chart_oracle_success": 0.0,
|
| 2562 |
"hidden_chart_oracle_utility": 0.2786632478237152,
|
| 2563 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2564 |
"predicted_margin": 0.22414376708358918,
|
| 2565 |
"proposal_oracle_success": 0.0,
|
| 2566 |
"proposal_oracle_utility": 0.13453228771686554,
|
|
@@ -2584,6 +3020,10 @@
|
|
| 2584 |
"hidden_chart_oracle_success": 0.0,
|
| 2585 |
"hidden_chart_oracle_utility": 0.2786632478237152,
|
| 2586 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2587 |
"predicted_margin": 0.27086090181173816,
|
| 2588 |
"proposal_oracle_success": 0.0,
|
| 2589 |
"proposal_oracle_utility": 0.06424945592880249,
|
|
@@ -2607,6 +3047,10 @@
|
|
| 2607 |
"hidden_chart_oracle_success": 0.0,
|
| 2608 |
"hidden_chart_oracle_utility": 0.2786632478237152,
|
| 2609 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2610 |
"predicted_margin": 0.2587694523429491,
|
| 2611 |
"proposal_oracle_success": 0.0,
|
| 2612 |
"proposal_oracle_utility": 0.14127866923809052,
|
|
@@ -2630,6 +3074,10 @@
|
|
| 2630 |
"hidden_chart_oracle_success": 0.6666666666666666,
|
| 2631 |
"hidden_chart_oracle_utility": 1.4079964955647786,
|
| 2632 |
"outcome_ptr": 0.6666666666666666,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2633 |
"predicted_margin": 0.16432581896894002,
|
| 2634 |
"proposal_oracle_success": 0.3333333333333333,
|
| 2635 |
"proposal_oracle_utility": 0.7655241700510184,
|
|
@@ -2653,6 +3101,10 @@
|
|
| 2653 |
"hidden_chart_oracle_success": 0.6666666666666666,
|
| 2654 |
"hidden_chart_oracle_utility": 1.4079964955647786,
|
| 2655 |
"outcome_ptr": 1.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2656 |
"predicted_margin": 0.1977895366681389,
|
| 2657 |
"proposal_oracle_success": 0.3333333333333333,
|
| 2658 |
"proposal_oracle_utility": 0.7756740699211756,
|
|
@@ -2676,6 +3128,10 @@
|
|
| 2676 |
"hidden_chart_oracle_success": 0.6666666666666666,
|
| 2677 |
"hidden_chart_oracle_utility": 1.4079964955647786,
|
| 2678 |
"outcome_ptr": 0.6666666666666666,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2679 |
"predicted_margin": 0.2141060722436662,
|
| 2680 |
"proposal_oracle_success": 0.3333333333333333,
|
| 2681 |
"proposal_oracle_utility": 0.7652813941240311,
|
|
@@ -2699,6 +3155,10 @@
|
|
| 2699 |
"hidden_chart_oracle_success": 0.25,
|
| 2700 |
"hidden_chart_oracle_utility": 0.6737305596470833,
|
| 2701 |
"outcome_ptr": 0.25,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2702 |
"predicted_margin": 0.203919201209952,
|
| 2703 |
"proposal_oracle_success": 0.5,
|
| 2704 |
"proposal_oracle_utility": 1.0471355519257486,
|
|
@@ -2722,6 +3182,10 @@
|
|
| 2722 |
"hidden_chart_oracle_success": 0.25,
|
| 2723 |
"hidden_chart_oracle_utility": 0.6737305596470833,
|
| 2724 |
"outcome_ptr": 0.25,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2725 |
"predicted_margin": 0.20002804565583177,
|
| 2726 |
"proposal_oracle_success": 0.0,
|
| 2727 |
"proposal_oracle_utility": 0.10832798760384321,
|
|
@@ -2745,6 +3209,10 @@
|
|
| 2745 |
"hidden_chart_oracle_success": 0.25,
|
| 2746 |
"hidden_chart_oracle_utility": 0.6737305596470833,
|
| 2747 |
"outcome_ptr": 0.25,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2748 |
"predicted_margin": 0.19413727025774286,
|
| 2749 |
"proposal_oracle_success": 0.25,
|
| 2750 |
"proposal_oracle_utility": 0.5676866680150852,
|
|
|
|
| 8 |
"hidden_chart_oracle_success": 1.0,
|
| 9 |
"hidden_chart_oracle_utility": 2.0,
|
| 10 |
"outcome_ptr": 0.0,
|
| 11 |
+
"pairwise_causal_calibration_accuracy": 0.5,
|
| 12 |
+
"pairwise_causal_calibration_confidence": 0.5098566341155361,
|
| 13 |
+
"pairwise_causal_calibration_ece": 0.009856634115536056,
|
| 14 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 15 |
"predicted_margin": 0.197805317803883,
|
| 16 |
"proposal_oracle_success": 1.0,
|
| 17 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 35 |
"hidden_chart_oracle_success": 1.0,
|
| 36 |
"hidden_chart_oracle_utility": 2.0,
|
| 37 |
"outcome_ptr": 0.0,
|
| 38 |
+
"pairwise_causal_calibration_accuracy": 0.5714285714285714,
|
| 39 |
+
"pairwise_causal_calibration_confidence": 0.5091112114195717,
|
| 40 |
+
"pairwise_causal_calibration_ece": 0.06231736000899968,
|
| 41 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 42 |
"predicted_margin": 0.1847762400277908,
|
| 43 |
"proposal_oracle_success": 0.0,
|
| 44 |
"proposal_oracle_utility": 0.7075161337852478,
|
|
|
|
| 62 |
"hidden_chart_oracle_success": 1.0,
|
| 63 |
"hidden_chart_oracle_utility": 2.0,
|
| 64 |
"outcome_ptr": 0.0,
|
| 65 |
+
"pairwise_causal_calibration_accuracy": 0.6111111111111112,
|
| 66 |
+
"pairwise_causal_calibration_confidence": 0.5089181730493006,
|
| 67 |
+
"pairwise_causal_calibration_ece": 0.10219293806181051,
|
| 68 |
+
"pairwise_causal_calibration_pairs": 18.0,
|
| 69 |
"predicted_margin": 0.14402410971307644,
|
| 70 |
"proposal_oracle_success": 1.0,
|
| 71 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 89 |
"hidden_chart_oracle_success": 1.0,
|
| 90 |
"hidden_chart_oracle_utility": 2.0,
|
| 91 |
"outcome_ptr": 0.0,
|
| 92 |
+
"pairwise_causal_calibration_accuracy": 0.39285714285714285,
|
| 93 |
+
"pairwise_causal_calibration_confidence": 0.5198284636786039,
|
| 94 |
+
"pairwise_causal_calibration_ece": 0.12697132082146106,
|
| 95 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 96 |
"predicted_margin": 0.246546487439865,
|
| 97 |
"proposal_oracle_success": 0.0,
|
| 98 |
"proposal_oracle_utility": 0.09061422944068909,
|
|
|
|
| 116 |
"hidden_chart_oracle_success": 1.0,
|
| 117 |
"hidden_chart_oracle_utility": 2.0,
|
| 118 |
"outcome_ptr": 0.0,
|
| 119 |
+
"pairwise_causal_calibration_accuracy": 0.5714285714285714,
|
| 120 |
+
"pairwise_causal_calibration_confidence": 0.5196427235751272,
|
| 121 |
+
"pairwise_causal_calibration_ece": 0.0517858478534442,
|
| 122 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 123 |
"predicted_margin": 0.23873008123815037,
|
| 124 |
"proposal_oracle_success": 0.0,
|
| 125 |
"proposal_oracle_utility": 0.14092671871185303,
|
|
|
|
| 143 |
"hidden_chart_oracle_success": 1.0,
|
| 144 |
"hidden_chart_oracle_utility": 2.0,
|
| 145 |
"outcome_ptr": 0.0,
|
| 146 |
+
"pairwise_causal_calibration_accuracy": 0.35714285714285715,
|
| 147 |
+
"pairwise_causal_calibration_confidence": 0.5198017998119995,
|
| 148 |
+
"pairwise_causal_calibration_ece": 0.16265894266914233,
|
| 149 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 150 |
"predicted_margin": 0.2642625319749369,
|
| 151 |
"proposal_oracle_success": 0.0,
|
| 152 |
"proposal_oracle_utility": 0.1624142825603485,
|
|
|
|
| 170 |
"hidden_chart_oracle_success": 1.0,
|
| 171 |
"hidden_chart_oracle_utility": 2.0,
|
| 172 |
"outcome_ptr": 0.0,
|
| 173 |
+
"pairwise_causal_calibration_accuracy": 0.7857142857142857,
|
| 174 |
+
"pairwise_causal_calibration_confidence": 0.5145081193786424,
|
| 175 |
+
"pairwise_causal_calibration_ece": 0.2712061663356433,
|
| 176 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 177 |
"predicted_margin": 0.2766048493279699,
|
| 178 |
"proposal_oracle_success": 0.0,
|
| 179 |
"proposal_oracle_utility": 0.20478898286819458,
|
|
|
|
| 197 |
"hidden_chart_oracle_success": 1.0,
|
| 198 |
"hidden_chart_oracle_utility": 2.0,
|
| 199 |
"outcome_ptr": 0.0,
|
| 200 |
+
"pairwise_causal_calibration_accuracy": 0.6428571428571429,
|
| 201 |
+
"pairwise_causal_calibration_confidence": 0.5069167821110796,
|
| 202 |
+
"pairwise_causal_calibration_ece": 0.1359403607460633,
|
| 203 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 204 |
"predicted_margin": 0.23010369929396896,
|
| 205 |
"proposal_oracle_success": 0.0,
|
| 206 |
"proposal_oracle_utility": 0.2086353600025177,
|
|
|
|
| 224 |
"hidden_chart_oracle_success": 1.0,
|
| 225 |
"hidden_chart_oracle_utility": 2.0,
|
| 226 |
"outcome_ptr": 1.0,
|
| 227 |
+
"pairwise_causal_calibration_accuracy": 0.25,
|
| 228 |
+
"pairwise_causal_calibration_confidence": 0.5078543932512505,
|
| 229 |
+
"pairwise_causal_calibration_ece": 0.2578543932512505,
|
| 230 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 231 |
"predicted_margin": 0.27542788251735256,
|
| 232 |
"proposal_oracle_success": 0.0,
|
| 233 |
"proposal_oracle_utility": 0.5664863586425781,
|
|
|
|
| 251 |
"hidden_chart_oracle_success": 1.0,
|
| 252 |
"hidden_chart_oracle_utility": 2.0,
|
| 253 |
"outcome_ptr": 1.0,
|
| 254 |
+
"pairwise_causal_calibration_accuracy": 0.2857142857142857,
|
| 255 |
+
"pairwise_causal_calibration_confidence": 0.5097695136941484,
|
| 256 |
+
"pairwise_causal_calibration_ece": 0.22405522797986266,
|
| 257 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 258 |
"predicted_margin": 0.6193304427041512,
|
| 259 |
"proposal_oracle_success": 0.0,
|
| 260 |
"proposal_oracle_utility": 0.06612107157707214,
|
|
|
|
| 278 |
"hidden_chart_oracle_success": 1.0,
|
| 279 |
"hidden_chart_oracle_utility": 2.0,
|
| 280 |
"outcome_ptr": 1.0,
|
| 281 |
+
"pairwise_causal_calibration_accuracy": 0.6785714285714286,
|
| 282 |
+
"pairwise_causal_calibration_confidence": 0.5085443341947974,
|
| 283 |
+
"pairwise_causal_calibration_ece": 0.17002709437663122,
|
| 284 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 285 |
"predicted_margin": 0.667791884975374,
|
| 286 |
"proposal_oracle_success": 0.0,
|
| 287 |
"proposal_oracle_utility": 0.05957312881946564,
|
|
|
|
| 305 |
"hidden_chart_oracle_success": 1.0,
|
| 306 |
"hidden_chart_oracle_utility": 2.0,
|
| 307 |
"outcome_ptr": 1.0,
|
| 308 |
+
"pairwise_causal_calibration_accuracy": 0.6428571428571429,
|
| 309 |
+
"pairwise_causal_calibration_confidence": 0.5078411572088926,
|
| 310 |
+
"pairwise_causal_calibration_ece": 0.13501598564825035,
|
| 311 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 312 |
"predicted_margin": 0.5738964308429505,
|
| 313 |
"proposal_oracle_success": 0.0,
|
| 314 |
"proposal_oracle_utility": 0.06987608969211578,
|
|
|
|
| 332 |
"hidden_chart_oracle_success": 1.0,
|
| 333 |
"hidden_chart_oracle_utility": 2.0,
|
| 334 |
"outcome_ptr": 0.0,
|
| 335 |
+
"pairwise_causal_calibration_accuracy": 0.42857142857142855,
|
| 336 |
+
"pairwise_causal_calibration_confidence": 0.5224226018004667,
|
| 337 |
+
"pairwise_causal_calibration_ece": 0.09385117322903819,
|
| 338 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 339 |
"predicted_margin": 0.26820123426100245,
|
| 340 |
"proposal_oracle_success": 0.0,
|
| 341 |
"proposal_oracle_utility": 0.056939899921417236,
|
|
|
|
| 359 |
"hidden_chart_oracle_success": 1.0,
|
| 360 |
"hidden_chart_oracle_utility": 2.0,
|
| 361 |
"outcome_ptr": 0.0,
|
| 362 |
+
"pairwise_causal_calibration_accuracy": 0.5714285714285714,
|
| 363 |
+
"pairwise_causal_calibration_confidence": 0.5227821807492782,
|
| 364 |
+
"pairwise_causal_calibration_ece": 0.04864639067929322,
|
| 365 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 366 |
"predicted_margin": 0.30466427900455206,
|
| 367 |
"proposal_oracle_success": 0.0,
|
| 368 |
"proposal_oracle_utility": 0.12235099077224731,
|
|
|
|
| 386 |
"hidden_chart_oracle_success": 1.0,
|
| 387 |
"hidden_chart_oracle_utility": 2.0,
|
| 388 |
"outcome_ptr": 1.0,
|
| 389 |
+
"pairwise_causal_calibration_accuracy": 0.25,
|
| 390 |
+
"pairwise_causal_calibration_confidence": 0.5174370315106026,
|
| 391 |
+
"pairwise_causal_calibration_ece": 0.2674370315106026,
|
| 392 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 393 |
"predicted_margin": 0.26540267342818874,
|
| 394 |
"proposal_oracle_success": 0.0,
|
| 395 |
"proposal_oracle_utility": 0.18062961101531982,
|
|
|
|
| 413 |
"hidden_chart_oracle_success": 1.0,
|
| 414 |
"hidden_chart_oracle_utility": 2.0,
|
| 415 |
"outcome_ptr": 1.0,
|
| 416 |
+
"pairwise_causal_calibration_accuracy": 0.25,
|
| 417 |
+
"pairwise_causal_calibration_confidence": 0.5088647318101668,
|
| 418 |
+
"pairwise_causal_calibration_ece": 0.2588647318101668,
|
| 419 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 420 |
"predicted_margin": 0.6712258551783686,
|
| 421 |
"proposal_oracle_success": 0.0,
|
| 422 |
"proposal_oracle_utility": 0.17992261052131653,
|
|
|
|
| 440 |
"hidden_chart_oracle_success": 1.0,
|
| 441 |
"hidden_chart_oracle_utility": 2.0,
|
| 442 |
"outcome_ptr": 1.0,
|
| 443 |
+
"pairwise_causal_calibration_accuracy": 0.5714285714285714,
|
| 444 |
+
"pairwise_causal_calibration_confidence": 0.508953471583655,
|
| 445 |
+
"pairwise_causal_calibration_ece": 0.06247509984491639,
|
| 446 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 447 |
"predicted_margin": 0.7232270300064332,
|
| 448 |
"proposal_oracle_success": 0.0,
|
| 449 |
"proposal_oracle_utility": 0.1777031123638153,
|
|
|
|
| 467 |
"hidden_chart_oracle_success": 1.0,
|
| 468 |
"hidden_chart_oracle_utility": 2.0,
|
| 469 |
"outcome_ptr": 1.0,
|
| 470 |
+
"pairwise_causal_calibration_accuracy": 0.8,
|
| 471 |
+
"pairwise_causal_calibration_confidence": 0.5098442555481146,
|
| 472 |
+
"pairwise_causal_calibration_ece": 0.2901557444518854,
|
| 473 |
+
"pairwise_causal_calibration_pairs": 25.0,
|
| 474 |
"predicted_margin": 0.5588708482544975,
|
| 475 |
"proposal_oracle_success": 1.0,
|
| 476 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 494 |
"hidden_chart_oracle_success": 1.0,
|
| 495 |
"hidden_chart_oracle_utility": 2.0,
|
| 496 |
"outcome_ptr": 0.0,
|
| 497 |
+
"pairwise_causal_calibration_ece": 0.0,
|
| 498 |
+
"pairwise_causal_calibration_pairs": 0.0,
|
| 499 |
"predicted_margin": 0.1142441575920736,
|
| 500 |
"proposal_oracle_success": 1.0,
|
| 501 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 519 |
"hidden_chart_oracle_success": 1.0,
|
| 520 |
"hidden_chart_oracle_utility": 2.0,
|
| 521 |
"outcome_ptr": 0.0,
|
| 522 |
+
"pairwise_causal_calibration_accuracy": 0.5,
|
| 523 |
+
"pairwise_causal_calibration_confidence": 0.5070611656440998,
|
| 524 |
+
"pairwise_causal_calibration_ece": 0.007061165644099754,
|
| 525 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 526 |
"predicted_margin": 0.1658630525898114,
|
| 527 |
"proposal_oracle_success": 0.0,
|
| 528 |
"proposal_oracle_utility": 0.09690030664205551,
|
|
|
|
| 546 |
"hidden_chart_oracle_success": 1.0,
|
| 547 |
"hidden_chart_oracle_utility": 2.0,
|
| 548 |
"outcome_ptr": 0.0,
|
| 549 |
+
"pairwise_causal_calibration_accuracy": 0.35714285714285715,
|
| 550 |
+
"pairwise_causal_calibration_confidence": 0.5065725175893429,
|
| 551 |
+
"pairwise_causal_calibration_ece": 0.1494296604464857,
|
| 552 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 553 |
"predicted_margin": 0.13791743075293594,
|
| 554 |
"proposal_oracle_success": 0.0,
|
| 555 |
"proposal_oracle_utility": 0.10843148082494736,
|
|
|
|
| 573 |
"hidden_chart_oracle_success": 0.0,
|
| 574 |
"hidden_chart_oracle_utility": 0.2857772707939148,
|
| 575 |
"outcome_ptr": 0.0,
|
| 576 |
+
"pairwise_causal_calibration_accuracy": 0.4642857142857143,
|
| 577 |
+
"pairwise_causal_calibration_confidence": 0.507901687873458,
|
| 578 |
+
"pairwise_causal_calibration_ece": 0.04361597358774372,
|
| 579 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 580 |
"predicted_margin": 0.20959728035122172,
|
| 581 |
"proposal_oracle_success": 0.0,
|
| 582 |
"proposal_oracle_utility": 0.008802552707493305,
|
|
|
|
| 600 |
"hidden_chart_oracle_success": 0.0,
|
| 601 |
"hidden_chart_oracle_utility": 0.2857772707939148,
|
| 602 |
"outcome_ptr": 0.0,
|
| 603 |
+
"pairwise_causal_calibration_accuracy": 0.6785714285714286,
|
| 604 |
+
"pairwise_causal_calibration_confidence": 0.5164579122807452,
|
| 605 |
+
"pairwise_causal_calibration_ece": 0.16211351629068338,
|
| 606 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 607 |
"predicted_margin": 0.21183340573001233,
|
| 608 |
"proposal_oracle_success": 0.0,
|
| 609 |
"proposal_oracle_utility": 0.03755499795079231,
|
|
|
|
| 627 |
"hidden_chart_oracle_success": 0.0,
|
| 628 |
"hidden_chart_oracle_utility": 0.2857772707939148,
|
| 629 |
"outcome_ptr": 0.0,
|
| 630 |
+
"pairwise_causal_calibration_accuracy": 0.17857142857142858,
|
| 631 |
+
"pairwise_causal_calibration_confidence": 0.5171743151724543,
|
| 632 |
+
"pairwise_causal_calibration_ece": 0.33860288660102567,
|
| 633 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 634 |
"predicted_margin": 0.18308305510200482,
|
| 635 |
"proposal_oracle_success": 0.0,
|
| 636 |
"proposal_oracle_utility": 0.03313249349594116,
|
|
|
|
| 654 |
"hidden_chart_oracle_success": 0.0,
|
| 655 |
"hidden_chart_oracle_utility": 0.1593424528837204,
|
| 656 |
"outcome_ptr": 0.0,
|
| 657 |
+
"pairwise_causal_calibration_accuracy": 0.5,
|
| 658 |
+
"pairwise_causal_calibration_confidence": 0.5097775167745581,
|
| 659 |
+
"pairwise_causal_calibration_ece": 0.00977751677455807,
|
| 660 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 661 |
"predicted_margin": 0.1969950204810189,
|
| 662 |
"proposal_oracle_success": 0.0,
|
| 663 |
"proposal_oracle_utility": 0.06010149046778679,
|
|
|
|
| 681 |
"hidden_chart_oracle_success": 0.0,
|
| 682 |
"hidden_chart_oracle_utility": 0.1593424528837204,
|
| 683 |
"outcome_ptr": 0.0,
|
| 684 |
+
"pairwise_causal_calibration_accuracy": 0.5,
|
| 685 |
+
"pairwise_causal_calibration_confidence": 0.5045038789520366,
|
| 686 |
+
"pairwise_causal_calibration_ece": 0.004503878952036633,
|
| 687 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 688 |
"predicted_margin": 0.20397936444755985,
|
| 689 |
"proposal_oracle_success": 0.0,
|
| 690 |
"proposal_oracle_utility": 0.030005255714058876,
|
|
|
|
| 708 |
"hidden_chart_oracle_success": 0.0,
|
| 709 |
"hidden_chart_oracle_utility": 0.1593424528837204,
|
| 710 |
"outcome_ptr": 0.0,
|
| 711 |
+
"pairwise_causal_calibration_accuracy": 0.6785714285714286,
|
| 712 |
+
"pairwise_causal_calibration_confidence": 0.5110427839399747,
|
| 713 |
+
"pairwise_causal_calibration_ece": 0.1675286446314539,
|
| 714 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 715 |
"predicted_margin": 0.27422519801828676,
|
| 716 |
"proposal_oracle_success": 0.0,
|
| 717 |
"proposal_oracle_utility": 0.09976043552160263,
|
|
|
|
| 735 |
"hidden_chart_oracle_success": 0.0,
|
| 736 |
"hidden_chart_oracle_utility": 0.20948036015033722,
|
| 737 |
"outcome_ptr": 1.0,
|
| 738 |
+
"pairwise_causal_calibration_accuracy": 0.6666666666666666,
|
| 739 |
+
"pairwise_causal_calibration_confidence": 0.5092043778610017,
|
| 740 |
+
"pairwise_causal_calibration_ece": 0.15746228880566493,
|
| 741 |
+
"pairwise_causal_calibration_pairs": 18.0,
|
| 742 |
"predicted_margin": 0.6561771188937511,
|
| 743 |
"proposal_oracle_success": 1.0,
|
| 744 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 762 |
"hidden_chart_oracle_success": 0.0,
|
| 763 |
"hidden_chart_oracle_utility": 0.20948036015033722,
|
| 764 |
"outcome_ptr": 1.0,
|
| 765 |
+
"pairwise_causal_calibration_accuracy": 0.32142857142857145,
|
| 766 |
+
"pairwise_causal_calibration_confidence": 0.509309423579836,
|
| 767 |
+
"pairwise_causal_calibration_ece": 0.18788085215126454,
|
| 768 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 769 |
"predicted_margin": 0.7595411023961984,
|
| 770 |
"proposal_oracle_success": 0.0,
|
| 771 |
"proposal_oracle_utility": 0.19969357550144196,
|
|
|
|
| 789 |
"hidden_chart_oracle_success": 0.0,
|
| 790 |
"hidden_chart_oracle_utility": 0.20948036015033722,
|
| 791 |
"outcome_ptr": 1.0,
|
| 792 |
+
"pairwise_causal_calibration_accuracy": 0.5555555555555556,
|
| 793 |
+
"pairwise_causal_calibration_confidence": 0.5057799962955093,
|
| 794 |
+
"pairwise_causal_calibration_ece": 0.04977555926004629,
|
| 795 |
+
"pairwise_causal_calibration_pairs": 27.0,
|
| 796 |
"predicted_margin": 0.6374969590449568,
|
| 797 |
"proposal_oracle_success": 1.0,
|
| 798 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 816 |
"hidden_chart_oracle_success": 1.0,
|
| 817 |
"hidden_chart_oracle_utility": 2.0,
|
| 818 |
"outcome_ptr": 1.0,
|
| 819 |
+
"pairwise_causal_calibration_accuracy": 0.17857142857142858,
|
| 820 |
+
"pairwise_causal_calibration_confidence": 0.5107026203890623,
|
| 821 |
+
"pairwise_causal_calibration_ece": 0.33213119181763373,
|
| 822 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 823 |
"predicted_margin": 0.2595476416603391,
|
| 824 |
"proposal_oracle_success": 1.0,
|
| 825 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 843 |
"hidden_chart_oracle_success": 1.0,
|
| 844 |
"hidden_chart_oracle_utility": 2.0,
|
| 845 |
"outcome_ptr": 1.0,
|
| 846 |
+
"pairwise_causal_calibration_accuracy": 0.36,
|
| 847 |
+
"pairwise_causal_calibration_confidence": 0.5100155711510672,
|
| 848 |
+
"pairwise_causal_calibration_ece": 0.15001557115106723,
|
| 849 |
+
"pairwise_causal_calibration_pairs": 25.0,
|
| 850 |
"predicted_margin": 0.276906017761115,
|
| 851 |
"proposal_oracle_success": 1.0,
|
| 852 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 870 |
"hidden_chart_oracle_success": 1.0,
|
| 871 |
"hidden_chart_oracle_utility": 2.0,
|
| 872 |
"outcome_ptr": 0.0,
|
| 873 |
+
"pairwise_causal_calibration_accuracy": 0.6071428571428571,
|
| 874 |
+
"pairwise_causal_calibration_confidence": 0.5135673267124184,
|
| 875 |
+
"pairwise_causal_calibration_ece": 0.09357553043043865,
|
| 876 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 877 |
"predicted_margin": 0.3504006480636836,
|
| 878 |
"proposal_oracle_success": 0.0,
|
| 879 |
"proposal_oracle_utility": 0.8133031129837036,
|
|
|
|
| 897 |
"hidden_chart_oracle_success": 1.0,
|
| 898 |
"hidden_chart_oracle_utility": 2.0,
|
| 899 |
"outcome_ptr": 0.0,
|
| 900 |
+
"pairwise_causal_calibration_accuracy": 0.42857142857142855,
|
| 901 |
+
"pairwise_causal_calibration_confidence": 0.5250784055152783,
|
| 902 |
+
"pairwise_causal_calibration_ece": 0.09650697694384974,
|
| 903 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 904 |
"predicted_margin": 0.3284611884666808,
|
| 905 |
"proposal_oracle_success": 0.0,
|
| 906 |
"proposal_oracle_utility": 0.16821590065956116,
|
|
|
|
| 924 |
"hidden_chart_oracle_success": 1.0,
|
| 925 |
"hidden_chart_oracle_utility": 2.0,
|
| 926 |
"outcome_ptr": 0.0,
|
| 927 |
+
"pairwise_causal_calibration_accuracy": 0.75,
|
| 928 |
+
"pairwise_causal_calibration_confidence": 0.5105908420311568,
|
| 929 |
+
"pairwise_causal_calibration_ece": 0.2394091579688432,
|
| 930 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 931 |
"predicted_margin": 0.2760379294226463,
|
| 932 |
"proposal_oracle_success": 0.0,
|
| 933 |
"proposal_oracle_utility": 0.07200749218463898,
|
|
|
|
| 951 |
"hidden_chart_oracle_success": 1.0,
|
| 952 |
"hidden_chart_oracle_utility": 2.0,
|
| 953 |
"outcome_ptr": 0.0,
|
| 954 |
+
"pairwise_causal_calibration_accuracy": 0.4642857142857143,
|
| 955 |
+
"pairwise_causal_calibration_confidence": 0.5231037401616476,
|
| 956 |
+
"pairwise_causal_calibration_ece": 0.05881802587593332,
|
| 957 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 958 |
"predicted_margin": 0.3311257655653472,
|
| 959 |
"proposal_oracle_success": 0.0,
|
| 960 |
"proposal_oracle_utility": 0.15220001339912415,
|
|
|
|
| 978 |
"hidden_chart_oracle_success": 0.0,
|
| 979 |
"hidden_chart_oracle_utility": 0.2974560856819153,
|
| 980 |
"outcome_ptr": 0.0,
|
| 981 |
+
"pairwise_causal_calibration_accuracy": 0.42857142857142855,
|
| 982 |
+
"pairwise_causal_calibration_confidence": 0.5082800251982494,
|
| 983 |
+
"pairwise_causal_calibration_ece": 0.0797085966268209,
|
| 984 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 985 |
"predicted_margin": 0.23866768065123248,
|
| 986 |
"proposal_oracle_success": 0.0,
|
| 987 |
"proposal_oracle_utility": 0.04792773723602295,
|
|
|
|
| 1005 |
"hidden_chart_oracle_success": 0.0,
|
| 1006 |
"hidden_chart_oracle_utility": 0.2974560856819153,
|
| 1007 |
"outcome_ptr": 1.0,
|
| 1008 |
+
"pairwise_causal_calibration_accuracy": 0.25,
|
| 1009 |
+
"pairwise_causal_calibration_confidence": 0.5088781157128205,
|
| 1010 |
+
"pairwise_causal_calibration_ece": 0.2588781157128205,
|
| 1011 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 1012 |
"predicted_margin": 0.3111189505567141,
|
| 1013 |
"proposal_oracle_success": 0.0,
|
| 1014 |
"proposal_oracle_utility": 0.2170039862394333,
|
|
|
|
| 1032 |
"hidden_chart_oracle_success": 0.0,
|
| 1033 |
"hidden_chart_oracle_utility": 0.2974560856819153,
|
| 1034 |
"outcome_ptr": 0.0,
|
| 1035 |
+
"pairwise_causal_calibration_accuracy": 0.5,
|
| 1036 |
+
"pairwise_causal_calibration_confidence": 0.5111297696744155,
|
| 1037 |
+
"pairwise_causal_calibration_ece": 0.01112976967441548,
|
| 1038 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 1039 |
"predicted_margin": 0.2819397366030672,
|
| 1040 |
"proposal_oracle_success": 0.0,
|
| 1041 |
"proposal_oracle_utility": 0.09991621971130371,
|
|
|
|
| 1059 |
"hidden_chart_oracle_success": 1.0,
|
| 1060 |
"hidden_chart_oracle_utility": 2.0,
|
| 1061 |
"outcome_ptr": 1.0,
|
| 1062 |
+
"pairwise_causal_calibration_accuracy": 0.5714285714285714,
|
| 1063 |
+
"pairwise_causal_calibration_confidence": 0.5158496642258744,
|
| 1064 |
+
"pairwise_causal_calibration_ece": 0.05557890720269698,
|
| 1065 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 1066 |
"predicted_margin": 0.26276318940133603,
|
| 1067 |
"proposal_oracle_success": 0.0,
|
| 1068 |
"proposal_oracle_utility": 0.20236456394195557,
|
|
|
|
| 1086 |
"hidden_chart_oracle_success": 1.0,
|
| 1087 |
"hidden_chart_oracle_utility": 2.0,
|
| 1088 |
"outcome_ptr": 1.0,
|
| 1089 |
+
"pairwise_causal_calibration_accuracy": 0.5357142857142857,
|
| 1090 |
+
"pairwise_causal_calibration_confidence": 0.5115243772227345,
|
| 1091 |
+
"pairwise_causal_calibration_ece": 0.024189908491551226,
|
| 1092 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 1093 |
"predicted_margin": 0.2479399896937514,
|
| 1094 |
"proposal_oracle_success": 0.0,
|
| 1095 |
"proposal_oracle_utility": 0.20639251172542572,
|
|
|
|
| 1113 |
"hidden_chart_oracle_success": 1.0,
|
| 1114 |
"hidden_chart_oracle_utility": 2.0,
|
| 1115 |
"outcome_ptr": 1.0,
|
| 1116 |
+
"pairwise_causal_calibration_accuracy": 0.4642857142857143,
|
| 1117 |
+
"pairwise_causal_calibration_confidence": 0.5130055389494003,
|
| 1118 |
+
"pairwise_causal_calibration_ece": 0.04871982466368596,
|
| 1119 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 1120 |
"predicted_margin": 0.28149350829315467,
|
| 1121 |
"proposal_oracle_success": 0.0,
|
| 1122 |
"proposal_oracle_utility": 0.5673043727874756,
|
|
|
|
| 1140 |
"hidden_chart_oracle_success": 1.0,
|
| 1141 |
"hidden_chart_oracle_utility": 2.0,
|
| 1142 |
"outcome_ptr": 0.0,
|
| 1143 |
+
"pairwise_causal_calibration_ece": 0.0,
|
| 1144 |
+
"pairwise_causal_calibration_pairs": 0.0,
|
| 1145 |
"predicted_margin": 0.29301731712998935,
|
| 1146 |
"proposal_oracle_success": 1.0,
|
| 1147 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 1165 |
"hidden_chart_oracle_success": 1.0,
|
| 1166 |
"hidden_chart_oracle_utility": 2.0,
|
| 1167 |
"outcome_ptr": 0.0,
|
| 1168 |
+
"pairwise_causal_calibration_ece": 0.0,
|
| 1169 |
+
"pairwise_causal_calibration_pairs": 0.0,
|
| 1170 |
"predicted_margin": 0.3803269800393895,
|
| 1171 |
"proposal_oracle_success": 1.0,
|
| 1172 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 1190 |
"hidden_chart_oracle_success": 1.0,
|
| 1191 |
"hidden_chart_oracle_utility": 2.0,
|
| 1192 |
"outcome_ptr": 0.0,
|
| 1193 |
+
"pairwise_causal_calibration_ece": 0.0,
|
| 1194 |
+
"pairwise_causal_calibration_pairs": 0.0,
|
| 1195 |
"predicted_margin": 0.3463013216157089,
|
| 1196 |
"proposal_oracle_success": 1.0,
|
| 1197 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 1215 |
"hidden_chart_oracle_success": 1.0,
|
| 1216 |
"hidden_chart_oracle_utility": 2.0,
|
| 1217 |
"outcome_ptr": 1.0,
|
| 1218 |
+
"pairwise_causal_calibration_accuracy": 0.14285714285714285,
|
| 1219 |
+
"pairwise_causal_calibration_confidence": 0.5040985707207823,
|
| 1220 |
+
"pairwise_causal_calibration_ece": 0.3612414278636395,
|
| 1221 |
+
"pairwise_causal_calibration_pairs": 7.0,
|
| 1222 |
"predicted_margin": 0.6291681545836397,
|
| 1223 |
"proposal_oracle_success": 1.0,
|
| 1224 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 1242 |
"hidden_chart_oracle_success": 1.0,
|
| 1243 |
"hidden_chart_oracle_utility": 2.0,
|
| 1244 |
"outcome_ptr": 1.0,
|
| 1245 |
+
"pairwise_causal_calibration_ece": 0.0,
|
| 1246 |
+
"pairwise_causal_calibration_pairs": 0.0,
|
| 1247 |
"predicted_margin": 0.6121226617843838,
|
| 1248 |
"proposal_oracle_success": 1.0,
|
| 1249 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 1267 |
"hidden_chart_oracle_success": 1.0,
|
| 1268 |
"hidden_chart_oracle_utility": 2.0,
|
| 1269 |
"outcome_ptr": 1.0,
|
| 1270 |
+
"pairwise_causal_calibration_accuracy": 0.2857142857142857,
|
| 1271 |
+
"pairwise_causal_calibration_confidence": 0.5056687364786325,
|
| 1272 |
+
"pairwise_causal_calibration_ece": 0.21995445076434683,
|
| 1273 |
+
"pairwise_causal_calibration_pairs": 7.0,
|
| 1274 |
"predicted_margin": 0.6378944690046493,
|
| 1275 |
"proposal_oracle_success": 1.0,
|
| 1276 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 1294 |
"hidden_chart_oracle_success": 1.0,
|
| 1295 |
"hidden_chart_oracle_utility": 2.0,
|
| 1296 |
"outcome_ptr": 1.0,
|
| 1297 |
+
"pairwise_causal_calibration_accuracy": 0.39285714285714285,
|
| 1298 |
+
"pairwise_causal_calibration_confidence": 0.5148884431169273,
|
| 1299 |
+
"pairwise_causal_calibration_ece": 0.12203130025978443,
|
| 1300 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 1301 |
"predicted_margin": 0.11067121611907713,
|
| 1302 |
"proposal_oracle_success": 1.0,
|
| 1303 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 1321 |
"hidden_chart_oracle_success": 1.0,
|
| 1322 |
"hidden_chart_oracle_utility": 2.0,
|
| 1323 |
"outcome_ptr": 0.0,
|
| 1324 |
+
"pairwise_causal_calibration_accuracy": 0.32142857142857145,
|
| 1325 |
+
"pairwise_causal_calibration_confidence": 0.5177759880984708,
|
| 1326 |
+
"pairwise_causal_calibration_ece": 0.19634741666989936,
|
| 1327 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 1328 |
"predicted_margin": 0.09669884825042896,
|
| 1329 |
"proposal_oracle_success": 0.0,
|
| 1330 |
"proposal_oracle_utility": 0.13508938252925873,
|
|
|
|
| 1348 |
"hidden_chart_oracle_success": 1.0,
|
| 1349 |
"hidden_chart_oracle_utility": 2.0,
|
| 1350 |
"outcome_ptr": 1.0,
|
| 1351 |
+
"pairwise_causal_calibration_accuracy": 0.5357142857142857,
|
| 1352 |
+
"pairwise_causal_calibration_confidence": 0.5107043594970093,
|
| 1353 |
+
"pairwise_causal_calibration_ece": 0.02500992621727638,
|
| 1354 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 1355 |
"predicted_margin": 0.0578055103471221,
|
| 1356 |
"proposal_oracle_success": 1.0,
|
| 1357 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 1375 |
"hidden_chart_oracle_success": 1.0,
|
| 1376 |
"hidden_chart_oracle_utility": 2.0,
|
| 1377 |
"outcome_ptr": 0.0,
|
| 1378 |
+
"pairwise_causal_calibration_accuracy": 0.39285714285714285,
|
| 1379 |
+
"pairwise_causal_calibration_confidence": 0.5108692392445339,
|
| 1380 |
+
"pairwise_causal_calibration_ece": 0.11801209638739102,
|
| 1381 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 1382 |
"predicted_margin": 0.14108783056103189,
|
| 1383 |
"proposal_oracle_success": 0.0,
|
| 1384 |
"proposal_oracle_utility": 0.02055191993713379,
|
|
|
|
| 1402 |
"hidden_chart_oracle_success": 1.0,
|
| 1403 |
"hidden_chart_oracle_utility": 2.0,
|
| 1404 |
"outcome_ptr": 0.0,
|
| 1405 |
+
"pairwise_causal_calibration_accuracy": 0.5714285714285714,
|
| 1406 |
+
"pairwise_causal_calibration_confidence": 0.5114463604438318,
|
| 1407 |
+
"pairwise_causal_calibration_ece": 0.05998221098473955,
|
| 1408 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 1409 |
"predicted_margin": 0.1226666983652919,
|
| 1410 |
"proposal_oracle_success": 0.0,
|
| 1411 |
"proposal_oracle_utility": 0.05916740372776985,
|
|
|
|
| 1429 |
"hidden_chart_oracle_success": 1.0,
|
| 1430 |
"hidden_chart_oracle_utility": 2.0,
|
| 1431 |
"outcome_ptr": 0.0,
|
| 1432 |
+
"pairwise_causal_calibration_accuracy": 0.32142857142857145,
|
| 1433 |
+
"pairwise_causal_calibration_confidence": 0.5089957841341125,
|
| 1434 |
+
"pairwise_causal_calibration_ece": 0.18756721270554105,
|
| 1435 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 1436 |
"predicted_margin": 0.07991407615376411,
|
| 1437 |
"proposal_oracle_success": 1.0,
|
| 1438 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 1456 |
"hidden_chart_oracle_success": 1.0,
|
| 1457 |
"hidden_chart_oracle_utility": 2.0,
|
| 1458 |
"outcome_ptr": 0.0,
|
| 1459 |
+
"pairwise_causal_calibration_accuracy": 0.24,
|
| 1460 |
+
"pairwise_causal_calibration_confidence": 0.5186386660397809,
|
| 1461 |
+
"pairwise_causal_calibration_ece": 0.27863866603978094,
|
| 1462 |
+
"pairwise_causal_calibration_pairs": 25.0,
|
| 1463 |
"predicted_margin": 0.25444049345463426,
|
| 1464 |
"proposal_oracle_success": 1.0,
|
| 1465 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 1483 |
"hidden_chart_oracle_success": 1.0,
|
| 1484 |
"hidden_chart_oracle_utility": 2.0,
|
| 1485 |
"outcome_ptr": 0.0,
|
| 1486 |
+
"pairwise_causal_calibration_accuracy": 0.4090909090909091,
|
| 1487 |
+
"pairwise_causal_calibration_confidence": 0.5119334766574462,
|
| 1488 |
+
"pairwise_causal_calibration_ece": 0.10284256756653704,
|
| 1489 |
+
"pairwise_causal_calibration_pairs": 22.0,
|
| 1490 |
"predicted_margin": 0.24790697988984683,
|
| 1491 |
"proposal_oracle_success": 1.0,
|
| 1492 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 1510 |
"hidden_chart_oracle_success": 1.0,
|
| 1511 |
"hidden_chart_oracle_utility": 2.0,
|
| 1512 |
"outcome_ptr": 0.0,
|
| 1513 |
+
"pairwise_causal_calibration_accuracy": 0.5,
|
| 1514 |
+
"pairwise_causal_calibration_confidence": 0.5121892620224119,
|
| 1515 |
+
"pairwise_causal_calibration_ece": 0.012189262022411884,
|
| 1516 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 1517 |
"predicted_margin": 0.3067677760628564,
|
| 1518 |
"proposal_oracle_success": 1.0,
|
| 1519 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 1537 |
"hidden_chart_oracle_success": 1.0,
|
| 1538 |
"hidden_chart_oracle_utility": 2.0,
|
| 1539 |
"outcome_ptr": 0.0,
|
| 1540 |
+
"pairwise_causal_calibration_accuracy": 0.5357142857142857,
|
| 1541 |
+
"pairwise_causal_calibration_confidence": 0.5079042501299456,
|
| 1542 |
+
"pairwise_causal_calibration_ece": 0.027810035584340143,
|
| 1543 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 1544 |
"predicted_margin": 0.20092600070845926,
|
| 1545 |
"proposal_oracle_success": 1.0,
|
| 1546 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 1564 |
"hidden_chart_oracle_success": 1.0,
|
| 1565 |
"hidden_chart_oracle_utility": 2.0,
|
| 1566 |
"outcome_ptr": 0.0,
|
| 1567 |
+
"pairwise_causal_calibration_accuracy": 0.5714285714285714,
|
| 1568 |
+
"pairwise_causal_calibration_confidence": 0.5038741047450654,
|
| 1569 |
+
"pairwise_causal_calibration_ece": 0.06755446668350595,
|
| 1570 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 1571 |
"predicted_margin": 0.16349292872459203,
|
| 1572 |
"proposal_oracle_success": 1.0,
|
| 1573 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 1591 |
"hidden_chart_oracle_success": 1.0,
|
| 1592 |
"hidden_chart_oracle_utility": 2.0,
|
| 1593 |
"outcome_ptr": 0.0,
|
| 1594 |
+
"pairwise_causal_calibration_accuracy": 0.7272727272727273,
|
| 1595 |
+
"pairwise_causal_calibration_confidence": 0.5094648157635078,
|
| 1596 |
+
"pairwise_causal_calibration_ece": 0.21780791150921952,
|
| 1597 |
+
"pairwise_causal_calibration_pairs": 22.0,
|
| 1598 |
"predicted_margin": 0.17995610986243743,
|
| 1599 |
"proposal_oracle_success": 1.0,
|
| 1600 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 1618 |
"hidden_chart_oracle_success": 1.0,
|
| 1619 |
"hidden_chart_oracle_utility": 2.0,
|
| 1620 |
"outcome_ptr": 0.0,
|
| 1621 |
+
"pairwise_causal_calibration_accuracy": 0.5185185185185185,
|
| 1622 |
+
"pairwise_causal_calibration_confidence": 0.520866835944586,
|
| 1623 |
+
"pairwise_causal_calibration_ece": 0.0023483174260675588,
|
| 1624 |
+
"pairwise_causal_calibration_pairs": 27.0,
|
| 1625 |
"predicted_margin": 0.11438425065440704,
|
| 1626 |
"proposal_oracle_success": 1.0,
|
| 1627 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 1645 |
"hidden_chart_oracle_success": 1.0,
|
| 1646 |
"hidden_chart_oracle_utility": 2.0,
|
| 1647 |
"outcome_ptr": 0.0,
|
| 1648 |
+
"pairwise_causal_calibration_accuracy": 0.5714285714285714,
|
| 1649 |
+
"pairwise_causal_calibration_confidence": 0.5261612748851421,
|
| 1650 |
+
"pairwise_causal_calibration_ece": 0.045267296543429314,
|
| 1651 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 1652 |
"predicted_margin": 0.17159672738556292,
|
| 1653 |
"proposal_oracle_success": 0.0,
|
| 1654 |
"proposal_oracle_utility": 0.15240873396396637,
|
|
|
|
| 1672 |
"hidden_chart_oracle_success": 1.0,
|
| 1673 |
"hidden_chart_oracle_utility": 2.0,
|
| 1674 |
"outcome_ptr": 0.0,
|
| 1675 |
+
"pairwise_causal_calibration_accuracy": 0.39285714285714285,
|
| 1676 |
+
"pairwise_causal_calibration_confidence": 0.5252405539802958,
|
| 1677 |
+
"pairwise_causal_calibration_ece": 0.13238341112315294,
|
| 1678 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 1679 |
"predicted_margin": 0.10074789459755354,
|
| 1680 |
"proposal_oracle_success": 0.0,
|
| 1681 |
"proposal_oracle_utility": 0.1992303878068924,
|
|
|
|
| 1699 |
"hidden_chart_oracle_success": 1.0,
|
| 1700 |
"hidden_chart_oracle_utility": 2.0,
|
| 1701 |
"outcome_ptr": 0.0,
|
| 1702 |
+
"pairwise_causal_calibration_accuracy": 0.5714285714285714,
|
| 1703 |
+
"pairwise_causal_calibration_confidence": 0.5089660423581928,
|
| 1704 |
+
"pairwise_causal_calibration_ece": 0.031231264535189318,
|
| 1705 |
+
"pairwise_causal_calibration_pairs": 14.0,
|
| 1706 |
"predicted_margin": 0.275573013269513,
|
| 1707 |
"proposal_oracle_success": 0.5,
|
| 1708 |
"proposal_oracle_utility": 1.0780495330691338,
|
|
|
|
| 1726 |
"hidden_chart_oracle_success": 1.0,
|
| 1727 |
"hidden_chart_oracle_utility": 2.0,
|
| 1728 |
"outcome_ptr": 0.0,
|
| 1729 |
+
"pairwise_causal_calibration_accuracy": 0.6785714285714286,
|
| 1730 |
+
"pairwise_causal_calibration_confidence": 0.5076812671130492,
|
| 1731 |
+
"pairwise_causal_calibration_ece": 0.08544508072918972,
|
| 1732 |
+
"pairwise_causal_calibration_pairs": 14.0,
|
| 1733 |
"predicted_margin": 0.2841151465083533,
|
| 1734 |
"proposal_oracle_success": 0.5,
|
| 1735 |
"proposal_oracle_utility": 1.0777441188693047,
|
|
|
|
| 1753 |
"hidden_chart_oracle_success": 1.0,
|
| 1754 |
"hidden_chart_oracle_utility": 2.0,
|
| 1755 |
"outcome_ptr": 0.0,
|
| 1756 |
+
"pairwise_causal_calibration_accuracy": 0.35714285714285715,
|
| 1757 |
+
"pairwise_causal_calibration_confidence": 0.5094631537502677,
|
| 1758 |
+
"pairwise_causal_calibration_ece": 0.0761601483037053,
|
| 1759 |
+
"pairwise_causal_calibration_pairs": 14.0,
|
| 1760 |
"predicted_margin": 0.2679532638886026,
|
| 1761 |
"proposal_oracle_success": 0.5,
|
| 1762 |
"proposal_oracle_utility": 1.053164016455412,
|
|
|
|
| 1780 |
"hidden_chart_oracle_success": 1.0,
|
| 1781 |
"hidden_chart_oracle_utility": 2.0,
|
| 1782 |
"outcome_ptr": 0.0,
|
| 1783 |
+
"pairwise_causal_calibration_accuracy": 0.35714285714285715,
|
| 1784 |
+
"pairwise_causal_calibration_confidence": 0.5193863539393203,
|
| 1785 |
+
"pairwise_causal_calibration_ece": 0.16224349679646316,
|
| 1786 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 1787 |
"predicted_margin": 0.20663971147287907,
|
| 1788 |
"proposal_oracle_success": 0.0,
|
| 1789 |
"proposal_oracle_utility": 0.0642656534910202,
|
|
|
|
| 1807 |
"hidden_chart_oracle_success": 1.0,
|
| 1808 |
"hidden_chart_oracle_utility": 2.0,
|
| 1809 |
"outcome_ptr": 0.0,
|
| 1810 |
+
"pairwise_causal_calibration_accuracy": 0.4642857142857143,
|
| 1811 |
+
"pairwise_causal_calibration_confidence": 0.5172736961846305,
|
| 1812 |
+
"pairwise_causal_calibration_ece": 0.05298798189891618,
|
| 1813 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 1814 |
"predicted_margin": 0.26282861689601783,
|
| 1815 |
"proposal_oracle_success": 0.0,
|
| 1816 |
"proposal_oracle_utility": 0.07914192974567413,
|
|
|
|
| 1834 |
"hidden_chart_oracle_success": 1.0,
|
| 1835 |
"hidden_chart_oracle_utility": 2.0,
|
| 1836 |
"outcome_ptr": 0.0,
|
| 1837 |
+
"pairwise_causal_calibration_accuracy": 0.5,
|
| 1838 |
+
"pairwise_causal_calibration_confidence": 0.5184867483888203,
|
| 1839 |
+
"pairwise_causal_calibration_ece": 0.01848674838882025,
|
| 1840 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 1841 |
"predicted_margin": 0.2871776156075902,
|
| 1842 |
"proposal_oracle_success": 0.0,
|
| 1843 |
"proposal_oracle_utility": 0.06764055788516998,
|
|
|
|
| 1861 |
"hidden_chart_oracle_success": 1.0,
|
| 1862 |
"hidden_chart_oracle_utility": 2.0,
|
| 1863 |
"outcome_ptr": 1.0,
|
| 1864 |
+
"pairwise_causal_calibration_accuracy": 0.5714285714285714,
|
| 1865 |
+
"pairwise_causal_calibration_confidence": 0.5093770540150734,
|
| 1866 |
+
"pairwise_causal_calibration_ece": 0.062051517413498014,
|
| 1867 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 1868 |
"predicted_margin": 0.2818040383848624,
|
| 1869 |
"proposal_oracle_success": 1.0,
|
| 1870 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 1888 |
"hidden_chart_oracle_success": 1.0,
|
| 1889 |
"hidden_chart_oracle_utility": 2.0,
|
| 1890 |
"outcome_ptr": 1.0,
|
| 1891 |
+
"pairwise_causal_calibration_accuracy": 0.7857142857142857,
|
| 1892 |
+
"pairwise_causal_calibration_confidence": 0.5065274413865767,
|
| 1893 |
+
"pairwise_causal_calibration_ece": 0.279186844327709,
|
| 1894 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 1895 |
"predicted_margin": 0.27950117722622586,
|
| 1896 |
"proposal_oracle_success": 0.0,
|
| 1897 |
"proposal_oracle_utility": 0.22683429718017578,
|
|
|
|
| 1915 |
"hidden_chart_oracle_success": 1.0,
|
| 1916 |
"hidden_chart_oracle_utility": 2.0,
|
| 1917 |
"outcome_ptr": 1.0,
|
| 1918 |
+
"pairwise_causal_calibration_accuracy": 0.7857142857142857,
|
| 1919 |
+
"pairwise_causal_calibration_confidence": 0.5094448174745808,
|
| 1920 |
+
"pairwise_causal_calibration_ece": 0.2762694682397049,
|
| 1921 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 1922 |
"predicted_margin": 0.29289339637189027,
|
| 1923 |
"proposal_oracle_success": 0.0,
|
| 1924 |
"proposal_oracle_utility": 0.216936856508255,
|
|
|
|
| 1942 |
"hidden_chart_oracle_success": 0.5,
|
| 1943 |
"hidden_chart_oracle_utility": 1.1139086410403252,
|
| 1944 |
"outcome_ptr": 1.0,
|
| 1945 |
+
"pairwise_causal_calibration_accuracy": 0.4951298701298701,
|
| 1946 |
+
"pairwise_causal_calibration_confidence": 0.507580765557335,
|
| 1947 |
+
"pairwise_causal_calibration_ece": 0.03747258161333558,
|
| 1948 |
+
"pairwise_causal_calibration_pairs": 25.0,
|
| 1949 |
"predicted_margin": 0.10924904877533874,
|
| 1950 |
"proposal_oracle_success": 0.5,
|
| 1951 |
"proposal_oracle_utility": 1.0352020561695099,
|
|
|
|
| 1969 |
"hidden_chart_oracle_success": 0.5,
|
| 1970 |
"hidden_chart_oracle_utility": 1.1139086410403252,
|
| 1971 |
"outcome_ptr": 1.0,
|
| 1972 |
+
"pairwise_causal_calibration_accuracy": 0.6785714285714286,
|
| 1973 |
+
"pairwise_causal_calibration_confidence": 0.5096338287511724,
|
| 1974 |
+
"pairwise_causal_calibration_ece": 0.16893759982025613,
|
| 1975 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 1976 |
"predicted_margin": 0.12740485358567616,
|
| 1977 |
"proposal_oracle_success": 0.5,
|
| 1978 |
"proposal_oracle_utility": 1.0799707099795341,
|
|
|
|
| 1996 |
"hidden_chart_oracle_success": 0.5,
|
| 1997 |
"hidden_chart_oracle_utility": 1.1139086410403252,
|
| 1998 |
"outcome_ptr": 1.0,
|
| 1999 |
+
"pairwise_causal_calibration_accuracy": 0.7421428571428572,
|
| 2000 |
+
"pairwise_causal_calibration_confidence": 0.5138432222487193,
|
| 2001 |
+
"pairwise_causal_calibration_ece": 0.22829963489413796,
|
| 2002 |
+
"pairwise_causal_calibration_pairs": 26.5,
|
| 2003 |
"predicted_margin": 0.18675708897764087,
|
| 2004 |
"proposal_oracle_success": 0.5,
|
| 2005 |
"proposal_oracle_utility": 1.1454492062330246,
|
|
|
|
| 2023 |
"hidden_chart_oracle_success": 1.0,
|
| 2024 |
"hidden_chart_oracle_utility": 2.0,
|
| 2025 |
"outcome_ptr": 0.0,
|
| 2026 |
+
"pairwise_causal_calibration_accuracy": 0.4642857142857143,
|
| 2027 |
+
"pairwise_causal_calibration_confidence": 0.517127112474433,
|
| 2028 |
+
"pairwise_causal_calibration_ece": 0.05284139818871869,
|
| 2029 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 2030 |
"predicted_margin": 0.21756370458471622,
|
| 2031 |
"proposal_oracle_success": 0.0,
|
| 2032 |
"proposal_oracle_utility": 0.20391923189163208,
|
|
|
|
| 2050 |
"hidden_chart_oracle_success": 1.0,
|
| 2051 |
"hidden_chart_oracle_utility": 2.0,
|
| 2052 |
"outcome_ptr": 0.0,
|
| 2053 |
+
"pairwise_causal_calibration_accuracy": 0.32142857142857145,
|
| 2054 |
+
"pairwise_causal_calibration_confidence": 0.5139732937786187,
|
| 2055 |
+
"pairwise_causal_calibration_ece": 0.19254472235004721,
|
| 2056 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 2057 |
"predicted_margin": 0.1801716769818562,
|
| 2058 |
"proposal_oracle_success": 0.0,
|
| 2059 |
"proposal_oracle_utility": 0.20632563531398773,
|
|
|
|
| 2077 |
"hidden_chart_oracle_success": 1.0,
|
| 2078 |
"hidden_chart_oracle_utility": 2.0,
|
| 2079 |
"outcome_ptr": 0.0,
|
| 2080 |
+
"pairwise_causal_calibration_accuracy": 0.4642857142857143,
|
| 2081 |
+
"pairwise_causal_calibration_confidence": 0.5182153571387064,
|
| 2082 |
+
"pairwise_causal_calibration_ece": 0.053929642852992066,
|
| 2083 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 2084 |
"predicted_margin": 0.20469744001592355,
|
| 2085 |
"proposal_oracle_success": 0.0,
|
| 2086 |
"proposal_oracle_utility": 0.20428544282913208,
|
|
|
|
| 2104 |
"hidden_chart_oracle_success": 1.0,
|
| 2105 |
"hidden_chart_oracle_utility": 2.0,
|
| 2106 |
"outcome_ptr": 0.0,
|
| 2107 |
+
"pairwise_causal_calibration_accuracy": 0.14285714285714285,
|
| 2108 |
+
"pairwise_causal_calibration_confidence": 0.5110298208425068,
|
| 2109 |
+
"pairwise_causal_calibration_ece": 0.36817267798536396,
|
| 2110 |
+
"pairwise_causal_calibration_pairs": 7.0,
|
| 2111 |
"predicted_margin": 0.2782043046573905,
|
| 2112 |
"proposal_oracle_success": 1.0,
|
| 2113 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 2131 |
"hidden_chart_oracle_success": 1.0,
|
| 2132 |
"hidden_chart_oracle_utility": 2.0,
|
| 2133 |
"outcome_ptr": 0.0,
|
| 2134 |
+
"pairwise_causal_calibration_accuracy": 0.2857142857142857,
|
| 2135 |
+
"pairwise_causal_calibration_confidence": 0.507869130131741,
|
| 2136 |
+
"pairwise_causal_calibration_ece": 0.22215484441745526,
|
| 2137 |
+
"pairwise_causal_calibration_pairs": 7.0,
|
| 2138 |
"predicted_margin": 0.2854092085136363,
|
| 2139 |
"proposal_oracle_success": 1.0,
|
| 2140 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 2158 |
"hidden_chart_oracle_success": 1.0,
|
| 2159 |
"hidden_chart_oracle_utility": 2.0,
|
| 2160 |
"outcome_ptr": 0.0,
|
| 2161 |
+
"pairwise_causal_calibration_accuracy": 0.32142857142857145,
|
| 2162 |
+
"pairwise_causal_calibration_confidence": 0.5063674605891497,
|
| 2163 |
+
"pairwise_causal_calibration_ece": 0.18493888916057827,
|
| 2164 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 2165 |
"predicted_margin": 0.29579179447511267,
|
| 2166 |
"proposal_oracle_success": 1.0,
|
| 2167 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 2185 |
"hidden_chart_oracle_success": 1.0,
|
| 2186 |
"hidden_chart_oracle_utility": 2.0,
|
| 2187 |
"outcome_ptr": 0.0,
|
| 2188 |
+
"pairwise_causal_calibration_accuracy": 0.14285714285714285,
|
| 2189 |
+
"pairwise_causal_calibration_confidence": 0.5052959089864312,
|
| 2190 |
+
"pairwise_causal_calibration_ece": 0.3624387661292883,
|
| 2191 |
+
"pairwise_causal_calibration_pairs": 7.0,
|
| 2192 |
"predicted_margin": 0.6870668145379031,
|
| 2193 |
"proposal_oracle_success": 1.0,
|
| 2194 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 2212 |
"hidden_chart_oracle_success": 1.0,
|
| 2213 |
"hidden_chart_oracle_utility": 2.0,
|
| 2214 |
"outcome_ptr": 0.0,
|
| 2215 |
+
"pairwise_causal_calibration_accuracy": 1.0,
|
| 2216 |
+
"pairwise_causal_calibration_confidence": 0.5089048821442496,
|
| 2217 |
+
"pairwise_causal_calibration_ece": 0.49109511785575044,
|
| 2218 |
+
"pairwise_causal_calibration_pairs": 7.0,
|
| 2219 |
"predicted_margin": 0.7609842811296632,
|
| 2220 |
"proposal_oracle_success": 1.0,
|
| 2221 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 2239 |
"hidden_chart_oracle_success": 1.0,
|
| 2240 |
"hidden_chart_oracle_utility": 2.0,
|
| 2241 |
"outcome_ptr": 0.0,
|
| 2242 |
+
"pairwise_causal_calibration_ece": 0.0,
|
| 2243 |
+
"pairwise_causal_calibration_pairs": 0.0,
|
| 2244 |
"predicted_margin": 0.6954287552895091,
|
| 2245 |
"proposal_oracle_success": 1.0,
|
| 2246 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 2264 |
"hidden_chart_oracle_success": 1.0,
|
| 2265 |
"hidden_chart_oracle_utility": 2.0,
|
| 2266 |
"outcome_ptr": 1.0,
|
| 2267 |
+
"pairwise_causal_calibration_accuracy": 0.2857142857142857,
|
| 2268 |
+
"pairwise_causal_calibration_confidence": 0.5192681731031695,
|
| 2269 |
+
"pairwise_causal_calibration_ece": 0.23355388738888383,
|
| 2270 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 2271 |
"predicted_margin": 0.2043317342807088,
|
| 2272 |
"proposal_oracle_success": 0.0,
|
| 2273 |
"proposal_oracle_utility": 0.5837622880935669,
|
|
|
|
| 2291 |
"hidden_chart_oracle_success": 1.0,
|
| 2292 |
"hidden_chart_oracle_utility": 2.0,
|
| 2293 |
"outcome_ptr": 1.0,
|
| 2294 |
+
"pairwise_causal_calibration_accuracy": 0.32142857142857145,
|
| 2295 |
+
"pairwise_causal_calibration_confidence": 0.5155934510473184,
|
| 2296 |
+
"pairwise_causal_calibration_ece": 0.1941648796187469,
|
| 2297 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 2298 |
"predicted_margin": 0.16033886964133004,
|
| 2299 |
"proposal_oracle_success": 0.0,
|
| 2300 |
"proposal_oracle_utility": 0.7209141850471497,
|
|
|
|
| 2318 |
"hidden_chart_oracle_success": 1.0,
|
| 2319 |
"hidden_chart_oracle_utility": 2.0,
|
| 2320 |
"outcome_ptr": 0.0,
|
| 2321 |
+
"pairwise_causal_calibration_accuracy": 0.14285714285714285,
|
| 2322 |
+
"pairwise_causal_calibration_confidence": 0.5184906548624142,
|
| 2323 |
+
"pairwise_causal_calibration_ece": 0.37563351200527134,
|
| 2324 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 2325 |
"predicted_margin": 0.19688713080112988,
|
| 2326 |
"proposal_oracle_success": 0.0,
|
| 2327 |
"proposal_oracle_utility": 0.21979393064975739,
|
|
|
|
| 2345 |
"hidden_chart_oracle_success": 1.0,
|
| 2346 |
"hidden_chart_oracle_utility": 2.0,
|
| 2347 |
"outcome_ptr": 1.0,
|
| 2348 |
+
"pairwise_causal_calibration_accuracy": 0.5357142857142857,
|
| 2349 |
+
"pairwise_causal_calibration_confidence": 0.5059629842951442,
|
| 2350 |
+
"pairwise_causal_calibration_ece": 0.029751301419141507,
|
| 2351 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 2352 |
"predicted_margin": 0.6471664922001317,
|
| 2353 |
"proposal_oracle_success": 0.0,
|
| 2354 |
"proposal_oracle_utility": 0.12932512164115906,
|
|
|
|
| 2372 |
"hidden_chart_oracle_success": 1.0,
|
| 2373 |
"hidden_chart_oracle_utility": 2.0,
|
| 2374 |
"outcome_ptr": 1.0,
|
| 2375 |
+
"pairwise_causal_calibration_accuracy": 0.9090909090909091,
|
| 2376 |
+
"pairwise_causal_calibration_confidence": 0.5089098400968083,
|
| 2377 |
+
"pairwise_causal_calibration_ece": 0.4001810689941008,
|
| 2378 |
+
"pairwise_causal_calibration_pairs": 22.0,
|
| 2379 |
"predicted_margin": 0.6675461655230028,
|
| 2380 |
"proposal_oracle_success": 1.0,
|
| 2381 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 2399 |
"hidden_chart_oracle_success": 1.0,
|
| 2400 |
"hidden_chart_oracle_utility": 2.0,
|
| 2401 |
"outcome_ptr": 1.0,
|
| 2402 |
+
"pairwise_causal_calibration_accuracy": 0.7777777777777778,
|
| 2403 |
+
"pairwise_causal_calibration_confidence": 0.5070230790958891,
|
| 2404 |
+
"pairwise_causal_calibration_ece": 0.27075469868188873,
|
| 2405 |
+
"pairwise_causal_calibration_pairs": 27.0,
|
| 2406 |
"predicted_margin": 0.6010746259173304,
|
| 2407 |
"proposal_oracle_success": 1.0,
|
| 2408 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 2426 |
"hidden_chart_oracle_success": 1.0,
|
| 2427 |
"hidden_chart_oracle_utility": 2.0,
|
| 2428 |
"outcome_ptr": 0.0,
|
| 2429 |
+
"pairwise_causal_calibration_accuracy": 0.5555555555555556,
|
| 2430 |
+
"pairwise_causal_calibration_confidence": 0.5104367146945467,
|
| 2431 |
+
"pairwise_causal_calibration_ece": 0.045118840861008924,
|
| 2432 |
+
"pairwise_causal_calibration_pairs": 18.0,
|
| 2433 |
"predicted_margin": 0.09578732958371006,
|
| 2434 |
"proposal_oracle_success": 1.0,
|
| 2435 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 2453 |
"hidden_chart_oracle_success": 1.0,
|
| 2454 |
"hidden_chart_oracle_utility": 2.0,
|
| 2455 |
"outcome_ptr": 0.0,
|
| 2456 |
+
"pairwise_causal_calibration_accuracy": 0.42857142857142855,
|
| 2457 |
+
"pairwise_causal_calibration_confidence": 0.5082796231877765,
|
| 2458 |
+
"pairwise_causal_calibration_ece": 0.07970819461634798,
|
| 2459 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 2460 |
"predicted_margin": 0.10249988539655566,
|
| 2461 |
"proposal_oracle_success": 0.0,
|
| 2462 |
"proposal_oracle_utility": 0.1290181279182434,
|
|
|
|
| 2480 |
"hidden_chart_oracle_success": 1.0,
|
| 2481 |
"hidden_chart_oracle_utility": 2.0,
|
| 2482 |
"outcome_ptr": 0.0,
|
| 2483 |
+
"pairwise_causal_calibration_accuracy": 0.5357142857142857,
|
| 2484 |
+
"pairwise_causal_calibration_confidence": 0.5113970881858346,
|
| 2485 |
+
"pairwise_causal_calibration_ece": 0.024317197528451073,
|
| 2486 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 2487 |
"predicted_margin": 0.09962893251516124,
|
| 2488 |
"proposal_oracle_success": 1.0,
|
| 2489 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 2507 |
"hidden_chart_oracle_success": 1.0,
|
| 2508 |
"hidden_chart_oracle_utility": 2.0,
|
| 2509 |
"outcome_ptr": 0.0,
|
| 2510 |
+
"pairwise_causal_calibration_accuracy": 0.6428571428571429,
|
| 2511 |
+
"pairwise_causal_calibration_confidence": 0.5093751939822863,
|
| 2512 |
+
"pairwise_causal_calibration_ece": 0.13348194887485665,
|
| 2513 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 2514 |
"predicted_margin": 0.1717547088237561,
|
| 2515 |
"proposal_oracle_success": 1.0,
|
| 2516 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 2534 |
"hidden_chart_oracle_success": 1.0,
|
| 2535 |
"hidden_chart_oracle_utility": 2.0,
|
| 2536 |
"outcome_ptr": 0.0,
|
| 2537 |
+
"pairwise_causal_calibration_accuracy": 0.3888888888888889,
|
| 2538 |
+
"pairwise_causal_calibration_confidence": 0.5098943896149679,
|
| 2539 |
+
"pairwise_causal_calibration_ece": 0.12100550072607902,
|
| 2540 |
+
"pairwise_causal_calibration_pairs": 18.0,
|
| 2541 |
"predicted_margin": 0.20877294341834293,
|
| 2542 |
"proposal_oracle_success": 1.0,
|
| 2543 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 2561 |
"hidden_chart_oracle_success": 1.0,
|
| 2562 |
"hidden_chart_oracle_utility": 2.0,
|
| 2563 |
"outcome_ptr": 0.0,
|
| 2564 |
+
"pairwise_causal_calibration_accuracy": 0.32142857142857145,
|
| 2565 |
+
"pairwise_causal_calibration_confidence": 0.5120271305387815,
|
| 2566 |
+
"pairwise_causal_calibration_ece": 0.19059855911021,
|
| 2567 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 2568 |
"predicted_margin": 0.13802477297102525,
|
| 2569 |
"proposal_oracle_success": 0.0,
|
| 2570 |
"proposal_oracle_utility": 0.5664259791374207,
|
|
|
|
| 2588 |
"hidden_chart_oracle_success": 0.0,
|
| 2589 |
"hidden_chart_oracle_utility": 0.16558238118886948,
|
| 2590 |
"outcome_ptr": 1.0,
|
| 2591 |
+
"pairwise_causal_calibration_accuracy": 0.4821428571428571,
|
| 2592 |
+
"pairwise_causal_calibration_confidence": 0.5113876468965113,
|
| 2593 |
+
"pairwise_causal_calibration_ece": 0.04992710785321339,
|
| 2594 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 2595 |
"predicted_margin": 0.15020438296667366,
|
| 2596 |
"proposal_oracle_success": 0.0,
|
| 2597 |
"proposal_oracle_utility": 0.04293955350294709,
|
|
|
|
| 2615 |
"hidden_chart_oracle_success": 0.0,
|
| 2616 |
"hidden_chart_oracle_utility": 0.16558238118886948,
|
| 2617 |
"outcome_ptr": 1.0,
|
| 2618 |
+
"pairwise_causal_calibration_accuracy": 0.4642857142857143,
|
| 2619 |
+
"pairwise_causal_calibration_confidence": 0.5116297285361195,
|
| 2620 |
+
"pairwise_causal_calibration_ece": 0.14403731951037171,
|
| 2621 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 2622 |
"predicted_margin": 0.17852089799806223,
|
| 2623 |
"proposal_oracle_success": 0.0,
|
| 2624 |
"proposal_oracle_utility": 0.02655014442279935,
|
|
|
|
| 2642 |
"hidden_chart_oracle_success": 0.0,
|
| 2643 |
"hidden_chart_oracle_utility": 0.16558238118886948,
|
| 2644 |
"outcome_ptr": 1.0,
|
| 2645 |
+
"pairwise_causal_calibration_accuracy": 0.5892857142857142,
|
| 2646 |
+
"pairwise_causal_calibration_confidence": 0.514313264785462,
|
| 2647 |
+
"pairwise_causal_calibration_ece": 0.07497244950025217,
|
| 2648 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 2649 |
"predicted_margin": 0.14307776706871508,
|
| 2650 |
"proposal_oracle_success": 0.0,
|
| 2651 |
"proposal_oracle_utility": 0.03389316890388727,
|
|
|
|
| 2669 |
"hidden_chart_oracle_success": 0.0,
|
| 2670 |
"hidden_chart_oracle_utility": 0.17056432366371155,
|
| 2671 |
"outcome_ptr": 1.0,
|
| 2672 |
+
"pairwise_causal_calibration_accuracy": 0.5,
|
| 2673 |
+
"pairwise_causal_calibration_confidence": 0.5110307445144335,
|
| 2674 |
+
"pairwise_causal_calibration_ece": 0.011030744514433533,
|
| 2675 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 2676 |
"predicted_margin": 0.11748877632482482,
|
| 2677 |
"proposal_oracle_success": 0.0,
|
| 2678 |
"proposal_oracle_utility": 0.09099660068750381,
|
|
|
|
| 2696 |
"hidden_chart_oracle_success": 0.0,
|
| 2697 |
"hidden_chart_oracle_utility": 0.17056432366371155,
|
| 2698 |
"outcome_ptr": 0.0,
|
| 2699 |
+
"pairwise_causal_calibration_accuracy": 0.5357142857142857,
|
| 2700 |
+
"pairwise_causal_calibration_confidence": 0.5087141999697643,
|
| 2701 |
+
"pairwise_causal_calibration_ece": 0.027000085744521396,
|
| 2702 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 2703 |
"predicted_margin": 0.1312853076823468,
|
| 2704 |
"proposal_oracle_success": 0.0,
|
| 2705 |
"proposal_oracle_utility": 0.08514270931482315,
|
|
|
|
| 2723 |
"hidden_chart_oracle_success": 0.0,
|
| 2724 |
"hidden_chart_oracle_utility": 0.17056432366371155,
|
| 2725 |
"outcome_ptr": 0.0,
|
| 2726 |
+
"pairwise_causal_calibration_accuracy": 0.6785714285714286,
|
| 2727 |
+
"pairwise_causal_calibration_confidence": 0.5078943863682678,
|
| 2728 |
+
"pairwise_causal_calibration_ece": 0.17067704220316082,
|
| 2729 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 2730 |
"predicted_margin": 0.1345855010442043,
|
| 2731 |
"proposal_oracle_success": 0.0,
|
| 2732 |
"proposal_oracle_utility": 0.08911379426717758,
|
|
|
|
| 2750 |
"hidden_chart_oracle_success": 1.0,
|
| 2751 |
"hidden_chart_oracle_utility": 2.0,
|
| 2752 |
"outcome_ptr": 0.0,
|
| 2753 |
+
"pairwise_causal_calibration_accuracy": 0.25,
|
| 2754 |
+
"pairwise_causal_calibration_confidence": 0.5196382790206474,
|
| 2755 |
+
"pairwise_causal_calibration_ece": 0.2696382790206474,
|
| 2756 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 2757 |
"predicted_margin": 0.24752873446742565,
|
| 2758 |
"proposal_oracle_success": 0.0,
|
| 2759 |
"proposal_oracle_utility": 0.1631268858909607,
|
|
|
|
| 2777 |
"hidden_chart_oracle_success": 1.0,
|
| 2778 |
"hidden_chart_oracle_utility": 2.0,
|
| 2779 |
"outcome_ptr": 0.0,
|
| 2780 |
+
"pairwise_causal_calibration_accuracy": 0.2857142857142857,
|
| 2781 |
+
"pairwise_causal_calibration_confidence": 0.5166281511611449,
|
| 2782 |
+
"pairwise_causal_calibration_ece": 0.2309138654468592,
|
| 2783 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 2784 |
"predicted_margin": 0.23791187673632117,
|
| 2785 |
"proposal_oracle_success": 0.0,
|
| 2786 |
"proposal_oracle_utility": 0.15427637100219727,
|
|
|
|
| 2804 |
"hidden_chart_oracle_success": 1.0,
|
| 2805 |
"hidden_chart_oracle_utility": 2.0,
|
| 2806 |
"outcome_ptr": 1.0,
|
| 2807 |
+
"pairwise_causal_calibration_accuracy": 0.5714285714285714,
|
| 2808 |
+
"pairwise_causal_calibration_confidence": 0.5152073709393564,
|
| 2809 |
+
"pairwise_causal_calibration_ece": 0.05622120048921497,
|
| 2810 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 2811 |
"predicted_margin": 0.24387011799339325,
|
| 2812 |
"proposal_oracle_success": 0.0,
|
| 2813 |
"proposal_oracle_utility": 0.19178248941898346,
|
|
|
|
| 2831 |
"hidden_chart_oracle_success": 1.0,
|
| 2832 |
"hidden_chart_oracle_utility": 2.0,
|
| 2833 |
"outcome_ptr": 1.0,
|
| 2834 |
+
"pairwise_causal_calibration_accuracy": 0.7272727272727273,
|
| 2835 |
+
"pairwise_causal_calibration_confidence": 0.5152932666576685,
|
| 2836 |
+
"pairwise_causal_calibration_ece": 0.2119794606150588,
|
| 2837 |
+
"pairwise_causal_calibration_pairs": 22.0,
|
| 2838 |
"predicted_margin": 0.0886735921808986,
|
| 2839 |
"proposal_oracle_success": 1.0,
|
| 2840 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 2858 |
"hidden_chart_oracle_success": 1.0,
|
| 2859 |
"hidden_chart_oracle_utility": 2.0,
|
| 2860 |
"outcome_ptr": 1.0,
|
| 2861 |
+
"pairwise_causal_calibration_accuracy": 0.6666666666666666,
|
| 2862 |
+
"pairwise_causal_calibration_confidence": 0.5278734108303779,
|
| 2863 |
+
"pairwise_causal_calibration_ece": 0.13879325583628876,
|
| 2864 |
+
"pairwise_causal_calibration_pairs": 18.0,
|
| 2865 |
"predicted_margin": 0.18825713114760673,
|
| 2866 |
"proposal_oracle_success": 1.0,
|
| 2867 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 2885 |
"hidden_chart_oracle_success": 1.0,
|
| 2886 |
"hidden_chart_oracle_utility": 2.0,
|
| 2887 |
"outcome_ptr": 1.0,
|
| 2888 |
+
"pairwise_causal_calibration_accuracy": 0.12,
|
| 2889 |
+
"pairwise_causal_calibration_confidence": 0.5211353121802774,
|
| 2890 |
+
"pairwise_causal_calibration_ece": 0.4011353121802774,
|
| 2891 |
+
"pairwise_causal_calibration_pairs": 25.0,
|
| 2892 |
"predicted_margin": 0.08486096062823595,
|
| 2893 |
"proposal_oracle_success": 1.0,
|
| 2894 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 2912 |
"hidden_chart_oracle_success": 1.0,
|
| 2913 |
"hidden_chart_oracle_utility": 2.0,
|
| 2914 |
"outcome_ptr": 1.0,
|
| 2915 |
+
"pairwise_causal_calibration_accuracy": 0.3076923076923077,
|
| 2916 |
+
"pairwise_causal_calibration_confidence": 0.511181830700846,
|
| 2917 |
+
"pairwise_causal_calibration_ece": 0.2034895230085383,
|
| 2918 |
+
"pairwise_causal_calibration_pairs": 13.0,
|
| 2919 |
"predicted_margin": 0.2713698658138815,
|
| 2920 |
"proposal_oracle_success": 1.0,
|
| 2921 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 2939 |
"hidden_chart_oracle_success": 1.0,
|
| 2940 |
"hidden_chart_oracle_utility": 2.0,
|
| 2941 |
"outcome_ptr": 1.0,
|
| 2942 |
+
"pairwise_causal_calibration_accuracy": 0.14285714285714285,
|
| 2943 |
+
"pairwise_causal_calibration_confidence": 0.5114740860753181,
|
| 2944 |
+
"pairwise_causal_calibration_ece": 0.36861694321817523,
|
| 2945 |
+
"pairwise_causal_calibration_pairs": 7.0,
|
| 2946 |
"predicted_margin": 0.30277388356719176,
|
| 2947 |
"proposal_oracle_success": 1.0,
|
| 2948 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 2966 |
"hidden_chart_oracle_success": 1.0,
|
| 2967 |
"hidden_chart_oracle_utility": 2.0,
|
| 2968 |
"outcome_ptr": 1.0,
|
| 2969 |
+
"pairwise_causal_calibration_accuracy": 0.6153846153846154,
|
| 2970 |
+
"pairwise_causal_calibration_confidence": 0.5096441739491412,
|
| 2971 |
+
"pairwise_causal_calibration_ece": 0.10574044143547423,
|
| 2972 |
+
"pairwise_causal_calibration_pairs": 13.0,
|
| 2973 |
"predicted_margin": 0.3320023318656702,
|
| 2974 |
"proposal_oracle_success": 1.0,
|
| 2975 |
"proposal_oracle_utility": 2.0,
|
|
|
|
| 2993 |
"hidden_chart_oracle_success": 0.0,
|
| 2994 |
"hidden_chart_oracle_utility": 0.2786632478237152,
|
| 2995 |
"outcome_ptr": 0.0,
|
| 2996 |
+
"pairwise_causal_calibration_accuracy": 0.4642857142857143,
|
| 2997 |
+
"pairwise_causal_calibration_confidence": 0.5082874271968991,
|
| 2998 |
+
"pairwise_causal_calibration_ece": 0.04400171291118482,
|
| 2999 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 3000 |
"predicted_margin": 0.22414376708358918,
|
| 3001 |
"proposal_oracle_success": 0.0,
|
| 3002 |
"proposal_oracle_utility": 0.13453228771686554,
|
|
|
|
| 3020 |
"hidden_chart_oracle_success": 0.0,
|
| 3021 |
"hidden_chart_oracle_utility": 0.2786632478237152,
|
| 3022 |
"outcome_ptr": 0.0,
|
| 3023 |
+
"pairwise_causal_calibration_accuracy": 0.6428571428571429,
|
| 3024 |
+
"pairwise_causal_calibration_confidence": 0.5105870204506016,
|
| 3025 |
+
"pairwise_causal_calibration_ece": 0.13227012240654135,
|
| 3026 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 3027 |
"predicted_margin": 0.27086090181173816,
|
| 3028 |
"proposal_oracle_success": 0.0,
|
| 3029 |
"proposal_oracle_utility": 0.06424945592880249,
|
|
|
|
| 3047 |
"hidden_chart_oracle_success": 0.0,
|
| 3048 |
"hidden_chart_oracle_utility": 0.2786632478237152,
|
| 3049 |
"outcome_ptr": 0.0,
|
| 3050 |
+
"pairwise_causal_calibration_accuracy": 0.21428571428571427,
|
| 3051 |
+
"pairwise_causal_calibration_confidence": 0.508636849456031,
|
| 3052 |
+
"pairwise_causal_calibration_ece": 0.29435113517031675,
|
| 3053 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 3054 |
"predicted_margin": 0.2587694523429491,
|
| 3055 |
"proposal_oracle_success": 0.0,
|
| 3056 |
"proposal_oracle_utility": 0.14127866923809052,
|
|
|
|
| 3074 |
"hidden_chart_oracle_success": 0.6666666666666666,
|
| 3075 |
"hidden_chart_oracle_utility": 1.4079964955647786,
|
| 3076 |
"outcome_ptr": 0.6666666666666666,
|
| 3077 |
+
"pairwise_causal_calibration_accuracy": 0.6331168831168831,
|
| 3078 |
+
"pairwise_causal_calibration_confidence": 0.5114539295387354,
|
| 3079 |
+
"pairwise_causal_calibration_ece": 0.21751195260366055,
|
| 3080 |
+
"pairwise_causal_calibration_pairs": 26.0,
|
| 3081 |
"predicted_margin": 0.16432581896894002,
|
| 3082 |
"proposal_oracle_success": 0.3333333333333333,
|
| 3083 |
"proposal_oracle_utility": 0.7655241700510184,
|
|
|
|
| 3101 |
"hidden_chart_oracle_success": 0.6666666666666666,
|
| 3102 |
"hidden_chart_oracle_utility": 1.4079964955647786,
|
| 3103 |
"outcome_ptr": 1.0,
|
| 3104 |
+
"pairwise_causal_calibration_accuracy": 0.5787545787545787,
|
| 3105 |
+
"pairwise_causal_calibration_confidence": 0.513396376538758,
|
| 3106 |
+
"pairwise_causal_calibration_ece": 0.20120295855494272,
|
| 3107 |
+
"pairwise_causal_calibration_pairs": 23.0,
|
| 3108 |
"predicted_margin": 0.1977895366681389,
|
| 3109 |
"proposal_oracle_success": 0.3333333333333333,
|
| 3110 |
"proposal_oracle_utility": 0.7756740699211756,
|
|
|
|
| 3128 |
"hidden_chart_oracle_success": 0.6666666666666666,
|
| 3129 |
"hidden_chart_oracle_utility": 1.4079964955647786,
|
| 3130 |
"outcome_ptr": 0.6666666666666666,
|
| 3131 |
+
"pairwise_causal_calibration_accuracy": 0.6017316017316018,
|
| 3132 |
+
"pairwise_causal_calibration_confidence": 0.5153965226212408,
|
| 3133 |
+
"pairwise_causal_calibration_ece": 0.08633507911036098,
|
| 3134 |
+
"pairwise_causal_calibration_pairs": 26.0,
|
| 3135 |
"predicted_margin": 0.2141060722436662,
|
| 3136 |
"proposal_oracle_success": 0.3333333333333333,
|
| 3137 |
"proposal_oracle_utility": 0.7652813941240311,
|
|
|
|
| 3155 |
"hidden_chart_oracle_success": 0.25,
|
| 3156 |
"hidden_chart_oracle_utility": 0.6737305596470833,
|
| 3157 |
"outcome_ptr": 0.25,
|
| 3158 |
+
"pairwise_causal_calibration_accuracy": 0.6190476190476191,
|
| 3159 |
+
"pairwise_causal_calibration_confidence": 0.5090917071150636,
|
| 3160 |
+
"pairwise_causal_calibration_ece": 0.10703103606299277,
|
| 3161 |
+
"pairwise_causal_calibration_pairs": 21.0,
|
| 3162 |
"predicted_margin": 0.203919201209952,
|
| 3163 |
"proposal_oracle_success": 0.5,
|
| 3164 |
"proposal_oracle_utility": 1.0471355519257486,
|
|
|
|
| 3182 |
"hidden_chart_oracle_success": 0.25,
|
| 3183 |
"hidden_chart_oracle_utility": 0.6737305596470833,
|
| 3184 |
"outcome_ptr": 0.25,
|
| 3185 |
+
"pairwise_causal_calibration_accuracy": 0.4642857142857143,
|
| 3186 |
+
"pairwise_causal_calibration_confidence": 0.5071819889949212,
|
| 3187 |
+
"pairwise_causal_calibration_ece": 0.10997339402687645,
|
| 3188 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 3189 |
"predicted_margin": 0.20002804565583177,
|
| 3190 |
"proposal_oracle_success": 0.0,
|
| 3191 |
"proposal_oracle_utility": 0.10832798760384321,
|
|
|
|
| 3209 |
"hidden_chart_oracle_success": 0.25,
|
| 3210 |
"hidden_chart_oracle_utility": 0.6737305596470833,
|
| 3211 |
"outcome_ptr": 0.25,
|
| 3212 |
+
"pairwise_causal_calibration_accuracy": 0.6339285714285715,
|
| 3213 |
+
"pairwise_causal_calibration_confidence": 0.5087171759695033,
|
| 3214 |
+
"pairwise_causal_calibration_ece": 0.21759707156370559,
|
| 3215 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 3216 |
"predicted_margin": 0.19413727025774286,
|
| 3217 |
"proposal_oracle_success": 0.25,
|
| 3218 |
"proposal_oracle_utility": 0.5676866680150852,
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_success_tanh_train_to_test/metrics_by_task.json
CHANGED
|
@@ -8,6 +8,10 @@
|
|
| 8 |
"hidden_chart_oracle_success": 1.0,
|
| 9 |
"hidden_chart_oracle_utility": 2.0,
|
| 10 |
"outcome_ptr": 0.13333333333333333,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
"predicted_margin": 0.18356824589060186,
|
| 12 |
"proposal_oracle_success": 0.4666666666666667,
|
| 13 |
"proposal_oracle_utility": 1.1608628551165263,
|
|
@@ -31,6 +35,10 @@
|
|
| 31 |
"hidden_chart_oracle_success": 1.0,
|
| 32 |
"hidden_chart_oracle_utility": 2.0,
|
| 33 |
"outcome_ptr": 0.0,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 34 |
"predicted_margin": 0.3118749611515581,
|
| 35 |
"proposal_oracle_success": 0.0,
|
| 36 |
"proposal_oracle_utility": 0.1308078020811081,
|
|
@@ -54,6 +62,10 @@
|
|
| 54 |
"hidden_chart_oracle_success": 0.52,
|
| 55 |
"hidden_chart_oracle_utility": 1.1467878860235214,
|
| 56 |
"outcome_ptr": 0.38666666666666666,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 57 |
"predicted_margin": 0.18544317093400783,
|
| 58 |
"proposal_oracle_success": 0.3333333333333333,
|
| 59 |
"proposal_oracle_utility": 0.7346437490296861,
|
|
@@ -77,6 +89,10 @@
|
|
| 77 |
"hidden_chart_oracle_success": 0.8333333333333334,
|
| 78 |
"hidden_chart_oracle_utility": 1.7015800600250561,
|
| 79 |
"outcome_ptr": 0.8333333333333334,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 80 |
"predicted_margin": 0.6558894495703831,
|
| 81 |
"proposal_oracle_success": 0.6111111111111112,
|
| 82 |
"proposal_oracle_utility": 1.2712341505620215,
|
|
@@ -100,6 +116,10 @@
|
|
| 100 |
"hidden_chart_oracle_success": 1.0,
|
| 101 |
"hidden_chart_oracle_utility": 2.0,
|
| 102 |
"outcome_ptr": 0.42424242424242425,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 103 |
"predicted_margin": 0.2716308071762482,
|
| 104 |
"proposal_oracle_success": 0.36363636363636365,
|
| 105 |
"proposal_oracle_utility": 0.8693077586817018,
|
|
|
|
| 8 |
"hidden_chart_oracle_success": 1.0,
|
| 9 |
"hidden_chart_oracle_utility": 2.0,
|
| 10 |
"outcome_ptr": 0.13333333333333333,
|
| 11 |
+
"pairwise_causal_calibration_accuracy": 0.4580086580086581,
|
| 12 |
+
"pairwise_causal_calibration_confidence": 0.5122062630509082,
|
| 13 |
+
"pairwise_causal_calibration_ece": 0.13568622647194786,
|
| 14 |
+
"pairwise_causal_calibration_pairs": 26.266666666666666,
|
| 15 |
"predicted_margin": 0.18356824589060186,
|
| 16 |
"proposal_oracle_success": 0.4666666666666667,
|
| 17 |
"proposal_oracle_utility": 1.1608628551165263,
|
|
|
|
| 35 |
"hidden_chart_oracle_success": 1.0,
|
| 36 |
"hidden_chart_oracle_utility": 2.0,
|
| 37 |
"outcome_ptr": 0.0,
|
| 38 |
+
"pairwise_causal_calibration_accuracy": 0.5476190476190476,
|
| 39 |
+
"pairwise_causal_calibration_confidence": 0.5195909959026942,
|
| 40 |
+
"pairwise_causal_calibration_ece": 0.13157805359620875,
|
| 41 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 42 |
"predicted_margin": 0.3118749611515581,
|
| 43 |
"proposal_oracle_success": 0.0,
|
| 44 |
"proposal_oracle_utility": 0.1308078020811081,
|
|
|
|
| 62 |
"hidden_chart_oracle_success": 0.52,
|
| 63 |
"hidden_chart_oracle_utility": 1.1467878860235214,
|
| 64 |
"outcome_ptr": 0.38666666666666666,
|
| 65 |
+
"pairwise_causal_calibration_accuracy": 0.527235771414876,
|
| 66 |
+
"pairwise_causal_calibration_confidence": 0.5116387196406088,
|
| 67 |
+
"pairwise_causal_calibration_ece": 0.11664272491716678,
|
| 68 |
+
"pairwise_causal_calibration_pairs": 24.133333333333333,
|
| 69 |
"predicted_margin": 0.18544317093400783,
|
| 70 |
"proposal_oracle_success": 0.3333333333333333,
|
| 71 |
"proposal_oracle_utility": 0.7346437490296861,
|
|
|
|
| 89 |
"hidden_chart_oracle_success": 0.8333333333333334,
|
| 90 |
"hidden_chart_oracle_utility": 1.7015800600250561,
|
| 91 |
"outcome_ptr": 0.8333333333333334,
|
| 92 |
+
"pairwise_causal_calibration_accuracy": 0.5353896103896104,
|
| 93 |
+
"pairwise_causal_calibration_confidence": 0.5077484539746286,
|
| 94 |
+
"pairwise_causal_calibration_ece": 0.20395163422426912,
|
| 95 |
+
"pairwise_causal_calibration_pairs": 19.055555555555557,
|
| 96 |
"predicted_margin": 0.6558894495703831,
|
| 97 |
"proposal_oracle_success": 0.6111111111111112,
|
| 98 |
"proposal_oracle_utility": 1.2712341505620215,
|
|
|
|
| 116 |
"hidden_chart_oracle_success": 1.0,
|
| 117 |
"hidden_chart_oracle_utility": 2.0,
|
| 118 |
"outcome_ptr": 0.42424242424242425,
|
| 119 |
+
"pairwise_causal_calibration_accuracy": 0.4395029213211031,
|
| 120 |
+
"pairwise_causal_calibration_confidence": 0.5138835458798818,
|
| 121 |
+
"pairwise_causal_calibration_ece": 0.16288926599179274,
|
| 122 |
+
"pairwise_causal_calibration_pairs": 24.818181818181817,
|
| 123 |
"predicted_margin": 0.2716308071762482,
|
| 124 |
"proposal_oracle_success": 0.36363636363636365,
|
| 125 |
"proposal_oracle_utility": 0.8693077586817018,
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_success_tanh_train_to_test/table.tex
CHANGED
|
@@ -1,8 +1,8 @@
|
|
| 1 |
% Auto-generated by scripts/eval_learned_dominance_selector.py
|
| 2 |
-
\begin{tabular}{
|
| 3 |
\toprule
|
| 4 |
-
Rows & Coverage & Fallback & Base succ. & Selected succ. & Oracle succ. & OutcomePTR & Succ. support gap & Succ. selector gap \\
|
| 5 |
\midrule
|
| 6 |
-
144 & 0.1806 & 0.8194 & 0.3333 & 0.3819 & 0.3819 & 0.4167 & 0.3681 & 0.1250 \\
|
| 7 |
\bottomrule
|
| 8 |
\end{tabular}
|
|
|
|
| 1 |
% Auto-generated by scripts/eval_learned_dominance_selector.py
|
| 2 |
+
\begin{tabular}{lrrrrrrrrr}
|
| 3 |
\toprule
|
| 4 |
+
Rows & Coverage & Fallback & Base succ. & Selected succ. & Oracle succ. & OutcomePTR & Succ. support gap & Succ. selector gap & Cal. ECE \\
|
| 5 |
\midrule
|
| 6 |
+
144 & 0.1806 & 0.8194 & 0.3333 & 0.3819 & 0.3819 & 0.4167 & 0.3681 & 0.1250 & 0.0055 \\
|
| 7 |
\bottomrule
|
| 8 |
\end{tabular}
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_success_tanh_train_to_test/train.log
CHANGED
|
@@ -1,4 +1,5 @@
|
|
| 1 |
trained ridge dominance calibrator on calibration measured rows only
|
|
|
|
| 2 |
selected_lambda=0.0
|
| 3 |
tau=0.315251
|
| 4 |
calibration_input=runs/ctt_base_context_obs_train_cal_tanh_rollout_comparison/combined_measured_candidates.json
|
|
|
|
| 1 |
trained ridge dominance calibrator on calibration measured rows only
|
| 2 |
+
fit_objective=pointwise
|
| 3 |
selected_lambda=0.0
|
| 4 |
tau=0.315251
|
| 5 |
calibration_input=runs/ctt_base_context_obs_train_cal_tanh_rollout_comparison/combined_measured_candidates.json
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_perdim_trainmax_train_to_test/command.txt
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
python scripts/eval_learned_dominance_selector.py --calibration-input runs/ctt_base_context_obs_train_cal_perdim_trainmax_rollout_comparison/combined_measured_candidates.json --calibration-target-index data/cil_charts_rgb_refs/train/index.json --eval-input runs/ctt_base_context_obs_test_perdim_trainmax_rollout_comparison/combined_measured_candidates.json --eval-target-index data/cil_charts_rgb_refs/test/index.json --checkpoint-template runs/ctt_residual_base_context_obs_seed{seed}/model.pt --out-dir runs/ctt_base_context_obs_learned_dominance_context_tangent_perdim_trainmax_train_to_test --feature-set context_tangent --target utility_margin --k 8 --bootstrap-samples 1000 --no-markdown-report
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_perdim_trainmax_train_to_test/config.yaml
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
bootstrap_samples: 1000
|
| 2 |
+
calibration_input: runs/ctt_base_context_obs_train_cal_perdim_trainmax_rollout_comparison/combined_measured_candidates.json
|
| 3 |
+
calibration_target_index: data/cil_charts_rgb_refs/train/index.json
|
| 4 |
+
checkpoint_template: runs/ctt_residual_base_context_obs_seed{seed}/model.pt
|
| 5 |
+
eval_input: runs/ctt_base_context_obs_test_perdim_trainmax_rollout_comparison/combined_measured_candidates.json
|
| 6 |
+
eval_target_index: data/cil_charts_rgb_refs/test/index.json
|
| 7 |
+
feature_set: context_tangent
|
| 8 |
+
fit_objective: pointwise
|
| 9 |
+
k: 8
|
| 10 |
+
no_markdown_report: True
|
| 11 |
+
out_dir: runs/ctt_base_context_obs_learned_dominance_context_tangent_perdim_trainmax_train_to_test
|
| 12 |
+
pairwise_weight: 1.0
|
| 13 |
+
ridge_lambdas: 0,0.01,0.1,1,10,100
|
| 14 |
+
selector_chart_feature_mode: base_context_obs_obj
|
| 15 |
+
source_index: None
|
| 16 |
+
success_bonus: 1.0
|
| 17 |
+
target: utility_margin
|
| 18 |
+
threshold_scope: global
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_perdim_trainmax_train_to_test/data_hash.txt
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"calibration_input": "130d86b8c5ba5dc901ee48835454693bf42c361803c01f6b2d80590cb2493491",
|
| 3 |
+
"calibration_target_index": "67973c50072aa87a3b97533b6292b08a8b4d88a95571dc22e91b6e8fd19e0d85",
|
| 4 |
+
"eval_input": "88d4bfe8c95f8d1579cbe59096cf665dc5f2dfdd99481a386a25e1f3cfa2b09d",
|
| 5 |
+
"eval_target_index": "dee1b5225485df6624747dd4c5074bcf514685b54017d05e698d2051638fbf36"
|
| 6 |
+
}
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_perdim_trainmax_train_to_test/eval.log
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
evaluated learned fallback rule on held-out measured rollout rows
|
| 2 |
+
eval_input=runs/ctt_base_context_obs_test_perdim_trainmax_rollout_comparison/combined_measured_candidates.json
|
| 3 |
+
num_eval_rows=144
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_perdim_trainmax_train_to_test/git_hash.txt
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
14e81292e2004925f81e0bb4810f6f144bfd87b7
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_perdim_trainmax_train_to_test/metrics.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_perdim_trainmax_train_to_test/metrics_by_seed.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_perdim_trainmax_train_to_test/metrics_by_task.json
ADDED
|
@@ -0,0 +1,137 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"LiftPegUpright-v1": {
|
| 3 |
+
"base_success": 0.0,
|
| 4 |
+
"base_utility": 0.4796089559793472,
|
| 5 |
+
"coverage": 0.9333333333333333,
|
| 6 |
+
"execute_generated": 0.9333333333333333,
|
| 7 |
+
"fallback_rate": 0.06666666666666667,
|
| 8 |
+
"hidden_chart_oracle_success": 1.0,
|
| 9 |
+
"hidden_chart_oracle_utility": 2.0,
|
| 10 |
+
"outcome_ptr": 0.3333333333333333,
|
| 11 |
+
"pairwise_causal_calibration_accuracy": 0.44400352733686066,
|
| 12 |
+
"pairwise_causal_calibration_confidence": 0.5297947934025979,
|
| 13 |
+
"pairwise_causal_calibration_ece": 0.17610616365730683,
|
| 14 |
+
"pairwise_causal_calibration_pairs": 27.933333333333334,
|
| 15 |
+
"predicted_margin": 0.044647164658061525,
|
| 16 |
+
"proposal_oracle_success": 0.13333333333333333,
|
| 17 |
+
"proposal_oracle_utility": 0.6601470321416855,
|
| 18 |
+
"selected_candidate_index": 4.8,
|
| 19 |
+
"selected_success": 0.0,
|
| 20 |
+
"selected_success_gain_over_base": 0.0,
|
| 21 |
+
"selected_utility": 0.46326595544815063,
|
| 22 |
+
"selected_utility_gain_over_base": -0.016343000531196594,
|
| 23 |
+
"selector_regret": 0.19712601602077484,
|
| 24 |
+
"success_selector_gap": 0.13333333333333333,
|
| 25 |
+
"success_support_gap": 0.8666666666666667,
|
| 26 |
+
"support_gap": 1.3398529678583144,
|
| 27 |
+
"tau": -0.09621934768667259
|
| 28 |
+
},
|
| 29 |
+
"PickCube-v1": {
|
| 30 |
+
"base_success": 1.0,
|
| 31 |
+
"base_utility": 2.0,
|
| 32 |
+
"coverage": 0.0,
|
| 33 |
+
"execute_generated": 0.0,
|
| 34 |
+
"fallback_rate": 1.0,
|
| 35 |
+
"hidden_chart_oracle_success": 1.0,
|
| 36 |
+
"hidden_chart_oracle_utility": 2.0,
|
| 37 |
+
"outcome_ptr": 0.0,
|
| 38 |
+
"pairwise_causal_calibration_accuracy": 0.6904761904761904,
|
| 39 |
+
"pairwise_causal_calibration_confidence": 0.5548326273106553,
|
| 40 |
+
"pairwise_causal_calibration_ece": 0.13565268355755775,
|
| 41 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 42 |
+
"predicted_margin": -0.5743526430578836,
|
| 43 |
+
"proposal_oracle_success": 0.3333333333333333,
|
| 44 |
+
"proposal_oracle_utility": 1.021850009759267,
|
| 45 |
+
"selected_candidate_index": 5.0,
|
| 46 |
+
"selected_success": 1.0,
|
| 47 |
+
"selected_success_gain_over_base": 0.0,
|
| 48 |
+
"selected_utility": 2.0,
|
| 49 |
+
"selected_utility_gain_over_base": 0.0,
|
| 50 |
+
"selector_regret": 0.0,
|
| 51 |
+
"success_selector_gap": 0.0,
|
| 52 |
+
"success_support_gap": 0.6666666666666666,
|
| 53 |
+
"support_gap": 0.9781499902407328,
|
| 54 |
+
"tau": -0.09621934768667256
|
| 55 |
+
},
|
| 56 |
+
"PullCube-v1": {
|
| 57 |
+
"base_success": 0.36,
|
| 58 |
+
"base_utility": 0.8032506120204925,
|
| 59 |
+
"coverage": 0.6666666666666666,
|
| 60 |
+
"execute_generated": 0.6666666666666666,
|
| 61 |
+
"fallback_rate": 0.3333333333333333,
|
| 62 |
+
"hidden_chart_oracle_success": 0.52,
|
| 63 |
+
"hidden_chart_oracle_utility": 1.1467878860235214,
|
| 64 |
+
"outcome_ptr": 0.32,
|
| 65 |
+
"pairwise_causal_calibration_accuracy": 0.5367965367965366,
|
| 66 |
+
"pairwise_causal_calibration_confidence": 0.5519987946284919,
|
| 67 |
+
"pairwise_causal_calibration_ece": 0.14975573664802355,
|
| 68 |
+
"pairwise_causal_calibration_pairs": 24.16,
|
| 69 |
+
"predicted_margin": 0.051817275904245494,
|
| 70 |
+
"proposal_oracle_success": 0.17333333333333334,
|
| 71 |
+
"proposal_oracle_utility": 0.46170220340291657,
|
| 72 |
+
"selected_candidate_index": 3.3866666666666667,
|
| 73 |
+
"selected_success": 0.30666666666666664,
|
| 74 |
+
"selected_success_gain_over_base": -0.05333333333333334,
|
| 75 |
+
"selected_utility": 0.6904245187963048,
|
| 76 |
+
"selected_utility_gain_over_base": -0.1128260932241877,
|
| 77 |
+
"selector_regret": 0.02100611601024866,
|
| 78 |
+
"success_selector_gap": 0.0,
|
| 79 |
+
"success_support_gap": 0.3466666666666667,
|
| 80 |
+
"support_gap": 0.6881442584097386,
|
| 81 |
+
"tau": -0.09621934768667265
|
| 82 |
+
},
|
| 83 |
+
"PushCube-v1": {
|
| 84 |
+
"base_success": 0.8333333333333334,
|
| 85 |
+
"base_utility": 1.6734557673335075,
|
| 86 |
+
"coverage": 0.0,
|
| 87 |
+
"execute_generated": 0.0,
|
| 88 |
+
"fallback_rate": 1.0,
|
| 89 |
+
"hidden_chart_oracle_success": 0.8333333333333334,
|
| 90 |
+
"hidden_chart_oracle_utility": 1.7015800600250561,
|
| 91 |
+
"outcome_ptr": 0.16666666666666666,
|
| 92 |
+
"pairwise_causal_calibration_accuracy": 0.4658470723176606,
|
| 93 |
+
"pairwise_causal_calibration_confidence": 0.5396225164665516,
|
| 94 |
+
"pairwise_causal_calibration_ece": 0.14011534783179128,
|
| 95 |
+
"pairwise_causal_calibration_pairs": 23.833333333333332,
|
| 96 |
+
"predicted_margin": -0.8998105005052524,
|
| 97 |
+
"proposal_oracle_success": 0.5555555555555556,
|
| 98 |
+
"proposal_oracle_utility": 1.1790093589160178,
|
| 99 |
+
"selected_candidate_index": 3.6666666666666665,
|
| 100 |
+
"selected_success": 0.8333333333333334,
|
| 101 |
+
"selected_success_gain_over_base": 0.0,
|
| 102 |
+
"selected_utility": 1.6734557673335075,
|
| 103 |
+
"selected_utility_gain_over_base": 0.0,
|
| 104 |
+
"selector_regret": 0.32654423266649246,
|
| 105 |
+
"success_selector_gap": 0.16666666666666666,
|
| 106 |
+
"success_support_gap": 0.3333333333333333,
|
| 107 |
+
"support_gap": 0.6220440144340197,
|
| 108 |
+
"tau": -0.09621934768667259
|
| 109 |
+
},
|
| 110 |
+
"StackCube-v1": {
|
| 111 |
+
"base_success": 0.09090909090909091,
|
| 112 |
+
"base_utility": 0.5003613992170854,
|
| 113 |
+
"coverage": 0.7272727272727273,
|
| 114 |
+
"execute_generated": 0.7272727272727273,
|
| 115 |
+
"fallback_rate": 0.2727272727272727,
|
| 116 |
+
"hidden_chart_oracle_success": 1.0,
|
| 117 |
+
"hidden_chart_oracle_utility": 2.0,
|
| 118 |
+
"outcome_ptr": 0.6060606060606061,
|
| 119 |
+
"pairwise_causal_calibration_accuracy": 0.5692640692640694,
|
| 120 |
+
"pairwise_causal_calibration_confidence": 0.5286780647437916,
|
| 121 |
+
"pairwise_causal_calibration_ece": 0.14904052207511279,
|
| 122 |
+
"pairwise_causal_calibration_pairs": 27.818181818181817,
|
| 123 |
+
"predicted_margin": 0.04387425666638764,
|
| 124 |
+
"proposal_oracle_success": 0.18181818181818182,
|
| 125 |
+
"proposal_oracle_utility": 0.634040683056369,
|
| 126 |
+
"selected_candidate_index": 3.9696969696969697,
|
| 127 |
+
"selected_success": 0.06060606060606061,
|
| 128 |
+
"selected_success_gain_over_base": -0.030303030303030304,
|
| 129 |
+
"selected_utility": 0.4272875478773406,
|
| 130 |
+
"selected_utility_gain_over_base": -0.0730738513397448,
|
| 131 |
+
"selector_regret": 0.24322165729421558,
|
| 132 |
+
"success_selector_gap": 0.15151515151515152,
|
| 133 |
+
"success_support_gap": 0.8181818181818182,
|
| 134 |
+
"support_gap": 1.365959316943631,
|
| 135 |
+
"tau": -0.09621934768667252
|
| 136 |
+
}
|
| 137 |
+
}
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_perdim_trainmax_train_to_test/split_hash.txt
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"calibration_target_index": {
|
| 3 |
+
"content_hash": "67f7b4a692f7c7e71da24378dc71e3399c5d35afc14c494869eba6087b765c42",
|
| 4 |
+
"retrieval_index_allowed": true,
|
| 5 |
+
"split": "train",
|
| 6 |
+
"split_hash": "867a94a1c67686c082765671a7bf8d74a9d633085e1b2619c83197c60bf58372"
|
| 7 |
+
},
|
| 8 |
+
"eval_target_index": {
|
| 9 |
+
"content_hash": "0c38d73dc9bc4ab756e12006671b5d12f1a23358b281e54b3f6a9ae3cbf6e9af",
|
| 10 |
+
"retrieval_index_allowed": false,
|
| 11 |
+
"split": "test",
|
| 12 |
+
"split_hash": "84151df9e88c219b76751677558cc5bbbc19082bf358cea7330158e9e403e25f"
|
| 13 |
+
}
|
| 14 |
+
}
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_perdim_trainmax_train_to_test/table.tex
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
% Auto-generated by scripts/eval_learned_dominance_selector.py
|
| 2 |
+
\begin{tabular}{lrrrrrrrrr}
|
| 3 |
+
\toprule
|
| 4 |
+
Rows & Coverage & Fallback & Base succ. & Selected succ. & Oracle succ. & OutcomePTR & Succ. support gap & Succ. selector gap & Cal. ECE \\
|
| 5 |
+
\midrule
|
| 6 |
+
144 & 0.6111 & 0.3889 & 0.3333 & 0.2986 & 0.2222 & 0.3611 & 0.5139 & 0.0694 & 0.0145 \\
|
| 7 |
+
\bottomrule
|
| 8 |
+
\end{tabular}
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_perdim_trainmax_train_to_test/train.log
ADDED
|
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
trained ridge dominance calibrator on calibration measured rows only
|
| 2 |
+
fit_objective=pointwise
|
| 3 |
+
selected_lambda=10.0
|
| 4 |
+
tau=-0.096219
|
| 5 |
+
calibration_input=runs/ctt_base_context_obs_train_cal_perdim_trainmax_rollout_comparison/combined_measured_candidates.json
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_success_perdim_trainmax_train_to_test/command.txt
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
python scripts/eval_learned_dominance_selector.py --calibration-input runs/ctt_base_context_obs_train_cal_perdim_trainmax_rollout_comparison/combined_measured_candidates.json --calibration-target-index data/cil_charts_rgb_refs/train/index.json --eval-input runs/ctt_base_context_obs_test_perdim_trainmax_rollout_comparison/combined_measured_candidates.json --eval-target-index data/cil_charts_rgb_refs/test/index.json --checkpoint-template runs/ctt_residual_base_context_obs_seed{seed}/model.pt --out-dir runs/ctt_base_context_obs_learned_dominance_context_tangent_success_perdim_trainmax_train_to_test --feature-set context_tangent --target success --k 8 --bootstrap-samples 1000 --no-markdown-report
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_success_perdim_trainmax_train_to_test/config.yaml
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
bootstrap_samples: 1000
|
| 2 |
+
calibration_input: runs/ctt_base_context_obs_train_cal_perdim_trainmax_rollout_comparison/combined_measured_candidates.json
|
| 3 |
+
calibration_target_index: data/cil_charts_rgb_refs/train/index.json
|
| 4 |
+
checkpoint_template: runs/ctt_residual_base_context_obs_seed{seed}/model.pt
|
| 5 |
+
eval_input: runs/ctt_base_context_obs_test_perdim_trainmax_rollout_comparison/combined_measured_candidates.json
|
| 6 |
+
eval_target_index: data/cil_charts_rgb_refs/test/index.json
|
| 7 |
+
feature_set: context_tangent
|
| 8 |
+
fit_objective: pointwise
|
| 9 |
+
k: 8
|
| 10 |
+
no_markdown_report: True
|
| 11 |
+
out_dir: runs/ctt_base_context_obs_learned_dominance_context_tangent_success_perdim_trainmax_train_to_test
|
| 12 |
+
pairwise_weight: 1.0
|
| 13 |
+
ridge_lambdas: 0,0.01,0.1,1,10,100
|
| 14 |
+
selector_chart_feature_mode: base_context_obs_obj
|
| 15 |
+
source_index: None
|
| 16 |
+
success_bonus: 1.0
|
| 17 |
+
target: success
|
| 18 |
+
threshold_scope: global
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_success_perdim_trainmax_train_to_test/data_hash.txt
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"calibration_input": "130d86b8c5ba5dc901ee48835454693bf42c361803c01f6b2d80590cb2493491",
|
| 3 |
+
"calibration_target_index": "67973c50072aa87a3b97533b6292b08a8b4d88a95571dc22e91b6e8fd19e0d85",
|
| 4 |
+
"eval_input": "88d4bfe8c95f8d1579cbe59096cf665dc5f2dfdd99481a386a25e1f3cfa2b09d",
|
| 5 |
+
"eval_target_index": "dee1b5225485df6624747dd4c5074bcf514685b54017d05e698d2051638fbf36"
|
| 6 |
+
}
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_success_perdim_trainmax_train_to_test/eval.log
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
evaluated learned fallback rule on held-out measured rollout rows
|
| 2 |
+
eval_input=runs/ctt_base_context_obs_test_perdim_trainmax_rollout_comparison/combined_measured_candidates.json
|
| 3 |
+
num_eval_rows=144
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_success_perdim_trainmax_train_to_test/git_hash.txt
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
14e81292e2004925f81e0bb4810f6f144bfd87b7
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_success_perdim_trainmax_train_to_test/metrics.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_success_perdim_trainmax_train_to_test/metrics_by_seed.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_success_perdim_trainmax_train_to_test/metrics_by_task.json
ADDED
|
@@ -0,0 +1,137 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"LiftPegUpright-v1": {
|
| 3 |
+
"base_success": 0.0,
|
| 4 |
+
"base_utility": 0.4796089559793472,
|
| 5 |
+
"coverage": 0.0,
|
| 6 |
+
"execute_generated": 0.0,
|
| 7 |
+
"fallback_rate": 1.0,
|
| 8 |
+
"hidden_chart_oracle_success": 1.0,
|
| 9 |
+
"hidden_chart_oracle_utility": 2.0,
|
| 10 |
+
"outcome_ptr": 0.3333333333333333,
|
| 11 |
+
"pairwise_causal_calibration_accuracy": 0.3862433862433862,
|
| 12 |
+
"pairwise_causal_calibration_confidence": 0.5110477943182259,
|
| 13 |
+
"pairwise_causal_calibration_ece": 0.17161536008124903,
|
| 14 |
+
"pairwise_causal_calibration_pairs": 27.933333333333334,
|
| 15 |
+
"predicted_margin": 0.11036931560956814,
|
| 16 |
+
"proposal_oracle_success": 0.13333333333333333,
|
| 17 |
+
"proposal_oracle_utility": 0.6601470321416855,
|
| 18 |
+
"selected_candidate_index": 4.733333333333333,
|
| 19 |
+
"selected_success": 0.0,
|
| 20 |
+
"selected_success_gain_over_base": 0.0,
|
| 21 |
+
"selected_utility": 0.4796089559793472,
|
| 22 |
+
"selected_utility_gain_over_base": 0.0,
|
| 23 |
+
"selector_regret": 0.18421521186828613,
|
| 24 |
+
"success_selector_gap": 0.13333333333333333,
|
| 25 |
+
"success_support_gap": 0.8666666666666667,
|
| 26 |
+
"support_gap": 1.3398529678583144,
|
| 27 |
+
"tau": 0.5618318497586812
|
| 28 |
+
},
|
| 29 |
+
"PickCube-v1": {
|
| 30 |
+
"base_success": 1.0,
|
| 31 |
+
"base_utility": 2.0,
|
| 32 |
+
"coverage": 0.0,
|
| 33 |
+
"execute_generated": 0.0,
|
| 34 |
+
"fallback_rate": 1.0,
|
| 35 |
+
"hidden_chart_oracle_success": 1.0,
|
| 36 |
+
"hidden_chart_oracle_utility": 2.0,
|
| 37 |
+
"outcome_ptr": 0.0,
|
| 38 |
+
"pairwise_causal_calibration_accuracy": 0.35714285714285715,
|
| 39 |
+
"pairwise_causal_calibration_confidence": 0.5281130065616054,
|
| 40 |
+
"pairwise_causal_calibration_ece": 0.24812974136863244,
|
| 41 |
+
"pairwise_causal_calibration_pairs": 28.0,
|
| 42 |
+
"predicted_margin": 0.3995110235202226,
|
| 43 |
+
"proposal_oracle_success": 0.3333333333333333,
|
| 44 |
+
"proposal_oracle_utility": 1.021850009759267,
|
| 45 |
+
"selected_candidate_index": 1.3333333333333333,
|
| 46 |
+
"selected_success": 1.0,
|
| 47 |
+
"selected_success_gain_over_base": 0.0,
|
| 48 |
+
"selected_utility": 2.0,
|
| 49 |
+
"selected_utility_gain_over_base": 0.0,
|
| 50 |
+
"selector_regret": 0.0,
|
| 51 |
+
"success_selector_gap": 0.0,
|
| 52 |
+
"success_support_gap": 0.6666666666666666,
|
| 53 |
+
"support_gap": 0.9781499902407328,
|
| 54 |
+
"tau": 0.5618318497586814
|
| 55 |
+
},
|
| 56 |
+
"PullCube-v1": {
|
| 57 |
+
"base_success": 0.36,
|
| 58 |
+
"base_utility": 0.8032506120204925,
|
| 59 |
+
"coverage": 0.08,
|
| 60 |
+
"execute_generated": 0.08,
|
| 61 |
+
"fallback_rate": 0.92,
|
| 62 |
+
"hidden_chart_oracle_success": 0.52,
|
| 63 |
+
"hidden_chart_oracle_utility": 1.1467878860235214,
|
| 64 |
+
"outcome_ptr": 0.32,
|
| 65 |
+
"pairwise_causal_calibration_accuracy": 0.5088661338661338,
|
| 66 |
+
"pairwise_causal_calibration_confidence": 0.522002123999376,
|
| 67 |
+
"pairwise_causal_calibration_ece": 0.13393386645915098,
|
| 68 |
+
"pairwise_causal_calibration_pairs": 24.16,
|
| 69 |
+
"predicted_margin": 0.26581590627488516,
|
| 70 |
+
"proposal_oracle_success": 0.17333333333333334,
|
| 71 |
+
"proposal_oracle_utility": 0.46170220340291657,
|
| 72 |
+
"selected_candidate_index": 2.88,
|
| 73 |
+
"selected_success": 0.3333333333333333,
|
| 74 |
+
"selected_success_gain_over_base": -0.02666666666666667,
|
| 75 |
+
"selected_utility": 0.7537987943490346,
|
| 76 |
+
"selected_utility_gain_over_base": -0.049451817671457926,
|
| 77 |
+
"selector_regret": 0.027659279704093934,
|
| 78 |
+
"success_selector_gap": 0.0,
|
| 79 |
+
"success_support_gap": 0.3466666666666667,
|
| 80 |
+
"support_gap": 0.6881442584097386,
|
| 81 |
+
"tau": 0.5618318497586821
|
| 82 |
+
},
|
| 83 |
+
"PushCube-v1": {
|
| 84 |
+
"base_success": 0.8333333333333334,
|
| 85 |
+
"base_utility": 1.6734557673335075,
|
| 86 |
+
"coverage": 0.3888888888888889,
|
| 87 |
+
"execute_generated": 0.3888888888888889,
|
| 88 |
+
"fallback_rate": 0.6111111111111112,
|
| 89 |
+
"hidden_chart_oracle_success": 0.8333333333333334,
|
| 90 |
+
"hidden_chart_oracle_utility": 1.7015800600250561,
|
| 91 |
+
"outcome_ptr": 0.16666666666666666,
|
| 92 |
+
"pairwise_causal_calibration_accuracy": 0.3967934244404833,
|
| 93 |
+
"pairwise_causal_calibration_confidence": 0.5182152964138562,
|
| 94 |
+
"pairwise_causal_calibration_ece": 0.20331974993456392,
|
| 95 |
+
"pairwise_causal_calibration_pairs": 23.833333333333332,
|
| 96 |
+
"predicted_margin": 0.46411142965639,
|
| 97 |
+
"proposal_oracle_success": 0.5555555555555556,
|
| 98 |
+
"proposal_oracle_utility": 1.1790093589160178,
|
| 99 |
+
"selected_candidate_index": 4.166666666666667,
|
| 100 |
+
"selected_success": 0.5,
|
| 101 |
+
"selected_success_gain_over_base": -0.3333333333333333,
|
| 102 |
+
"selected_utility": 1.0624081848396196,
|
| 103 |
+
"selected_utility_gain_over_base": -0.611047582493888,
|
| 104 |
+
"selector_regret": 0.7349132572611173,
|
| 105 |
+
"success_selector_gap": 0.3888888888888889,
|
| 106 |
+
"success_support_gap": 0.3333333333333333,
|
| 107 |
+
"support_gap": 0.6220440144340197,
|
| 108 |
+
"tau": 0.5618318497586812
|
| 109 |
+
},
|
| 110 |
+
"StackCube-v1": {
|
| 111 |
+
"base_success": 0.09090909090909091,
|
| 112 |
+
"base_utility": 0.5003613992170854,
|
| 113 |
+
"coverage": 0.0,
|
| 114 |
+
"execute_generated": 0.0,
|
| 115 |
+
"fallback_rate": 1.0,
|
| 116 |
+
"hidden_chart_oracle_success": 1.0,
|
| 117 |
+
"hidden_chart_oracle_utility": 2.0,
|
| 118 |
+
"outcome_ptr": 0.6060606060606061,
|
| 119 |
+
"pairwise_causal_calibration_accuracy": 0.5877922077922076,
|
| 120 |
+
"pairwise_causal_calibration_confidence": 0.5114535745455024,
|
| 121 |
+
"pairwise_causal_calibration_ece": 0.12935577712039273,
|
| 122 |
+
"pairwise_causal_calibration_pairs": 27.818181818181817,
|
| 123 |
+
"predicted_margin": 0.13404695454329726,
|
| 124 |
+
"proposal_oracle_success": 0.18181818181818182,
|
| 125 |
+
"proposal_oracle_utility": 0.634040683056369,
|
| 126 |
+
"selected_candidate_index": 4.151515151515151,
|
| 127 |
+
"selected_success": 0.09090909090909091,
|
| 128 |
+
"selected_success_gain_over_base": 0.0,
|
| 129 |
+
"selected_utility": 0.5003613992170854,
|
| 130 |
+
"selected_utility_gain_over_base": 0.0,
|
| 131 |
+
"selector_regret": 0.18030828102068466,
|
| 132 |
+
"success_selector_gap": 0.12121212121212122,
|
| 133 |
+
"success_support_gap": 0.8181818181818182,
|
| 134 |
+
"support_gap": 1.365959316943631,
|
| 135 |
+
"tau": 0.5618318497586813
|
| 136 |
+
}
|
| 137 |
+
}
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_success_perdim_trainmax_train_to_test/split_hash.txt
ADDED
|
@@ -0,0 +1,14 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"calibration_target_index": {
|
| 3 |
+
"content_hash": "67f7b4a692f7c7e71da24378dc71e3399c5d35afc14c494869eba6087b765c42",
|
| 4 |
+
"retrieval_index_allowed": true,
|
| 5 |
+
"split": "train",
|
| 6 |
+
"split_hash": "867a94a1c67686c082765671a7bf8d74a9d633085e1b2619c83197c60bf58372"
|
| 7 |
+
},
|
| 8 |
+
"eval_target_index": {
|
| 9 |
+
"content_hash": "0c38d73dc9bc4ab756e12006671b5d12f1a23358b281e54b3f6a9ae3cbf6e9af",
|
| 10 |
+
"retrieval_index_allowed": false,
|
| 11 |
+
"split": "test",
|
| 12 |
+
"split_hash": "84151df9e88c219b76751677558cc5bbbc19082bf358cea7330158e9e403e25f"
|
| 13 |
+
}
|
| 14 |
+
}
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_success_perdim_trainmax_train_to_test/table.tex
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
% Auto-generated by scripts/eval_learned_dominance_selector.py
|
| 2 |
+
\begin{tabular}{lrrrrrrrrr}
|
| 3 |
+
\toprule
|
| 4 |
+
Rows & Coverage & Fallback & Base succ. & Selected succ. & Oracle succ. & OutcomePTR & Succ. support gap & Succ. selector gap & Cal. ECE \\
|
| 5 |
+
\midrule
|
| 6 |
+
144 & 0.0903 & 0.9097 & 0.3333 & 0.2778 & 0.2222 & 0.3611 & 0.5139 & 0.0903 & 0.0152 \\
|
| 7 |
+
\bottomrule
|
| 8 |
+
\end{tabular}
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_success_perdim_trainmax_train_to_test/train.log
ADDED
|
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
trained ridge dominance calibrator on calibration measured rows only
|
| 2 |
+
fit_objective=pointwise
|
| 3 |
+
selected_lambda=0.0
|
| 4 |
+
tau=0.561832
|
| 5 |
+
calibration_input=runs/ctt_base_context_obs_train_cal_perdim_trainmax_rollout_comparison/combined_measured_candidates.json
|
workspace/runs/ctt_base_context_obs_learned_dominance_context_tangent_success_tanh_train_to_test/command.txt
CHANGED
|
@@ -1 +1 @@
|
|
| 1 |
-
python scripts/eval_learned_dominance_selector.py --calibration-input runs/ctt_base_context_obs_train_cal_tanh_rollout_comparison/combined_measured_candidates.json --calibration-target-index data/cil_charts_rgb_refs/train/index.json --eval-input runs/ctt_base_context_obs_test_tanh_rollout_comparison/combined_measured_candidates.json --eval-target-index data/cil_charts_rgb_refs/test/index.json --checkpoint-template runs/ctt_residual_base_context_obs_seed{seed}/model.pt --out-dir runs/ctt_base_context_obs_learned_dominance_context_tangent_success_tanh_train_to_test --feature-set context_tangent --target success --k 8 --bootstrap-samples 1000
|
|
|
|
| 1 |
+
python scripts/eval_learned_dominance_selector.py --calibration-input runs/ctt_base_context_obs_train_cal_tanh_rollout_comparison/combined_measured_candidates.json --calibration-target-index data/cil_charts_rgb_refs/train/index.json --eval-input runs/ctt_base_context_obs_test_tanh_rollout_comparison/combined_measured_candidates.json --eval-target-index data/cil_charts_rgb_refs/test/index.json --checkpoint-template runs/ctt_residual_base_context_obs_seed{seed}/model.pt --out-dir runs/ctt_base_context_obs_learned_dominance_context_tangent_success_tanh_train_to_test --feature-set context_tangent --target success --k 8 --bootstrap-samples 1000 --no-markdown-report
|