pavan01729 commited on
Commit
7faabc2
·
verified ·
1 Parent(s): 2032738

Upload folder using huggingface_hub

Browse files
Files changed (25) hide show
  1. sld_26.0/panel_counter_v19_rsft/README.md +44 -0
  2. sld_26.0/panel_counter_v19_rsft/eval/phase0_serve.log +4 -0
  3. sld_26.0/panel_counter_v19_rsft/eval/phase2_rsft.log +0 -0
  4. sld_26.0/panel_counter_v19_rsft/eval/phase3_eval.log +61 -0
  5. sld_26.0/panel_counter_v19_rsft/eval/phase3_serve_down.log +6 -0
  6. sld_26.0/panel_counter_v19_rsft/eval/phase3_serve_up.log +7 -0
  7. sld_26.0/panel_counter_v19_rsft/eval/phase4_smoke.log +0 -0
  8. sld_26.0/panel_counter_v19_rsft/eval/phase5_grpo_20260703-182138.log +0 -0
  9. sld_26.0/panel_counter_v19_rsft/eval/phase6_eval.log +61 -0
  10. sld_26.0/panel_counter_v19_rsft/eval/phase6_select.log +16 -0
  11. sld_26.0/panel_counter_v19_rsft/eval/phase6_serve_candidates.log +7 -0
  12. sld_26.0/panel_counter_v19_rsft/eval/phase6_serve_winner.log +9 -0
  13. sld_26.0/panel_counter_v19_rsft/eval/run_20260703-172758.log +0 -0
  14. sld_26.0/panel_counter_v19_rsft/eval/sft_v19_pure30_eval.jsonl +3 -0
  15. sld_26.0/panel_counter_v19_rsft/eval/v19_grpo_best_ckpt.txt +1 -0
  16. sld_26.0/panel_counter_v19_rsft/eval/v19_rsft_best_ckpt.txt +1 -0
  17. sld_26.0/panel_counter_v19_rsft/eval/v19a_eval3.json +84 -0
  18. sld_26.0/panel_counter_v19_rsft/eval/v19a_holdA.json +958 -0
  19. sld_26.0/panel_counter_v19_rsft/eval/v19a_holdB.json +662 -0
  20. sld_26.0/panel_counter_v19_rsft/eval/v19b_eval3.json +84 -0
  21. sld_26.0/panel_counter_v19_rsft/eval/v19b_holdA.json +958 -0
  22. sld_26.0/panel_counter_v19_rsft/eval/v19b_holdB.json +662 -0
  23. sld_26.0/panel_counter_v19_rsft/eval/v19b_selection.json +32 -0
  24. sld_26.0/panel_counter_v19_rsft/eval/vllm_serve.log +0 -0
  25. sld_26.0/panel_counter_v19_rsft/training_logging_full.jsonl +24 -0
sld_26.0/panel_counter_v19_rsft/README.md ADDED
@@ -0,0 +1,44 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # panel_counter_v19_rsft (archived, superseded)
2
+
3
+ Source: `sld_26.0/training/checkpoints/panel_counter_v19_rsft`
4
+ Final checkpoint archived: `checkpoint-28` (step 28) from run `v0-20260703-172808`
5
+ Training run segments (resumed sequentially): ['v0-20260703-172808']
6
+
7
+ This is an **inference-only** archive: adapter weights + configs + full training log +
8
+ eval results. Optimizer/scheduler/RNG state (needed only to resume this exact run) was
9
+ dropped to save space; this run is superseded by later versions and not intended to be
10
+ resumed.
11
+
12
+ ## Training summary
13
+ ```
14
+ {
15
+ "loss_first": 0.61602575,
16
+ "loss_last": 0.00833158
17
+ }
18
+ ```
19
+ Total logged steps across all resumed segments: 24
20
+
21
+ ## Eval files included
22
+ - eval/phase0_serve.log
23
+ - eval/phase2_rsft.log
24
+ - eval/phase3_eval.log
25
+ - eval/phase3_serve_down.log
26
+ - eval/phase3_serve_up.log
27
+ - eval/phase4_smoke.log
28
+ - eval/phase5_grpo_20260703-182138.log
29
+ - eval/phase6_eval.log
30
+ - eval/phase6_select.log
31
+ - eval/phase6_serve_candidates.log
32
+ - eval/phase6_serve_winner.log
33
+ - eval/run_20260703-172758.log
34
+ - eval/v19a_eval3.json
35
+ - eval/v19a_holdA.json
36
+ - eval/v19a_holdB.json
37
+ - eval/v19b_eval3.json
38
+ - eval/v19b_holdA.json
39
+ - eval/v19b_holdB.json
40
+ - eval/v19b_selection.json
41
+ - eval/vllm_serve.log
42
+ - eval/sft_v19_pure30_eval.jsonl
43
+ - eval/v19_grpo_best_ckpt.txt
44
+ - eval/v19_rsft_best_ckpt.txt
sld_26.0/panel_counter_v19_rsft/eval/phase0_serve.log ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
2
+ [serve_v19] target gpu-memory-utilization=stop extra_lora=none
3
+ [serve_v19] no existing vLLM process found
4
+ [serve_v19] stop requested — vLLM killed, not relaunching
sld_26.0/panel_counter_v19_rsft/eval/phase2_rsft.log ADDED
The diff for this file is too large to render. See raw diff
 
sld_26.0/panel_counter_v19_rsft/eval/phase3_eval.log ADDED
@@ -0,0 +1,61 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [eval-v17] sft_v19_pure30_eval.jsonl: 3 records variant=v19a model=panel_counter_v19a_rsft
2
+ [ 3/3] 8-pl/PS6-1-S1-2 cc=True fc=True ps=True ms=1.00 rwd=1.00
3
+
4
+ [eval-v17] === v19a on sft_v19_pure30_eval.jsonl ===
5
+ total : 3
6
+ CC exact : 3/3 = 100.0%
7
+ FC exact : 2/3 = 66.7%
8
+ PS exact : 2/3 = 66.7%
9
+ multiset : 0.925
10
+ all-3 exact : 2/3 = 66.7%
11
+ reward : 0.735
12
+ per bucket:
13
+ 8-pl: n=3 CC=100.0% FC=66.7% PS=66.7% ms=0.93
14
+ → wrote /workspace/training/_v19_logs/v19a_eval3.json
15
+ [eval-v17] sft_v17_holdA.jsonl: 51 records variant=v19a model=panel_counter_v19a_rsft
16
+ [ 5/51] stm_holdout/D2 cc=True fc=False ps=False ms=0.67 rwd=0.18
17
+ [ 10/51] stm_holdout/HH3 cc=True fc=False ps=False ms=0.89 rwd=0.23
18
+ [ 15/51] stm_holdout/N1 cc=True fc=False ps=False ms=0.65 rwd=0.18
19
+ [ 20/51] stm_holdout/UU13 cc=True fc=False ps=False ms=0.86 rwd=0.22
20
+ [ 25/51] stm_holdout/UUSP6 cc=True fc=True ps=True ms=1.00 rwd=1.00
21
+ [ 30/51] 8-pl/P2-1-08-1 cc=True fc=False ps=True ms=0.80 rwd=0.51
22
+ [ 35/51] 8-pl/P6-1-02-1 cc=True fc=False ps=True ms=0.71 rwd=0.49
23
+ [ 40/51] 8-pl/PD6-1-S1-2 cc=True fc=True ps=True ms=1.00 rwd=1.00
24
+ [ 45/51] 8-pl/PS2-1-08-1 cc=True fc=True ps=True ms=1.00 rwd=1.00
25
+ [ 50/51] 8-pl/PU2-1-S1-1 cc=True fc=True ps=True ms=1.00 rwd=1.00
26
+ [ 51/51] 8-pl/PU6-1-S1-1 cc=True fc=False ps=False ms=0.89 rwd=0.23
27
+
28
+ [eval-v17] === v19a on sft_v17_holdA.jsonl ===
29
+ total : 51
30
+ CC exact : 47/51 = 92.2%
31
+ FC exact : 12/51 = 23.5%
32
+ PS exact : 25/51 = 49.0%
33
+ multiset : 0.724
34
+ all-3 exact : 10/51 = 19.6%
35
+ reward : 0.440
36
+ per bucket:
37
+ stm_holdout: n=25 CC=84.0% FC=12.0% PS=28.0% ms=0.61
38
+ 8-pl: n=26 CC=100.0% FC=34.6% PS=69.2% ms=0.83
39
+ → wrote /workspace/training/_v19_logs/v19a_holdA.json
40
+ [eval-v17] sft_v17_holdB.jsonl: 34 records variant=v19a model=panel_counter_v19a_rsft
41
+ [ 5/34] omni/1C1 cc=True fc=False ps=False ms=0.84 rwd=0.22
42
+ [ 10/34] omni/1E5 cc=True fc=False ps=False ms=0.94 rwd=0.24
43
+ [ 15/34] omni/1VW cc=True fc=False ps=False ms=0.79 rwd=0.21
44
+ [ 20/34] omni/1W5 cc=True fc=False ps=False ms=0.82 rwd=0.21
45
+ [ 25/34] omni/2VW cc=True fc=False ps=True ms=0.97 rwd=0.54
46
+ [ 30/34] omni/DP-DVB3 cc=True fc=False ps=False ms=0.73 rwd=0.20
47
+ [ 34/34] spring_valley/Panel 1 cc=True fc=False ps=False ms=0.57 rwd=0.16
48
+
49
+ [eval-v17] === v19a on sft_v17_holdB.jsonl ===
50
+ total : 34
51
+ CC exact : 33/34 = 97.1%
52
+ FC exact : 6/34 = 17.6%
53
+ PS exact : 7/34 = 20.6%
54
+ multiset : 0.781
55
+ all-3 exact : 3/34 = 8.8%
56
+ reward : 0.330
57
+ per bucket:
58
+ calgary: n=4 CC=100.0% FC=25.0% PS=25.0% ms=0.39
59
+ omni: n=28 CC=96.4% FC=17.9% PS=21.4% ms=0.85
60
+ spring_valley: n=2 CC=100.0% FC=0.0% PS=0.0% ms=0.57
61
+ → wrote /workspace/training/_v19_logs/v19a_holdB.json
sld_26.0/panel_counter_v19_rsft/eval/phase3_serve_down.log ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
2
+ [serve_v19] target gpu-memory-utilization=stop extra_lora=none
3
+ [serve_v19] killing existing vLLM pids: 969266
4
+ 969371
5
+ [serve_v19] SIGKILL fallback for pid 969266
6
+ [serve_v19] stop requested — vLLM killed, not relaunching
sld_26.0/panel_counter_v19_rsft/eval/phase3_serve_up.log ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
2
+ [serve_v19] target gpu-memory-utilization=0.6 extra_lora=panel_counter_v19a_rsft=/workspace/training/checkpoints/panel_counter_v19_rsft/v0-20260703-172808/checkpoint-16
3
+ [serve_v19] no existing vLLM process found
4
+ [serve_v19] launching: --gpu-memory-utilization 0.6 --lora-modules panel_counter_v15_sft=/workspace/training/checkpoints/panel_counter_v15_sft/v0-20260629-140516/checkpoint-63 panel_counter_v19a_rsft=/workspace/training/checkpoints/panel_counter_v19_rsft/v0-20260703-172808/checkpoint-16
5
+ [serve_v19] waiting for /v1/models to report healthy (launch pid 969266)...
6
+ [serve_v19] vLLM healthy after 100s
7
+ models: ['Qwen/Qwen3.5-9B', 'panel_counter_v15_sft', 'panel_counter_v19a_rsft']
sld_26.0/panel_counter_v19_rsft/eval/phase4_smoke.log ADDED
The diff for this file is too large to render. See raw diff
 
sld_26.0/panel_counter_v19_rsft/eval/phase5_grpo_20260703-182138.log ADDED
The diff for this file is too large to render. See raw diff
 
sld_26.0/panel_counter_v19_rsft/eval/phase6_eval.log ADDED
@@ -0,0 +1,61 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [eval-v17] sft_v19_pure30_eval.jsonl: 3 records variant=v19b model=panel_counter_v19b_grpo
2
+ [ 3/3] 8-pl/PS6-1-S1-2 cc=True fc=True ps=True ms=1.00 rwd=1.00
3
+
4
+ [eval-v17] === v19b on sft_v19_pure30_eval.jsonl ===
5
+ total : 3
6
+ CC exact : 3/3 = 100.0%
7
+ FC exact : 2/3 = 66.7%
8
+ PS exact : 2/3 = 66.7%
9
+ multiset : 0.951
10
+ all-3 exact : 2/3 = 66.7%
11
+ reward : 0.740
12
+ per bucket:
13
+ 8-pl: n=3 CC=100.0% FC=66.7% PS=66.7% ms=0.95
14
+ → wrote /workspace/training/_v19_logs/v19b_eval3.json
15
+ [eval-v17] sft_v17_holdA.jsonl: 51 records variant=v19b model=panel_counter_v19b_grpo
16
+ [ 5/51] stm_holdout/D2 cc=True fc=False ps=False ms=0.78 rwd=0.21
17
+ [ 10/51] stm_holdout/HH3 cc=True fc=False ps=False ms=0.80 rwd=0.21
18
+ [ 15/51] stm_holdout/N1 cc=True fc=True ps=True ms=0.87 rwd=0.97
19
+ [ 20/51] stm_holdout/UU13 cc=True fc=True ps=False ms=0.79 rwd=0.56
20
+ [ 25/51] stm_holdout/UUSP6 cc=True fc=True ps=False ms=0.71 rwd=0.54
21
+ [ 30/51] 8-pl/P2-1-08-1 cc=True fc=False ps=True ms=0.80 rwd=0.51
22
+ [ 35/51] 8-pl/P6-1-02-1 cc=True fc=False ps=True ms=0.81 rwd=0.51
23
+ [ 40/51] 8-pl/PD6-1-S1-2 cc=True fc=True ps=True ms=1.00 rwd=1.00
24
+ [ 45/51] 8-pl/PS2-1-08-1 cc=True fc=True ps=True ms=1.00 rwd=1.00
25
+ [ 50/51] 8-pl/PU2-1-S1-1 cc=True fc=False ps=True ms=0.67 rwd=0.48
26
+ [ 51/51] 8-pl/PU6-1-S1-1 cc=True fc=True ps=False ms=1.00 rwd=0.60
27
+
28
+ [eval-v17] === v19b on sft_v17_holdA.jsonl ===
29
+ total : 51
30
+ CC exact : 50/51 = 98.0%
31
+ FC exact : 18/51 = 35.3%
32
+ PS exact : 31/51 = 60.8%
33
+ multiset : 0.812
34
+ all-3 exact : 12/51 = 23.5%
35
+ reward : 0.541
36
+ per bucket:
37
+ stm_holdout: n=25 CC=96.0% FC=36.0% PS=44.0% ms=0.78
38
+ 8-pl: n=26 CC=100.0% FC=34.6% PS=76.9% ms=0.84
39
+ → wrote /workspace/training/_v19_logs/v19b_holdA.json
40
+ [eval-v17] sft_v17_holdB.jsonl: 34 records variant=v19b model=panel_counter_v19b_grpo
41
+ [ 5/34] omni/1C1 cc=True fc=False ps=False ms=0.78 rwd=0.21
42
+ [ 10/34] omni/1E5 cc=True fc=True ps=False ms=1.00 rwd=0.60
43
+ [ 15/34] omni/1VW cc=True fc=False ps=False ms=0.79 rwd=0.21
44
+ [ 20/34] omni/1W5 cc=True fc=False ps=False ms=0.82 rwd=0.21
45
+ [ 25/34] omni/2VW cc=True fc=False ps=True ms=0.97 rwd=0.54
46
+ [ 30/34] omni/DP-DVB3 cc=True fc=False ps=False ms=0.75 rwd=0.20
47
+ [ 34/34] spring_valley/Panel 1 cc=True fc=False ps=False ms=0.34 rwd=0.12
48
+
49
+ [eval-v17] === v19b on sft_v17_holdB.jsonl ===
50
+ total : 34
51
+ CC exact : 34/34 = 100.0%
52
+ FC exact : 7/34 = 20.6%
53
+ PS exact : 9/34 = 26.5%
54
+ multiset : 0.758
55
+ all-3 exact : 3/34 = 8.8%
56
+ reward : 0.359
57
+ per bucket:
58
+ calgary: n=4 CC=100.0% FC=25.0% PS=25.0% ms=0.39
59
+ omni: n=28 CC=100.0% FC=21.4% PS=28.6% ms=0.84
60
+ spring_valley: n=2 CC=100.0% FC=0.0% PS=0.0% ms=0.34
61
+ → wrote /workspace/training/_v19_logs/v19b_holdB.json
sld_26.0/panel_counter_v19_rsft/eval/phase6_select.log ADDED
@@ -0,0 +1,16 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [select-v19b] candidates: [(115, '/workspace/training/checkpoints/panel_counter_v19_grpo/v0-20260703-182148/checkpoint-115'), (120, '/workspace/training/checkpoints/panel_counter_v19_grpo/v0-20260703-182148/checkpoint-120'), (125, '/workspace/training/checkpoints/panel_counter_v19_grpo/v0-20260703-182148/checkpoint-125')]
2
+ [panel_counter_v19b_cand_115] P2-1-S1-1: reward=0.2341
3
+ [panel_counter_v19b_cand_115] PS2-1-01-1: reward=0.9867
4
+ [panel_counter_v19b_cand_115] PS6-1-S1-2: reward=1.0000
5
+ [select-v19b] panel_counter_v19b_cand_115 (step 115): mean_reward=0.7403
6
+ [panel_counter_v19b_cand_120] P2-1-S1-1: reward=0.2310
7
+ [panel_counter_v19b_cand_120] PS2-1-01-1: reward=0.9867
8
+ [panel_counter_v19b_cand_120] PS6-1-S1-2: reward=1.0000
9
+ [select-v19b] panel_counter_v19b_cand_120 (step 120): mean_reward=0.7392
10
+ [panel_counter_v19b_cand_125] P2-1-S1-1: reward=0.2341
11
+ [panel_counter_v19b_cand_125] PS2-1-01-1: reward=0.9867
12
+ [panel_counter_v19b_cand_125] PS6-1-S1-2: reward=1.0000
13
+ [select-v19b] panel_counter_v19b_cand_125 (step 125): mean_reward=0.7403
14
+
15
+ [select-v19b] WINNER: panel_counter_v19b_cand_115 step=115 mean_reward=0.7403
16
+ [select-v19b] wrote best checkpoint path to /workspace/training/v19_grpo_best_ckpt.txt
sld_26.0/panel_counter_v19_rsft/eval/phase6_serve_candidates.log ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
2
+ [serve_v19] target gpu-memory-utilization=0.6 extra_lora=panel_counter_v19b_cand_115=/workspace/training/checkpoints/panel_counter_v19_grpo/v0-20260703-182148/checkpoint-115 panel_counter_v19b_cand_120=/workspace/training/checkpoints/panel_counter_v19_grpo/v0-20260703-182148/checkpoint-120 panel_counter_v19b_cand_125=/workspace/training/checkpoints/panel_counter_v19_grpo/v0-20260703-182148/checkpoint-125
3
+ [serve_v19] no existing vLLM process found
4
+ [serve_v19] launching: --gpu-memory-utilization 0.6 --lora-modules panel_counter_v15_sft=/workspace/training/checkpoints/panel_counter_v15_sft/v0-20260629-140516/checkpoint-63 panel_counter_v19b_cand_115=/workspace/training/checkpoints/panel_counter_v19_grpo/v0-20260703-182148/checkpoint-115 panel_counter_v19b_cand_120=/workspace/training/checkpoints/panel_counter_v19_grpo/v0-20260703-182148/checkpoint-120 panel_counter_v19b_cand_125=/workspace/training/checkpoints/panel_counter_v19_grpo/v0-20260703-182148/checkpoint-125
5
+ [serve_v19] waiting for /v1/models to report healthy (launch pid 1097143)...
6
+ [serve_v19] vLLM healthy after 100s
7
+ models: ['Qwen/Qwen3.5-9B', 'panel_counter_v15_sft', 'panel_counter_v19b_cand_115', 'panel_counter_v19b_cand_120', 'panel_counter_v19b_cand_125']
sld_26.0/panel_counter_v19_rsft/eval/phase6_serve_winner.log ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
2
+ [serve_v19] target gpu-memory-utilization=0.6 extra_lora=panel_counter_v19a_rsft=/workspace/training/checkpoints/panel_counter_v19_rsft/v0-20260703-172808/checkpoint-16 panel_counter_v19b_grpo=/workspace/training/checkpoints/panel_counter_v19_grpo/v0-20260703-182148/checkpoint-115
3
+ [serve_v19] killing existing vLLM pids: 1097143
4
+ 1097246
5
+ [serve_v19] SIGKILL fallback for pid 1097143
6
+ [serve_v19] launching: --gpu-memory-utilization 0.6 --lora-modules panel_counter_v15_sft=/workspace/training/checkpoints/panel_counter_v15_sft/v0-20260629-140516/checkpoint-63 panel_counter_v19a_rsft=/workspace/training/checkpoints/panel_counter_v19_rsft/v0-20260703-172808/checkpoint-16 panel_counter_v19b_grpo=/workspace/training/checkpoints/panel_counter_v19_grpo/v0-20260703-182148/checkpoint-115
7
+ [serve_v19] waiting for /v1/models to report healthy (launch pid 1098035)...
8
+ [serve_v19] vLLM healthy after 100s
9
+ models: ['Qwen/Qwen3.5-9B', 'panel_counter_v15_sft', 'panel_counter_v19a_rsft', 'panel_counter_v19b_grpo']
sld_26.0/panel_counter_v19_rsft/eval/run_20260703-172758.log ADDED
The diff for this file is too large to render. See raw diff
 
sld_26.0/panel_counter_v19_rsft/eval/sft_v19_pure30_eval.jsonl ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ {"messages": [{"role": "user", "content": "<image>Analyze this electrical panel schedule image and count all circuits.\n\nFor each position in the panel, identify whether it contains:\n- An ACTIVE breaker (connected to a load)\n- A SPARE breaker (physical breaker installed but not connected)\n- A SPACE (empty slot, no breaker)\n\nFor each breaker, determine:\n- Amperage rating (e.g., 15A, 20A, 30A, 40A, 60A)\n- Pole configuration: 1P (1 slot), 2P (spans 2 adjacent slots), or 3P (spans 3 slots)\n- The circuit description/load name\n- Whether it has GFI/GFCI protection\n\nThink step by step. Scan left side (odd circuits) and right side (even circuits) row by row from top to bottom. Then provide final counts.\n\nOutput your final answer as JSON:\n{\"Feeder_Count\": N, \"Feeder_Details\": \"NxAMPA/POLE; ...\", \"Prepared_Spaces\": \"N (spec)\" or \"0\", \"Circuit_Count\": \"N Circuit\"}"}, {"role": "assistant", "content": "## Reasoning\n\nHeader Analysis: Panel P2-1-S1-1, 120/208V 3PH 4W, 400A Main. Visual Symbol Analysis: 2-Pole Breakers (P=2): Left side: CCT 37,39 (VB-1-SS2-01), 41,43 (VB-1-SS2-03), 45,47 (VT-1-SS2-02), 49,51 (VT-1-SS2-01), 53,55 (VB-1-SS1-02), 57,59 (VT-1-SS1-03 ET 05), 61,63 (VT-1-SS1-02) = 7x15A/2P. Right side: CCT 38,40 (VB-1-SS2-02), 42,44 (VB-1-SS2-05), 46,48 (VB-1-SS2-04), 50,52 (VT-1-SS2-03), 54,56 (VB-1-SS1-01), 58,60 (VB-1-SS1-03), 62,64 (VT-1-SS1-04) = 7x15A/2P, plus CCT 66,68 (CO-1-SS1-01) = 1x50A/2P. Total: 14x15A/2P + 1x50A/2P. 1-Pole Breakers: 20A/1P: CCT 5,7,9,11,13,15,17,23,25,31,33,35 (left) + CCT 2,4,6,10,12,14,18,20,22,24,26,32 (right) = 24x20A/1P. 15A/1P: CCT 1,3,19,21,27,29,57,65,67,69 (left) + CCT 8,16,28,30,34,68,70,72,74,76,78,80,82,84 (right, including LIBRE spares) = 24x15A/1P. Spaces: CCT 73,75,77,79,81,83 = 6 spaces. Total Feeders: 14 + 1 + 24 + 24 = 63. Total Circuits: 84.\n\n\n## Breaker_Groups\n\n- 1x 50A/2P\n- 14x 15A/2P\n- 24x 20A/1P\n- 24x 15A/1P\n\n\n## Counting\n\nFeeder breakdown: 14x15A/2P; 1x50A/2P; 24x20A/1P; 24x15A/1P\nTotal feeders: 63\nPrepared spaces (empty slots): 6\nCircuit count: 84\nVerification: 78 occupied + 6 spaces = 84 = 84\n\n\n## Final Answer\n\n```json\n{\n \"Feeder_Count\": 63,\n \"Feeder_Details\": \"14x15A/2P; 1x50A/2P; 24x20A/1P; 24x15A/1P\",\n \"Prepared_Spaces\": \"6\",\n \"Circuit_Count\": \"84 Circuit\"\n}\n```"}], "images": ["/workspace/training/images_v19/8-pl-du-commerce-bat-b1-pur-immobilia-galleria_P2-1-S1-1.png"], "_meta": {"project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria", "panel_id": "P2-1-S1-1", "tag": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria/P2-1-S1-1", "source": "v3_corrected_pure30", "pseudo_reasoning_chars": 899}}
2
+ {"messages": [{"role": "user", "content": "<image>Analyze this electrical panel schedule image and count all circuits.\n\nFor each position in the panel, identify whether it contains:\n- An ACTIVE breaker (connected to a load)\n- A SPARE breaker (physical breaker installed but not connected)\n- A SPACE (empty slot, no breaker)\n\nFor each breaker, determine:\n- Amperage rating (e.g., 15A, 20A, 30A, 40A, 60A)\n- Pole configuration: 1P (1 slot), 2P (spans 2 adjacent slots), or 3P (spans 3 slots)\n- The circuit description/load name\n- Whether it has GFI/GFCI protection\n\nThink step by step. Scan left side (odd circuits) and right side (even circuits) row by row from top to bottom. Then provide final counts.\n\nOutput your final answer as JSON:\n{\"Feeder_Count\": N, \"Feeder_Details\": \"NxAMPA/POLE; ...\", \"Prepared_Spaces\": \"N (spec)\" or \"0\", \"Circuit_Count\": \"N Circuit\"}"}, {"role": "assistant", "content": "## Reasoning\n\nHeader Analysis: Panel PS2-1-01-1, 120/208V 3PH 4W, 125A Main. Visual Symbol Analysis: 3-Pole Breaker: CCT 1,3,5 (PS2-1-04-1, 60A) = 1 breaker. 1-Pole Breakers: 20A/1P: CCT 11 (ESCALIER #3) = 1 breaker. 15A/1P: Active Left (7,9,13,15,17,19,21,23,25,27) = 10. Active Right (2,4,6,8,10,12,14,16,18,20,22,24,26) = 13. Spares (28,30,32,34) = 4. Total 15A/1P = 27. Spaces: Left (29,31,33,35,37,39,41) = 7. Right (36,38,40,42) = 4. Total Spaces = 11. Total Feeders: 1 + 1 + 27 = 29. Total Circuits: 42.\n\n\n## Breaker_Groups\n\n- 1x 60A/3P\n- 1x 20A/1P\n- 27x 15A/1P\n\n\n## Counting\n\nFeeder breakdown: 1x60A/3P; 1x20A/1P; 27x15A/1P\nTotal feeders: 29\nPrepared spaces (empty slots): 11\nCircuit count: 42\nVerification: 31 occupied + 11 spaces = 42 = 42\n\n\n## Final Answer\n\n```json\n{\n \"Feeder_Count\": 29,\n \"Feeder_Details\": \"1x60A/3P; 1x20A/1P; 27x15A/1P\",\n \"Prepared_Spaces\": \"11\",\n \"Circuit_Count\": \"42 Circuit\"\n}\n```"}], "images": ["/workspace/training/images_v19/8-pl-du-commerce-bat-b1-pur-immobilia-galleria_PS2-1-01-1.png"], "_meta": {"project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria", "panel_id": "PS2-1-01-1", "tag": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria/PS2-1-01-1", "source": "v3_corrected_pure30", "pseudo_reasoning_chars": 496}}
3
+ {"messages": [{"role": "user", "content": "<image>Analyze this electrical panel schedule image and count all circuits.\n\nFor each position in the panel, identify whether it contains:\n- An ACTIVE breaker (connected to a load)\n- A SPARE breaker (physical breaker installed but not connected)\n- A SPACE (empty slot, no breaker)\n\nFor each breaker, determine:\n- Amperage rating (e.g., 15A, 20A, 30A, 40A, 60A)\n- Pole configuration: 1P (1 slot), 2P (spans 2 adjacent slots), or 3P (spans 3 slots)\n- The circuit description/load name\n- Whether it has GFI/GFCI protection\n\nThink step by step. Scan left side (odd circuits) and right side (even circuits) row by row from top to bottom. Then provide final counts.\n\nOutput your final answer as JSON:\n{\"Feeder_Count\": N, \"Feeder_Details\": \"NxAMPA/POLE; ...\", \"Prepared_Spaces\": \"N (spec)\" or \"0\", \"Circuit_Count\": \"N Circuit\"}"}, {"role": "assistant", "content": "## Reasoning\n\nHeader Analysis: Panel PS6-1-S1-2, 347/600V 3PH 4W, 225A Main. Visual Symbol Analysis: 3-Pole Breakers: Left side CCT 1,3,5 (PS2-1-01-1 VIA TS-1-01-1, 60A). Right side CCT 2,4,6 (UAN-1-T-01 VENTILATEUR, 25A). Total 3P: 1x60A/3P + 1x25A/3P. Spaces: CCT 7-42 are all empty. Total 36 spaces. Total Feeders: 2. Total Circuits: 42.\n\n\n## Breaker_Groups\n\n- 1x 60A/3P\n- 1x 25A/3P\n\n\n## Counting\n\nFeeder breakdown: 1x25A/3P; 1x60A/3P\nTotal feeders: 2\nPrepared spaces (empty slots): 36\nCircuit count: 42\nVerification: 6 occupied + 36 spaces = 42 = 42\n\n\n## Final Answer\n\n```json\n{\n \"Feeder_Count\": 2,\n \"Feeder_Details\": \"1x25A/3P; 1x60A/3P\",\n \"Prepared_Spaces\": \"36\",\n \"Circuit_Count\": \"42 Circuit\"\n}\n```"}], "images": ["/workspace/training/images_v19/8-pl-du-commerce-bat-b1-pur-immobilia-galleria_PS6-1-S1-2.png"], "_meta": {"project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria", "panel_id": "PS6-1-S1-2", "tag": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria/PS6-1-S1-2", "source": "v3_corrected_pure30", "pseudo_reasoning_chars": 326}}
sld_26.0/panel_counter_v19_rsft/eval/v19_grpo_best_ckpt.txt ADDED
@@ -0,0 +1 @@
 
 
1
+ /workspace/training/checkpoints/panel_counter_v19_grpo/v0-20260703-182148/checkpoint-115
sld_26.0/panel_counter_v19_rsft/eval/v19_rsft_best_ckpt.txt ADDED
@@ -0,0 +1 @@
 
 
1
+ /workspace/training/checkpoints/panel_counter_v19_rsft/v0-20260703-172808/checkpoint-16
sld_26.0/panel_counter_v19_rsft/eval/v19a_eval3.json ADDED
@@ -0,0 +1,84 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "variant": "v19a",
3
+ "model": "panel_counter_v19a_rsft",
4
+ "checkpoint": "",
5
+ "holdout_file": "sft_v19_pure30_eval.jsonl",
6
+ "total": 3,
7
+ "aggregate_cc_em": 1.0,
8
+ "aggregate_fc_em": 0.6667,
9
+ "aggregate_ps_em": 0.6667,
10
+ "aggregate_groups_exact": 0.6667,
11
+ "aggregate_self_consistent": 0.6667,
12
+ "aggregate_multiset_jaccard": 0.9254,
13
+ "aggregate_reward_mean": 0.7351,
14
+ "all_three_counts_exact": 2,
15
+ "all_three_counts_exact_rate": 0.6667,
16
+ "per_bucket": {
17
+ "8-pl": {
18
+ "n": 3,
19
+ "cc_em": 1.0,
20
+ "fc_em": 0.6667,
21
+ "ps_em": 0.6667,
22
+ "groups_exact": 0.6667,
23
+ "self_consistent": 0.6667,
24
+ "multiset_jaccard_mean": 0.9254,
25
+ "reward_mean": 0.7351
26
+ }
27
+ },
28
+ "per_panel": [
29
+ {
30
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
31
+ "panel_id": "P2-1-S1-1",
32
+ "bucket": "8-pl",
33
+ "cc_ok": true,
34
+ "fc_ok": false,
35
+ "ps_ok": false,
36
+ "groups_exact": false,
37
+ "self_consistent": false,
38
+ "multiset_jaccard": 0.7761,
39
+ "reward": 0.2052,
40
+ "pred_fc": 56,
41
+ "gold_fc": 63,
42
+ "pred_ps": 7,
43
+ "gold_ps": 6,
44
+ "pred_cc": 84,
45
+ "gold_cc": 84
46
+ },
47
+ {
48
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
49
+ "panel_id": "PS2-1-01-1",
50
+ "bucket": "8-pl",
51
+ "cc_ok": true,
52
+ "fc_ok": true,
53
+ "ps_ok": true,
54
+ "groups_exact": true,
55
+ "self_consistent": true,
56
+ "multiset_jaccard": 1.0,
57
+ "reward": 1.0,
58
+ "pred_fc": 29,
59
+ "gold_fc": 29,
60
+ "pred_ps": 11,
61
+ "gold_ps": 11,
62
+ "pred_cc": 42,
63
+ "gold_cc": 42
64
+ },
65
+ {
66
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
67
+ "panel_id": "PS6-1-S1-2",
68
+ "bucket": "8-pl",
69
+ "cc_ok": true,
70
+ "fc_ok": true,
71
+ "ps_ok": true,
72
+ "groups_exact": true,
73
+ "self_consistent": true,
74
+ "multiset_jaccard": 1.0,
75
+ "reward": 1.0,
76
+ "pred_fc": 2,
77
+ "gold_fc": 2,
78
+ "pred_ps": 36,
79
+ "gold_ps": 36,
80
+ "pred_cc": 42,
81
+ "gold_cc": 42
82
+ }
83
+ ]
84
+ }
sld_26.0/panel_counter_v19_rsft/eval/v19a_holdA.json ADDED
@@ -0,0 +1,958 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "variant": "v19a",
3
+ "model": "panel_counter_v19a_rsft",
4
+ "checkpoint": "",
5
+ "holdout_file": "sft_v17_holdA.jsonl",
6
+ "total": 51,
7
+ "aggregate_cc_em": 0.9216,
8
+ "aggregate_fc_em": 0.2353,
9
+ "aggregate_ps_em": 0.4902,
10
+ "aggregate_groups_exact": 0.2157,
11
+ "aggregate_self_consistent": 0.2549,
12
+ "aggregate_multiset_jaccard": 0.7242,
13
+ "aggregate_reward_mean": 0.4399,
14
+ "all_three_counts_exact": 10,
15
+ "all_three_counts_exact_rate": 0.1961,
16
+ "per_bucket": {
17
+ "stm_holdout": {
18
+ "n": 25,
19
+ "cc_em": 0.84,
20
+ "fc_em": 0.12,
21
+ "ps_em": 0.28,
22
+ "groups_exact": 0.12,
23
+ "self_consistent": 0.16,
24
+ "multiset_jaccard_mean": 0.6142,
25
+ "reward_mean": 0.2988
26
+ },
27
+ "8-pl": {
28
+ "n": 26,
29
+ "cc_em": 1.0,
30
+ "fc_em": 0.3462,
31
+ "ps_em": 0.6923,
32
+ "groups_exact": 0.3077,
33
+ "self_consistent": 0.3462,
34
+ "multiset_jaccard_mean": 0.83,
35
+ "reward_mean": 0.5756
36
+ }
37
+ },
38
+ "per_panel": [
39
+ {
40
+ "project": "stm-lot-ci22",
41
+ "panel_id": "B2",
42
+ "bucket": "stm_holdout",
43
+ "cc_ok": false,
44
+ "fc_ok": false,
45
+ "ps_ok": false,
46
+ "groups_exact": false,
47
+ "self_consistent": false,
48
+ "multiset_jaccard": 0.0,
49
+ "reward": 0.0,
50
+ "pred_fc": null,
51
+ "gold_fc": 7,
52
+ "pred_ps": null,
53
+ "gold_ps": 23,
54
+ "pred_cc": null,
55
+ "gold_cc": 30
56
+ },
57
+ {
58
+ "project": "stm-lot-ci22",
59
+ "panel_id": "BB2",
60
+ "bucket": "stm_holdout",
61
+ "cc_ok": true,
62
+ "fc_ok": true,
63
+ "ps_ok": false,
64
+ "groups_exact": true,
65
+ "self_consistent": false,
66
+ "multiset_jaccard": 1.0,
67
+ "reward": 0.6,
68
+ "pred_fc": 12,
69
+ "gold_fc": 12,
70
+ "pred_ps": 42,
71
+ "gold_ps": 28,
72
+ "pred_cc": 42,
73
+ "gold_cc": 42
74
+ },
75
+ {
76
+ "project": "stm-lot-ci22",
77
+ "panel_id": "C1",
78
+ "bucket": "stm_holdout",
79
+ "cc_ok": true,
80
+ "fc_ok": false,
81
+ "ps_ok": true,
82
+ "groups_exact": false,
83
+ "self_consistent": false,
84
+ "multiset_jaccard": 0.8077,
85
+ "reward": 0.5115,
86
+ "pred_fc": 23,
87
+ "gold_fc": 24,
88
+ "pred_ps": 15,
89
+ "gold_ps": 15,
90
+ "pred_cc": 42,
91
+ "gold_cc": 42
92
+ },
93
+ {
94
+ "project": "stm-lot-ci22",
95
+ "panel_id": "CC4",
96
+ "bucket": "stm_holdout",
97
+ "cc_ok": true,
98
+ "fc_ok": false,
99
+ "ps_ok": false,
100
+ "groups_exact": false,
101
+ "self_consistent": false,
102
+ "multiset_jaccard": 0.8,
103
+ "reward": 0.21,
104
+ "pred_fc": 20,
105
+ "gold_fc": 16,
106
+ "pred_ps": 22,
107
+ "gold_ps": 24,
108
+ "pred_cc": 42,
109
+ "gold_cc": 42
110
+ },
111
+ {
112
+ "project": "stm-lot-ci22",
113
+ "panel_id": "D2",
114
+ "bucket": "stm_holdout",
115
+ "cc_ok": true,
116
+ "fc_ok": false,
117
+ "ps_ok": false,
118
+ "groups_exact": false,
119
+ "self_consistent": false,
120
+ "multiset_jaccard": 0.6667,
121
+ "reward": 0.1833,
122
+ "pred_fc": 8,
123
+ "gold_fc": 7,
124
+ "pred_ps": 30,
125
+ "gold_ps": 33,
126
+ "pred_cc": 42,
127
+ "gold_cc": 42
128
+ },
129
+ {
130
+ "project": "stm-lot-ci22",
131
+ "panel_id": "F1",
132
+ "bucket": "stm_holdout",
133
+ "cc_ok": true,
134
+ "fc_ok": false,
135
+ "ps_ok": true,
136
+ "groups_exact": false,
137
+ "self_consistent": true,
138
+ "multiset_jaccard": 0.25,
139
+ "reward": 0.4,
140
+ "pred_fc": 2,
141
+ "gold_fc": 3,
142
+ "pred_ps": 38,
143
+ "gold_ps": 38,
144
+ "pred_cc": 42,
145
+ "gold_cc": 42
146
+ },
147
+ {
148
+ "project": "stm-lot-ci22",
149
+ "panel_id": "F8",
150
+ "bucket": "stm_holdout",
151
+ "cc_ok": true,
152
+ "fc_ok": true,
153
+ "ps_ok": true,
154
+ "groups_exact": true,
155
+ "self_consistent": true,
156
+ "multiset_jaccard": 1.0,
157
+ "reward": 1.0,
158
+ "pred_fc": 3,
159
+ "gold_fc": 3,
160
+ "pred_ps": 38,
161
+ "gold_ps": 38,
162
+ "pred_cc": 42,
163
+ "gold_cc": 42
164
+ },
165
+ {
166
+ "project": "stm-lot-ci22",
167
+ "panel_id": "FF2",
168
+ "bucket": "stm_holdout",
169
+ "cc_ok": false,
170
+ "fc_ok": false,
171
+ "ps_ok": false,
172
+ "groups_exact": false,
173
+ "self_consistent": false,
174
+ "multiset_jaccard": 0.0,
175
+ "reward": 0.0,
176
+ "pred_fc": null,
177
+ "gold_fc": 16,
178
+ "pred_ps": null,
179
+ "gold_ps": 12,
180
+ "pred_cc": null,
181
+ "gold_cc": 54
182
+ },
183
+ {
184
+ "project": "stm-lot-ci22",
185
+ "panel_id": "FF4",
186
+ "bucket": "stm_holdout",
187
+ "cc_ok": true,
188
+ "fc_ok": false,
189
+ "ps_ok": false,
190
+ "groups_exact": false,
191
+ "self_consistent": false,
192
+ "multiset_jaccard": 0.5294,
193
+ "reward": 0.1559,
194
+ "pred_fc": 34,
195
+ "gold_fc": 18,
196
+ "pred_ps": 25,
197
+ "gold_ps": 24,
198
+ "pred_cc": 42,
199
+ "gold_cc": 42
200
+ },
201
+ {
202
+ "project": "stm-lot-ci22",
203
+ "panel_id": "HH3",
204
+ "bucket": "stm_holdout",
205
+ "cc_ok": true,
206
+ "fc_ok": false,
207
+ "ps_ok": false,
208
+ "groups_exact": false,
209
+ "self_consistent": false,
210
+ "multiset_jaccard": 0.8889,
211
+ "reward": 0.2278,
212
+ "pred_fc": 9,
213
+ "gold_fc": 8,
214
+ "pred_ps": 14,
215
+ "gold_ps": 20,
216
+ "pred_cc": 30,
217
+ "gold_cc": 30
218
+ },
219
+ {
220
+ "project": "stm-lot-ci22",
221
+ "panel_id": "K7",
222
+ "bucket": "stm_holdout",
223
+ "cc_ok": false,
224
+ "fc_ok": false,
225
+ "ps_ok": false,
226
+ "groups_exact": false,
227
+ "self_consistent": false,
228
+ "multiset_jaccard": 0.0,
229
+ "reward": 0.0,
230
+ "pred_fc": null,
231
+ "gold_fc": 11,
232
+ "pred_ps": null,
233
+ "gold_ps": 19,
234
+ "pred_cc": null,
235
+ "gold_cc": 30
236
+ },
237
+ {
238
+ "project": "stm-lot-ci22",
239
+ "panel_id": "KK2",
240
+ "bucket": "stm_holdout",
241
+ "cc_ok": true,
242
+ "fc_ok": false,
243
+ "ps_ok": false,
244
+ "groups_exact": false,
245
+ "self_consistent": false,
246
+ "multiset_jaccard": 0.875,
247
+ "reward": 0.225,
248
+ "pred_fc": 8,
249
+ "gold_fc": 7,
250
+ "pred_ps": 16,
251
+ "gold_ps": 21,
252
+ "pred_cc": 30,
253
+ "gold_cc": 30
254
+ },
255
+ {
256
+ "project": "stm-lot-ci22",
257
+ "panel_id": "KK3",
258
+ "bucket": "stm_holdout",
259
+ "cc_ok": true,
260
+ "fc_ok": false,
261
+ "ps_ok": false,
262
+ "groups_exact": false,
263
+ "self_consistent": false,
264
+ "multiset_jaccard": 0.5833,
265
+ "reward": 0.1667,
266
+ "pred_fc": 8,
267
+ "gold_fc": 11,
268
+ "pred_ps": 32,
269
+ "gold_ps": 29,
270
+ "pred_cc": 42,
271
+ "gold_cc": 42
272
+ },
273
+ {
274
+ "project": "stm-lot-ci22",
275
+ "panel_id": "KK5",
276
+ "bucket": "stm_holdout",
277
+ "cc_ok": true,
278
+ "fc_ok": false,
279
+ "ps_ok": true,
280
+ "groups_exact": false,
281
+ "self_consistent": false,
282
+ "multiset_jaccard": 0.7,
283
+ "reward": 0.49,
284
+ "pred_fc": 9,
285
+ "gold_fc": 8,
286
+ "pred_ps": 34,
287
+ "gold_ps": 34,
288
+ "pred_cc": 54,
289
+ "gold_cc": 54
290
+ },
291
+ {
292
+ "project": "stm-lot-ci22",
293
+ "panel_id": "N1",
294
+ "bucket": "stm_holdout",
295
+ "cc_ok": true,
296
+ "fc_ok": false,
297
+ "ps_ok": false,
298
+ "groups_exact": false,
299
+ "self_consistent": false,
300
+ "multiset_jaccard": 0.65,
301
+ "reward": 0.18,
302
+ "pred_fc": 19,
303
+ "gold_fc": 14,
304
+ "pred_ps": 26,
305
+ "gold_ps": 27,
306
+ "pred_cc": 42,
307
+ "gold_cc": 42
308
+ },
309
+ {
310
+ "project": "stm-lot-ci22",
311
+ "panel_id": "P2",
312
+ "bucket": "stm_holdout",
313
+ "cc_ok": true,
314
+ "fc_ok": false,
315
+ "ps_ok": true,
316
+ "groups_exact": false,
317
+ "self_consistent": false,
318
+ "multiset_jaccard": 0.7778,
319
+ "reward": 0.5056,
320
+ "pred_fc": 17,
321
+ "gold_fc": 15,
322
+ "pred_ps": 14,
323
+ "gold_ps": 14,
324
+ "pred_cc": 30,
325
+ "gold_cc": 30
326
+ },
327
+ {
328
+ "project": "stm-lot-ci22",
329
+ "panel_id": "SS2",
330
+ "bucket": "stm_holdout",
331
+ "cc_ok": true,
332
+ "fc_ok": false,
333
+ "ps_ok": true,
334
+ "groups_exact": false,
335
+ "self_consistent": false,
336
+ "multiset_jaccard": 0.4375,
337
+ "reward": 0.4375,
338
+ "pred_fc": 12,
339
+ "gold_fc": 11,
340
+ "pred_ps": 27,
341
+ "gold_ps": 27,
342
+ "pred_cc": 54,
343
+ "gold_cc": 54
344
+ },
345
+ {
346
+ "project": "stm-lot-ci22",
347
+ "panel_id": "U5",
348
+ "bucket": "stm_holdout",
349
+ "cc_ok": true,
350
+ "fc_ok": false,
351
+ "ps_ok": false,
352
+ "groups_exact": false,
353
+ "self_consistent": true,
354
+ "multiset_jaccard": 0.9167,
355
+ "reward": 0.2333,
356
+ "pred_fc": 12,
357
+ "gold_fc": 11,
358
+ "pred_ps": 29,
359
+ "gold_ps": 30,
360
+ "pred_cc": 42,
361
+ "gold_cc": 42
362
+ },
363
+ {
364
+ "project": "stm-lot-ci22",
365
+ "panel_id": "U9",
366
+ "bucket": "stm_holdout",
367
+ "cc_ok": true,
368
+ "fc_ok": false,
369
+ "ps_ok": false,
370
+ "groups_exact": false,
371
+ "self_consistent": false,
372
+ "multiset_jaccard": 0.7857,
373
+ "reward": 0.2071,
374
+ "pred_fc": 13,
375
+ "gold_fc": 12,
376
+ "pred_ps": 29,
377
+ "gold_ps": 28,
378
+ "pred_cc": 42,
379
+ "gold_cc": 42
380
+ },
381
+ {
382
+ "project": "stm-lot-ci22",
383
+ "panel_id": "UU13",
384
+ "bucket": "stm_holdout",
385
+ "cc_ok": true,
386
+ "fc_ok": false,
387
+ "ps_ok": false,
388
+ "groups_exact": false,
389
+ "self_consistent": false,
390
+ "multiset_jaccard": 0.8571,
391
+ "reward": 0.2214,
392
+ "pred_fc": 13,
393
+ "gold_fc": 12,
394
+ "pred_ps": 8,
395
+ "gold_ps": 10,
396
+ "pred_cc": 42,
397
+ "gold_cc": 42
398
+ },
399
+ {
400
+ "project": "stm-lot-ci22",
401
+ "panel_id": "UU3",
402
+ "bucket": "stm_holdout",
403
+ "cc_ok": true,
404
+ "fc_ok": false,
405
+ "ps_ok": false,
406
+ "groups_exact": false,
407
+ "self_consistent": false,
408
+ "multiset_jaccard": 0.8,
409
+ "reward": 0.21,
410
+ "pred_fc": 4,
411
+ "gold_fc": 5,
412
+ "pred_ps": 15,
413
+ "gold_ps": 19,
414
+ "pred_cc": 30,
415
+ "gold_cc": 30
416
+ },
417
+ {
418
+ "project": "stm-lot-ci22",
419
+ "panel_id": "UU5",
420
+ "bucket": "stm_holdout",
421
+ "cc_ok": true,
422
+ "fc_ok": false,
423
+ "ps_ok": false,
424
+ "groups_exact": false,
425
+ "self_consistent": false,
426
+ "multiset_jaccard": 0.5,
427
+ "reward": 0.15,
428
+ "pred_fc": 12,
429
+ "gold_fc": 9,
430
+ "pred_ps": 18,
431
+ "gold_ps": 19,
432
+ "pred_cc": 42,
433
+ "gold_cc": 42
434
+ },
435
+ {
436
+ "project": "stm-lot-ci22",
437
+ "panel_id": "UU9",
438
+ "bucket": "stm_holdout",
439
+ "cc_ok": false,
440
+ "fc_ok": false,
441
+ "ps_ok": false,
442
+ "groups_exact": false,
443
+ "self_consistent": false,
444
+ "multiset_jaccard": 0.0,
445
+ "reward": 0.0,
446
+ "pred_fc": null,
447
+ "gold_fc": 9,
448
+ "pred_ps": null,
449
+ "gold_ps": 19,
450
+ "pred_cc": null,
451
+ "gold_cc": 42
452
+ },
453
+ {
454
+ "project": "stm-lot-ci22",
455
+ "panel_id": "UUSP13",
456
+ "bucket": "stm_holdout",
457
+ "cc_ok": true,
458
+ "fc_ok": false,
459
+ "ps_ok": false,
460
+ "groups_exact": false,
461
+ "self_consistent": false,
462
+ "multiset_jaccard": 0.5283,
463
+ "reward": 0.1557,
464
+ "pred_fc": 53,
465
+ "gold_fc": 28,
466
+ "pred_ps": 0,
467
+ "gold_ps": 24,
468
+ "pred_cc": 54,
469
+ "gold_cc": 54
470
+ },
471
+ {
472
+ "project": "stm-lot-ci22",
473
+ "panel_id": "UUSP6",
474
+ "bucket": "stm_holdout",
475
+ "cc_ok": true,
476
+ "fc_ok": true,
477
+ "ps_ok": true,
478
+ "groups_exact": true,
479
+ "self_consistent": true,
480
+ "multiset_jaccard": 1.0,
481
+ "reward": 1.0,
482
+ "pred_fc": 6,
483
+ "gold_fc": 6,
484
+ "pred_ps": 18,
485
+ "gold_ps": 18,
486
+ "pred_cc": 30,
487
+ "gold_cc": 30
488
+ },
489
+ {
490
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
491
+ "panel_id": "P2-1-01-1",
492
+ "bucket": "8-pl",
493
+ "cc_ok": true,
494
+ "fc_ok": false,
495
+ "ps_ok": true,
496
+ "groups_exact": false,
497
+ "self_consistent": false,
498
+ "multiset_jaccard": 0.4512,
499
+ "reward": 0.4402,
500
+ "pred_fc": 65,
501
+ "gold_fc": 54,
502
+ "pred_ps": 10,
503
+ "gold_ps": 10,
504
+ "pred_cc": 72,
505
+ "gold_cc": 72
506
+ },
507
+ {
508
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
509
+ "panel_id": "P2-1-01-2",
510
+ "bucket": "8-pl",
511
+ "cc_ok": true,
512
+ "fc_ok": true,
513
+ "ps_ok": true,
514
+ "groups_exact": true,
515
+ "self_consistent": true,
516
+ "multiset_jaccard": 1.0,
517
+ "reward": 1.0,
518
+ "pred_fc": 24,
519
+ "gold_fc": 24,
520
+ "pred_ps": 12,
521
+ "gold_ps": 12,
522
+ "pred_cc": 42,
523
+ "gold_cc": 42
524
+ },
525
+ {
526
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
527
+ "panel_id": "P2-1-02-1",
528
+ "bucket": "8-pl",
529
+ "cc_ok": true,
530
+ "fc_ok": false,
531
+ "ps_ok": true,
532
+ "groups_exact": false,
533
+ "self_consistent": false,
534
+ "multiset_jaccard": 0.875,
535
+ "reward": 0.525,
536
+ "pred_fc": 32,
537
+ "gold_fc": 28,
538
+ "pred_ps": 6,
539
+ "gold_ps": 6,
540
+ "pred_cc": 42,
541
+ "gold_cc": 42
542
+ },
543
+ {
544
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
545
+ "panel_id": "P2-1-04-1",
546
+ "bucket": "8-pl",
547
+ "cc_ok": true,
548
+ "fc_ok": false,
549
+ "ps_ok": true,
550
+ "groups_exact": false,
551
+ "self_consistent": false,
552
+ "multiset_jaccard": 0.7324,
553
+ "reward": 0.4965,
554
+ "pred_fc": 71,
555
+ "gold_fc": 52,
556
+ "pred_ps": 2,
557
+ "gold_ps": 2,
558
+ "pred_cc": 72,
559
+ "gold_cc": 72
560
+ },
561
+ {
562
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
563
+ "panel_id": "P2-1-08-1",
564
+ "bucket": "8-pl",
565
+ "cc_ok": true,
566
+ "fc_ok": false,
567
+ "ps_ok": true,
568
+ "groups_exact": false,
569
+ "self_consistent": false,
570
+ "multiset_jaccard": 0.8,
571
+ "reward": 0.51,
572
+ "pred_fc": 56,
573
+ "gold_fc": 52,
574
+ "pred_ps": 2,
575
+ "gold_ps": 2,
576
+ "pred_cc": 72,
577
+ "gold_cc": 72
578
+ },
579
+ {
580
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
581
+ "panel_id": "P2-1-12-1",
582
+ "bucket": "8-pl",
583
+ "cc_ok": true,
584
+ "fc_ok": false,
585
+ "ps_ok": true,
586
+ "groups_exact": false,
587
+ "self_consistent": false,
588
+ "multiset_jaccard": 0.5248,
589
+ "reward": 0.455,
590
+ "pred_fc": 101,
591
+ "gold_fc": 53,
592
+ "pred_ps": 6,
593
+ "gold_ps": 6,
594
+ "pred_cc": 72,
595
+ "gold_cc": 72
596
+ },
597
+ {
598
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
599
+ "panel_id": "P2-1-S1-1",
600
+ "bucket": "8-pl",
601
+ "cc_ok": true,
602
+ "fc_ok": false,
603
+ "ps_ok": false,
604
+ "groups_exact": false,
605
+ "self_consistent": false,
606
+ "multiset_jaccard": 0.7126,
607
+ "reward": 0.1925,
608
+ "pred_fc": 86,
609
+ "gold_fc": 63,
610
+ "pred_ps": 7,
611
+ "gold_ps": 6,
612
+ "pred_cc": 84,
613
+ "gold_cc": 84
614
+ },
615
+ {
616
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
617
+ "panel_id": "P2-1-TT-1",
618
+ "bucket": "8-pl",
619
+ "cc_ok": true,
620
+ "fc_ok": false,
621
+ "ps_ok": false,
622
+ "groups_exact": false,
623
+ "self_consistent": false,
624
+ "multiset_jaccard": 0.7674,
625
+ "reward": 0.2035,
626
+ "pred_fc": 40,
627
+ "gold_fc": 36,
628
+ "pred_ps": 16,
629
+ "gold_ps": 18,
630
+ "pred_cc": 72,
631
+ "gold_cc": 72
632
+ },
633
+ {
634
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
635
+ "panel_id": "P6-1-01-1",
636
+ "bucket": "8-pl",
637
+ "cc_ok": true,
638
+ "fc_ok": false,
639
+ "ps_ok": true,
640
+ "groups_exact": false,
641
+ "self_consistent": false,
642
+ "multiset_jaccard": 0.8,
643
+ "reward": 0.51,
644
+ "pred_fc": 26,
645
+ "gold_fc": 20,
646
+ "pred_ps": 30,
647
+ "gold_ps": 30,
648
+ "pred_cc": 72,
649
+ "gold_cc": 72
650
+ },
651
+ {
652
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
653
+ "panel_id": "P6-1-02-1",
654
+ "bucket": "8-pl",
655
+ "cc_ok": true,
656
+ "fc_ok": false,
657
+ "ps_ok": true,
658
+ "groups_exact": false,
659
+ "self_consistent": false,
660
+ "multiset_jaccard": 0.7143,
661
+ "reward": 0.4929,
662
+ "pred_fc": 15,
663
+ "gold_fc": 21,
664
+ "pred_ps": 3,
665
+ "gold_ps": 3,
666
+ "pred_cc": 60,
667
+ "gold_cc": 60
668
+ },
669
+ {
670
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
671
+ "panel_id": "P6-1-S1-1",
672
+ "bucket": "8-pl",
673
+ "cc_ok": true,
674
+ "fc_ok": false,
675
+ "ps_ok": false,
676
+ "groups_exact": false,
677
+ "self_consistent": false,
678
+ "multiset_jaccard": 0.6957,
679
+ "reward": 0.1891,
680
+ "pred_fc": 16,
681
+ "gold_fc": 23,
682
+ "pred_ps": 30,
683
+ "gold_ps": 29,
684
+ "pred_cc": 72,
685
+ "gold_cc": 72
686
+ },
687
+ {
688
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
689
+ "panel_id": "P6-1-TT-1",
690
+ "bucket": "8-pl",
691
+ "cc_ok": true,
692
+ "fc_ok": true,
693
+ "ps_ok": true,
694
+ "groups_exact": false,
695
+ "self_consistent": true,
696
+ "multiset_jaccard": 0.8125,
697
+ "reward": 0.9625,
698
+ "pred_fc": 29,
699
+ "gold_fc": 29,
700
+ "pred_ps": 3,
701
+ "gold_ps": 3,
702
+ "pred_cc": 84,
703
+ "gold_cc": 84
704
+ },
705
+ {
706
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
707
+ "panel_id": "PB-1-S1-1",
708
+ "bucket": "8-pl",
709
+ "cc_ok": true,
710
+ "fc_ok": false,
711
+ "ps_ok": true,
712
+ "groups_exact": false,
713
+ "self_consistent": false,
714
+ "multiset_jaccard": 0.5,
715
+ "reward": 0.45,
716
+ "pred_fc": 56,
717
+ "gold_fc": 28,
718
+ "pred_ps": 28,
719
+ "gold_ps": 28,
720
+ "pred_cc": 84,
721
+ "gold_cc": 84
722
+ },
723
+ {
724
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
725
+ "panel_id": "PB6-1-S1-1",
726
+ "bucket": "8-pl",
727
+ "cc_ok": true,
728
+ "fc_ok": true,
729
+ "ps_ok": true,
730
+ "groups_exact": true,
731
+ "self_consistent": true,
732
+ "multiset_jaccard": 1.0,
733
+ "reward": 1.0,
734
+ "pred_fc": 4,
735
+ "gold_fc": 4,
736
+ "pred_ps": 30,
737
+ "gold_ps": 30,
738
+ "pred_cc": 42,
739
+ "gold_cc": 42
740
+ },
741
+ {
742
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
743
+ "panel_id": "PD6-1-S1-2",
744
+ "bucket": "8-pl",
745
+ "cc_ok": true,
746
+ "fc_ok": true,
747
+ "ps_ok": true,
748
+ "groups_exact": true,
749
+ "self_consistent": true,
750
+ "multiset_jaccard": 1.0,
751
+ "reward": 1.0,
752
+ "pred_fc": 4,
753
+ "gold_fc": 4,
754
+ "pred_ps": 30,
755
+ "gold_ps": 30,
756
+ "pred_cc": 42,
757
+ "gold_cc": 42
758
+ },
759
+ {
760
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
761
+ "panel_id": "PD6-1-S1-4",
762
+ "bucket": "8-pl",
763
+ "cc_ok": true,
764
+ "fc_ok": false,
765
+ "ps_ok": false,
766
+ "groups_exact": false,
767
+ "self_consistent": false,
768
+ "multiset_jaccard": 0.8,
769
+ "reward": 0.21,
770
+ "pred_fc": 7,
771
+ "gold_fc": 9,
772
+ "pred_ps": 24,
773
+ "gold_ps": 23,
774
+ "pred_cc": 42,
775
+ "gold_cc": 42
776
+ },
777
+ {
778
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
779
+ "panel_id": "PDS6-1-S1-1",
780
+ "bucket": "8-pl",
781
+ "cc_ok": true,
782
+ "fc_ok": true,
783
+ "ps_ok": false,
784
+ "groups_exact": true,
785
+ "self_consistent": false,
786
+ "multiset_jaccard": 1.0,
787
+ "reward": 0.6,
788
+ "pred_fc": 5,
789
+ "gold_fc": 5,
790
+ "pred_ps": 24,
791
+ "gold_ps": 27,
792
+ "pred_cc": 42,
793
+ "gold_cc": 42
794
+ },
795
+ {
796
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
797
+ "panel_id": "PS2-1-01-1",
798
+ "bucket": "8-pl",
799
+ "cc_ok": true,
800
+ "fc_ok": true,
801
+ "ps_ok": true,
802
+ "groups_exact": true,
803
+ "self_consistent": true,
804
+ "multiset_jaccard": 1.0,
805
+ "reward": 1.0,
806
+ "pred_fc": 29,
807
+ "gold_fc": 29,
808
+ "pred_ps": 11,
809
+ "gold_ps": 11,
810
+ "pred_cc": 42,
811
+ "gold_cc": 42
812
+ },
813
+ {
814
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
815
+ "panel_id": "PS2-1-04-1",
816
+ "bucket": "8-pl",
817
+ "cc_ok": true,
818
+ "fc_ok": false,
819
+ "ps_ok": false,
820
+ "groups_exact": false,
821
+ "self_consistent": true,
822
+ "multiset_jaccard": 0.9615,
823
+ "reward": 0.2423,
824
+ "pred_fc": 25,
825
+ "gold_fc": 26,
826
+ "pred_ps": 15,
827
+ "gold_ps": 14,
828
+ "pred_cc": 42,
829
+ "gold_cc": 42
830
+ },
831
+ {
832
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
833
+ "panel_id": "PS2-1-08-1",
834
+ "bucket": "8-pl",
835
+ "cc_ok": true,
836
+ "fc_ok": true,
837
+ "ps_ok": true,
838
+ "groups_exact": true,
839
+ "self_consistent": true,
840
+ "multiset_jaccard": 1.0,
841
+ "reward": 1.0,
842
+ "pred_fc": 25,
843
+ "gold_fc": 25,
844
+ "pred_ps": 15,
845
+ "gold_ps": 15,
846
+ "pred_cc": 42,
847
+ "gold_cc": 42
848
+ },
849
+ {
850
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
851
+ "panel_id": "PS2-1-12-1",
852
+ "bucket": "8-pl",
853
+ "cc_ok": true,
854
+ "fc_ok": false,
855
+ "ps_ok": false,
856
+ "groups_exact": false,
857
+ "self_consistent": false,
858
+ "multiset_jaccard": 0.8696,
859
+ "reward": 0.2239,
860
+ "pred_fc": 23,
861
+ "gold_fc": 20,
862
+ "pred_ps": 20,
863
+ "gold_ps": 22,
864
+ "pred_cc": 42,
865
+ "gold_cc": 42
866
+ },
867
+ {
868
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
869
+ "panel_id": "PS2-1-S1-1",
870
+ "bucket": "8-pl",
871
+ "cc_ok": true,
872
+ "fc_ok": false,
873
+ "ps_ok": true,
874
+ "groups_exact": false,
875
+ "self_consistent": false,
876
+ "multiset_jaccard": 0.8,
877
+ "reward": 0.51,
878
+ "pred_fc": 16,
879
+ "gold_fc": 20,
880
+ "pred_ps": 19,
881
+ "gold_ps": 19,
882
+ "pred_cc": 42,
883
+ "gold_cc": 42
884
+ },
885
+ {
886
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
887
+ "panel_id": "PS6-1-S1-1",
888
+ "bucket": "8-pl",
889
+ "cc_ok": true,
890
+ "fc_ok": false,
891
+ "ps_ok": true,
892
+ "groups_exact": false,
893
+ "self_consistent": false,
894
+ "multiset_jaccard": 0.875,
895
+ "reward": 0.525,
896
+ "pred_fc": 14,
897
+ "gold_fc": 16,
898
+ "pred_ps": 18,
899
+ "gold_ps": 18,
900
+ "pred_cc": 42,
901
+ "gold_cc": 42
902
+ },
903
+ {
904
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
905
+ "panel_id": "PS6-1-S1-2",
906
+ "bucket": "8-pl",
907
+ "cc_ok": true,
908
+ "fc_ok": true,
909
+ "ps_ok": true,
910
+ "groups_exact": true,
911
+ "self_consistent": true,
912
+ "multiset_jaccard": 1.0,
913
+ "reward": 1.0,
914
+ "pred_fc": 2,
915
+ "gold_fc": 2,
916
+ "pred_ps": 36,
917
+ "gold_ps": 36,
918
+ "pred_cc": 42,
919
+ "gold_cc": 42
920
+ },
921
+ {
922
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
923
+ "panel_id": "PU2-1-S1-1",
924
+ "bucket": "8-pl",
925
+ "cc_ok": true,
926
+ "fc_ok": true,
927
+ "ps_ok": true,
928
+ "groups_exact": true,
929
+ "self_consistent": true,
930
+ "multiset_jaccard": 1.0,
931
+ "reward": 1.0,
932
+ "pred_fc": 8,
933
+ "gold_fc": 8,
934
+ "pred_ps": 34,
935
+ "gold_ps": 34,
936
+ "pred_cc": 42,
937
+ "gold_cc": 42
938
+ },
939
+ {
940
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
941
+ "panel_id": "PU6-1-S1-1",
942
+ "bucket": "8-pl",
943
+ "cc_ok": true,
944
+ "fc_ok": false,
945
+ "ps_ok": false,
946
+ "groups_exact": false,
947
+ "self_consistent": false,
948
+ "multiset_jaccard": 0.8889,
949
+ "reward": 0.2278,
950
+ "pred_fc": 7,
951
+ "gold_fc": 9,
952
+ "pred_ps": 24,
953
+ "gold_ps": 23,
954
+ "pred_cc": 42,
955
+ "gold_cc": 42
956
+ }
957
+ ]
958
+ }
sld_26.0/panel_counter_v19_rsft/eval/v19a_holdB.json ADDED
@@ -0,0 +1,662 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "variant": "v19a",
3
+ "model": "panel_counter_v19a_rsft",
4
+ "checkpoint": "",
5
+ "holdout_file": "sft_v17_holdB.jsonl",
6
+ "total": 34,
7
+ "aggregate_cc_em": 0.9706,
8
+ "aggregate_fc_em": 0.1765,
9
+ "aggregate_ps_em": 0.2059,
10
+ "aggregate_groups_exact": 0.0294,
11
+ "aggregate_self_consistent": 0.1765,
12
+ "aggregate_multiset_jaccard": 0.7811,
13
+ "aggregate_reward_mean": 0.33,
14
+ "all_three_counts_exact": 3,
15
+ "all_three_counts_exact_rate": 0.0882,
16
+ "per_bucket": {
17
+ "calgary": {
18
+ "n": 4,
19
+ "cc_em": 1.0,
20
+ "fc_em": 0.25,
21
+ "ps_em": 0.25,
22
+ "groups_exact": 0.0,
23
+ "self_consistent": 0.75,
24
+ "multiset_jaccard_mean": 0.3929,
25
+ "reward_mean": 0.2911
26
+ },
27
+ "omni": {
28
+ "n": 28,
29
+ "cc_em": 0.9643,
30
+ "fc_em": 0.1786,
31
+ "ps_em": 0.2143,
32
+ "groups_exact": 0.0357,
33
+ "self_consistent": 0.1071,
34
+ "multiset_jaccard_mean": 0.8517,
35
+ "reward_mean": 0.3474
36
+ },
37
+ "spring_valley": {
38
+ "n": 2,
39
+ "cc_em": 1.0,
40
+ "fc_em": 0.0,
41
+ "ps_em": 0.0,
42
+ "groups_exact": 0.0,
43
+ "self_consistent": 0.0,
44
+ "multiset_jaccard_mean": 0.569,
45
+ "reward_mean": 0.1638
46
+ }
47
+ },
48
+ "per_panel": [
49
+ {
50
+ "project": "calgary-winter-club-renovation-pn-25-01-0297",
51
+ "panel_id": "AD",
52
+ "bucket": "calgary",
53
+ "cc_ok": true,
54
+ "fc_ok": true,
55
+ "ps_ok": true,
56
+ "groups_exact": false,
57
+ "self_consistent": true,
58
+ "multiset_jaccard": 0.9765,
59
+ "reward": 0.9953,
60
+ "pred_fc": 84,
61
+ "gold_fc": 84,
62
+ "pred_ps": 0,
63
+ "gold_ps": 0,
64
+ "pred_cc": 84,
65
+ "gold_cc": 84
66
+ },
67
+ {
68
+ "project": "calgary-winter-club-renovation-pn-25-01-0297",
69
+ "panel_id": "M2F",
70
+ "bucket": "calgary",
71
+ "cc_ok": true,
72
+ "fc_ok": false,
73
+ "ps_ok": false,
74
+ "groups_exact": false,
75
+ "self_consistent": false,
76
+ "multiset_jaccard": 0.5952,
77
+ "reward": 0.169,
78
+ "pred_fc": 25,
79
+ "gold_fc": 42,
80
+ "pred_ps": 16,
81
+ "gold_ps": 0,
82
+ "pred_cc": 42,
83
+ "gold_cc": 42
84
+ },
85
+ {
86
+ "project": "calgary-winter-club-renovation-pn-25-01-0297",
87
+ "panel_id": "M2P",
88
+ "bucket": "calgary",
89
+ "cc_ok": true,
90
+ "fc_ok": false,
91
+ "ps_ok": false,
92
+ "groups_exact": false,
93
+ "self_consistent": true,
94
+ "multiset_jaccard": 0.0,
95
+ "reward": 0.0,
96
+ "pred_fc": 0,
97
+ "gold_fc": 42,
98
+ "pred_ps": 42,
99
+ "gold_ps": 0,
100
+ "pred_cc": 42,
101
+ "gold_cc": 42
102
+ },
103
+ {
104
+ "project": "calgary-winter-club-renovation-pn-25-01-0297",
105
+ "panel_id": "M6A",
106
+ "bucket": "calgary",
107
+ "cc_ok": true,
108
+ "fc_ok": false,
109
+ "ps_ok": false,
110
+ "groups_exact": false,
111
+ "self_consistent": true,
112
+ "multiset_jaccard": 0.0,
113
+ "reward": 0.0,
114
+ "pred_fc": 0,
115
+ "gold_fc": 42,
116
+ "pred_ps": 42,
117
+ "gold_ps": 0,
118
+ "pred_cc": 42,
119
+ "gold_cc": 42
120
+ },
121
+ {
122
+ "project": "omni-seaforth-ltc",
123
+ "panel_id": "1C1",
124
+ "bucket": "omni",
125
+ "cc_ok": true,
126
+ "fc_ok": false,
127
+ "ps_ok": false,
128
+ "groups_exact": false,
129
+ "self_consistent": false,
130
+ "multiset_jaccard": 0.8444,
131
+ "reward": 0.2189,
132
+ "pred_fc": 39,
133
+ "gold_fc": 45,
134
+ "pred_ps": 18,
135
+ "gold_ps": 15,
136
+ "pred_cc": 72,
137
+ "gold_cc": 72
138
+ },
139
+ {
140
+ "project": "omni-seaforth-ltc",
141
+ "panel_id": "1C2",
142
+ "bucket": "omni",
143
+ "cc_ok": true,
144
+ "fc_ok": true,
145
+ "ps_ok": true,
146
+ "groups_exact": true,
147
+ "self_consistent": true,
148
+ "multiset_jaccard": 1.0,
149
+ "reward": 1.0,
150
+ "pred_fc": 15,
151
+ "gold_fc": 15,
152
+ "pred_ps": 20,
153
+ "gold_ps": 20,
154
+ "pred_cc": 42,
155
+ "gold_cc": 42
156
+ },
157
+ {
158
+ "project": "omni-seaforth-ltc",
159
+ "panel_id": "1C3",
160
+ "bucket": "omni",
161
+ "cc_ok": true,
162
+ "fc_ok": false,
163
+ "ps_ok": false,
164
+ "groups_exact": false,
165
+ "self_consistent": false,
166
+ "multiset_jaccard": 0.8372,
167
+ "reward": 0.2174,
168
+ "pred_fc": 37,
169
+ "gold_fc": 42,
170
+ "pred_ps": 21,
171
+ "gold_ps": 10,
172
+ "pred_cc": 72,
173
+ "gold_cc": 72
174
+ },
175
+ {
176
+ "project": "omni-seaforth-ltc",
177
+ "panel_id": "1E3",
178
+ "bucket": "omni",
179
+ "cc_ok": true,
180
+ "fc_ok": true,
181
+ "ps_ok": false,
182
+ "groups_exact": false,
183
+ "self_consistent": false,
184
+ "multiset_jaccard": 0.7288,
185
+ "reward": 0.5458,
186
+ "pred_fc": 51,
187
+ "gold_fc": 51,
188
+ "pred_ps": 15,
189
+ "gold_ps": 19,
190
+ "pred_cc": 72,
191
+ "gold_cc": 72
192
+ },
193
+ {
194
+ "project": "omni-seaforth-ltc",
195
+ "panel_id": "1E4",
196
+ "bucket": "omni",
197
+ "cc_ok": true,
198
+ "fc_ok": false,
199
+ "ps_ok": true,
200
+ "groups_exact": false,
201
+ "self_consistent": false,
202
+ "multiset_jaccard": 0.9783,
203
+ "reward": 0.5457,
204
+ "pred_fc": 46,
205
+ "gold_fc": 45,
206
+ "pred_ps": 26,
207
+ "gold_ps": 26,
208
+ "pred_cc": 72,
209
+ "gold_cc": 72
210
+ },
211
+ {
212
+ "project": "omni-seaforth-ltc",
213
+ "panel_id": "1E5",
214
+ "bucket": "omni",
215
+ "cc_ok": true,
216
+ "fc_ok": false,
217
+ "ps_ok": false,
218
+ "groups_exact": false,
219
+ "self_consistent": false,
220
+ "multiset_jaccard": 0.9423,
221
+ "reward": 0.2385,
222
+ "pred_fc": 52,
223
+ "gold_fc": 49,
224
+ "pred_ps": 21,
225
+ "gold_ps": 22,
226
+ "pred_cc": 72,
227
+ "gold_cc": 72
228
+ },
229
+ {
230
+ "project": "omni-seaforth-ltc",
231
+ "panel_id": "1E6",
232
+ "bucket": "omni",
233
+ "cc_ok": true,
234
+ "fc_ok": false,
235
+ "ps_ok": false,
236
+ "groups_exact": false,
237
+ "self_consistent": false,
238
+ "multiset_jaccard": 0.8077,
239
+ "reward": 0.2115,
240
+ "pred_fc": 42,
241
+ "gold_fc": 52,
242
+ "pred_ps": 17,
243
+ "gold_ps": 20,
244
+ "pred_cc": 72,
245
+ "gold_cc": 72
246
+ },
247
+ {
248
+ "project": "omni-seaforth-ltc",
249
+ "panel_id": "1E7",
250
+ "bucket": "omni",
251
+ "cc_ok": true,
252
+ "fc_ok": false,
253
+ "ps_ok": false,
254
+ "groups_exact": false,
255
+ "self_consistent": false,
256
+ "multiset_jaccard": 0.7692,
257
+ "reward": 0.2038,
258
+ "pred_fc": 41,
259
+ "gold_fc": 51,
260
+ "pred_ps": 19,
261
+ "gold_ps": 20,
262
+ "pred_cc": 72,
263
+ "gold_cc": 72
264
+ },
265
+ {
266
+ "project": "omni-seaforth-ltc",
267
+ "panel_id": "1VC",
268
+ "bucket": "omni",
269
+ "cc_ok": true,
270
+ "fc_ok": false,
271
+ "ps_ok": false,
272
+ "groups_exact": false,
273
+ "self_consistent": true,
274
+ "multiset_jaccard": 0.8462,
275
+ "reward": 0.2192,
276
+ "pred_fc": 23,
277
+ "gold_fc": 25,
278
+ "pred_ps": 18,
279
+ "gold_ps": 16,
280
+ "pred_cc": 42,
281
+ "gold_cc": 42
282
+ },
283
+ {
284
+ "project": "omni-seaforth-ltc",
285
+ "panel_id": "1VE",
286
+ "bucket": "omni",
287
+ "cc_ok": true,
288
+ "fc_ok": true,
289
+ "ps_ok": true,
290
+ "groups_exact": false,
291
+ "self_consistent": true,
292
+ "multiset_jaccard": 0.9412,
293
+ "reward": 0.9882,
294
+ "pred_fc": 33,
295
+ "gold_fc": 33,
296
+ "pred_ps": 9,
297
+ "gold_ps": 9,
298
+ "pred_cc": 42,
299
+ "gold_cc": 42
300
+ },
301
+ {
302
+ "project": "omni-seaforth-ltc",
303
+ "panel_id": "1VW",
304
+ "bucket": "omni",
305
+ "cc_ok": true,
306
+ "fc_ok": false,
307
+ "ps_ok": false,
308
+ "groups_exact": false,
309
+ "self_consistent": false,
310
+ "multiset_jaccard": 0.7879,
311
+ "reward": 0.2076,
312
+ "pred_fc": 26,
313
+ "gold_fc": 33,
314
+ "pred_ps": 10,
315
+ "gold_ps": 9,
316
+ "pred_cc": 42,
317
+ "gold_cc": 42
318
+ },
319
+ {
320
+ "project": "omni-seaforth-ltc",
321
+ "panel_id": "1W1",
322
+ "bucket": "omni",
323
+ "cc_ok": true,
324
+ "fc_ok": true,
325
+ "ps_ok": false,
326
+ "groups_exact": false,
327
+ "self_consistent": false,
328
+ "multiset_jaccard": 0.7879,
329
+ "reward": 0.5576,
330
+ "pred_fc": 29,
331
+ "gold_fc": 29,
332
+ "pred_ps": 4,
333
+ "gold_ps": 5,
334
+ "pred_cc": 42,
335
+ "gold_cc": 42
336
+ },
337
+ {
338
+ "project": "omni-seaforth-ltc",
339
+ "panel_id": "1W2",
340
+ "bucket": "omni",
341
+ "cc_ok": true,
342
+ "fc_ok": true,
343
+ "ps_ok": false,
344
+ "groups_exact": false,
345
+ "self_consistent": false,
346
+ "multiset_jaccard": 0.7049,
347
+ "reward": 0.541,
348
+ "pred_fc": 52,
349
+ "gold_fc": 52,
350
+ "pred_ps": 4,
351
+ "gold_ps": 19,
352
+ "pred_cc": 72,
353
+ "gold_cc": 72
354
+ },
355
+ {
356
+ "project": "omni-seaforth-ltc",
357
+ "panel_id": "1W3",
358
+ "bucket": "omni",
359
+ "cc_ok": true,
360
+ "fc_ok": false,
361
+ "ps_ok": false,
362
+ "groups_exact": false,
363
+ "self_consistent": false,
364
+ "multiset_jaccard": 0.9434,
365
+ "reward": 0.2387,
366
+ "pred_fc": 51,
367
+ "gold_fc": 52,
368
+ "pred_ps": 17,
369
+ "gold_ps": 18,
370
+ "pred_cc": 72,
371
+ "gold_cc": 72
372
+ },
373
+ {
374
+ "project": "omni-seaforth-ltc",
375
+ "panel_id": "1W4",
376
+ "bucket": "omni",
377
+ "cc_ok": true,
378
+ "fc_ok": false,
379
+ "ps_ok": false,
380
+ "groups_exact": false,
381
+ "self_consistent": false,
382
+ "multiset_jaccard": 0.9608,
383
+ "reward": 0.2422,
384
+ "pred_fc": 51,
385
+ "gold_fc": 49,
386
+ "pred_ps": 21,
387
+ "gold_ps": 22,
388
+ "pred_cc": 72,
389
+ "gold_cc": 72
390
+ },
391
+ {
392
+ "project": "omni-seaforth-ltc",
393
+ "panel_id": "1W5",
394
+ "bucket": "omni",
395
+ "cc_ok": true,
396
+ "fc_ok": false,
397
+ "ps_ok": false,
398
+ "groups_exact": false,
399
+ "self_consistent": false,
400
+ "multiset_jaccard": 0.8222,
401
+ "reward": 0.2144,
402
+ "pred_fc": 37,
403
+ "gold_fc": 45,
404
+ "pred_ps": 27,
405
+ "gold_ps": 26,
406
+ "pred_cc": 72,
407
+ "gold_cc": 72
408
+ },
409
+ {
410
+ "project": "omni-seaforth-ltc",
411
+ "panel_id": "1W6",
412
+ "bucket": "omni",
413
+ "cc_ok": true,
414
+ "fc_ok": false,
415
+ "ps_ok": false,
416
+ "groups_exact": false,
417
+ "self_consistent": false,
418
+ "multiset_jaccard": 0.9815,
419
+ "reward": 0.2463,
420
+ "pred_fc": 53,
421
+ "gold_fc": 54,
422
+ "pred_ps": 20,
423
+ "gold_ps": 16,
424
+ "pred_cc": 72,
425
+ "gold_cc": 72
426
+ },
427
+ {
428
+ "project": "omni-seaforth-ltc",
429
+ "panel_id": "1W7",
430
+ "bucket": "omni",
431
+ "cc_ok": true,
432
+ "fc_ok": false,
433
+ "ps_ok": true,
434
+ "groups_exact": false,
435
+ "self_consistent": false,
436
+ "multiset_jaccard": 0.9623,
437
+ "reward": 0.5425,
438
+ "pred_fc": 53,
439
+ "gold_fc": 51,
440
+ "pred_ps": 20,
441
+ "gold_ps": 20,
442
+ "pred_cc": 72,
443
+ "gold_cc": 72
444
+ },
445
+ {
446
+ "project": "omni-seaforth-ltc",
447
+ "panel_id": "2VC",
448
+ "bucket": "omni",
449
+ "cc_ok": true,
450
+ "fc_ok": false,
451
+ "ps_ok": false,
452
+ "groups_exact": false,
453
+ "self_consistent": false,
454
+ "multiset_jaccard": 0.9048,
455
+ "reward": 0.231,
456
+ "pred_fc": 19,
457
+ "gold_fc": 21,
458
+ "pred_ps": 12,
459
+ "gold_ps": 21,
460
+ "pred_cc": 42,
461
+ "gold_cc": 42
462
+ },
463
+ {
464
+ "project": "omni-seaforth-ltc",
465
+ "panel_id": "2VE",
466
+ "bucket": "omni",
467
+ "cc_ok": true,
468
+ "fc_ok": false,
469
+ "ps_ok": false,
470
+ "groups_exact": false,
471
+ "self_consistent": false,
472
+ "multiset_jaccard": 0.7941,
473
+ "reward": 0.2088,
474
+ "pred_fc": 27,
475
+ "gold_fc": 34,
476
+ "pred_ps": 11,
477
+ "gold_ps": 8,
478
+ "pred_cc": 42,
479
+ "gold_cc": 42
480
+ },
481
+ {
482
+ "project": "omni-seaforth-ltc",
483
+ "panel_id": "2VW",
484
+ "bucket": "omni",
485
+ "cc_ok": true,
486
+ "fc_ok": false,
487
+ "ps_ok": true,
488
+ "groups_exact": false,
489
+ "self_consistent": false,
490
+ "multiset_jaccard": 0.9706,
491
+ "reward": 0.5441,
492
+ "pred_fc": 33,
493
+ "gold_fc": 34,
494
+ "pred_ps": 8,
495
+ "gold_ps": 8,
496
+ "pred_cc": 42,
497
+ "gold_cc": 42
498
+ },
499
+ {
500
+ "project": "omni-seaforth-ltc",
501
+ "panel_id": "DP-CA1",
502
+ "bucket": "omni",
503
+ "cc_ok": true,
504
+ "fc_ok": false,
505
+ "ps_ok": false,
506
+ "groups_exact": false,
507
+ "self_consistent": false,
508
+ "multiset_jaccard": 0.7857,
509
+ "reward": 0.2071,
510
+ "pred_fc": 12,
511
+ "gold_fc": 13,
512
+ "pred_ps": 30,
513
+ "gold_ps": 15,
514
+ "pred_cc": 54,
515
+ "gold_cc": 54
516
+ },
517
+ {
518
+ "project": "omni-seaforth-ltc",
519
+ "panel_id": "DP-DVA1",
520
+ "bucket": "omni",
521
+ "cc_ok": true,
522
+ "fc_ok": false,
523
+ "ps_ok": false,
524
+ "groups_exact": false,
525
+ "self_consistent": false,
526
+ "multiset_jaccard": 0.875,
527
+ "reward": 0.225,
528
+ "pred_fc": 8,
529
+ "gold_fc": 15,
530
+ "pred_ps": 42,
531
+ "gold_ps": 9,
532
+ "pred_cc": 54,
533
+ "gold_cc": 54
534
+ },
535
+ {
536
+ "project": "omni-seaforth-ltc",
537
+ "panel_id": "DP-DVB1",
538
+ "bucket": "omni",
539
+ "cc_ok": true,
540
+ "fc_ok": false,
541
+ "ps_ok": false,
542
+ "groups_exact": false,
543
+ "self_consistent": false,
544
+ "multiset_jaccard": 0.6154,
545
+ "reward": 0.1731,
546
+ "pred_fc": 8,
547
+ "gold_fc": 13,
548
+ "pred_ps": 29,
549
+ "gold_ps": 15,
550
+ "pred_cc": 54,
551
+ "gold_cc": 54
552
+ },
553
+ {
554
+ "project": "omni-seaforth-ltc",
555
+ "panel_id": "DP-DVB2",
556
+ "bucket": "omni",
557
+ "cc_ok": false,
558
+ "fc_ok": false,
559
+ "ps_ok": false,
560
+ "groups_exact": false,
561
+ "self_consistent": false,
562
+ "multiset_jaccard": 0.7059,
563
+ "reward": 0.0,
564
+ "pred_fc": null,
565
+ "gold_fc": 15,
566
+ "pred_ps": null,
567
+ "gold_ps": 9,
568
+ "pred_cc": null,
569
+ "gold_cc": 54
570
+ },
571
+ {
572
+ "project": "omni-seaforth-ltc",
573
+ "panel_id": "DP-DVB3",
574
+ "bucket": "omni",
575
+ "cc_ok": true,
576
+ "fc_ok": false,
577
+ "ps_ok": false,
578
+ "groups_exact": false,
579
+ "self_consistent": false,
580
+ "multiset_jaccard": 0.7333,
581
+ "reward": 0.1967,
582
+ "pred_fc": 13,
583
+ "gold_fc": 12,
584
+ "pred_ps": 35,
585
+ "gold_ps": 18,
586
+ "pred_cc": 54,
587
+ "gold_cc": 54
588
+ },
589
+ {
590
+ "project": "omni-seaforth-ltc",
591
+ "panel_id": "DP-DVB4",
592
+ "bucket": "omni",
593
+ "cc_ok": true,
594
+ "fc_ok": false,
595
+ "ps_ok": false,
596
+ "groups_exact": false,
597
+ "self_consistent": false,
598
+ "multiset_jaccard": 0.9286,
599
+ "reward": 0.2357,
600
+ "pred_fc": 14,
601
+ "gold_fc": 13,
602
+ "pred_ps": 11,
603
+ "gold_ps": 15,
604
+ "pred_cc": 54,
605
+ "gold_cc": 54
606
+ },
607
+ {
608
+ "project": "omni-seaforth-ltc",
609
+ "panel_id": "DP-VB",
610
+ "bucket": "omni",
611
+ "cc_ok": true,
612
+ "fc_ok": false,
613
+ "ps_ok": true,
614
+ "groups_exact": false,
615
+ "self_consistent": false,
616
+ "multiset_jaccard": 0.8889,
617
+ "reward": 0.5278,
618
+ "pred_fc": 8,
619
+ "gold_fc": 9,
620
+ "pred_ps": 15,
621
+ "gold_ps": 15,
622
+ "pred_cc": 42,
623
+ "gold_cc": 42
624
+ },
625
+ {
626
+ "project": "spring-valley-sow-nursery-barn-pn-4223-001",
627
+ "panel_id": "B(208)",
628
+ "bucket": "spring_valley",
629
+ "cc_ok": true,
630
+ "fc_ok": false,
631
+ "ps_ok": false,
632
+ "groups_exact": false,
633
+ "self_consistent": false,
634
+ "multiset_jaccard": 0.569,
635
+ "reward": 0.1638,
636
+ "pred_fc": 55,
637
+ "gold_fc": 36,
638
+ "pred_ps": 40,
639
+ "gold_ps": 15,
640
+ "pred_cc": 72,
641
+ "gold_cc": 72
642
+ },
643
+ {
644
+ "project": "spring-valley-sow-nursery-barn-pn-4223-001",
645
+ "panel_id": "Panel 1",
646
+ "bucket": "spring_valley",
647
+ "cc_ok": true,
648
+ "fc_ok": false,
649
+ "ps_ok": false,
650
+ "groups_exact": false,
651
+ "self_consistent": false,
652
+ "multiset_jaccard": 0.569,
653
+ "reward": 0.1638,
654
+ "pred_fc": 55,
655
+ "gold_fc": 36,
656
+ "pred_ps": 40,
657
+ "gold_ps": 15,
658
+ "pred_cc": 72,
659
+ "gold_cc": 72
660
+ }
661
+ ]
662
+ }
sld_26.0/panel_counter_v19_rsft/eval/v19b_eval3.json ADDED
@@ -0,0 +1,84 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "variant": "v19b",
3
+ "model": "panel_counter_v19b_grpo",
4
+ "checkpoint": "",
5
+ "holdout_file": "sft_v19_pure30_eval.jsonl",
6
+ "total": 3,
7
+ "aggregate_cc_em": 1.0,
8
+ "aggregate_fc_em": 0.6667,
9
+ "aggregate_ps_em": 0.6667,
10
+ "aggregate_groups_exact": 0.3333,
11
+ "aggregate_self_consistent": 0.6667,
12
+ "aggregate_multiset_jaccard": 0.9513,
13
+ "aggregate_reward_mean": 0.7403,
14
+ "all_three_counts_exact": 2,
15
+ "all_three_counts_exact_rate": 0.6667,
16
+ "per_bucket": {
17
+ "8-pl": {
18
+ "n": 3,
19
+ "cc_em": 1.0,
20
+ "fc_em": 0.6667,
21
+ "ps_em": 0.6667,
22
+ "groups_exact": 0.3333,
23
+ "self_consistent": 0.6667,
24
+ "multiset_jaccard_mean": 0.9513,
25
+ "reward_mean": 0.7403
26
+ }
27
+ },
28
+ "per_panel": [
29
+ {
30
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
31
+ "panel_id": "P2-1-S1-1",
32
+ "bucket": "8-pl",
33
+ "cc_ok": true,
34
+ "fc_ok": false,
35
+ "ps_ok": false,
36
+ "groups_exact": false,
37
+ "self_consistent": false,
38
+ "multiset_jaccard": 0.9206,
39
+ "reward": 0.2341,
40
+ "pred_fc": 58,
41
+ "gold_fc": 63,
42
+ "pred_ps": 7,
43
+ "gold_ps": 6,
44
+ "pred_cc": 84,
45
+ "gold_cc": 84
46
+ },
47
+ {
48
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
49
+ "panel_id": "PS2-1-01-1",
50
+ "bucket": "8-pl",
51
+ "cc_ok": true,
52
+ "fc_ok": true,
53
+ "ps_ok": true,
54
+ "groups_exact": false,
55
+ "self_consistent": true,
56
+ "multiset_jaccard": 0.9333,
57
+ "reward": 0.9867,
58
+ "pred_fc": 29,
59
+ "gold_fc": 29,
60
+ "pred_ps": 11,
61
+ "gold_ps": 11,
62
+ "pred_cc": 42,
63
+ "gold_cc": 42
64
+ },
65
+ {
66
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
67
+ "panel_id": "PS6-1-S1-2",
68
+ "bucket": "8-pl",
69
+ "cc_ok": true,
70
+ "fc_ok": true,
71
+ "ps_ok": true,
72
+ "groups_exact": true,
73
+ "self_consistent": true,
74
+ "multiset_jaccard": 1.0,
75
+ "reward": 1.0,
76
+ "pred_fc": 2,
77
+ "gold_fc": 2,
78
+ "pred_ps": 36,
79
+ "gold_ps": 36,
80
+ "pred_cc": 42,
81
+ "gold_cc": 42
82
+ }
83
+ ]
84
+ }
sld_26.0/panel_counter_v19_rsft/eval/v19b_holdA.json ADDED
@@ -0,0 +1,958 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "variant": "v19b",
3
+ "model": "panel_counter_v19b_grpo",
4
+ "checkpoint": "",
5
+ "holdout_file": "sft_v17_holdA.jsonl",
6
+ "total": 51,
7
+ "aggregate_cc_em": 0.9804,
8
+ "aggregate_fc_em": 0.3529,
9
+ "aggregate_ps_em": 0.6078,
10
+ "aggregate_groups_exact": 0.2549,
11
+ "aggregate_self_consistent": 0.2745,
12
+ "aggregate_multiset_jaccard": 0.8116,
13
+ "aggregate_reward_mean": 0.5407,
14
+ "all_three_counts_exact": 12,
15
+ "all_three_counts_exact_rate": 0.2353,
16
+ "per_bucket": {
17
+ "stm_holdout": {
18
+ "n": 25,
19
+ "cc_em": 0.96,
20
+ "fc_em": 0.36,
21
+ "ps_em": 0.44,
22
+ "groups_exact": 0.2,
23
+ "self_consistent": 0.24,
24
+ "multiset_jaccard_mean": 0.7817,
25
+ "reward_mean": 0.4823
26
+ },
27
+ "8-pl": {
28
+ "n": 26,
29
+ "cc_em": 1.0,
30
+ "fc_em": 0.3462,
31
+ "ps_em": 0.7692,
32
+ "groups_exact": 0.3077,
33
+ "self_consistent": 0.3077,
34
+ "multiset_jaccard_mean": 0.8403,
35
+ "reward_mean": 0.5969
36
+ }
37
+ },
38
+ "per_panel": [
39
+ {
40
+ "project": "stm-lot-ci22",
41
+ "panel_id": "B2",
42
+ "bucket": "stm_holdout",
43
+ "cc_ok": true,
44
+ "fc_ok": true,
45
+ "ps_ok": true,
46
+ "groups_exact": true,
47
+ "self_consistent": true,
48
+ "multiset_jaccard": 1.0,
49
+ "reward": 1.0,
50
+ "pred_fc": 7,
51
+ "gold_fc": 7,
52
+ "pred_ps": 23,
53
+ "gold_ps": 23,
54
+ "pred_cc": 30,
55
+ "gold_cc": 30
56
+ },
57
+ {
58
+ "project": "stm-lot-ci22",
59
+ "panel_id": "BB2",
60
+ "bucket": "stm_holdout",
61
+ "cc_ok": true,
62
+ "fc_ok": true,
63
+ "ps_ok": false,
64
+ "groups_exact": true,
65
+ "self_consistent": false,
66
+ "multiset_jaccard": 1.0,
67
+ "reward": 0.6,
68
+ "pred_fc": 12,
69
+ "gold_fc": 12,
70
+ "pred_ps": 24,
71
+ "gold_ps": 28,
72
+ "pred_cc": 42,
73
+ "gold_cc": 42
74
+ },
75
+ {
76
+ "project": "stm-lot-ci22",
77
+ "panel_id": "C1",
78
+ "bucket": "stm_holdout",
79
+ "cc_ok": true,
80
+ "fc_ok": false,
81
+ "ps_ok": true,
82
+ "groups_exact": false,
83
+ "self_consistent": false,
84
+ "multiset_jaccard": 0.8148,
85
+ "reward": 0.513,
86
+ "pred_fc": 25,
87
+ "gold_fc": 24,
88
+ "pred_ps": 15,
89
+ "gold_ps": 15,
90
+ "pred_cc": 42,
91
+ "gold_cc": 42
92
+ },
93
+ {
94
+ "project": "stm-lot-ci22",
95
+ "panel_id": "CC4",
96
+ "bucket": "stm_holdout",
97
+ "cc_ok": true,
98
+ "fc_ok": false,
99
+ "ps_ok": false,
100
+ "groups_exact": false,
101
+ "self_consistent": false,
102
+ "multiset_jaccard": 0.8,
103
+ "reward": 0.21,
104
+ "pred_fc": 20,
105
+ "gold_fc": 16,
106
+ "pred_ps": 22,
107
+ "gold_ps": 24,
108
+ "pred_cc": 42,
109
+ "gold_cc": 42
110
+ },
111
+ {
112
+ "project": "stm-lot-ci22",
113
+ "panel_id": "D2",
114
+ "bucket": "stm_holdout",
115
+ "cc_ok": true,
116
+ "fc_ok": false,
117
+ "ps_ok": false,
118
+ "groups_exact": false,
119
+ "self_consistent": false,
120
+ "multiset_jaccard": 0.7778,
121
+ "reward": 0.2056,
122
+ "pred_fc": 8,
123
+ "gold_fc": 7,
124
+ "pred_ps": 34,
125
+ "gold_ps": 33,
126
+ "pred_cc": 42,
127
+ "gold_cc": 42
128
+ },
129
+ {
130
+ "project": "stm-lot-ci22",
131
+ "panel_id": "F1",
132
+ "bucket": "stm_holdout",
133
+ "cc_ok": true,
134
+ "fc_ok": false,
135
+ "ps_ok": false,
136
+ "groups_exact": false,
137
+ "self_consistent": false,
138
+ "multiset_jaccard": 0.75,
139
+ "reward": 0.2,
140
+ "pred_fc": 4,
141
+ "gold_fc": 3,
142
+ "pred_ps": 40,
143
+ "gold_ps": 38,
144
+ "pred_cc": 42,
145
+ "gold_cc": 42
146
+ },
147
+ {
148
+ "project": "stm-lot-ci22",
149
+ "panel_id": "F8",
150
+ "bucket": "stm_holdout",
151
+ "cc_ok": true,
152
+ "fc_ok": true,
153
+ "ps_ok": true,
154
+ "groups_exact": true,
155
+ "self_consistent": true,
156
+ "multiset_jaccard": 1.0,
157
+ "reward": 1.0,
158
+ "pred_fc": 3,
159
+ "gold_fc": 3,
160
+ "pred_ps": 38,
161
+ "gold_ps": 38,
162
+ "pred_cc": 42,
163
+ "gold_cc": 42
164
+ },
165
+ {
166
+ "project": "stm-lot-ci22",
167
+ "panel_id": "FF2",
168
+ "bucket": "stm_holdout",
169
+ "cc_ok": true,
170
+ "fc_ok": false,
171
+ "ps_ok": true,
172
+ "groups_exact": false,
173
+ "self_consistent": false,
174
+ "multiset_jaccard": 0.5909,
175
+ "reward": 0.4682,
176
+ "pred_fc": 19,
177
+ "gold_fc": 16,
178
+ "pred_ps": 12,
179
+ "gold_ps": 12,
180
+ "pred_cc": 54,
181
+ "gold_cc": 54
182
+ },
183
+ {
184
+ "project": "stm-lot-ci22",
185
+ "panel_id": "FF4",
186
+ "bucket": "stm_holdout",
187
+ "cc_ok": true,
188
+ "fc_ok": false,
189
+ "ps_ok": false,
190
+ "groups_exact": false,
191
+ "self_consistent": false,
192
+ "multiset_jaccard": 0.4286,
193
+ "reward": 0.1357,
194
+ "pred_fc": 42,
195
+ "gold_fc": 18,
196
+ "pred_ps": 27,
197
+ "gold_ps": 24,
198
+ "pred_cc": 42,
199
+ "gold_cc": 42
200
+ },
201
+ {
202
+ "project": "stm-lot-ci22",
203
+ "panel_id": "HH3",
204
+ "bucket": "stm_holdout",
205
+ "cc_ok": true,
206
+ "fc_ok": false,
207
+ "ps_ok": false,
208
+ "groups_exact": false,
209
+ "self_consistent": false,
210
+ "multiset_jaccard": 0.8,
211
+ "reward": 0.21,
212
+ "pred_fc": 10,
213
+ "gold_fc": 8,
214
+ "pred_ps": 14,
215
+ "gold_ps": 20,
216
+ "pred_cc": 30,
217
+ "gold_cc": 30
218
+ },
219
+ {
220
+ "project": "stm-lot-ci22",
221
+ "panel_id": "K7",
222
+ "bucket": "stm_holdout",
223
+ "cc_ok": true,
224
+ "fc_ok": false,
225
+ "ps_ok": false,
226
+ "groups_exact": false,
227
+ "self_consistent": true,
228
+ "multiset_jaccard": 0.9167,
229
+ "reward": 0.2333,
230
+ "pred_fc": 12,
231
+ "gold_fc": 11,
232
+ "pred_ps": 18,
233
+ "gold_ps": 19,
234
+ "pred_cc": 30,
235
+ "gold_cc": 30
236
+ },
237
+ {
238
+ "project": "stm-lot-ci22",
239
+ "panel_id": "KK2",
240
+ "bucket": "stm_holdout",
241
+ "cc_ok": true,
242
+ "fc_ok": true,
243
+ "ps_ok": true,
244
+ "groups_exact": true,
245
+ "self_consistent": true,
246
+ "multiset_jaccard": 1.0,
247
+ "reward": 1.0,
248
+ "pred_fc": 7,
249
+ "gold_fc": 7,
250
+ "pred_ps": 21,
251
+ "gold_ps": 21,
252
+ "pred_cc": 30,
253
+ "gold_cc": 30
254
+ },
255
+ {
256
+ "project": "stm-lot-ci22",
257
+ "panel_id": "KK3",
258
+ "bucket": "stm_holdout",
259
+ "cc_ok": true,
260
+ "fc_ok": true,
261
+ "ps_ok": false,
262
+ "groups_exact": false,
263
+ "self_consistent": false,
264
+ "multiset_jaccard": 0.8333,
265
+ "reward": 0.5667,
266
+ "pred_fc": 11,
267
+ "gold_fc": 11,
268
+ "pred_ps": 26,
269
+ "gold_ps": 29,
270
+ "pred_cc": 42,
271
+ "gold_cc": 42
272
+ },
273
+ {
274
+ "project": "stm-lot-ci22",
275
+ "panel_id": "KK5",
276
+ "bucket": "stm_holdout",
277
+ "cc_ok": true,
278
+ "fc_ok": false,
279
+ "ps_ok": true,
280
+ "groups_exact": false,
281
+ "self_consistent": false,
282
+ "multiset_jaccard": 0.7,
283
+ "reward": 0.49,
284
+ "pred_fc": 9,
285
+ "gold_fc": 8,
286
+ "pred_ps": 34,
287
+ "gold_ps": 34,
288
+ "pred_cc": 54,
289
+ "gold_cc": 54
290
+ },
291
+ {
292
+ "project": "stm-lot-ci22",
293
+ "panel_id": "N1",
294
+ "bucket": "stm_holdout",
295
+ "cc_ok": true,
296
+ "fc_ok": true,
297
+ "ps_ok": true,
298
+ "groups_exact": false,
299
+ "self_consistent": false,
300
+ "multiset_jaccard": 0.8667,
301
+ "reward": 0.9733,
302
+ "pred_fc": 14,
303
+ "gold_fc": 14,
304
+ "pred_ps": 27,
305
+ "gold_ps": 27,
306
+ "pred_cc": 42,
307
+ "gold_cc": 42
308
+ },
309
+ {
310
+ "project": "stm-lot-ci22",
311
+ "panel_id": "P2",
312
+ "bucket": "stm_holdout",
313
+ "cc_ok": true,
314
+ "fc_ok": false,
315
+ "ps_ok": true,
316
+ "groups_exact": false,
317
+ "self_consistent": false,
318
+ "multiset_jaccard": 0.7778,
319
+ "reward": 0.5056,
320
+ "pred_fc": 17,
321
+ "gold_fc": 15,
322
+ "pred_ps": 14,
323
+ "gold_ps": 14,
324
+ "pred_cc": 30,
325
+ "gold_cc": 30
326
+ },
327
+ {
328
+ "project": "stm-lot-ci22",
329
+ "panel_id": "SS2",
330
+ "bucket": "stm_holdout",
331
+ "cc_ok": false,
332
+ "fc_ok": false,
333
+ "ps_ok": false,
334
+ "groups_exact": false,
335
+ "self_consistent": false,
336
+ "multiset_jaccard": 0.0,
337
+ "reward": 0.0,
338
+ "pred_fc": null,
339
+ "gold_fc": 11,
340
+ "pred_ps": null,
341
+ "gold_ps": 27,
342
+ "pred_cc": null,
343
+ "gold_cc": 54
344
+ },
345
+ {
346
+ "project": "stm-lot-ci22",
347
+ "panel_id": "U5",
348
+ "bucket": "stm_holdout",
349
+ "cc_ok": true,
350
+ "fc_ok": true,
351
+ "ps_ok": true,
352
+ "groups_exact": true,
353
+ "self_consistent": true,
354
+ "multiset_jaccard": 1.0,
355
+ "reward": 1.0,
356
+ "pred_fc": 11,
357
+ "gold_fc": 11,
358
+ "pred_ps": 30,
359
+ "gold_ps": 30,
360
+ "pred_cc": 42,
361
+ "gold_cc": 42
362
+ },
363
+ {
364
+ "project": "stm-lot-ci22",
365
+ "panel_id": "U9",
366
+ "bucket": "stm_holdout",
367
+ "cc_ok": true,
368
+ "fc_ok": false,
369
+ "ps_ok": true,
370
+ "groups_exact": false,
371
+ "self_consistent": false,
372
+ "multiset_jaccard": 0.7857,
373
+ "reward": 0.5071,
374
+ "pred_fc": 13,
375
+ "gold_fc": 12,
376
+ "pred_ps": 28,
377
+ "gold_ps": 28,
378
+ "pred_cc": 42,
379
+ "gold_cc": 42
380
+ },
381
+ {
382
+ "project": "stm-lot-ci22",
383
+ "panel_id": "UU13",
384
+ "bucket": "stm_holdout",
385
+ "cc_ok": true,
386
+ "fc_ok": true,
387
+ "ps_ok": false,
388
+ "groups_exact": false,
389
+ "self_consistent": false,
390
+ "multiset_jaccard": 0.7857,
391
+ "reward": 0.5571,
392
+ "pred_fc": 12,
393
+ "gold_fc": 12,
394
+ "pred_ps": 8,
395
+ "gold_ps": 10,
396
+ "pred_cc": 42,
397
+ "gold_cc": 42
398
+ },
399
+ {
400
+ "project": "stm-lot-ci22",
401
+ "panel_id": "UU3",
402
+ "bucket": "stm_holdout",
403
+ "cc_ok": true,
404
+ "fc_ok": false,
405
+ "ps_ok": false,
406
+ "groups_exact": false,
407
+ "self_consistent": true,
408
+ "multiset_jaccard": 0.8,
409
+ "reward": 0.21,
410
+ "pred_fc": 4,
411
+ "gold_fc": 5,
412
+ "pred_ps": 22,
413
+ "gold_ps": 19,
414
+ "pred_cc": 30,
415
+ "gold_cc": 30
416
+ },
417
+ {
418
+ "project": "stm-lot-ci22",
419
+ "panel_id": "UU5",
420
+ "bucket": "stm_holdout",
421
+ "cc_ok": true,
422
+ "fc_ok": false,
423
+ "ps_ok": true,
424
+ "groups_exact": false,
425
+ "self_consistent": false,
426
+ "multiset_jaccard": 0.6667,
427
+ "reward": 0.4833,
428
+ "pred_fc": 11,
429
+ "gold_fc": 9,
430
+ "pred_ps": 19,
431
+ "gold_ps": 19,
432
+ "pred_cc": 42,
433
+ "gold_cc": 42
434
+ },
435
+ {
436
+ "project": "stm-lot-ci22",
437
+ "panel_id": "UU9",
438
+ "bucket": "stm_holdout",
439
+ "cc_ok": true,
440
+ "fc_ok": false,
441
+ "ps_ok": false,
442
+ "groups_exact": false,
443
+ "self_consistent": false,
444
+ "multiset_jaccard": 0.8,
445
+ "reward": 0.21,
446
+ "pred_fc": 10,
447
+ "gold_fc": 9,
448
+ "pred_ps": 18,
449
+ "gold_ps": 19,
450
+ "pred_cc": 42,
451
+ "gold_cc": 42
452
+ },
453
+ {
454
+ "project": "stm-lot-ci22",
455
+ "panel_id": "UUSP13",
456
+ "bucket": "stm_holdout",
457
+ "cc_ok": true,
458
+ "fc_ok": false,
459
+ "ps_ok": false,
460
+ "groups_exact": false,
461
+ "self_consistent": false,
462
+ "multiset_jaccard": 0.9333,
463
+ "reward": 0.2367,
464
+ "pred_fc": 30,
465
+ "gold_fc": 28,
466
+ "pred_ps": 23,
467
+ "gold_ps": 24,
468
+ "pred_cc": 54,
469
+ "gold_cc": 54
470
+ },
471
+ {
472
+ "project": "stm-lot-ci22",
473
+ "panel_id": "UUSP6",
474
+ "bucket": "stm_holdout",
475
+ "cc_ok": true,
476
+ "fc_ok": true,
477
+ "ps_ok": false,
478
+ "groups_exact": false,
479
+ "self_consistent": false,
480
+ "multiset_jaccard": 0.7143,
481
+ "reward": 0.5429,
482
+ "pred_fc": 6,
483
+ "gold_fc": 6,
484
+ "pred_ps": 26,
485
+ "gold_ps": 18,
486
+ "pred_cc": 30,
487
+ "gold_cc": 30
488
+ },
489
+ {
490
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
491
+ "panel_id": "P2-1-01-1",
492
+ "bucket": "8-pl",
493
+ "cc_ok": true,
494
+ "fc_ok": false,
495
+ "ps_ok": true,
496
+ "groups_exact": false,
497
+ "self_consistent": false,
498
+ "multiset_jaccard": 0.5286,
499
+ "reward": 0.4557,
500
+ "pred_fc": 53,
501
+ "gold_fc": 54,
502
+ "pred_ps": 10,
503
+ "gold_ps": 10,
504
+ "pred_cc": 72,
505
+ "gold_cc": 72
506
+ },
507
+ {
508
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
509
+ "panel_id": "P2-1-01-2",
510
+ "bucket": "8-pl",
511
+ "cc_ok": true,
512
+ "fc_ok": false,
513
+ "ps_ok": true,
514
+ "groups_exact": false,
515
+ "self_consistent": false,
516
+ "multiset_jaccard": 0.9167,
517
+ "reward": 0.5333,
518
+ "pred_fc": 22,
519
+ "gold_fc": 24,
520
+ "pred_ps": 12,
521
+ "gold_ps": 12,
522
+ "pred_cc": 42,
523
+ "gold_cc": 42
524
+ },
525
+ {
526
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
527
+ "panel_id": "P2-1-02-1",
528
+ "bucket": "8-pl",
529
+ "cc_ok": true,
530
+ "fc_ok": true,
531
+ "ps_ok": true,
532
+ "groups_exact": true,
533
+ "self_consistent": true,
534
+ "multiset_jaccard": 1.0,
535
+ "reward": 1.0,
536
+ "pred_fc": 28,
537
+ "gold_fc": 28,
538
+ "pred_ps": 6,
539
+ "gold_ps": 6,
540
+ "pred_cc": 42,
541
+ "gold_cc": 42
542
+ },
543
+ {
544
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
545
+ "panel_id": "P2-1-04-1",
546
+ "bucket": "8-pl",
547
+ "cc_ok": true,
548
+ "fc_ok": false,
549
+ "ps_ok": true,
550
+ "groups_exact": false,
551
+ "self_consistent": false,
552
+ "multiset_jaccard": 0.6341,
553
+ "reward": 0.4768,
554
+ "pred_fc": 82,
555
+ "gold_fc": 52,
556
+ "pred_ps": 2,
557
+ "gold_ps": 2,
558
+ "pred_cc": 72,
559
+ "gold_cc": 72
560
+ },
561
+ {
562
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
563
+ "panel_id": "P2-1-08-1",
564
+ "bucket": "8-pl",
565
+ "cc_ok": true,
566
+ "fc_ok": false,
567
+ "ps_ok": true,
568
+ "groups_exact": false,
569
+ "self_consistent": false,
570
+ "multiset_jaccard": 0.8,
571
+ "reward": 0.51,
572
+ "pred_fc": 56,
573
+ "gold_fc": 52,
574
+ "pred_ps": 2,
575
+ "gold_ps": 2,
576
+ "pred_cc": 72,
577
+ "gold_cc": 72
578
+ },
579
+ {
580
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
581
+ "panel_id": "P2-1-12-1",
582
+ "bucket": "8-pl",
583
+ "cc_ok": true,
584
+ "fc_ok": false,
585
+ "ps_ok": true,
586
+ "groups_exact": false,
587
+ "self_consistent": false,
588
+ "multiset_jaccard": 0.6625,
589
+ "reward": 0.4825,
590
+ "pred_fc": 80,
591
+ "gold_fc": 53,
592
+ "pred_ps": 6,
593
+ "gold_ps": 6,
594
+ "pred_cc": 72,
595
+ "gold_cc": 72
596
+ },
597
+ {
598
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
599
+ "panel_id": "P2-1-S1-1",
600
+ "bucket": "8-pl",
601
+ "cc_ok": true,
602
+ "fc_ok": false,
603
+ "ps_ok": false,
604
+ "groups_exact": false,
605
+ "self_consistent": false,
606
+ "multiset_jaccard": 0.9206,
607
+ "reward": 0.2341,
608
+ "pred_fc": 58,
609
+ "gold_fc": 63,
610
+ "pred_ps": 7,
611
+ "gold_ps": 6,
612
+ "pred_cc": 84,
613
+ "gold_cc": 84
614
+ },
615
+ {
616
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
617
+ "panel_id": "P2-1-TT-1",
618
+ "bucket": "8-pl",
619
+ "cc_ok": true,
620
+ "fc_ok": false,
621
+ "ps_ok": false,
622
+ "groups_exact": false,
623
+ "self_consistent": false,
624
+ "multiset_jaccard": 0.4762,
625
+ "reward": 0.1452,
626
+ "pred_fc": 23,
627
+ "gold_fc": 36,
628
+ "pred_ps": 6,
629
+ "gold_ps": 18,
630
+ "pred_cc": 72,
631
+ "gold_cc": 72
632
+ },
633
+ {
634
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
635
+ "panel_id": "P6-1-01-1",
636
+ "bucket": "8-pl",
637
+ "cc_ok": true,
638
+ "fc_ok": false,
639
+ "ps_ok": true,
640
+ "groups_exact": false,
641
+ "self_consistent": false,
642
+ "multiset_jaccard": 0.8636,
643
+ "reward": 0.5227,
644
+ "pred_fc": 19,
645
+ "gold_fc": 20,
646
+ "pred_ps": 30,
647
+ "gold_ps": 30,
648
+ "pred_cc": 72,
649
+ "gold_cc": 72
650
+ },
651
+ {
652
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
653
+ "panel_id": "P6-1-02-1",
654
+ "bucket": "8-pl",
655
+ "cc_ok": true,
656
+ "fc_ok": false,
657
+ "ps_ok": true,
658
+ "groups_exact": false,
659
+ "self_consistent": false,
660
+ "multiset_jaccard": 0.8095,
661
+ "reward": 0.5119,
662
+ "pred_fc": 14,
663
+ "gold_fc": 21,
664
+ "pred_ps": 3,
665
+ "gold_ps": 3,
666
+ "pred_cc": 60,
667
+ "gold_cc": 60
668
+ },
669
+ {
670
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
671
+ "panel_id": "P6-1-S1-1",
672
+ "bucket": "8-pl",
673
+ "cc_ok": true,
674
+ "fc_ok": false,
675
+ "ps_ok": true,
676
+ "groups_exact": false,
677
+ "self_consistent": false,
678
+ "multiset_jaccard": 0.5217,
679
+ "reward": 0.4543,
680
+ "pred_fc": 12,
681
+ "gold_fc": 23,
682
+ "pred_ps": 29,
683
+ "gold_ps": 29,
684
+ "pred_cc": 72,
685
+ "gold_cc": 72
686
+ },
687
+ {
688
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
689
+ "panel_id": "P6-1-TT-1",
690
+ "bucket": "8-pl",
691
+ "cc_ok": true,
692
+ "fc_ok": false,
693
+ "ps_ok": true,
694
+ "groups_exact": false,
695
+ "self_consistent": false,
696
+ "multiset_jaccard": 0.8621,
697
+ "reward": 0.5224,
698
+ "pred_fc": 24,
699
+ "gold_fc": 29,
700
+ "pred_ps": 3,
701
+ "gold_ps": 3,
702
+ "pred_cc": 84,
703
+ "gold_cc": 84
704
+ },
705
+ {
706
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
707
+ "panel_id": "PB-1-S1-1",
708
+ "bucket": "8-pl",
709
+ "cc_ok": true,
710
+ "fc_ok": true,
711
+ "ps_ok": true,
712
+ "groups_exact": true,
713
+ "self_consistent": true,
714
+ "multiset_jaccard": 1.0,
715
+ "reward": 1.0,
716
+ "pred_fc": 28,
717
+ "gold_fc": 28,
718
+ "pred_ps": 28,
719
+ "gold_ps": 28,
720
+ "pred_cc": 84,
721
+ "gold_cc": 84
722
+ },
723
+ {
724
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
725
+ "panel_id": "PB6-1-S1-1",
726
+ "bucket": "8-pl",
727
+ "cc_ok": true,
728
+ "fc_ok": true,
729
+ "ps_ok": true,
730
+ "groups_exact": true,
731
+ "self_consistent": true,
732
+ "multiset_jaccard": 1.0,
733
+ "reward": 1.0,
734
+ "pred_fc": 4,
735
+ "gold_fc": 4,
736
+ "pred_ps": 30,
737
+ "gold_ps": 30,
738
+ "pred_cc": 42,
739
+ "gold_cc": 42
740
+ },
741
+ {
742
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
743
+ "panel_id": "PD6-1-S1-2",
744
+ "bucket": "8-pl",
745
+ "cc_ok": true,
746
+ "fc_ok": true,
747
+ "ps_ok": true,
748
+ "groups_exact": true,
749
+ "self_consistent": true,
750
+ "multiset_jaccard": 1.0,
751
+ "reward": 1.0,
752
+ "pred_fc": 4,
753
+ "gold_fc": 4,
754
+ "pred_ps": 30,
755
+ "gold_ps": 30,
756
+ "pred_cc": 42,
757
+ "gold_cc": 42
758
+ },
759
+ {
760
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
761
+ "panel_id": "PD6-1-S1-4",
762
+ "bucket": "8-pl",
763
+ "cc_ok": true,
764
+ "fc_ok": false,
765
+ "ps_ok": false,
766
+ "groups_exact": false,
767
+ "self_consistent": false,
768
+ "multiset_jaccard": 0.8,
769
+ "reward": 0.21,
770
+ "pred_fc": 6,
771
+ "gold_fc": 9,
772
+ "pred_ps": 24,
773
+ "gold_ps": 23,
774
+ "pred_cc": 42,
775
+ "gold_cc": 42
776
+ },
777
+ {
778
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
779
+ "panel_id": "PDS6-1-S1-1",
780
+ "bucket": "8-pl",
781
+ "cc_ok": true,
782
+ "fc_ok": true,
783
+ "ps_ok": false,
784
+ "groups_exact": true,
785
+ "self_consistent": false,
786
+ "multiset_jaccard": 1.0,
787
+ "reward": 0.6,
788
+ "pred_fc": 5,
789
+ "gold_fc": 5,
790
+ "pred_ps": 24,
791
+ "gold_ps": 27,
792
+ "pred_cc": 42,
793
+ "gold_cc": 42
794
+ },
795
+ {
796
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
797
+ "panel_id": "PS2-1-01-1",
798
+ "bucket": "8-pl",
799
+ "cc_ok": true,
800
+ "fc_ok": true,
801
+ "ps_ok": true,
802
+ "groups_exact": false,
803
+ "self_consistent": true,
804
+ "multiset_jaccard": 0.9333,
805
+ "reward": 0.9867,
806
+ "pred_fc": 29,
807
+ "gold_fc": 29,
808
+ "pred_ps": 11,
809
+ "gold_ps": 11,
810
+ "pred_cc": 42,
811
+ "gold_cc": 42
812
+ },
813
+ {
814
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
815
+ "panel_id": "PS2-1-04-1",
816
+ "bucket": "8-pl",
817
+ "cc_ok": true,
818
+ "fc_ok": false,
819
+ "ps_ok": false,
820
+ "groups_exact": false,
821
+ "self_consistent": true,
822
+ "multiset_jaccard": 0.9615,
823
+ "reward": 0.2423,
824
+ "pred_fc": 25,
825
+ "gold_fc": 26,
826
+ "pred_ps": 15,
827
+ "gold_ps": 14,
828
+ "pred_cc": 42,
829
+ "gold_cc": 42
830
+ },
831
+ {
832
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
833
+ "panel_id": "PS2-1-08-1",
834
+ "bucket": "8-pl",
835
+ "cc_ok": true,
836
+ "fc_ok": true,
837
+ "ps_ok": true,
838
+ "groups_exact": true,
839
+ "self_consistent": true,
840
+ "multiset_jaccard": 1.0,
841
+ "reward": 1.0,
842
+ "pred_fc": 25,
843
+ "gold_fc": 25,
844
+ "pred_ps": 15,
845
+ "gold_ps": 15,
846
+ "pred_cc": 42,
847
+ "gold_cc": 42
848
+ },
849
+ {
850
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
851
+ "panel_id": "PS2-1-12-1",
852
+ "bucket": "8-pl",
853
+ "cc_ok": true,
854
+ "fc_ok": false,
855
+ "ps_ok": true,
856
+ "groups_exact": false,
857
+ "self_consistent": false,
858
+ "multiset_jaccard": 0.75,
859
+ "reward": 0.5,
860
+ "pred_fc": 15,
861
+ "gold_fc": 20,
862
+ "pred_ps": 22,
863
+ "gold_ps": 22,
864
+ "pred_cc": 42,
865
+ "gold_cc": 42
866
+ },
867
+ {
868
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
869
+ "panel_id": "PS2-1-S1-1",
870
+ "bucket": "8-pl",
871
+ "cc_ok": true,
872
+ "fc_ok": false,
873
+ "ps_ok": true,
874
+ "groups_exact": false,
875
+ "self_consistent": false,
876
+ "multiset_jaccard": 0.8,
877
+ "reward": 0.51,
878
+ "pred_fc": 16,
879
+ "gold_fc": 20,
880
+ "pred_ps": 19,
881
+ "gold_ps": 19,
882
+ "pred_cc": 42,
883
+ "gold_cc": 42
884
+ },
885
+ {
886
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
887
+ "panel_id": "PS6-1-S1-1",
888
+ "bucket": "8-pl",
889
+ "cc_ok": true,
890
+ "fc_ok": false,
891
+ "ps_ok": true,
892
+ "groups_exact": false,
893
+ "self_consistent": false,
894
+ "multiset_jaccard": 0.9412,
895
+ "reward": 0.5382,
896
+ "pred_fc": 17,
897
+ "gold_fc": 16,
898
+ "pred_ps": 18,
899
+ "gold_ps": 18,
900
+ "pred_cc": 42,
901
+ "gold_cc": 42
902
+ },
903
+ {
904
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
905
+ "panel_id": "PS6-1-S1-2",
906
+ "bucket": "8-pl",
907
+ "cc_ok": true,
908
+ "fc_ok": true,
909
+ "ps_ok": true,
910
+ "groups_exact": true,
911
+ "self_consistent": true,
912
+ "multiset_jaccard": 1.0,
913
+ "reward": 1.0,
914
+ "pred_fc": 2,
915
+ "gold_fc": 2,
916
+ "pred_ps": 36,
917
+ "gold_ps": 36,
918
+ "pred_cc": 42,
919
+ "gold_cc": 42
920
+ },
921
+ {
922
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
923
+ "panel_id": "PU2-1-S1-1",
924
+ "bucket": "8-pl",
925
+ "cc_ok": true,
926
+ "fc_ok": false,
927
+ "ps_ok": true,
928
+ "groups_exact": false,
929
+ "self_consistent": false,
930
+ "multiset_jaccard": 0.6667,
931
+ "reward": 0.4833,
932
+ "pred_fc": 12,
933
+ "gold_fc": 8,
934
+ "pred_ps": 34,
935
+ "gold_ps": 34,
936
+ "pred_cc": 42,
937
+ "gold_cc": 42
938
+ },
939
+ {
940
+ "project": "8-pl-du-commerce-bat-b1-pur-immobilia-galleria",
941
+ "panel_id": "PU6-1-S1-1",
942
+ "bucket": "8-pl",
943
+ "cc_ok": true,
944
+ "fc_ok": true,
945
+ "ps_ok": false,
946
+ "groups_exact": true,
947
+ "self_consistent": false,
948
+ "multiset_jaccard": 1.0,
949
+ "reward": 0.6,
950
+ "pred_fc": 9,
951
+ "gold_fc": 9,
952
+ "pred_ps": 24,
953
+ "gold_ps": 23,
954
+ "pred_cc": 42,
955
+ "gold_cc": 42
956
+ }
957
+ ]
958
+ }
sld_26.0/panel_counter_v19_rsft/eval/v19b_holdB.json ADDED
@@ -0,0 +1,662 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "variant": "v19b",
3
+ "model": "panel_counter_v19b_grpo",
4
+ "checkpoint": "",
5
+ "holdout_file": "sft_v17_holdB.jsonl",
6
+ "total": 34,
7
+ "aggregate_cc_em": 1.0,
8
+ "aggregate_fc_em": 0.2059,
9
+ "aggregate_ps_em": 0.2647,
10
+ "aggregate_groups_exact": 0.0882,
11
+ "aggregate_self_consistent": 0.1765,
12
+ "aggregate_multiset_jaccard": 0.7581,
13
+ "aggregate_reward_mean": 0.359,
14
+ "all_three_counts_exact": 3,
15
+ "all_three_counts_exact_rate": 0.0882,
16
+ "per_bucket": {
17
+ "calgary": {
18
+ "n": 4,
19
+ "cc_em": 1.0,
20
+ "fc_em": 0.25,
21
+ "ps_em": 0.25,
22
+ "groups_exact": 0.0,
23
+ "self_consistent": 0.75,
24
+ "multiset_jaccard_mean": 0.3929,
25
+ "reward_mean": 0.2911
26
+ },
27
+ "omni": {
28
+ "n": 28,
29
+ "cc_em": 1.0,
30
+ "fc_em": 0.2143,
31
+ "ps_em": 0.2857,
32
+ "groups_exact": 0.1071,
33
+ "self_consistent": 0.1071,
34
+ "multiset_jaccard_mean": 0.8402,
35
+ "reward_mean": 0.3859
36
+ },
37
+ "spring_valley": {
38
+ "n": 2,
39
+ "cc_em": 1.0,
40
+ "fc_em": 0.0,
41
+ "ps_em": 0.0,
42
+ "groups_exact": 0.0,
43
+ "self_consistent": 0.0,
44
+ "multiset_jaccard_mean": 0.3377,
45
+ "reward_mean": 0.1175
46
+ }
47
+ },
48
+ "per_panel": [
49
+ {
50
+ "project": "calgary-winter-club-renovation-pn-25-01-0297",
51
+ "panel_id": "AD",
52
+ "bucket": "calgary",
53
+ "cc_ok": true,
54
+ "fc_ok": true,
55
+ "ps_ok": true,
56
+ "groups_exact": false,
57
+ "self_consistent": true,
58
+ "multiset_jaccard": 0.9765,
59
+ "reward": 0.9953,
60
+ "pred_fc": 84,
61
+ "gold_fc": 84,
62
+ "pred_ps": 0,
63
+ "gold_ps": 0,
64
+ "pred_cc": 84,
65
+ "gold_cc": 84
66
+ },
67
+ {
68
+ "project": "calgary-winter-club-renovation-pn-25-01-0297",
69
+ "panel_id": "M2F",
70
+ "bucket": "calgary",
71
+ "cc_ok": true,
72
+ "fc_ok": false,
73
+ "ps_ok": false,
74
+ "groups_exact": false,
75
+ "self_consistent": false,
76
+ "multiset_jaccard": 0.5952,
77
+ "reward": 0.169,
78
+ "pred_fc": 25,
79
+ "gold_fc": 42,
80
+ "pred_ps": 16,
81
+ "gold_ps": 0,
82
+ "pred_cc": 42,
83
+ "gold_cc": 42
84
+ },
85
+ {
86
+ "project": "calgary-winter-club-renovation-pn-25-01-0297",
87
+ "panel_id": "M2P",
88
+ "bucket": "calgary",
89
+ "cc_ok": true,
90
+ "fc_ok": false,
91
+ "ps_ok": false,
92
+ "groups_exact": false,
93
+ "self_consistent": true,
94
+ "multiset_jaccard": 0.0,
95
+ "reward": 0.0,
96
+ "pred_fc": 0,
97
+ "gold_fc": 42,
98
+ "pred_ps": 42,
99
+ "gold_ps": 0,
100
+ "pred_cc": 42,
101
+ "gold_cc": 42
102
+ },
103
+ {
104
+ "project": "calgary-winter-club-renovation-pn-25-01-0297",
105
+ "panel_id": "M6A",
106
+ "bucket": "calgary",
107
+ "cc_ok": true,
108
+ "fc_ok": false,
109
+ "ps_ok": false,
110
+ "groups_exact": false,
111
+ "self_consistent": true,
112
+ "multiset_jaccard": 0.0,
113
+ "reward": 0.0,
114
+ "pred_fc": 0,
115
+ "gold_fc": 42,
116
+ "pred_ps": 42,
117
+ "gold_ps": 0,
118
+ "pred_cc": 42,
119
+ "gold_cc": 42
120
+ },
121
+ {
122
+ "project": "omni-seaforth-ltc",
123
+ "panel_id": "1C1",
124
+ "bucket": "omni",
125
+ "cc_ok": true,
126
+ "fc_ok": false,
127
+ "ps_ok": false,
128
+ "groups_exact": false,
129
+ "self_consistent": false,
130
+ "multiset_jaccard": 0.7778,
131
+ "reward": 0.2056,
132
+ "pred_fc": 35,
133
+ "gold_fc": 45,
134
+ "pred_ps": 17,
135
+ "gold_ps": 15,
136
+ "pred_cc": 72,
137
+ "gold_cc": 72
138
+ },
139
+ {
140
+ "project": "omni-seaforth-ltc",
141
+ "panel_id": "1C2",
142
+ "bucket": "omni",
143
+ "cc_ok": true,
144
+ "fc_ok": true,
145
+ "ps_ok": true,
146
+ "groups_exact": true,
147
+ "self_consistent": true,
148
+ "multiset_jaccard": 1.0,
149
+ "reward": 1.0,
150
+ "pred_fc": 15,
151
+ "gold_fc": 15,
152
+ "pred_ps": 20,
153
+ "gold_ps": 20,
154
+ "pred_cc": 42,
155
+ "gold_cc": 42
156
+ },
157
+ {
158
+ "project": "omni-seaforth-ltc",
159
+ "panel_id": "1C3",
160
+ "bucket": "omni",
161
+ "cc_ok": true,
162
+ "fc_ok": false,
163
+ "ps_ok": false,
164
+ "groups_exact": false,
165
+ "self_consistent": false,
166
+ "multiset_jaccard": 0.8636,
167
+ "reward": 0.2227,
168
+ "pred_fc": 40,
169
+ "gold_fc": 42,
170
+ "pred_ps": 24,
171
+ "gold_ps": 10,
172
+ "pred_cc": 72,
173
+ "gold_cc": 72
174
+ },
175
+ {
176
+ "project": "omni-seaforth-ltc",
177
+ "panel_id": "1E3",
178
+ "bucket": "omni",
179
+ "cc_ok": true,
180
+ "fc_ok": false,
181
+ "ps_ok": true,
182
+ "groups_exact": false,
183
+ "self_consistent": false,
184
+ "multiset_jaccard": 0.7458,
185
+ "reward": 0.4992,
186
+ "pred_fc": 52,
187
+ "gold_fc": 51,
188
+ "pred_ps": 19,
189
+ "gold_ps": 19,
190
+ "pred_cc": 72,
191
+ "gold_cc": 72
192
+ },
193
+ {
194
+ "project": "omni-seaforth-ltc",
195
+ "panel_id": "1E4",
196
+ "bucket": "omni",
197
+ "cc_ok": true,
198
+ "fc_ok": false,
199
+ "ps_ok": false,
200
+ "groups_exact": false,
201
+ "self_consistent": false,
202
+ "multiset_jaccard": 0.8444,
203
+ "reward": 0.2189,
204
+ "pred_fc": 38,
205
+ "gold_fc": 45,
206
+ "pred_ps": 34,
207
+ "gold_ps": 26,
208
+ "pred_cc": 72,
209
+ "gold_cc": 72
210
+ },
211
+ {
212
+ "project": "omni-seaforth-ltc",
213
+ "panel_id": "1E5",
214
+ "bucket": "omni",
215
+ "cc_ok": true,
216
+ "fc_ok": true,
217
+ "ps_ok": false,
218
+ "groups_exact": true,
219
+ "self_consistent": false,
220
+ "multiset_jaccard": 1.0,
221
+ "reward": 0.6,
222
+ "pred_fc": 49,
223
+ "gold_fc": 49,
224
+ "pred_ps": 21,
225
+ "gold_ps": 22,
226
+ "pred_cc": 72,
227
+ "gold_cc": 72
228
+ },
229
+ {
230
+ "project": "omni-seaforth-ltc",
231
+ "panel_id": "1E6",
232
+ "bucket": "omni",
233
+ "cc_ok": true,
234
+ "fc_ok": false,
235
+ "ps_ok": false,
236
+ "groups_exact": false,
237
+ "self_consistent": false,
238
+ "multiset_jaccard": 0.8077,
239
+ "reward": 0.2115,
240
+ "pred_fc": 42,
241
+ "gold_fc": 52,
242
+ "pred_ps": 19,
243
+ "gold_ps": 20,
244
+ "pred_cc": 72,
245
+ "gold_cc": 72
246
+ },
247
+ {
248
+ "project": "omni-seaforth-ltc",
249
+ "panel_id": "1E7",
250
+ "bucket": "omni",
251
+ "cc_ok": true,
252
+ "fc_ok": false,
253
+ "ps_ok": true,
254
+ "groups_exact": false,
255
+ "self_consistent": false,
256
+ "multiset_jaccard": 0.7692,
257
+ "reward": 0.5038,
258
+ "pred_fc": 41,
259
+ "gold_fc": 51,
260
+ "pred_ps": 20,
261
+ "gold_ps": 20,
262
+ "pred_cc": 72,
263
+ "gold_cc": 72
264
+ },
265
+ {
266
+ "project": "omni-seaforth-ltc",
267
+ "panel_id": "1VC",
268
+ "bucket": "omni",
269
+ "cc_ok": true,
270
+ "fc_ok": false,
271
+ "ps_ok": false,
272
+ "groups_exact": false,
273
+ "self_consistent": false,
274
+ "multiset_jaccard": 0.8462,
275
+ "reward": 0.2192,
276
+ "pred_fc": 23,
277
+ "gold_fc": 25,
278
+ "pred_ps": 17,
279
+ "gold_ps": 16,
280
+ "pred_cc": 42,
281
+ "gold_cc": 42
282
+ },
283
+ {
284
+ "project": "omni-seaforth-ltc",
285
+ "panel_id": "1VE",
286
+ "bucket": "omni",
287
+ "cc_ok": true,
288
+ "fc_ok": true,
289
+ "ps_ok": false,
290
+ "groups_exact": false,
291
+ "self_consistent": false,
292
+ "multiset_jaccard": 0.9412,
293
+ "reward": 0.5882,
294
+ "pred_fc": 33,
295
+ "gold_fc": 33,
296
+ "pred_ps": 10,
297
+ "gold_ps": 9,
298
+ "pred_cc": 42,
299
+ "gold_cc": 42
300
+ },
301
+ {
302
+ "project": "omni-seaforth-ltc",
303
+ "panel_id": "1VW",
304
+ "bucket": "omni",
305
+ "cc_ok": true,
306
+ "fc_ok": false,
307
+ "ps_ok": false,
308
+ "groups_exact": false,
309
+ "self_consistent": false,
310
+ "multiset_jaccard": 0.7879,
311
+ "reward": 0.2076,
312
+ "pred_fc": 26,
313
+ "gold_fc": 33,
314
+ "pred_ps": 10,
315
+ "gold_ps": 9,
316
+ "pred_cc": 42,
317
+ "gold_cc": 42
318
+ },
319
+ {
320
+ "project": "omni-seaforth-ltc",
321
+ "panel_id": "1W1",
322
+ "bucket": "omni",
323
+ "cc_ok": true,
324
+ "fc_ok": false,
325
+ "ps_ok": false,
326
+ "groups_exact": false,
327
+ "self_consistent": false,
328
+ "multiset_jaccard": 0.7419,
329
+ "reward": 0.1984,
330
+ "pred_fc": 22,
331
+ "gold_fc": 29,
332
+ "pred_ps": 7,
333
+ "gold_ps": 5,
334
+ "pred_cc": 42,
335
+ "gold_cc": 42
336
+ },
337
+ {
338
+ "project": "omni-seaforth-ltc",
339
+ "panel_id": "1W2",
340
+ "bucket": "omni",
341
+ "cc_ok": true,
342
+ "fc_ok": false,
343
+ "ps_ok": true,
344
+ "groups_exact": false,
345
+ "self_consistent": false,
346
+ "multiset_jaccard": 0.7925,
347
+ "reward": 0.5085,
348
+ "pred_fc": 43,
349
+ "gold_fc": 52,
350
+ "pred_ps": 19,
351
+ "gold_ps": 19,
352
+ "pred_cc": 72,
353
+ "gold_cc": 72
354
+ },
355
+ {
356
+ "project": "omni-seaforth-ltc",
357
+ "panel_id": "1W3",
358
+ "bucket": "omni",
359
+ "cc_ok": true,
360
+ "fc_ok": false,
361
+ "ps_ok": false,
362
+ "groups_exact": false,
363
+ "self_consistent": false,
364
+ "multiset_jaccard": 0.7925,
365
+ "reward": 0.2085,
366
+ "pred_fc": 43,
367
+ "gold_fc": 52,
368
+ "pred_ps": 28,
369
+ "gold_ps": 18,
370
+ "pred_cc": 72,
371
+ "gold_cc": 72
372
+ },
373
+ {
374
+ "project": "omni-seaforth-ltc",
375
+ "panel_id": "1W4",
376
+ "bucket": "omni",
377
+ "cc_ok": true,
378
+ "fc_ok": false,
379
+ "ps_ok": false,
380
+ "groups_exact": false,
381
+ "self_consistent": false,
382
+ "multiset_jaccard": 0.9423,
383
+ "reward": 0.2385,
384
+ "pred_fc": 52,
385
+ "gold_fc": 49,
386
+ "pred_ps": 17,
387
+ "gold_ps": 22,
388
+ "pred_cc": 72,
389
+ "gold_cc": 72
390
+ },
391
+ {
392
+ "project": "omni-seaforth-ltc",
393
+ "panel_id": "1W5",
394
+ "bucket": "omni",
395
+ "cc_ok": true,
396
+ "fc_ok": false,
397
+ "ps_ok": false,
398
+ "groups_exact": false,
399
+ "self_consistent": false,
400
+ "multiset_jaccard": 0.8222,
401
+ "reward": 0.2144,
402
+ "pred_fc": 37,
403
+ "gold_fc": 45,
404
+ "pred_ps": 37,
405
+ "gold_ps": 26,
406
+ "pred_cc": 72,
407
+ "gold_cc": 72
408
+ },
409
+ {
410
+ "project": "omni-seaforth-ltc",
411
+ "panel_id": "1W6",
412
+ "bucket": "omni",
413
+ "cc_ok": true,
414
+ "fc_ok": false,
415
+ "ps_ok": false,
416
+ "groups_exact": false,
417
+ "self_consistent": false,
418
+ "multiset_jaccard": 0.9474,
419
+ "reward": 0.2395,
420
+ "pred_fc": 57,
421
+ "gold_fc": 54,
422
+ "pred_ps": 17,
423
+ "gold_ps": 16,
424
+ "pred_cc": 72,
425
+ "gold_cc": 72
426
+ },
427
+ {
428
+ "project": "omni-seaforth-ltc",
429
+ "panel_id": "1W7",
430
+ "bucket": "omni",
431
+ "cc_ok": true,
432
+ "fc_ok": false,
433
+ "ps_ok": true,
434
+ "groups_exact": false,
435
+ "self_consistent": false,
436
+ "multiset_jaccard": 0.9623,
437
+ "reward": 0.5425,
438
+ "pred_fc": 53,
439
+ "gold_fc": 51,
440
+ "pred_ps": 20,
441
+ "gold_ps": 20,
442
+ "pred_cc": 72,
443
+ "gold_cc": 72
444
+ },
445
+ {
446
+ "project": "omni-seaforth-ltc",
447
+ "panel_id": "2VC",
448
+ "bucket": "omni",
449
+ "cc_ok": true,
450
+ "fc_ok": false,
451
+ "ps_ok": false,
452
+ "groups_exact": false,
453
+ "self_consistent": false,
454
+ "multiset_jaccard": 0.8636,
455
+ "reward": 0.2227,
456
+ "pred_fc": 20,
457
+ "gold_fc": 21,
458
+ "pred_ps": 12,
459
+ "gold_ps": 21,
460
+ "pred_cc": 42,
461
+ "gold_cc": 42
462
+ },
463
+ {
464
+ "project": "omni-seaforth-ltc",
465
+ "panel_id": "2VE",
466
+ "bucket": "omni",
467
+ "cc_ok": true,
468
+ "fc_ok": false,
469
+ "ps_ok": true,
470
+ "groups_exact": false,
471
+ "self_consistent": false,
472
+ "multiset_jaccard": 0.8529,
473
+ "reward": 0.5206,
474
+ "pred_fc": 29,
475
+ "gold_fc": 34,
476
+ "pred_ps": 8,
477
+ "gold_ps": 8,
478
+ "pred_cc": 42,
479
+ "gold_cc": 42
480
+ },
481
+ {
482
+ "project": "omni-seaforth-ltc",
483
+ "panel_id": "2VW",
484
+ "bucket": "omni",
485
+ "cc_ok": true,
486
+ "fc_ok": false,
487
+ "ps_ok": true,
488
+ "groups_exact": false,
489
+ "self_consistent": false,
490
+ "multiset_jaccard": 0.9706,
491
+ "reward": 0.5441,
492
+ "pred_fc": 33,
493
+ "gold_fc": 34,
494
+ "pred_ps": 8,
495
+ "gold_ps": 8,
496
+ "pred_cc": 42,
497
+ "gold_cc": 42
498
+ },
499
+ {
500
+ "project": "omni-seaforth-ltc",
501
+ "panel_id": "DP-CA1",
502
+ "bucket": "omni",
503
+ "cc_ok": true,
504
+ "fc_ok": false,
505
+ "ps_ok": false,
506
+ "groups_exact": false,
507
+ "self_consistent": true,
508
+ "multiset_jaccard": 0.7857,
509
+ "reward": 0.2071,
510
+ "pred_fc": 12,
511
+ "gold_fc": 13,
512
+ "pred_ps": 18,
513
+ "gold_ps": 15,
514
+ "pred_cc": 54,
515
+ "gold_cc": 54
516
+ },
517
+ {
518
+ "project": "omni-seaforth-ltc",
519
+ "panel_id": "DP-DVA1",
520
+ "bucket": "omni",
521
+ "cc_ok": true,
522
+ "fc_ok": true,
523
+ "ps_ok": false,
524
+ "groups_exact": false,
525
+ "self_consistent": false,
526
+ "multiset_jaccard": 0.5238,
527
+ "reward": 0.5048,
528
+ "pred_fc": 15,
529
+ "gold_fc": 15,
530
+ "pred_ps": 20,
531
+ "gold_ps": 9,
532
+ "pred_cc": 54,
533
+ "gold_cc": 54
534
+ },
535
+ {
536
+ "project": "omni-seaforth-ltc",
537
+ "panel_id": "DP-DVB1",
538
+ "bucket": "omni",
539
+ "cc_ok": true,
540
+ "fc_ok": true,
541
+ "ps_ok": false,
542
+ "groups_exact": false,
543
+ "self_consistent": false,
544
+ "multiset_jaccard": 0.9286,
545
+ "reward": 0.5857,
546
+ "pred_fc": 13,
547
+ "gold_fc": 13,
548
+ "pred_ps": 20,
549
+ "gold_ps": 15,
550
+ "pred_cc": 54,
551
+ "gold_cc": 54
552
+ },
553
+ {
554
+ "project": "omni-seaforth-ltc",
555
+ "panel_id": "DP-DVB2",
556
+ "bucket": "omni",
557
+ "cc_ok": true,
558
+ "fc_ok": false,
559
+ "ps_ok": false,
560
+ "groups_exact": false,
561
+ "self_consistent": false,
562
+ "multiset_jaccard": 0.6,
563
+ "reward": 0.17,
564
+ "pred_fc": 9,
565
+ "gold_fc": 15,
566
+ "pred_ps": 16,
567
+ "gold_ps": 9,
568
+ "pred_cc": 54,
569
+ "gold_cc": 54
570
+ },
571
+ {
572
+ "project": "omni-seaforth-ltc",
573
+ "panel_id": "DP-DVB3",
574
+ "bucket": "omni",
575
+ "cc_ok": true,
576
+ "fc_ok": false,
577
+ "ps_ok": false,
578
+ "groups_exact": false,
579
+ "self_consistent": false,
580
+ "multiset_jaccard": 0.75,
581
+ "reward": 0.2,
582
+ "pred_fc": 16,
583
+ "gold_fc": 12,
584
+ "pred_ps": 27,
585
+ "gold_ps": 18,
586
+ "pred_cc": 54,
587
+ "gold_cc": 54
588
+ },
589
+ {
590
+ "project": "omni-seaforth-ltc",
591
+ "panel_id": "DP-DVB4",
592
+ "bucket": "omni",
593
+ "cc_ok": true,
594
+ "fc_ok": false,
595
+ "ps_ok": false,
596
+ "groups_exact": false,
597
+ "self_consistent": false,
598
+ "multiset_jaccard": 0.8667,
599
+ "reward": 0.2233,
600
+ "pred_fc": 15,
601
+ "gold_fc": 13,
602
+ "pred_ps": 13,
603
+ "gold_ps": 15,
604
+ "pred_cc": 54,
605
+ "gold_cc": 54
606
+ },
607
+ {
608
+ "project": "omni-seaforth-ltc",
609
+ "panel_id": "DP-VB",
610
+ "bucket": "omni",
611
+ "cc_ok": true,
612
+ "fc_ok": true,
613
+ "ps_ok": true,
614
+ "groups_exact": true,
615
+ "self_consistent": true,
616
+ "multiset_jaccard": 1.0,
617
+ "reward": 1.0,
618
+ "pred_fc": 9,
619
+ "gold_fc": 9,
620
+ "pred_ps": 15,
621
+ "gold_ps": 15,
622
+ "pred_cc": 42,
623
+ "gold_cc": 42
624
+ },
625
+ {
626
+ "project": "spring-valley-sow-nursery-barn-pn-4223-001",
627
+ "panel_id": "B(208)",
628
+ "bucket": "spring_valley",
629
+ "cc_ok": true,
630
+ "fc_ok": false,
631
+ "ps_ok": false,
632
+ "groups_exact": false,
633
+ "self_consistent": false,
634
+ "multiset_jaccard": 0.3377,
635
+ "reward": 0.1175,
636
+ "pred_fc": 67,
637
+ "gold_fc": 36,
638
+ "pred_ps": 36,
639
+ "gold_ps": 15,
640
+ "pred_cc": 72,
641
+ "gold_cc": 72
642
+ },
643
+ {
644
+ "project": "spring-valley-sow-nursery-barn-pn-4223-001",
645
+ "panel_id": "Panel 1",
646
+ "bucket": "spring_valley",
647
+ "cc_ok": true,
648
+ "fc_ok": false,
649
+ "ps_ok": false,
650
+ "groups_exact": false,
651
+ "self_consistent": false,
652
+ "multiset_jaccard": 0.3377,
653
+ "reward": 0.1175,
654
+ "pred_fc": 67,
655
+ "gold_fc": 36,
656
+ "pred_ps": 36,
657
+ "gold_ps": 15,
658
+ "pred_cc": 72,
659
+ "gold_cc": 72
660
+ }
661
+ ]
662
+ }
sld_26.0/panel_counter_v19_rsft/eval/v19b_selection.json ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "panel_counter_v19b_cand_115": {
3
+ "step": 115,
4
+ "checkpoint": "/workspace/training/checkpoints/panel_counter_v19_grpo/v0-20260703-182148/checkpoint-115",
5
+ "mean_reward": 0.7403,
6
+ "scores": [
7
+ 0.23412698412698413,
8
+ 0.9866666666666666,
9
+ 0.9999999999999999
10
+ ]
11
+ },
12
+ "panel_counter_v19b_cand_120": {
13
+ "step": 120,
14
+ "checkpoint": "/workspace/training/checkpoints/panel_counter_v19_grpo/v0-20260703-182148/checkpoint-120",
15
+ "mean_reward": 0.7392,
16
+ "scores": [
17
+ 0.23095238095238096,
18
+ 0.9866666666666666,
19
+ 0.9999999999999999
20
+ ]
21
+ },
22
+ "panel_counter_v19b_cand_125": {
23
+ "step": 125,
24
+ "checkpoint": "/workspace/training/checkpoints/panel_counter_v19_grpo/v0-20260703-182148/checkpoint-125",
25
+ "mean_reward": 0.7403,
26
+ "scores": [
27
+ 0.23412698412698413,
28
+ 0.9866666666666666,
29
+ 0.9999999999999999
30
+ ]
31
+ }
32
+ }
sld_26.0/panel_counter_v19_rsft/eval/vllm_serve.log ADDED
The diff for this file is too large to render. See raw diff
 
sld_26.0/panel_counter_v19_rsft/training_logging_full.jsonl ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {"loss": 0.61602575, "grad_norm": 1.85162568, "learning_rate": 1.667e-05, "token_acc": 0.89180798, "epoch": 0.2962963, "global_step/max_steps": "1/60", "elapsed_time": "27s", "remaining_time": "26m 10s", "memory(GiB)": 28.87, "train_speed(s/it)": 26.615115}
2
+ {"loss": 0.53196263, "grad_norm": 1.58778095, "learning_rate": 3.333e-05, "token_acc": 0.89859402, "epoch": 0.59259259, "global_step/max_steps": "2/60", "elapsed_time": "54s", "remaining_time": "25m 59s", "memory(GiB)": 31.08, "train_speed(s/it)": 26.87452}
3
+ {"loss": 0.3858228, "grad_norm": 0.91004008, "learning_rate": 4.996e-05, "token_acc": 0.9030533, "epoch": 1.0, "global_step/max_steps": "4/60", "elapsed_time": "1m 28s", "remaining_time": "20m 29s", "memory(GiB)": 31.08, "train_speed(s/it)": 21.943331}
4
+ {"eval_loss": 0.22256579, "eval_runtime": 2.8712, "eval_samples_per_second": 1.045, "eval_steps_per_second": 1.045, "eval_token_acc": 0.94058296, "epoch": 1.0, "global_step/max_steps": "4/60", "elapsed_time": "1m 31s", "remaining_time": "21m 9s", "memory(GiB)": 31.08, "train_speed(s/it)": 22.661474}
5
+ {"loss": 0.22709587, "grad_norm": 0.47737247, "learning_rate": 4.966e-05, "token_acc": 0.93690029, "epoch": 1.59259259, "global_step/max_steps": "6/60", "elapsed_time": "2m 25s", "remaining_time": "21m 43s", "memory(GiB)": 31.08, "train_speed(s/it)": 24.135171}
6
+ {"loss": 0.14378929, "grad_norm": 0.4572224, "learning_rate": 4.906e-05, "token_acc": 0.95678607, "epoch": 2.0, "global_step/max_steps": "8/60", "elapsed_time": "2m 60s", "remaining_time": "19m 28s", "memory(GiB)": 31.08, "train_speed(s/it)": 22.457967}
7
+ {"eval_loss": 0.12477306, "eval_runtime": 2.9109, "eval_samples_per_second": 1.031, "eval_steps_per_second": 1.031, "eval_token_acc": 0.96188341, "epoch": 2.0, "global_step/max_steps": "8/60", "elapsed_time": "3m 3s", "remaining_time": "19m 47s", "memory(GiB)": 31.08, "train_speed(s/it)": 22.822008}
8
+ {"loss": 0.10015421, "grad_norm": 0.36381683, "learning_rate": 4.816e-05, "token_acc": 0.97019325, "epoch": 2.59259259, "global_step/max_steps": "10/60", "elapsed_time": "3m 56s", "remaining_time": "19m 41s", "memory(GiB)": 31.08, "train_speed(s/it)": 23.622184}
9
+ {"loss": 0.11009409, "grad_norm": 0.77840304, "learning_rate": 4.699e-05, "token_acc": 0.96423638, "epoch": 3.0, "global_step/max_steps": "12/60", "elapsed_time": "4m 32s", "remaining_time": "18m 8s", "memory(GiB)": 31.08, "train_speed(s/it)": 22.670276}
10
+ {"eval_loss": 0.09810275, "eval_runtime": 2.8914, "eval_samples_per_second": 1.038, "eval_steps_per_second": 1.038, "eval_token_acc": 0.97141256, "epoch": 3.0, "global_step/max_steps": "12/60", "elapsed_time": "4m 35s", "remaining_time": "18m 20s", "memory(GiB)": 31.08, "train_speed(s/it)": 22.911346}
11
+ {"loss": 0.0627116, "grad_norm": 0.28772321, "learning_rate": 4.554e-05, "token_acc": 0.97947244, "epoch": 3.59259259, "global_step/max_steps": "14/60", "elapsed_time": "5m 29s", "remaining_time": "18m 3s", "memory(GiB)": 31.08, "train_speed(s/it)": 23.535489}
12
+ {"loss": 0.0672082, "grad_norm": 0.49678728, "learning_rate": 4.385e-05, "token_acc": 0.97706963, "epoch": 4.0, "global_step/max_steps": "16/60", "elapsed_time": "6m 4s", "remaining_time": "16m 41s", "memory(GiB)": 31.08, "train_speed(s/it)": 22.760119}
13
+ {"eval_loss": 0.09332478, "eval_runtime": 2.8818, "eval_samples_per_second": 1.041, "eval_steps_per_second": 1.041, "eval_token_acc": 0.97309417, "epoch": 4.0, "global_step/max_steps": "16/60", "elapsed_time": "6m 7s", "remaining_time": "16m 49s", "memory(GiB)": 31.08, "train_speed(s/it)": 22.940319}
14
+ {"loss": 0.03888388, "grad_norm": 0.21333447, "learning_rate": 4.193e-05, "token_acc": 0.98892405, "epoch": 4.59259259, "global_step/max_steps": "18/60", "elapsed_time": "6m 59s", "remaining_time": "16m 18s", "memory(GiB)": 31.08, "train_speed(s/it)": 23.275892}
15
+ {"loss": 0.04424371, "grad_norm": 0.57744259, "learning_rate": 3.981e-05, "token_acc": 0.98538482, "epoch": 5.0, "global_step/max_steps": "20/60", "elapsed_time": "7m 36s", "remaining_time": "15m 11s", "memory(GiB)": 31.08, "train_speed(s/it)": 22.786175}
16
+ {"eval_loss": 0.09925357, "eval_runtime": 2.8917, "eval_samples_per_second": 1.037, "eval_steps_per_second": 1.037, "eval_token_acc": 0.97477578, "epoch": 5.0, "global_step/max_steps": "20/60", "elapsed_time": "7m 39s", "remaining_time": "15m 17s", "memory(GiB)": 31.08, "train_speed(s/it)": 22.93083}
17
+ {"loss": 0.03528717, "grad_norm": 0.26964709, "learning_rate": 3.75e-05, "token_acc": 0.9895858, "epoch": 5.59259259, "global_step/max_steps": "22/60", "elapsed_time": "8m 32s", "remaining_time": "14m 44s", "memory(GiB)": 31.08, "train_speed(s/it)": 23.265631}
18
+ {"loss": 0.01588555, "grad_norm": 0.42894074, "learning_rate": 3.504e-05, "token_acc": 0.9959856, "epoch": 6.0, "global_step/max_steps": "24/60", "elapsed_time": "9m 8s", "remaining_time": "13m 42s", "memory(GiB)": 31.08, "train_speed(s/it)": 22.835372}
19
+ {"eval_loss": 0.12231197, "eval_runtime": 2.8791, "eval_samples_per_second": 1.042, "eval_steps_per_second": 1.042, "eval_token_acc": 0.97197309, "epoch": 6.0, "global_step/max_steps": "24/60", "elapsed_time": "9m 11s", "remaining_time": "13m 46s", "memory(GiB)": 31.08, "train_speed(s/it)": 22.955397}
20
+ {"loss": 0.01676361, "grad_norm": 0.21449812, "learning_rate": 3.246e-05, "token_acc": 0.99478426, "epoch": 6.59259259, "global_step/max_steps": "26/60", "elapsed_time": "10m 3s", "remaining_time": "13m 9s", "memory(GiB)": 31.08, "train_speed(s/it)": 23.208665}
21
+ {"loss": 0.00833158, "grad_norm": 0.34633994, "learning_rate": 2.979e-05, "token_acc": 0.99765128, "epoch": 7.0, "global_step/max_steps": "28/60", "elapsed_time": "10m 39s", "remaining_time": "12m 11s", "memory(GiB)": 31.08, "train_speed(s/it)": 22.836013}
22
+ {"eval_loss": 0.14736782, "eval_runtime": 2.8925, "eval_samples_per_second": 1.037, "eval_steps_per_second": 1.037, "eval_token_acc": 0.97141256, "epoch": 7.0, "global_step/max_steps": "28/60", "elapsed_time": "10m 42s", "remaining_time": "12m 14s", "memory(GiB)": 31.08, "train_speed(s/it)": 22.939367}
23
+ {"train_runtime": 644.4696, "train_samples_per_second": 0.628, "train_steps_per_second": 0.093, "total_flos": 1.996718609150496e+16, "train_loss": 0.13073327, "epoch": 7.0, "global_step/max_steps": "28/60", "elapsed_time": "10m 44s", "remaining_time": "12m 17s", "memory(GiB)": 31.08, "train_speed(s/it)": 23.016628}
24
+ {"model_parameter_info": "PeftModelForCausalLM: 9582.9271M Params (173.1133M Trainable [1.8065%]), 0.0001M Buffers.", "last_model_checkpoint": "/workspace/training/checkpoints/panel_counter_v19_rsft/v0-20260703-172808/checkpoint-28", "best_model_checkpoint": "/workspace/training/checkpoints/panel_counter_v19_rsft/v0-20260703-172808/checkpoint-16", "best_metric": 0.09332478, "global_step": 28, "log_history": [{"loss": 0.6160257458686829, "grad_norm": 1.851625680923462, "learning_rate": 1.6666666666666667e-05, "token_acc": 0.8918079770718892, "epoch": 0.2962962962962963, "step": 1}, {"loss": 0.5319626331329346, "grad_norm": 1.5877809524536133, "learning_rate": 3.3333333333333335e-05, "token_acc": 0.8985940246045694, "epoch": 0.5925925925925926, "step": 2}, {"loss": 0.3858228027820587, "grad_norm": 0.9100400805473328, "learning_rate": 4.996203791083291e-05, "token_acc": 0.9030533034328101, "epoch": 1.0, "step": 4}, {"eval_loss": 0.22256578505039215, "eval_runtime": 2.8712, "eval_samples_per_second": 1.045, "eval_steps_per_second": 1.045, "eval_token_acc": 0.9405829596412556, "epoch": 1.0, "step": 4}, {"loss": 0.22709587216377258, "grad_norm": 0.4773724675178528, "learning_rate": 4.965903258506806e-05, "token_acc": 0.9369002920317063, "epoch": 1.5925925925925926, "step": 6}, {"loss": 0.14378929138183594, "grad_norm": 0.4572224020957947, "learning_rate": 4.905670000773126e-05, "token_acc": 0.95678606638186, "epoch": 2.0, "step": 8}, {"eval_loss": 0.1247730553150177, "eval_runtime": 2.9109, "eval_samples_per_second": 1.031, "eval_steps_per_second": 1.031, "eval_token_acc": 0.9618834080717489, "epoch": 2.0, "step": 8}, {"loss": 0.10015420615673065, "grad_norm": 0.3638168275356293, "learning_rate": 4.8162351680370044e-05, "token_acc": 0.9701932525384868, "epoch": 2.5925925925925926, "step": 10}, {"loss": 0.1100940853357315, "grad_norm": 0.7784030437469482, "learning_rate": 4.698684378016222e-05, "token_acc": 0.9642363775901766, "epoch": 3.0, "step": 12}, {"eval_loss": 0.09810274839401245, "eval_runtime": 2.8914, "eval_samples_per_second": 1.038, "eval_steps_per_second": 1.038, "eval_token_acc": 0.9714125560538116, "epoch": 3.0, "step": 12}, {"loss": 0.06271159648895264, "grad_norm": 0.28772321343421936, "learning_rate": 4.554444538063113e-05, "token_acc": 0.9794724417530535, "epoch": 3.5925925925925926, "step": 14}, {"loss": 0.0672082006931305, "grad_norm": 0.49678727984428406, "learning_rate": 4.385266524442241e-05, "token_acc": 0.9770696341257799, "epoch": 4.0, "step": 16}, {"eval_loss": 0.09332478046417236, "eval_runtime": 2.8818, "eval_samples_per_second": 1.041, "eval_steps_per_second": 1.041, "eval_token_acc": 0.9730941704035875, "epoch": 4.0, "step": 16}, {"loss": 0.03888387978076935, "grad_norm": 0.21333447098731995, "learning_rate": 4.193203929064353e-05, "token_acc": 0.9889240506329114, "epoch": 4.592592592592593, "step": 18}, {"loss": 0.0442437082529068, "grad_norm": 0.5774425864219666, "learning_rate": 3.9805881316624506e-05, "token_acc": 0.9853848216680076, "epoch": 5.0, "step": 20}, {"eval_loss": 0.0992535725235939, "eval_runtime": 2.8917, "eval_samples_per_second": 1.037, "eval_steps_per_second": 1.037, "eval_token_acc": 0.9747757847533632, "epoch": 5.0, "step": 20}, {"loss": 0.035287171602249146, "grad_norm": 0.26964709162712097, "learning_rate": 3.7500000000000003e-05, "token_acc": 0.989585798816568, "epoch": 5.592592592592593, "step": 22}, {"loss": 0.015885546803474426, "grad_norm": 0.4289407432079315, "learning_rate": 3.504238561632424e-05, "token_acc": 0.9959856035437431, "epoch": 6.0, "step": 24}, {"eval_loss": 0.12231197208166122, "eval_runtime": 2.8791, "eval_samples_per_second": 1.042, "eval_steps_per_second": 1.042, "eval_token_acc": 0.9719730941704036, "epoch": 6.0, "step": 24}, {"loss": 0.016763607040047646, "grad_norm": 0.2144981175661087, "learning_rate": 3.246287027504237e-05, "token_acc": 0.9947842579421526, "epoch": 6.592592592592593, "step": 26}, {"loss": 0.008331576362252235, "grad_norm": 0.3463399410247803, "learning_rate": 2.9792765798093465e-05, "token_acc": 0.9976512848853274, "epoch": 7.0, "step": 28}, {"eval_loss": 0.14736782014369965, "eval_runtime": 2.8925, "eval_samples_per_second": 1.037, "eval_steps_per_second": 1.037, "eval_token_acc": 0.9714125560538116, "epoch": 7.0, "step": 28}, {"train_runtime": 644.4696, "train_samples_per_second": 0.628, "train_steps_per_second": 0.093, "total_flos": 1.996718609150496e+16, "train_loss": 0.13073326673890864, "epoch": 7.0, "step": 28}], "memory": 31.08203125}