Upload Gemma 4 31B token-level span-max probe (layer 15/60, ex_AUC=0.692, tok_AUC=0.730)

README.md

@@ -0,0 +1,101 @@
+---
+library_name: numpy
+base_model: google/gemma-4-31B-it
+tags:
+- gemma
+- gemma-4-31b
+- linear-probe
+- token-level
+- span-max
+- vulnerability-detection
+---
+
+# GemmaForge Token-Level Probe — Gemma 4 31B (span-max)
+
+Frozen `google/gemma-4-31B-it` (60 decoder layers, hidden_size 5376) with a
+**linear span-max probe** on per-token hidden states of decoder layer
+**15**. Trained with the Obeso, Arditi et al. 2025
+(arXiv 2509.03531 §3) span-max loss on `data/dataset.jsonl` (N=1374,
+687 pos / 687 neg, token-level char-range labels propagated from SVEN diffs).
+
+Companion models:
+- [`peaktwilight/gemmaforge-gemma4-probe`](https://huggingface.co/peaktwilight/gemmaforge-gemma4-probe) — sample-level (last-token) probe on Gemma 4 E2B
+- [`peaktwilight/gemmaforge-31b-probe`](https://huggingface.co/peaktwilight/gemmaforge-31b-probe) — sample-level (last-token) probe on Gemma 4 31B
+
+## Files
+
+- `probe_spanmax_31b.npz` — `(w, b, layer)`; `sigmoid(w @ hidden[layer + 1][0, t, :] + b)` is the per-token risk
+- `probe_spanmax_31b_card.json` — per-layer token/example AUC, training config
+- `token_probs_31b.npz` — per-row token probabilities (`probs_row_NNNN`) on `data/dataset.jsonl`
+- `token_offsets_31b.npz` — per-row `(T, 2)` char offsets (`offsets_row_NNNN`)
+- `spans.json` — positive `(example_id, tok_start, tok_end)` triples for the trainer's span-max pool
+- `token_report_31b.md` / `token_report_31b.json` — full three-level eval report
+
+## Training setup
+
+| | |
+|---|---|
+| Base model | `google/gemma-4-31B-it` |
+| Decoder layers / hidden | 60 / 5376 |
+| Layers probed | [15, 30, 45, 59] (25/50/75/100% depth) |
+| Winning layer | 15 |
+| Loss | span-max (alpha=10, omega 0→1 linear) |
+| Optimizer / epochs / batch | AdamW (lr=1e-3) / 30 / 8 examples |
+| Activation dtype on disk | float16 |
+
+### Layer sweep (winner = best example-level AUC)
+
+| Layer | tok_AUC | ex_AUC |
+|---:|---:|---:|
+| 15 | 0.730 | 0.692  **<-- winner** |
+| 30 | 0.721 | 0.657 |
+| 45 | 0.752 | 0.647 |
+| 59 | 0.683 | 0.676 |
+
+## Eval headline (`data/dataset.jsonl`, N=1374, pos=687)
+
+| Split | `all` AUC | `proximal_all` AUC | `span_max` AUC | `dilated_span_max` AUC |
+|---|---:|---:|---:|---:|
+| `random_stratified` | 0.879 | 0.729 | 0.669 | 0.565 |
+| `group_repo` | 0.812 | 0.714 | 0.623 | 0.495 |
+| `heldout_cwe::CWE-089` | 0.960 | 0.713 | 0.944 | 0.941 |
+| `heldout_cwe::CWE-125` | 0.813 | 0.714 | 0.512 | 0.332 |
+| `heldout_cwe::CWE-078` | 0.881 | 0.748 | 0.724 | 0.686 |
+| `heldout_cwe::CWE-476` | 0.794 | 0.706 | 0.497 | 0.285 |
+| `heldout_cwe::CWE-079` | 0.796 | 0.677 | 0.492 | 0.299 |
+| `heldout_lang::test=c` | 0.791 | 0.697 | 0.497 | 0.292 |
+| `heldout_lang::test=cpp` | 0.778 | 0.696 | 0.494 | 0.307 |
+| `heldout_lang::test=python` | 0.896 | 0.711 | 0.824 | 0.781 |
+
+Notes:
+- `all` measures the probe on every token (streaming-UI view); best for per-token highlighting.
+- `span_max` collapses to one decision per example — directly comparable to sample-level probes.
+- `span` is NaN on this corpus: SVEN has no sanitizer-annotated negatives, so the span level is single-class. The protocol synthesises a whole-file negative span for label=0 examples inside `span_max` only.
+
+## Reproduce inference
+
+```python
+from huggingface_hub import hf_hub_download
+import numpy as np
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+probe = np.load(hf_hub_download("mmtf/gemmaforge-31b-token-probe", "probe_spanmax_31b.npz"))
+w, b, layer = probe["w"], float(probe["b"]), int(probe["layer"])
+
+model = AutoModelForCausalLM.from_pretrained(
+    "google/gemma-4-31B-it", torch_dtype=torch.bfloat16,
+    device_map="auto", attn_implementation="eager",
+)
+tok = AutoTokenizer.from_pretrained("google/gemma-4-31B-it")
+
+ids = tok("def vuln(x): return os.system(x)", return_tensors="pt").input_ids.to(model.device)
+with torch.inference_mode():
+    out = model(ids, output_hidden_states=True, use_cache=False)
+h = out.hidden_states[layer + 1][0].float().cpu().numpy()
+per_token_risk = 1.0 / (1.0 + np.exp(-(h @ w + b)))
+```
+
+## Pipeline
+
+Full training + eval pipeline: <https://github.com/peaktwilight/gemmaforge>

probe_spanmax_31b.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6969587e79e7d950bcd0eca9e255df8c86405db19075c865fc02325a3b85895d
+size 20642

probe_spanmax_31b_card.json

@@ -0,0 +1,51 @@
+{
+  "model_id": "google/gemma-4-31B-it",
+  "loss": "span-max (Obeso, Arditi et al. 2025, arXiv 2509.03531 \u00a73)",
+  "alpha": 10.0,
+  "omega_schedule": "linear 0->1 over all training steps",
+  "epochs": 30,
+  "lr": 0.001,
+  "batch_examples": 8,
+  "label_window": 0,
+  "best_layer": 15,
+  "best_token_auc": 0.7302596080868795,
+  "best_example_auc": 0.6923811544991512,
+  "num_hidden_layers": 60,
+  "hidden_size": 5376,
+  "layer_candidates": [
+    15,
+    30,
+    45,
+    59
+  ],
+  "all_layers": [
+    {
+      "layer": 15,
+      "token_auc": 0.7302596080868795,
+      "example_auc": 0.6923811544991512,
+      "n_train_examples": 1236,
+      "n_eval_examples": 138
+    },
+    {
+      "layer": 30,
+      "token_auc": 0.7208410982876844,
+      "example_auc": 0.6571519524617997,
+      "n_train_examples": 1236,
+      "n_eval_examples": 138
+    },
+    {
+      "layer": 45,
+      "token_auc": 0.7521584403948716,
+      "example_auc": 0.64696519524618,
+      "n_train_examples": 1236,
+      "n_eval_examples": 138
+    },
+    {
+      "layer": 59,
+      "token_auc": 0.6833787970168794,
+      "example_auc": 0.676358234295416,
+      "n_train_examples": 1236,
+      "n_eval_examples": 138
+    }
+  ]
+}

spans.json

@@ -0,0 +1 @@
+[[0, 99, 99], [0, 100, 102], [0, 104, 104], [0, 105, 106], [2, 379, 395], [3, 598, 598], [3, 609, 609], [3, 611, 611], [3, 626, 626], [3, 628, 628], [3, 644, 644], [3, 646, 646], [3, 690, 690], [3, 701, 701], [3, 703, 703], [4, 107, 107], [4, 115, 115], [5, 53, 55], [5, 63, 73], [5, 76, 76], [5, 135, 137], [5, 143, 154], [5, 157, 157], [6, 45, 46], [6, 61, 61], [6, 61, 61], [6, 62, 62], [6, 64, 64], [6, 65, 65], [6, 65, 65], [6, 71, 71], [8, 154, 154], [8, 155, 155], [8, 162, 162], [8, 163, 164], [8, 276, 285], [8, 289, 289], [8, 290, 290], [8, 297, 299], [8, 918, 924], [8, 925, 925], [8, 929, 929], [8, 930, 930], [8, 935, 937], [8, 969, 970], [8, 977, 985], [8, 989, 996], [8, 1000, 1000], [8, 1001, 1003], [8, 1003, 1003], [8, 1008, 1010], [9, 356, 363], [13, 24, 36], [16, 67, 68], [17, 125, 126], [17, 128, 130], [17, 162, 162], [17, 272, 273], [17, 275, 276], [17, 286, 286], [17, 287, 291], [17, 292, 296], [20, 61, 82], [20, 84, 84], [21, 61, 63], [21, 67, 67], [24, 130, 154], [24, 202, 227], [30, 35, 35], [30, 61, 66], [30, 68, 68], [36, 76, 76], [36, 83, 84], [39, 343, 344], [39, 344, 345], [40, 12, 17], [40, 20, 20], [40, 20, 20], [40, 20, 20], [40, 21, 22], [44, 97, 98], [44, 99, 102], [44, 112, 113], [44, 119, 121], [45, 25, 25], [45, 29, 29], [45, 32, 33], [45, 33, 33], [45, 33, 35], [45, 39, 41], [45, 44, 44], [45, 49, 51], [45, 54, 61], [45, 63, 64], [45, 68, 68], [56, 10, 10], [56, 55, 56], [56, 56, 56], [56, 57, 57], [56, 58, 58], [56, 59, 60], [56, 61, 61], [61, 147, 148], [61, 160, 161], [65, 66, 66], [66, 214, 234], [67, 101, 103], [67, 112, 112], [67, 148, 150], [67, 155, 157], [67, 162, 162], [67, 163, 163], [67, 169, 169], [71, 84, 85], [71, 85, 85], [71, 96, 96], [71, 97, 98], [71, 100, 100], [71, 103, 103], [71, 109, 112], [71, 251, 251], [71, 252, 252], [71, 258, 258], [71, 260, 261], [71, 270, 272], [72, 420, 458], [73, 279, 284], [79, 34, 36], [79, 39, 41], [79, 52, 53], [79, 55, 55], [79, 55, 55], [79, 55, 55], [79, 61, 67], [83, 99, 156], [83, 170, 170], [84, 74, 75], [84, 75, 76], [84, 82, 82], [85, 20, 21], [85, 23, 23], [86, 59, 75], [86, 80, 84], [86, 94, 98], [86, 130, 130], [86, 149, 157], [86, 160, 161], [86, 164, 171], [86, 173, 173], [86, 174, 174], [86, 175, 175], [86, 176, 176], [86, 180, 180], [86, 180, 180], [86, 181, 181], [86, 183, 183], [86, 188, 190], [86, 192, 192], [86, 194, 194], [86, 194, 196], [86, 196, 198], [86, 198, 198], [86, 198, 198], [86, 199, 200], [86, 217, 224], [86, 224, 226], [86, 227, 227], [86, 243, 248], [86, 249, 250], [86, 251, 252], [86, 253, 254], [86, 256, 256], [86, 259, 260], [86, 262, 263], [86, 274, 274], [86, 283, 283], [86, 285, 286], [86, 291, 297], [86, 298, 344], [86, 347, 352], [86, 372, 372], [86, 377, 377], [86, 384, 384], [86, 388, 390], [86, 391, 399], [86, 407, 407], [87, 411, 411], [87, 555, 555], [87, 659, 659], [91, 64, 81], [93, 10, 10], [93, 11, 13], [93, 14, 14], [93, 16, 16], [93, 24, 24], [94, 22, 23], [94, 25, 26], [94, 77, 77], [94, 77, 77], [94, 79, 79], [94, 79, 79], [94, 81, 81], [94, 82, 82], [94, 86, 86], [94, 93, 94], [96, 82, 83], [99, 10, 10], [99, 18, 18], [99, 19, 19], [99, 21, 21], [99, 21, 23], [99, 23, 23], [99, 23, 24], [99, 25, 25], [103, 128, 150], [104, 229, 260], [104, 755, 755], [104, 828, 828], [109, 200, 222], [109, 352, 354], [109, 355, 356], [109, 357, 357], [109, 357, 358], [109, 359, 359], [109, 360, 363], [109, 363, 363], [109, 365, 365], [109, 366, 366], [109, 368, 368], [109, 369, 373], [109, 375, 375], [109, 378, 378], [109, 380, 380], [109, 380, 380], [109, 382, 382], [109, 383, 383], [109, 384, 385], [109, 387, 387], [109, 388, 389], [109, 391, 391], [109, 394, 395], [109, 396, 397], [109, 397, 397], [109, 397, 398], [111, 549, 551], [111, 551, 551], [111, 609, 611], [111, 611, 611], [111, 684, 686], [111, 686, 686], [111, 744, 746], [111, 746, 746], [111, 824, 826], [111, 826, 826], [111, 916, 918], [111, 918, 918], [111, 990, 992], [111, 992, 992], [112, 11, 14], [112, 18, 22], [115, 329, 329], [115, 344, 344], [115, 345, 345], [117, 30, 34], [117, 36, 36], [117, 37, 37], [117, 41, 42], [118, 860, 881], [119, 32, 34], [119, 47, 50], [120, 75, 89], [120, 124, 138], [121, 51, 57], [121, 70, 79], [122, 402, 403], [123, 62, 62], [123, 69, 70], [125, 78, 97], [125, 120, 120], [125, 120, 120], [125, 138, 146], [129, 19, 21], [129, 23, 28], [129, 35, 37], [133, 25, 25], [133, 29, 29], [133, 32, 33], [133, 33, 33], [133, 33, 35], [133, 39, 41], [133, 43, 46], [133, 49, 53], [133, 53, 53], [133, 54, 56], [133, 56, 56], [133, 56, 60], [133, 60, 61], [133, 66, 66], [134, 154, 154], [134, 155, 155], [134, 164, 165], [134, 166, 166], [134, 167, 173], [134, 174, 174], [134, 181, 181], [134, 183, 183], [134, 193, 196], [134, 197, 197], [134, 200, 200], [134, 200, 204], [134, 205, 213], [134, 214, 220], [134, 221, 223], [134, 224, 225], [136, 63, 79], [140, 96, 96], [140, 97, 98], [140, 102, 104], [141, 55, 56], [141, 58, 58], [142, 92, 95], [142, 98, 101], [142, 104, 107], [142, 124, 124], [143, 112, 121], [146, 154, 176], [148, 124, 128], [148, 443, 450], [151, 111, 112], [151, 157, 158], [151, 171, 172], [151, 218, 219], [151, 262, 264], [151, 325, 326], [151, 353, 363], [151, 365, 370], [151, 372, 372], [151, 377, 384], [151, 385, 386], [151, 387, 387], [151, 388, 388], [151, 388, 388], [151, 389, 389], [151, 390, 390], [151, 390, 391], [151, 391, 392], [151, 393, 393], [151, 394, 394], [151, 501, 513], [153, 31, 31], [153, 31, 31], [156, 63, 63], [156, 65, 65], [156, 65, 66], [156, 66, 66], [156, 76, 77], [156, 83, 91], [160, 92, 94], [160, 113, 118], [160, 121, 121], [160, 123, 123], [160, 124, 124], [160, 127, 127], [160, 127, 127], [160, 129, 129], [160, 131, 131], [160, 147, 153], [164, 51, 51], [167, 37, 38], [167, 38, 39], [167, 43, 43], [170, 274, 284], [173, 316, 318], [173, 361, 363], [173, 460, 462], [173, 485, 487], [173, 522, 524], [173, 546, 548], [174, 70, 72], [174, 73, 74], [174, 75, 76], [176, 174, 189], [176, 628, 643], [182, 75, 77], [182, 85, 86], [182, 394, 396], [182, 399, 401], [182, 409, 411], [182, 413, 414], [182, 419, 426], [183, 168, 168], [183, 171, 171], [185, 46, 47], [185, 58, 58], [185, 60, 61], [185, 87, 88], [185, 99, 99], [185, 101, 102], [187, 465, 466], [187, 469, 471], [190, 590, 601], [192, 1003, 1023], [193, 454, 494], [197, 312, 319], [197, 329, 335], [197, 482, 488], [197, 643, 671], [197, 719, 726], [197, 746, 752], [201, 465, 466], [201, 467, 467], [201, 541, 554], [201, 627, 634], [201, 637, 666], [201, 668, 669], [201, 670, 670], [204, 119, 119], [204, 119, 121], [204, 121, 129], [204, 131, 132], [204, 146, 149], [204, 152, 152], [207, 43, 58], [209, 40, 49], [213, 19, 19], [213, 21, 21], [213, 23, 23], [213, 25, 25], [213, 26, 26], [213, 27, 30], [213, 38, 38], [213, 56, 56], [214, 505, 513], [217, 69, 71], [217, 72, 72], [218, 82, 83], [218, 87, 91], [218, 108, 108], [218, 108, 108], [218, 109, 110], [218, 110, 110], [218, 117, 117], [218, 117, 117], [218, 118, 118], [220, 642, 654], [222, 40, 40], [222, 42, 43], [227, 359, 360], [227, 361, 362], [227, 363, 367], [227, 375, 376], [227, 377, 378], [227, 379, 381], [230, 46, 47], [230, 49, 49], [232, 207, 207], [232, 220, 220], [232, 229, 230], [232, 232, 232], [232, 236, 236], [232, 364, 365], [232, 367, 367], [232, 412, 414], [232, 415, 415], [232, 463, 464], [232, 466, 466], [232, 487, 488], [232, 490, 490], [232, 528, 529], [232, 531, 531], [233, 50, 50], [233, 654, 654], [233, 655, 657], [235, 45, 54], [235, 158, 165], [238, 21, 21], [238, 27, 29], [238, 31, 31], [238, 32, 38], [240, 62, 64], [240, 68, 68], [241, 928, 939], [243, 103, 104], [243, 108, 110], [243, 121, 121], [243, 122, 123], [243, 124, 124], [243, 127, 127], [244, 173, 174], [244, 178, 178], [244, 183, 183], [244, 184, 184], [244, 187, 187], [246, 156, 157], [246, 157, 161], [247, 471, 471], [247, 473, 473], [254, 343, 359], [256, 68, 70], [256, 73, 73], [256, 81, 81], [260, 28, 29], [260, 31, 31], [260, 32, 32], [260, 34, 35], [261, 228, 237], [263, 362, 374], [267, 53, 75], [267, 110, 135], [270, 22, 24], [270, 29, 29], [270, 30, 30], [270, 32, 32], [272, 188, 237], [273, 34, 36], [273, 37, 41], [274, 290, 290], [274, 291, 293], [274, 294, 294], [276, 54, 71], [276, 88, 100], [277, 65, 65], [277, 66, 66], [277, 73, 75], [280, 74, 76], [280, 82, 83], [281, 222, 222], [281, 223, 223], [281, 224, 224], [281, 225, 225], [282, 51, 51], [282, 68, 68], [285, 14, 14], [285, 14, 16], [285, 29, 29], [285, 32, 32], [285, 32, 34], [285, 37, 37], [285, 40, 49], [286, 130, 130], [286, 278, 279], [286, 521, 521], [288, 47, 47], [288, 55, 61], [291, 39, 39], [291, 40, 40], [291, 42, 42], [291, 42, 42], [291, 46, 47], [291, 47, 47], [291, 49, 50], [291, 51, 51], [294, 111, 112], [294, 116, 117], [294, 117, 117], [299, 9, 19], [300, 352, 357], [301, 587, 589], [301, 612, 612], [301, 612, 612], [301, 625, 631], [302, 282, 283], [302, 296, 296], [302, 331, 332], [302, 345, 345], [302, 388, 390], [302, 400, 400], [302, 432, 434], [302, 441, 441], [302, 459, 461], [302, 468, 468], [302, 494, 494], [302, 507, 507], [302, 511, 512], [302, 528, 528], [302, 540, 540], [302, 544, 545], [302, 597, 597], [302, 610, 610], [302, 614, 615], [302, 631, 631], [302, 643, 643], [302, 647, 648], [304, 42, 42], [304, 79, 79], [304, 111, 111], [308, 151, 151], [308, 151, 151], [308, 151, 151], [308, 152, 153], [308, 158, 160], [308, 161, 163], [308, 164, 165], [308, 168, 169], [308, 171, 172], [308, 175, 181], [309, 467, 467], [309, 472, 472], [309, 480, 480], [310, 77, 79], [310, 80, 80], [315, 19, 19], [316, 98, 98], [316, 112, 112], [316, 116, 116], [316, 117, 117], [317, 29, 30], [317, 202, 203], [317, 350, 351], [319, 329, 330], [319, 332, 332], [320, 233, 237], [320, 238, 239], [322, 76, 78], [322, 97, 102], [322, 105, 105], [322, 106, 106], [322, 109, 109], [322, 109, 109], [322, 111, 111], [322, 113, 113], [322, 127, 133], [324, 616, 630], [327, 134, 135], [327, 137, 138], [327, 169, 169], [328, 107, 118], [328, 145, 162], [328, 199, 216], [328, 405, 424], [329, 91, 106], [330, 903, 917], [331, 34, 35], [331, 35, 38], [331, 39, 43], [331, 43, 47], [333, 197, 197], [333, 208, 214], [336, 56, 58], [336, 64, 70], [339, 41, 42], [340, 58, 58], [340, 108, 146], [344, 31, 33], [344, 37, 39], [344, 41, 41], [345, 24, 24], [345, 26, 26], [345, 28, 28], [345, 33, 33], [347, 48, 48], [347, 52, 53], [351, 53, 53], [351, 57, 62], [351, 63, 64], [351, 65, 71], [351, 73, 76], [351, 78, 80], [351, 81, 81], [351, 83, 83], [351, 85, 85], [351, 90, 90], [351, 94, 98], [352, 368, 368], [352, 368, 369], [352, 371, 372], [352, 372, 372], [352, 374, 374], [352, 376, 376], [352, 377, 377], [352, 380, 384], [352, 385, 386], [352, 388, 389], [352, 390, 392], [352, 393, 395], [352, 400, 405], [352, 407, 408], [352, 410, 413], [352, 414, 415], [352, 417, 420], [354, 220, 224], [355, 32, 32], [355, 58, 59], [355, 111, 112], [355, 139, 139], [355, 143, 143], [355, 146, 146], [355, 147, 147], [355, 147, 149], [355, 153, 155], [355, 156, 156], [355, 158, 158], [355, 159, 159], [355, 161, 162], [355, 168, 168], [355, 177, 177], [355, 180, 180], [355, 185, 185], [355, 185, 185], [355, 185, 185], [355, 185, 186], [355, 191, 191], [355, 192, 196], [355, 197, 199], [356, 103, 122], [357, 534, 535], [357, 535, 539], [359, 27, 27], [360, 60, 75], [360, 80, 80], [360, 81, 81], [360, 82, 82], [360, 83, 87], [361, 55, 57], [361, 58, 62], [364, 224, 224], [364, 225, 226], [364, 227, 227], [364, 231, 231], [369, 66, 80], [369, 318, 330], [370, 20, 28], [375, 831, 860], [377, 130, 135], [377, 136, 142], [377, 144, 144], [379, 233, 235], [379, 243, 243], [379, 245, 245], [379, 246, 246], [379, 250, 256], [385, 66, 68], [385, 85, 88], [387, 55, 74], [387, 218, 232], [387, 371, 385], [390, 56, 56], [392, 51, 60], [393, 17, 18], [393, 24, 27], [393, 30, 32], [393, 33, 34], [393, 37, 38], [393, 41, 42], [394, 238, 240], [394, 243, 245], [394, 249, 250], [394, 250, 252], [394, 258, 258], [394, 552, 554], [394, 557, 559], [394, 563, 564], [394, 564, 566], [394, 572, 572], [396, 39, 42], [396, 43, 43], [396, 44, 48], [396, 53, 55], [396, 57, 57], [396, 59, 59], [396, 59, 59], [396, 61, 63], [396, 63, 63], [396, 63, 64], [396, 65, 65], [396, 66, 66], [396, 67, 67], [396, 69, 72], [396, 72, 72], [399, 79, 81], [399, 94, 94], [402, 268, 279], [402, 617, 640], [404, 125, 127], [404, 133, 155], [404, 162, 162], [406, 18, 19], [406, 20, 20], [406, 20, 21], [406, 27, 30], [406, 88, 91], [413, 61, 61], [413, 62, 68], [413, 74, 74], [413, 75, 75], [413, 82, 84], [415, 23, 25], [415, 26, 26], [417, 938, 1007], [418, 57, 60], [418, 62, 64], [418, 66, 67], [418, 73, 73], [418, 160, 160], [418, 167, 167], [418, 167, 168], [418, 169, 169], [418, 169, 169], [418, 170, 172], [418, 176, 179], [418, 181, 188], [422, 35, 36], [422, 36, 39], [422, 45, 49], [422, 49, 53], [429, 236, 236], [429, 238, 238], [429, 240, 240], [431, 85, 85], [431, 95, 101], [433, 115, 115], [433, 116, 117], [433, 118, 118], [433, 122, 122], [435, 71, 83], [438, 82, 82], [438, 88, 88], [438, 89, 89], [438, 95, 95], [438, 96, 96], [438, 99, 105], [438, 106, 106], [438, 112, 112], [438, 113, 113], [438, 117, 117], [438, 118, 118], [440, 40, 42], [440, 47, 49], [440, 55, 57], [440, 62, 64], [440, 69, 71], [440, 75, 75], [440, 91, 99], [441, 32, 34], [441, 45, 45], [441, 53, 56], [441, 62, 68], [442, 68, 80], [443, 43, 44], [443, 46, 46], [445, 68, 68], [445, 77, 78], [445, 80, 80], [445, 87, 87], [445, 103, 103], [445, 103, 105], [448, 323, 342], [449, 157, 157], [449, 159, 159], [451, 42, 44], [451, 45, 49], [454, 160, 185], [455, 32, 32], [455, 34, 41], [455, 57, 63], [455, 64, 67], [455, 67, 67], [455, 75, 97], [455, 232, 232], [455, 277, 277], [456, 140, 141], [456, 143, 144], [456, 147, 147], [456, 148, 149], [456, 174, 174], [456, 279, 279], [456, 280, 281], [456, 286, 289], [460, 47, 47], [460, 48, 48], [460, 51, 51], [461, 247, 259], [462, 93, 93], [462, 94, 95], [465, 6, 6], [465, 8, 8], [465, 12, 12], [465, 23, 24], [465, 28, 28], [465, 43, 44], [465, 48, 48], [467, 211, 212], [467, 213, 214], [467, 214, 214], [467, 215, 215], [467, 217, 217], [470, 67, 74], [472, 198, 198], [477, 108, 108], [479, 53, 53], [480, 161, 161], [485, 54, 55], [485, 57, 57], [487, 196, 197], [488, 83, 97], [492, 24, 24], [492, 25, 25], [492, 28, 28], [494, 71, 71], [494, 98, 98], [494, 104, 104], [494, 109, 109], [495, 48, 48], [495, 56, 59], [495, 63, 66], [495, 67, 69], [495, 73, 73], [496, 73, 75], [496, 83, 84], [499, 505, 518], [500, 724, 724], [500, 724, 724], [500, 746, 748], [500, 757, 760], [500, 770, 770], [500, 770, 772], [500, 773, 773], [500, 773, 773], [500, 773, 773], [500, 780, 780], [500, 784, 784], [500, 787, 787], [500, 788, 788], [500, 790, 790], [500, 792, 792], [500, 793, 793], [500, 794, 794], [500, 795, 796], [500, 797, 797], [500, 798, 798], [500, 799, 799], [500, 800, 800], [500, 800, 801], [500, 804, 806], [500, 807, 810], [500, 810, 810], [500, 814, 814], [500, 816, 816], [500, 817, 818], [500, 819, 819], [500, 822, 822], [500, 824, 827], [500, 828, 829], [500, 831, 831], [500, 831, 833], [500, 833, 834], [500, 835, 835], [500, 836, 837], [500, 837, 838], [503, 155, 155], [503, 156, 156], [503, 158, 158], [503, 166, 166], [503, 166, 167], [503, 170, 170], [503, 171, 172], [503, 179, 179], [505, 49, 49], [505, 119, 121], [506, 39, 41], [506, 41, 43], [508, 374, 391], [510, 562, 562], [510, 564, 565], [510, 565, 568], [510, 568, 570], [510, 596, 596], [510, 596, 596], [510, 597, 598], [510, 599, 599], [510, 600, 600], [510, 601, 602], [510, 633, 633], [511, 157, 162], [511, 164, 166], [511, 167, 168], [511, 168, 173], [511, 173, 174], [511, 190, 195], [511, 195, 195], [511, 197, 198], [511, 395, 398], [511, 400, 416], [512, 23, 24], [512, 26, 26], [513, 124, 146], [518, 80, 86], [519, 32, 32], [519, 57, 58], [519, 64, 64], [519, 65, 65], [519, 71, 71], [519, 72, 72], [519, 76, 76], [519, 77, 77], [519, 83, 83], [519, 84, 84], [519, 90, 90], [519, 90, 91], [521, 303, 306], [521, 307, 310], [523, 452, 483], [524, 10, 21], [525, 398, 400], [525, 400, 405], [525, 412, 414], [525, 414, 417], [525, 426, 430], [525, 430, 433], [525, 443, 443], [525, 449, 449], [527, 76, 79], [527, 80, 82], [527, 83, 85], [527, 87, 92], [527, 93, 102], [528, 86, 86], [528, 170, 171], [530, 12, 12], [530, 12, 14], [530, 25, 25], [530, 27, 27], [530, 28, 30], [530, 32, 33], [530, 34, 43], [532, 26, 27], [532, 27, 30], [532, 31, 35], [532, 35, 39], [533, 99, 99], [533, 100, 100], [533, 109, 111], [534, 97, 105], [535, 432, 432], [535, 435, 435], [535, 437, 438], [535, 439, 439], [535, 440, 440], [535, 445, 445], [535, 446, 446], [539, 532, 532], [540, 346, 373], [540, 409, 436], [542, 37, 38], [542, 40, 40], [548, 159, 164], [549, 227, 227], [549, 238, 238], [549, 263, 263], [555, 195, 195], [560, 64, 67], [560, 69, 69], [561, 278, 337], [564, 15, 15], [564, 15, 15], [564, 15, 16], [564, 17, 17], [564, 17, 17], [564, 18, 20], [564, 21, 21], [564, 22, 22], [564, 23, 23], [564, 24, 26], [564, 26, 28], [564, 28, 28], [564, 28, 29], [564, 29, 30], [564, 33, 33], [564, 33, 33], [564, 46, 46], [564, 48, 50], [564, 51, 51], [564, 52, 54], [564, 54, 54], [564, 55, 55], [564, 56, 56], [564, 56, 56], [564, 57, 57], [564, 58, 58], [564, 60, 60], [564, 61, 61], [564, 63, 63], [564, 69, 69], [564, 72, 72], [564, 73, 73], [564, 188, 189], [564, 191, 191], [565, 947, 947], [565, 952, 952], [565, 952, 952], [566, 922, 952], [569, 71, 71], [569, 72, 72], [569, 81, 83], [571, 34, 34], [571, 38, 38], [571, 50, 51], [571, 52, 54], [571, 56, 56], [571, 58, 60], [571, 119, 120], [571, 121, 131], [571, 133, 134], [571, 138, 138], [571, 138, 138], [571, 140, 140], [571, 140, 140], [571, 144, 144], [571, 145, 145], [571, 146, 146], [571, 146, 146], [571, 146, 146], [572, 21, 37], [572, 46, 46], [572, 55, 55], [572, 57, 58], [572, 61, 65], [572, 72, 76], [572, 78, 78], [572, 80, 80], [572, 80, 82], [572, 83, 83], [572, 84, 85], [572, 87, 91], [572, 91, 91], [572, 92, 94], [572, 97, 97], [572, 97, 97], [572, 98, 98], [572, 99, 101], [574, 116, 117], [574, 121, 122], [574, 126, 127], [574, 131, 132], [574, 139, 139], [574, 155, 163], [575, 175, 175], [578, 286, 291], [578, 584, 590], [578, 591, 591], [578, 595, 595], [578, 596, 596], [578, 601, 603], [578, 640, 648], [578, 652, 659], [578, 663, 663], [578, 664, 664], [578, 669, 671], [578, 698, 706], [578, 710, 717], [578, 721, 721], [578, 722, 722], [578, 727, 729], [578, 730, 730], [578, 749, 749], [578, 827, 827], [578, 828, 828], [578, 835, 837], [578, 908, 915], [578, 921, 921], [578, 922, 922], [578, 929, 931], [578, 945, 954], [578, 960, 960], [578, 961, 961], [578, 968, 970], [578, 984, 984], [578, 985, 985], [578, 994, 1002], [578, 1009, 1011], [578, 1023, 1023], [579, 263, 263], [579, 264, 266], [580, 32, 49], [581, 316, 316], [581, 325, 336], [581, 337, 338], [581, 339, 355], [581, 357, 358], [581, 359, 359], [581, 365, 365], [581, 367, 367], [581, 367, 369], [581, 370, 370], [581, 375, 376], [581, 376, 377], [581, 381, 385], [581, 552, 552], [581, 556, 558], [581, 564, 564], [581, 566, 566], [581, 566, 568], [581, 570, 570], [581, 571, 572], [581, 573, 573], [582, 451, 457], [582, 469, 469], [583, 28, 28], [583, 30, 30], [583, 35, 35], [587, 32, 33], [587, 33, 34], [587, 36, 36], [588, 54, 54], [588, 57, 57], [588, 58, 58], [588, 59, 59], [588, 59, 59], [588, 59, 59], [589, 19, 37], [592, 99, 110], [592, 137, 154], [592, 191, 208], [593, 194, 194], [593, 197, 198], [593, 300, 301], [593, 302, 302], [593, 303, 303], [593, 304, 304], [597, 105, 105], [597, 108, 109], [598, 78, 89], [598, 370, 379], [602, 9, 9], [602, 9, 9], [604, 118, 119], [604, 119, 120], [604, 122, 122], [604, 318, 319], [604, 319, 319], [604, 320, 320], [604, 324, 324], [604, 523, 524], [604, 524, 524], [604, 525, 525], [604, 531, 531], [604, 800, 801], [604, 801, 802], [604, 808, 808], [604, 920, 920], [604, 921, 921], [604, 922, 922], [604, 926, 929], [608, 571, 571], [610, 142, 142], [614, 336, 357], [618, 592, 610], [619, 86, 87], [619, 89, 103], [619, 109, 109], [619, 109, 109], [619, 117, 117], [619, 119, 119], [621, 437, 437], [624, 50, 50], [624, 54, 55], [626, 78, 92], [630, 33, 74], [630, 77, 77], [630, 85, 92], [631, 168, 186], [637, 143, 143], [639, 201, 202], [639, 205, 206], [639, 209, 210], [639, 213, 214], [639, 218, 218], [639, 220, 220], [639, 227, 227], [639, 229, 229], [639, 230, 230], [639, 232, 233], [639, 249, 249], [640, 31, 31], [640, 39, 39], [640, 41, 41], [640, 42, 42], [640, 46, 46], [647, 44, 45], [647, 46, 46], [647, 47, 51], [647, 60, 63], [647, 65, 66], [647, 67, 68], [647, 77, 77], [647, 78, 78], [647, 78, 78], [649, 77, 78], [649, 78, 79], [649, 81, 81], [651, 277, 277], [651, 279, 279], [651, 279, 281], [651, 281, 281], [651, 345, 345], [651, 347, 347], [651, 347, 349], [651, 349, 349], [653, 113, 117], [653, 453, 453], [653, 453, 455], [653, 455, 455], [653, 461, 463], [653, 471, 481], [654, 47, 49], [654, 49, 51], [654, 55, 56], [657, 123, 123], [657, 123, 123], [657, 123, 123], [657, 128, 128], [660, 32, 32], [660, 40, 41], [660, 42, 45], [661, 460, 460], [661, 460, 463], [663, 275, 277], [663, 277, 277], [663, 277, 277], [663, 289, 289], [663, 291, 291], [663, 291, 293], [663, 293, 293], [664, 41, 43], [664, 47, 48], [666, 23, 23], [666, 23, 23], [666, 27, 27], [666, 35, 37], [666, 39, 39], [666, 40, 46], [667, 613, 613], [669, 70, 70], [669, 70, 70], [669, 70, 70], [669, 77, 118], [670, 64, 66], [670, 74, 76], [670, 78, 78], [670, 164, 166], [670, 170, 172], [670, 176, 177], [673, 114, 114], [673, 119, 123], [673, 124, 125], [673, 133, 136], [674, 689, 709], [674, 913, 914], [674, 928, 928], [674, 931, 931], [677, 187, 187], [677, 187, 189], [677, 189, 189], [677, 189, 189], [677, 192, 192], [677, 193, 194], [677, 196, 196], [677, 206, 206], [677, 211, 212], [677, 229, 240], [677, 242, 242], [677, 247, 248], [677, 292, 303], [677, 305, 305], [677, 306, 307], [677, 338, 349], [677, 351, 351], [677, 352, 353], [679, 36, 36], [683, 339, 340], [683, 378, 379], [684, 179, 179], [684, 192, 192], [684, 196, 197], [684, 294, 294], [684, 319, 321], [684, 325, 325], [687, 60, 60], [687, 64, 64], [687, 65, 68], [687, 80, 81], [687, 86, 88], [687, 89, 89], [687, 103, 104], [687, 107, 107], [687, 117, 118], [687, 120, 122], [687, 126, 127], [687, 157, 158], [691, 526, 526], [691, 527, 527], [691, 529, 535], [691, 536, 536], [691, 541, 541], [695, 238, 309], [695, 331, 350], [696, 166, 205], [697, 71, 72], [697, 74, 75], [697, 166, 166], [697, 219, 219], [697, 219, 221], [697, 221, 221], [697, 221, 221], [697, 222, 227], [697, 231, 232], [697, 232, 239], [697, 243, 243], [697, 244, 244], [697, 245, 248], [702, 226, 229], [703, 138, 142], [704, 17, 34], [705, 82, 82], [705, 83, 83], [705, 91, 91], [705, 94, 97], [705, 98, 98], [705, 100, 100], [705, 102, 102], [705, 111, 111], [705, 112, 112], [705, 118, 118], [706, 736, 741], [707, 83, 83], [707, 87, 88], [707, 91, 91], [707, 92, 94], [707, 98, 98], [707, 102, 102], [707, 145, 153], [707, 157, 157], [707, 159, 159], [712, 277, 304], [712, 599, 626], [712, 929, 956], [713, 63, 65], [713, 78, 78], [714, 253, 253], [714, 285, 285], [717, 120, 190], [720, 594, 618], [720, 935, 951], [722, 121, 123], [722, 136, 136], [725, 795, 797], [726, 24, 25], [728, 26, 26], [728, 27, 27], [728, 30, 30], [730, 51, 71], [733, 50, 51], [733, 53, 53], [734, 909, 909], [741, 246, 262], [742, 242, 244], [742, 245, 246], [742, 247, 247], [742, 248, 248], [742, 249, 249], [742, 250, 250], [742, 251, 251], [742, 252, 252], [742, 255, 257], [742, 268, 268], [742, 282, 282], [742, 297, 298], [742, 298, 298], [742, 300, 300], [743, 124, 125], [743, 125, 126], [743, 226, 226], [743, 244, 245], [744, 39, 39], [744, 41, 42], [747, 74, 74], [747, 75, 75], [747, 84, 86], [748, 27, 28], [748, 33, 37], [754, 56, 56], [754, 58, 64], [756, 854, 854], [757, 56, 71], [760, 101, 101], [760, 102, 115], [760, 115, 116], [760, 116, 116], [760, 117, 120], [760, 120, 121], [760, 121, 123], [760, 123, 123], [760, 124, 124], [760, 125, 126], [760, 126, 133], [760, 134, 135], [760, 136, 136], [760, 137, 141], [760, 141, 141], [760, 142, 142], [760, 143, 143], [760, 143, 149], [760, 150, 154], [760, 154, 159], [760, 159, 166], [760, 167, 167], [760, 168, 169], [760, 170, 172], [760, 176, 176], [760, 177, 182], [760, 182, 188], [760, 188, 191], [760, 191, 203], [760, 204, 204], [760, 205, 208], [760, 208, 209], [760, 210, 212], [760, 213, 215], [760, 216, 220], [760, 221, 223], [760, 223, 230], [760, 230, 234], [760, 235, 240], [760, 240, 241], [760, 241, 247], [760, 248, 257], [760, 258, 260], [760, 261, 263], [760, 263, 267], [760, 267, 271], [760, 272, 274], [760, 274, 275], [760, 276, 285], [760, 285, 285], [760, 286, 286], [760, 287, 289], [760, 289, 289], [760, 289, 289], [760, 290, 290], [760, 290, 299], [760, 300, 302], [760, 302, 303], [760, 303, 303], [760, 305, 305], [760, 306, 307], [760, 311, 329], [760, 331, 337], [760, 339, 339], [760, 339, 341], [760, 341, 344], [760, 345, 348], [760, 349, 350], [760, 354, 355], [760, 355, 358], [760, 360, 361], [760, 362, 362], [760, 362, 363], [760, 364, 364], [760, 365, 366], [760, 368, 368], [760, 368, 368], [760, 369, 370], [760, 371, 371], [760, 373, 373], [760, 374, 374], [760, 375, 375], [762, 161, 183], [766, 160, 190], [770, 115, 116], [770, 118, 125], [770, 128, 131], [770, 132, 132], [770, 144, 144], [770, 146, 146], [770, 147, 147], [772, 179, 179], [772, 180, 181], [772, 183, 183], [772, 184, 185], [772, 202, 202], [772, 203, 203], [772, 207, 207], [772, 211, 211], [773, 183, 185], [773, 227, 229], [774, 261, 261], [774, 269, 270], [774, 270, 274], [774, 276, 278], [774, 281, 283], [774, 284, 284], [774, 287, 287], [774, 288, 289], [774, 290, 290], [774, 291, 298], [774, 302, 302], [774, 373, 373], [774, 382, 385], [774, 387, 387], [774, 388, 389], [774, 389, 389], [774, 394, 394], [775, 44, 45], [775, 45, 45], [775, 45, 46], [775, 48, 48], [776, 71, 71], [776, 71, 71], [776, 72, 74], [776, 85, 85], [776, 88, 88], [776, 89, 89], [776, 91, 91], [776, 92, 92], [777, 179, 197], [780, 903, 963], [781, 33, 33], [781, 34, 36], [781, 49, 50], [789, 59, 61], [789, 62, 62], [792, 43, 43], [792, 44, 44], [792, 46, 46], [792, 46, 46], [792, 50, 51], [792, 51, 51], [792, 53, 53], [792, 53, 53], [792, 55, 55], [792, 55, 55], [792, 57, 57], [792, 57, 57], [792, 61, 62], [792, 62, 62], [792, 64, 64], [792, 64, 64], [792, 69, 70], [792, 71, 71], [800, 608, 608], [800, 612, 612], [800, 614, 614], [800, 618, 618], [802, 265, 279], [804, 24, 25], [804, 27, 27], [804, 29, 30], [806, 42, 42], [806, 72, 72], [806, 123, 123], [807, 86, 112], [807, 262, 292], [808, 47, 47], [808, 48, 48], [808, 55, 57], [811, 90, 90], [815, 19, 21], [815, 26, 26], [816, 321, 348], [817, 67, 67], [817, 68, 68], [817, 69, 70], [817, 75, 82], [817, 87, 87], [819, 101, 101], [819, 105, 110], [819, 112, 113], [819, 116, 120], [819, 131, 131], [819, 145, 146], [821, 423, 450], [822, 27, 29], [822, 34, 35], [822, 35, 38], [822, 39, 40], [822, 42, 45], [822, 48, 51], [829, 191, 209], [832, 50, 50], [832, 52, 55], [832, 56, 58], [832, 59, 61], [832, 62, 64], [832, 65, 67], [832, 70, 70], [832, 71, 71], [832, 71, 71], [832, 89, 97], [833, 21, 21], [833, 24, 25], [833, 26, 27], [833, 28, 30], [833, 30, 32], [833, 32, 32], [833, 32, 33], [833, 33, 33], [833, 34, 34], [833, 34, 35], [833, 40, 40], [833, 42, 46], [833, 49, 50], [833, 50, 51], [833, 51, 52], [833, 54, 54], [833, 55, 56], [833, 57, 57], [833, 57, 57], [833, 58, 58], [833, 58, 58], [833, 59, 59], [833, 60, 60], [833, 60, 60], [833, 61, 62], [833, 62, 62], [833, 62, 63], [833, 66, 66], [833, 68, 74], [833, 77, 77], [833, 77, 77], [833, 78, 79], [833, 80, 80], [833, 81, 82], [833, 82, 82], [833, 92, 92], [833, 92, 92], [833, 103, 106], [833, 120, 121], [833, 132, 132], [833, 139, 139], [833, 141, 141], [833, 142, 142], [833, 152, 183], [833, 190, 191], [833, 193, 194], [833, 195, 199], [833, 201, 201], [834, 61, 61], [838, 228, 228], [839, 438, 439], [841, 342, 359], [844, 241, 252], [844, 495, 506], [844, 518, 532], [846, 390, 390], [846, 390, 390], [846, 395, 396], [846, 398, 398], [846, 482, 482], [846, 483, 483], [846, 484, 484], [846, 484, 484], [846, 484, 484], [846, 489, 498], [846, 499, 500], [846, 508, 508], [846, 513, 513], [846, 513, 513], [846, 518, 519], [846, 521, 521], [847, 198, 198], [847, 208, 211], [849, 79, 81], [849, 84, 85], [849, 95, 97], [849, 100, 100], [849, 108, 110], [849, 113, 114], [849, 124, 126], [849, 129, 129], [853, 58, 59], [853, 59, 62], [854, 50, 50], [854, 58, 64], [856, 210, 224], [857, 20, 22], [857, 30, 30], [857, 32, 32], [857, 33, 33], [857, 41, 47], [857, 69, 71], [857, 77, 77], [857, 79, 79], [857, 83, 83], [857, 85, 85], [857, 86, 89], [857, 109, 115], [857, 123, 123], [857, 125, 125], [857, 126, 129], [857, 137, 143], [858, 42, 43], [858, 48, 50], [859, 521, 521], [859, 553, 554], [859, 577, 583], [861, 40, 42], [861, 56, 56], [861, 58, 58], [861, 64, 70], [861, 82, 84], [861, 94, 94], [861, 95, 96], [861, 97, 97], [861, 103, 109], [862, 47, 47], [862, 48, 48], [862, 49, 50], [862, 54, 57], [863, 37, 37], [863, 38, 38], [863, 39, 39], [863, 39, 39], [863, 39, 39], [863, 44, 44], [863, 45, 45], [863, 79, 82], [863, 90, 90], [863, 91, 91], [863, 91, 93], [863, 94, 96], [863, 96, 97], [863, 99, 100], [863, 100, 100], [863, 102, 102], [863, 103, 108], [863, 108, 108], [863, 108, 110], [863, 112, 113], [863, 127, 128], [863, 129, 129], [863, 129, 129], [863, 129, 129], [863, 134, 135], [863, 137, 137], [864, 39, 39], [864, 40, 41], [864, 53, 54], [866, 29, 40], [870, 112, 112], [870, 113, 113], [870, 113, 113], [870, 120, 120], [870, 128, 129], [870, 133, 134], [870, 135, 135], [870, 135, 135], [870, 135, 135], [870, 142, 142], [870, 152, 153], [870, 158, 158], [870, 159, 159], [870, 159, 159], [870, 164, 166], [870, 167, 170], [870, 171, 171], [870, 172, 173], [870, 173, 173], [870, 176, 176], [870, 180, 180], [870, 181, 181], [870, 181, 181], [870, 186, 188], [870, 189, 190], [870, 191, 191], [870, 192, 192], [870, 192, 192], [872, 40, 42], [872, 51, 51], [872, 52, 53], [872, 54, 54], [872, 60, 66], [875, 106, 117], [875, 144, 161], [875, 198, 215], [876, 22, 22], [876, 24, 24], [876, 26, 26], [876, 31, 31], [877, 21, 22], [877, 26, 27], [877, 29, 29], [877, 30, 30], [877, 34, 34], [879, 86, 86], [883, 32, 57], [884, 74, 82], [886, 85, 88], [886, 89, 89], [886, 90, 94], [886, 99, 99], [886, 103, 109], [886, 111, 111], [886, 111, 115], [886, 115, 115], [886, 116, 119], [886, 119, 120], [886, 123, 123], [886, 124, 124], [886, 124, 124], [886, 125, 127], [886, 128, 128], [886, 130, 133], [886, 133, 133], [890, 93, 93], [890, 93, 93], [890, 108, 111], [890, 114, 117], [890, 120, 120], [890, 120, 120], [890, 142, 151], [891, 106, 121], [892, 9, 9], [892, 19, 19], [892, 20, 20], [892, 21, 21], [892, 22, 24], [892, 26, 26], [892, 27, 27], [892, 28, 28], [892, 28, 28], [892, 30, 30], [892, 31, 32], [892, 33, 33], [892, 33, 34], [892, 34, 34], [892, 35, 35], [892, 35, 36], [892, 37, 37], [892, 38, 38], [892, 39, 40], [892, 42, 42], [892, 43, 43], [892, 44, 44], [892, 45, 45], [892, 45, 45], [892, 45, 45], [892, 46, 46], [892, 46, 46], [892, 48, 48], [892, 50, 53], [892, 54, 55], [892, 56, 57], [892, 57, 57], [892, 58, 58], [892, 59, 59], [892, 60, 60], [892, 61, 62], [892, 63, 63], [892, 63, 63], [892, 65, 65], [892, 65, 65], [892, 124, 124], [892, 140, 140], [892, 188, 188], [892, 223, 223], [893, 986, 1002], [895, 14, 19], [895, 22, 22], [895, 22, 22], [895, 22, 22], [895, 23, 24], [896, 137, 140], [896, 142, 142], [897, 306, 309], [897, 310, 311], [897, 313, 320], [897, 402, 405], [897, 406, 407], [897, 409, 416], [897, 483, 486], [897, 487, 490], [898, 246, 264], [898, 276, 290], [898, 400, 422], [900, 105, 105], [900, 106, 107], [900, 110, 112], [901, 257, 267], [903, 582, 582], [903, 584, 585], [903, 904, 904], [903, 913, 914], [906, 46, 93], [907, 48, 54], [907, 66, 67], [909, 315, 342], [910, 45, 47], [910, 58, 58], [910, 60, 60], [910, 66, 72], [910, 129, 131], [910, 137, 137], [910, 139, 139], [910, 145, 145], [910, 146, 147], [910, 148, 151], [910, 159, 165], [912, 19, 20], [912, 25, 26], [914, 72, 73], [920, 45, 47], [920, 51, 52], [920, 55, 56], [923, 192, 237], [924, 37, 37], [924, 39, 39], [924, 44, 44], [924, 67, 67], [924, 68, 68], [924, 68, 68], [924, 70, 71], [924, 119, 119], [924, 120, 121], [924, 121, 121], [924, 122, 122], [924, 124, 125], [924, 125, 125], [924, 125, 125], [924, 127, 127], [924, 129, 130], [924, 131, 131], [924, 142, 143], [924, 145, 147], [924, 147, 148], [924, 150, 151], [924, 153, 153], [924, 153, 153], [924, 153, 153], [924, 154, 154], [924, 154, 155], [924, 156, 156], [924, 157, 158], [924, 159, 159], [924, 162, 162], [924, 163, 165], [924, 167, 169], [924, 171, 174], [924, 174, 175], [924, 177, 178], [924, 179, 180], [924, 181, 181], [924, 182, 182], [924, 183, 183], [924, 185, 187], [924, 190, 190], [924, 191, 191], [924, 191, 191], [924, 193, 195], [927, 59, 61], [927, 65, 65], [927, 68, 68], [928, 36, 39], [928, 64, 64], [928, 64, 64], [928, 71, 86], [929, 25, 26], [930, 33, 34], [930, 42, 44], [930, 46, 47], [933, 50, 51], [933, 53, 53], [933, 55, 56], [935, 36, 37], [935, 43, 51], [935, 56, 56], [935, 56, 58], [936, 26, 26], [937, 203, 208], [937, 303, 308], [939, 287, 321], [940, 787, 800], [942, 106, 117], [942, 144, 161], [942, 198, 215], [942, 251, 270], [942, 320, 339], [944, 191, 194], [948, 124, 130], [949, 156, 183], [954, 9, 11], [954, 11, 13], [954, 31, 121], [955, 140, 150], [956, 668, 670], [959, 72, 74], [959, 80, 81], [960, 257, 297], [965, 23, 28], [965, 31, 31], [965, 31, 31], [965, 31, 31], [965, 32, 33], [966, 199, 201], [966, 209, 209], [966, 240, 242], [966, 247, 247], [967, 953, 953], [969, 35, 46], [969, 127, 128], [970, 32, 55], [972, 652, 684], [974, 52, 53], [974, 125, 125], [974, 128, 131], [974, 146, 149], [974, 178, 178], [974, 179, 181], [974, 181, 181], [974, 186, 187], [974, 187, 189], [975, 184, 205], [975, 217, 234], [976, 408, 410], [976, 432, 434], [976, 443, 445], [976, 476, 478], [976, 486, 487], [976, 488, 490], [976, 490, 490], [986, 299, 309], [986, 395, 395], [986, 397, 403], [988, 208, 225], [989, 291, 304], [990, 52, 57], [992, 110, 124], [994, 605, 691], [995, 53, 54], [995, 69, 70], [995, 72, 72], [995, 73, 73], [995, 78, 79], [996, 90, 90], [996, 90, 90], [996, 91, 93], [996, 101, 101], [996, 104, 104], [996, 105, 105], [996, 109, 109], [997, 47, 48], [997, 957, 958], [997, 981, 982], [998, 77, 78], [998, 82, 83], [998, 87, 88], [998, 95, 95], [998, 113, 121], [998, 140, 142], [998, 147, 149], [998, 156, 158], [998, 163, 165], [998, 169, 171], [998, 176, 178], [998, 182, 182], [998, 221, 229], [999, 56, 58], [999, 63, 63], [999, 76, 80], [999, 82, 85], [999, 92, 92], [1001, 51, 52], [1001, 54, 54], [1002, 99, 100], [1002, 100, 101], [1002, 103, 103], [1002, 155, 156], [1002, 156, 157], [1002, 161, 161], [1002, 317, 318], [1002, 318, 319], [1002, 325, 325], [1003, 65, 67], [1003, 75, 76], [1003, 172, 174], [1003, 178, 180], [1003, 184, 185], [1004, 35, 35], [1004, 36, 39], [1007, 43, 43], [1012, 15, 32], [1014, 823, 842], [1016, 265, 267], [1016, 278, 280], [1016, 353, 355], [1016, 365, 367], [1016, 420, 426], [1017, 239, 273], [1019, 33, 39], [1019, 49, 49], [1019, 84, 84], [1019, 86, 87], [1019, 88, 89], [1019, 89, 89], [1019, 90, 101], [1019, 106, 122], [1026, 829, 830], [1026, 830, 830], [1026, 831, 831], [1027, 40, 44], [1031, 440, 456], [1032, 93, 100], [1033, 136, 151], [1034, 32, 33], [1034, 41, 42], [1034, 45, 61], [1038, 146, 146], [1038, 164, 174], [1038, 189, 189], [1038, 191, 198], [1038, 203, 203], [1038, 205, 212], [1038, 220, 228], [1038, 235, 235], [1039, 487, 490], [1039, 491, 491], [1039, 492, 492], [1040, 32, 32], [1040, 39, 39], [1040, 46, 49], [1040, 50, 50], [1040, 51, 51], [1040, 66, 66], [1040, 66, 66], [1040, 68, 68], [1040, 71, 73], [1040, 76, 78], [1040, 78, 80], [1040, 82, 83], [1040, 84, 85], [1040, 88, 89], [1040, 90, 90], [1040, 92, 92], [1040, 92, 102], [1040, 105, 105], [1041, 45, 48], [1041, 49, 49], [1041, 69, 72], [1041, 73, 73], [1041, 77, 77], [1041, 89, 104], [1044, 51, 53], [1044, 54, 58], [1045, 189, 192], [1045, 199, 200], [1045, 226, 229], [1045, 236, 237], [1045, 308, 309], [1045, 313, 315], [1045, 319, 325], [1045, 395, 398], [1045, 436, 439], [1045, 528, 529], [1045, 533, 535], [1045, 539, 545], [1050, 37, 38], [1050, 40, 40], [1053, 14, 21], [1054, 115, 128], [1058, 13, 23], [1060, 47, 49], [1060, 53, 53], [1064, 104, 144], [1065, 51, 73], [1066, 43, 46], [1066, 48, 48], [1066, 49, 52], [1068, 431, 431], [1069, 69, 71], [1069, 75, 76], [1077, 561, 579], [1078, 17, 17], [1078, 26, 27], [1078, 29, 29], [1078, 30, 30], [1078, 47, 47], [1078, 57, 58], [1078, 60, 61], [1079, 113, 115], [1079, 118, 119], [1083, 235, 235], [1083, 238, 238], [1083, 239, 239], [1083, 243, 243], [1083, 248, 250], [1086, 162, 183], [1089, 31, 34], [1089, 34, 35], [1089, 35, 38], [1089, 39, 39], [1089, 43, 46], [1092, 46, 46], [1092, 47, 48], [1092, 49, 49], [1092, 50, 50], [1092, 51, 51], [1092, 53, 53], [1092, 90, 90], [1092, 92, 92], [1092, 97, 97], [1092, 99, 99], [1092, 101, 103], [1092, 108, 108], [1092, 109, 109], [1092, 111, 112], [1092, 114, 115], [1094, 118, 118], [1094, 118, 120], [1094, 125, 128], [1095, 89, 94], [1095, 96, 96], [1098, 269, 271], [1098, 324, 326], [1099, 133, 152], [1100, 1, 1], [1100, 1, 1], [1102, 68, 69], [1102, 69, 69], [1102, 69, 70], [1102, 72, 72], [1103, 28, 30], [1106, 63, 63], [1110, 93, 94], [1110, 94, 96], [1110, 97, 97], [1110, 99, 99], [1110, 103, 103], [1110, 545, 545], [1110, 547, 547], [1110, 547, 552], [1110, 554, 554], [1110, 555, 555], [1110, 556, 556], [1110, 558, 558], [1110, 559, 559], [1110, 559, 560], [1110, 562, 562], [1110, 563, 563], [1110, 563, 564], [1110, 566, 566], [1110, 567, 567], [1110, 567, 568], [1110, 572, 572], [1110, 614, 615], [1110, 615, 615], [1110, 615, 616], [1110, 618, 618], [1113, 110, 112], [1113, 113, 117], [1113, 117, 117], [1113, 119, 123], [1113, 125, 129], [1113, 131, 141], [1113, 143, 155], [1114, 92, 93], [1114, 108, 109], [1116, 85, 87], [1116, 88, 88], [1117, 142, 144], [1117, 145, 146], [1117, 148, 155], [1121, 37, 40], [1121, 42, 42], [1121, 43, 43], [1121, 52, 53], [1126, 41, 42], [1126, 42, 43], [1126, 47, 47], [1127, 32, 40], [1127, 54, 63], [1129, 349, 349], [1132, 577, 612], [1136, 56, 63], [1136, 71, 72], [1138, 49, 50], [1138, 58, 58], [1140, 202, 202], [1140, 215, 215], [1145, 43, 46], [1145, 47, 49], [1145, 51, 56], [1145, 57, 57], [1145, 57, 60], [1146, 112, 112], [1146, 219, 219], [1147, 356, 364], [1150, 47, 49], [1150, 50, 54], [1151, 26, 26], [1151, 27, 27], [1151, 28, 28], [1151, 31, 31], [1151, 33, 33], [1151, 36, 36], [1151, 38, 38], [1151, 41, 42], [1152, 53, 55], [1152, 56, 62], [1153, 761, 761], [1153, 761, 761], [1153, 762, 763], [1153, 763, 763], [1153, 764, 764], [1153, 767, 768], [1155, 84, 100], [1156, 99, 101], [1156, 105, 105], [1157, 28, 28], [1157, 29, 30], [1157, 41, 44], [1159, 34, 36], [1159, 37, 38], [1159, 41, 42], [1162, 38, 44], [1162, 44, 49], [1162, 49, 50], [1162, 51, 57], [1162, 58, 58], [1162, 58, 60], [1162, 60, 60], [1162, 62, 69], [1162, 70, 71], [1162, 71, 72], [1162, 73, 77], [1163, 212, 213], [1164, 114, 116], [1164, 117, 118], [1164, 118, 118], [1167, 288, 331], [1169, 32, 33], [1169, 41, 42], [1169, 45, 61], [1170, 48, 56], [1172, 197, 217], [1174, 17, 18], [1174, 23, 24], [1174, 24, 24], [1174, 67, 67], [1174, 68, 68], [1174, 68, 69], [1174, 70, 70], [1174, 71, 71], [1174, 72, 72], [1174, 72, 72], [1174, 73, 73], [1174, 73, 74], [1174, 74, 74], [1174, 74, 75], [1174, 75, 76], [1174, 76, 76], [1174, 76, 76], [1174, 79, 79], [1174, 80, 80], [1174, 81, 83], [1174, 85, 85], [1174, 92, 96], [1174, 97, 97], [1174, 98, 99], [1174, 101, 101], [1174, 101, 101], [1174, 101, 110], [1174, 111, 113], [1174, 118, 118], [1174, 118, 118], [1174, 119, 119], [1174, 119, 120], [1179, 45, 45], [1179, 49, 50], [1180, 19, 21], [1181, 42, 91], [1183, 29, 49], [1184, 55, 55], [1184, 58, 58], [1184, 59, 59], [1184, 60, 60], [1184, 61, 61], [1184, 63, 63], [1184, 66, 66], [1184, 67, 67], [1184, 67, 68], [1184, 70, 70], [1184, 71, 71], [1184, 72, 72], [1184, 72, 72], [1184, 73, 73], [1184, 73, 73], [1184, 74, 74], [1184, 75, 75], [1184, 76, 76], [1184, 88, 94], [1184, 96, 96], [1187, 532, 532], [1187, 532, 532], [1187, 533, 533], [1187, 549, 552], [1187, 554, 560], [1188, 88, 105], [1189, 183, 191], [1190, 414, 443], [1193, 33, 34], [1193, 34, 35], [1193, 37, 37], [1195, 282, 290], [1196, 219, 240], [1196, 247, 249], [1196, 253, 255], [1199, 44, 44], [1199, 46, 47], [1201, 74, 74], [1202, 585, 596], [1203, 6, 6], [1203, 11, 11], [1203, 12, 12], [1203, 14, 14], [1203, 15, 15], [1203, 16, 17], [1203, 17, 17], [1203, 18, 19], [1203, 20, 20], [1203, 24, 24], [1203, 25, 25], [1203, 26, 26], [1203, 27, 27], [1203, 28, 29], [1203, 30, 31], [1203, 32, 32], [1203, 33, 33], [1203, 36, 36], [1203, 36, 36], [1203, 37, 37], [1203, 56, 56], [1203, 57, 57], [1210, 149, 149], [1210, 150, 150], [1210, 151, 151], [1210, 153, 157], [1210, 157, 157], [1210, 159, 160], [1210, 163, 163], [1210, 164, 164], [1210, 165, 167], [1210, 167, 169], [1210, 172, 172], [1210, 173, 173], [1210, 178, 178], [1213, 40, 40], [1213, 41, 43], [1213, 47, 53], [1216, 116, 116], [1216, 124, 124], [1216, 128, 128], [1216, 130, 134], [1217, 182, 182], [1217, 182, 182], [1217, 191, 192], [1218, 612, 619], [1219, 21, 26], [1219, 29, 29], [1219, 29, 29], [1219, 29, 29], [1219, 30, 31], [1221, 38, 39], [1221, 43, 44], [1221, 52, 58], [1223, 725, 740], [1225, 429, 430], [1229, 192, 193], [1229, 193, 194], [1229, 196, 196], [1229, 477, 478], [1229, 478, 479], [1229, 481, 481], [1232, 64, 66], [1232, 75, 76], [1232, 171, 173], [1232, 177, 179], [1232, 183, 184], [1236, 45, 47], [1236, 59, 59], [1236, 61, 61], [1236, 67, 73], [1236, 131, 133], [1236, 139, 139], [1236, 141, 141], [1236, 148, 148], [1236, 149, 150], [1236, 151, 154], [1236, 163, 169], [1236, 179, 181], [1236, 195, 195], [1236, 197, 197], [1236, 203, 209], [1237, 244, 257], [1239, 208, 209], [1244, 333, 359], [1244, 394, 400], [1247, 977, 978], [1248, 156, 156], [1248, 158, 158], [1250, 318, 318], [1250, 320, 325], [1250, 327, 331], [1250, 335, 335], [1251, 81, 82], [1251, 86, 87], [1251, 91, 92], [1251, 96, 97], [1251, 104, 104], [1251, 126, 134], [1252, 50, 50], [1252, 51, 51], [1252, 57, 63], [1252, 72, 74], [1253, 71, 73], [1253, 74, 75], [1253, 76, 77], [1253, 78, 78], [1253, 79, 80], [1257, 265, 265], [1257, 768, 768], [1257, 775, 775], [1257, 779, 780], [1257, 786, 786], [1257, 791, 791], [1257, 851, 852], [1261, 259, 259], [1261, 261, 261], [1262, 293, 293], [1262, 557, 557], [1263, 29, 31], [1263, 35, 36], [1263, 39, 40], [1264, 155, 189], [1266, 38, 42], [1267, 133, 140], [1269, 37, 37], [1269, 47, 47], [1270, 194, 196], [1270, 206, 208], [1270, 212, 224], [1272, 9, 9], [1272, 9, 9], [1273, 103, 103], [1273, 114, 114], [1273, 129, 129], [1273, 140, 140], [1278, 45, 47], [1278, 48, 48], [1279, 145, 145], [1279, 160, 160], [1279, 161, 161], [1279, 170, 172], [1281, 106, 117], [1281, 322, 377], [1281, 414, 534], [1282, 195, 206], [1282, 224, 228], [1282, 232, 232], [1282, 242, 242], [1282, 291, 302], [1282, 320, 324], [1282, 328, 328], [1282, 338, 338], [1283, 434, 460], [1287, 32, 32], [1287, 33, 35], [1287, 38, 38], [1290, 220, 222], [1290, 223, 227], [1290, 231, 232], [1290, 234, 234], [1290, 234, 234], [1290, 235, 237], [1290, 239, 241], [1290, 287, 300], [1299, 166, 171], [1300, 29, 29], [1300, 29, 29], [1300, 29, 30], [1303, 71, 74], [1303, 81, 81], [1304, 35, 35], [1304, 37, 37], [1304, 38, 43], [1306, 88, 89], [1306, 150, 151], [1307, 696, 784], [1308, 618, 622], [1312, 355, 366], [1313, 316, 323], [1316, 74, 77], [1316, 79, 79], [1316, 80, 80], [1316, 86, 88], [1316, 92, 93], [1316, 148, 151], [1316, 153, 153], [1316, 154, 154], [1316, 157, 159], [1316, 164, 165], [1316, 197, 200], [1316, 202, 202], [1316, 203, 203], [1316, 209, 211], [1316, 215, 216], [1318, 24, 25], [1318, 26, 26], [1318, 26, 27], [1318, 33, 36], [1318, 58, 61], [1318, 123, 123], [1321, 319, 330], [1322, 20, 22], [1322, 23, 24], [1322, 27, 28], [1323, 273, 274], [1323, 276, 279], [1324, 461, 462], [1324, 467, 480], [1324, 496, 497], [1324, 498, 498], [1324, 528, 530], [1324, 532, 535], [1324, 536, 536], [1324, 538, 539], [1324, 542, 549], [1324, 557, 558], [1324, 558, 566], [1324, 576, 577], [1324, 582, 595], [1324, 611, 612], [1324, 613, 613], [1324, 643, 645], [1324, 647, 650], [1324, 651, 651], [1324, 653, 654], [1324, 657, 664], [1324, 672, 673], [1324, 673, 681], [1327, 55, 55], [1327, 57, 57], [1327, 58, 59], [1336, 36, 37], [1336, 47, 49], [1336, 49, 49], [1336, 133, 135], [1336, 135, 135], [1337, 105, 119], [1339, 52, 54], [1339, 56, 57], [1341, 222, 227], [1344, 231, 235], [1344, 236, 237], [1345, 161, 173], [1346, 37, 39], [1346, 40, 40], [1350, 25, 37], [1350, 161, 162], [1350, 164, 164], [1350, 165, 174], [1350, 175, 225], [1350, 228, 275], [1353, 170, 171], [1355, 68, 69], [1355, 73, 74], [1355, 78, 79], [1355, 86, 95], [1355, 125, 126], [1355, 136, 137], [1355, 143, 143], [1355, 144, 145], [1355, 150, 161], [1357, 429, 458], [1358, 155, 155], [1359, 68, 68], [1359, 85, 85], [1359, 86, 87], [1359, 95, 95], [1360, 18, 23], [1360, 26, 26], [1360, 26, 26], [1360, 26, 26], [1360, 27, 28], [1363, 91, 91], [1363, 96, 99], [1363, 135, 136], [1366, 226, 226], [1366, 247, 262], [1366, 268, 268], [1368, 233, 233], [1368, 236, 236], [1368, 237, 237], [1368, 241, 241], [1368, 246, 248], [1369, 0, 7], [1369, 17, 30], [1371, 35, 35], [1371, 36, 43], [1372, 70, 87]]

token_offsets_31b.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:67e86c38f653f1908686561f070eb21f4f48be97ff525f38a94720d498b93d40
+size 5273246

token_probs_31b.npz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8abf2638c7c6a0241cd13ce3558344387210e63cc5ce739ca1123eb348e76327
+size 2812506

token_report_31b.json

@@ -0,0 +1,1232 @@
+{
+  "dataset_path": "data/dataset.jsonl",
+  "n_examples": 1374,
+  "n_pos_examples": 687,
+  "splits": [
+    {
+      "split_name": "random_stratified",
+      "n_train_examples": 1099,
+      "n_test_examples": 275,
+      "note": "leaky baseline; pairs share group key",
+      "all_metrics": {
+        "auc": 0.8792314700570182,
+        "auc_ci_lo": 0.8721397795400918,
+        "auc_ci_hi": 0.8867151288516568,
+        "accuracy": 0.9788393320109631,
+        "precision": 0.3694556451612903,
+        "recall": 0.3570384802727716,
+        "f1": 0.36314094624721327,
+        "recall_at_fpr_0.05": 0.5742815392109109,
+        "recall_at_fpr_0.10": 0.6765708718947882,
+        "brier": 0.022547235239081524,
+        "ece": 0.06328650542424526,
+        "n_total": 121499,
+        "n_pos": 2053
+      },
+      "span_metrics": {
+        "auc": NaN,
+        "auc_ci_lo": NaN,
+        "auc_ci_hi": NaN,
+        "accuracy": 0.3570384802727716,
+        "precision": 1.0,
+        "recall": 0.3570384802727716,
+        "f1": 0.5262024407753051,
+        "recall_at_fpr_0.05": NaN,
+        "recall_at_fpr_0.10": NaN,
+        "brier": NaN,
+        "ece": NaN,
+        "n_total": 2053,
+        "n_pos": 2053
+      },
+      "span_max_metrics": {
+        "auc": 0.6688617370147044,
+        "auc_ci_lo": 0.6029174107142857,
+        "auc_ci_hi": 0.7261501149872097,
+        "accuracy": 0.5563636363636364,
+        "precision": 0.5380952380952381,
+        "recall": 0.8188405797101449,
+        "f1": 0.6494252873563219,
+        "recall_at_fpr_0.05": 0.30434782608695654,
+        "recall_at_fpr_0.10": 0.35507246376811596,
+        "brier": 0.2665282933950874,
+        "ece": 0.1896156802773476,
+        "n_total": 275,
+        "n_pos": 138
+      },
+      "dilated_span_max_metrics": {
+        "auc": 0.5649000317359567,
+        "auc_ci_lo": 0.4928443594944609,
+        "auc_ci_hi": 0.6299806310920555,
+        "accuracy": 0.4909090909090909,
+        "precision": 0.4947916666666667,
+        "recall": 0.6884057971014492,
+        "f1": 0.5757575757575758,
+        "recall_at_fpr_0.05": 0.2898550724637681,
+        "recall_at_fpr_0.10": 0.3333333333333333,
+        "brier": 0.31105206073042857,
+        "ece": 0.2701450151069598,
+        "n_total": 275,
+        "n_pos": 138,
+        "proximity_window": 10
+      },
+      "proximal_all_metrics": {
+        "auc": 0.7293584774236471,
+        "auc_ci_lo": 0.722318787540874,
+        "auc_ci_hi": 0.73628668777408,
+        "accuracy": 0.9511600918526079,
+        "precision": 0.48185483870967744,
+        "recall": 0.16308427157966565,
+        "f1": 0.24369105276574052,
+        "recall_at_fpr_0.05": 0.31866257250085295,
+        "recall_at_fpr_0.10": 0.41265779597407026,
+        "brier": 0.04482944852432753,
+        "ece": 0.031936453160440616,
+        "n_total": 121499,
+        "n_pos": 5862,
+        "proximity_window": 10
+      },
+      "baseline_aucs": {
+        "random": {
+          "all_auc": 0.40577955531169196,
+          "span_auc": NaN,
+          "span_max_auc": 0.5145985401459854,
+          "dilated_span_max_auc": 0.5145985401459854
+        },
+        "length": {
+          "all_auc": 0.24271243668783954,
+          "span_auc": NaN,
+          "span_max_auc": 0.47238971754998416,
+          "dilated_span_max_auc": 0.47238971754998416
+        },
+        "regex": {
+          "all_auc": 0.49685148970626436,
+          "span_auc": NaN,
+          "span_max_auc": 0.48275679678408967,
+          "dilated_span_max_auc": 0.48275679678408967
+        }
+      }
+    },
+    {
+      "split_name": "group_repo",
+      "n_train_examples": 1100,
+      "n_test_examples": 274,
+      "note": "GroupShuffleSplit on pair_group_key (n_groups=677)",
+      "all_metrics": {
+        "auc": 0.8124500562118311,
+        "auc_ci_lo": 0.804129282586531,
+        "auc_ci_hi": 0.8207606743492433,
+        "accuracy": 0.9773988035117707,
+        "precision": 0.30012610340479196,
+        "recall": 0.20923076923076922,
+        "f1": 0.24656824656824658,
+        "recall_at_fpr_0.05": 0.41714285714285715,
+        "recall_at_fpr_0.10": 0.52,
+        "brier": 0.0242680309899385,
+        "ece": 0.059394872129765026,
+        "n_total": 128710,
+        "n_pos": 2275
+      },
+      "span_metrics": {
+        "auc": NaN,
+        "auc_ci_lo": NaN,
+        "auc_ci_hi": NaN,
+        "accuracy": 0.20923076923076922,
+        "precision": 1.0,
+        "recall": 0.20923076923076922,
+        "f1": 0.3460559796437659,
+        "recall_at_fpr_0.05": NaN,
+        "recall_at_fpr_0.10": NaN,
+        "brier": NaN,
+        "ece": NaN,
+        "n_total": 2275,
+        "n_pos": 2275
+      },
+      "span_max_metrics": {
+        "auc": 0.6225957696201183,
+        "auc_ci_lo": 0.5504153984914414,
+        "auc_ci_hi": 0.6886499253993129,
+        "accuracy": 0.5109489051094891,
+        "precision": 0.5068493150684932,
+        "recall": 0.8102189781021898,
+        "f1": 0.6235955056179775,
+        "recall_at_fpr_0.05": 0.22627737226277372,
+        "recall_at_fpr_0.10": 0.3357664233576642,
+        "brier": 0.2774446507987361,
+        "ece": 0.19170649763006364,
+        "n_total": 274,
+        "n_pos": 137
+      },
+      "dilated_span_max_metrics": {
+        "auc": 0.49483190367094676,
+        "auc_ci_lo": 0.42128598583424265,
+        "auc_ci_hi": 0.5687968367672156,
+        "accuracy": 0.41605839416058393,
+        "precision": 0.44041450777202074,
+        "recall": 0.6204379562043796,
+        "f1": 0.5151515151515151,
+        "recall_at_fpr_0.05": 0.21897810218978103,
+        "recall_at_fpr_0.10": 0.30656934306569344,
+        "brier": 0.32347205328339323,
+        "ece": 0.28945193242580786,
+        "n_total": 274,
+        "n_pos": 137,
+        "proximity_window": 10
+      },
+      "proximal_all_metrics": {
+        "auc": 0.7140756497499194,
+        "auc_ci_lo": 0.7077808622623438,
+        "auc_ci_hi": 0.7204944211748613,
+        "accuracy": 0.9491026338279854,
+        "precision": 0.4281210592686003,
+        "recall": 0.10738573461964257,
+        "f1": 0.17170312302440258,
+        "recall_at_fpr_0.05": 0.2734461489799146,
+        "recall_at_fpr_0.10": 0.37450577257630874,
+        "brier": 0.04667598901391797,
+        "ece": 0.02808592692994872,
+        "n_total": 128710,
+        "n_pos": 6323,
+        "proximity_window": 10
+      },
+      "baseline_aucs": {
+        "random": {
+          "all_auc": 0.4676345096750839,
+          "span_auc": NaN,
+          "span_max_auc": 0.47109595609782084,
+          "dilated_span_max_auc": 0.47109595609782084
+        },
+        "length": {
+          "all_auc": 0.3009325505830429,
+          "span_auc": NaN,
+          "span_max_auc": 0.4887846981725186,
+          "dilated_span_max_auc": 0.4887846981725186
+        },
+        "regex": {
+          "all_auc": 0.4593372575145028,
+          "span_auc": NaN,
+          "span_max_auc": 0.5040758697852842,
+          "dilated_span_max_auc": 0.5040758697852842
+        }
+      }
+    },
+    {
+      "split_name": "heldout_cwe::CWE-089",
+      "n_train_examples": 966,
+      "n_test_examples": 408,
+      "note": "train all-except / test only",
+      "all_metrics": {
+        "auc": 0.9600743560563135,
+        "auc_ci_lo": 0.9564255645486055,
+        "auc_ci_hi": 0.9636205093679034,
+        "accuracy": 0.9700066876127746,
+        "precision": 0.471442720369759,
+        "recall": 0.647912885662432,
+        "f1": 0.5457672463214217,
+        "recall_at_fpr_0.05": 0.793557168784029,
+        "recall_at_fpr_0.10": 0.8938294010889292,
+        "brier": 0.027861295785212913,
+        "ece": 0.07089718008997592,
+        "n_total": 79251,
+        "n_pos": 2204
+      },
+      "span_metrics": {
+        "auc": NaN,
+        "auc_ci_lo": NaN,
+        "auc_ci_hi": NaN,
+        "accuracy": 0.647912885662432,
+        "precision": 1.0,
+        "recall": 0.647912885662432,
+        "f1": 0.7863436123348018,
+        "recall_at_fpr_0.05": NaN,
+        "recall_at_fpr_0.10": NaN,
+        "brier": NaN,
+        "ece": NaN,
+        "n_total": 2204,
+        "n_pos": 2204
+      },
+      "span_max_metrics": {
+        "auc": 0.9439638600538255,
+        "auc_ci_lo": 0.9228041603762412,
+        "auc_ci_hi": 0.9640548297188922,
+        "accuracy": 0.6127450980392157,
+        "precision": 0.56353591160221,
+        "recall": 1.0,
+        "f1": 0.7208480565371025,
+        "recall_at_fpr_0.05": 0.6764705882352942,
+        "recall_at_fpr_0.10": 0.7794117647058824,
+        "brier": 0.22893340965732042,
+        "ece": 0.2857507723474911,
+        "n_total": 408,
+        "n_pos": 204
+      },
+      "dilated_span_max_metrics": {
+        "auc": 0.940840061514802,
+        "auc_ci_lo": 0.9185573696332674,
+        "auc_ci_hi": 0.9616658869197932,
+        "accuracy": 0.6102941176470589,
+        "precision": 0.5623268698060941,
+        "recall": 0.9950980392156863,
+        "f1": 0.7185840707964601,
+        "recall_at_fpr_0.05": 0.6715686274509803,
+        "recall_at_fpr_0.10": 0.7696078431372549,
+        "brier": 0.22963713933612706,
+        "ece": 0.28457983574081286,
+        "n_total": 408,
+        "n_pos": 204,
+        "proximity_window": 10
+      },
+      "proximal_all_metrics": {
+        "auc": 0.7134300147418045,
+        "auc_ci_lo": 0.7074353874396364,
+        "auc_ci_hi": 0.7201310540765753,
+        "accuracy": 0.8937174294330671,
+        "precision": 0.6526906569825025,
+        "recall": 0.21148908857509627,
+        "f1": 0.3194635210471035,
+        "recall_at_fpr_0.05": 0.32648694908001713,
+        "recall_at_fpr_0.10": 0.4100342319212666,
+        "brier": 0.0899336034885583,
+        "ece": 0.038984753727046566,
+        "n_total": 79251,
+        "n_pos": 9348,
+        "proximity_window": 10
+      },
+      "baseline_aucs": {
+        "random": {
+          "all_auc": 0.5014989995853523,
+          "span_auc": NaN,
+          "span_max_auc": 0.503844675124952,
+          "dilated_span_max_auc": 0.503844675124952
+        },
+        "length": {
+          "all_auc": 0.3690502440858159,
+          "span_auc": NaN,
+          "span_max_auc": 0.4987624951941561,
+          "dilated_span_max_auc": 0.4987624951941561
+        },
+        "regex": {
+          "all_auc": 0.5040802280230723,
+          "span_auc": NaN,
+          "span_max_auc": 0.518094002306805,
+          "dilated_span_max_auc": 0.518094002306805
+        }
+      }
+    },
+    {
+      "split_name": "heldout_cwe::CWE-125",
+      "n_train_examples": 1162,
+      "n_test_examples": 212,
+      "note": "train all-except / test only",
+      "all_metrics": {
+        "auc": 0.8130219698774679,
+        "auc_ci_lo": 0.8014375109497583,
+        "auc_ci_hi": 0.8240126189462291,
+        "accuracy": 0.9897590628182502,
+        "precision": 0.12827225130890052,
+        "recall": 0.03670411985018727,
+        "f1": 0.057076295864880604,
+        "recall_at_fpr_0.05": 0.3707865168539326,
+        "recall_at_fpr_0.10": 0.5086142322097378,
+        "brier": 0.013408697711262478,
+        "ece": 0.04723283682029318,
+        "n_total": 158091,
+        "n_pos": 1335
+      },
+      "span_metrics": {
+        "auc": NaN,
+        "auc_ci_lo": NaN,
+        "auc_ci_hi": NaN,
+        "accuracy": 0.03670411985018727,
+        "precision": 1.0,
+        "recall": 0.03670411985018727,
+        "f1": 0.0708092485549133,
+        "recall_at_fpr_0.05": NaN,
+        "recall_at_fpr_0.10": NaN,
+        "brier": NaN,
+        "ece": NaN,
+        "n_total": 1335,
+        "n_pos": 1335
+      },
+      "span_max_metrics": {
+        "auc": 0.5116589533641865,
+        "auc_ci_lo": 0.43433818912713607,
+        "auc_ci_hi": 0.5850843829570703,
+        "accuracy": 0.5,
+        "precision": 0.5,
+        "recall": 0.5377358490566038,
+        "f1": 0.5181818181818182,
+        "recall_at_fpr_0.05": 0.04716981132075472,
+        "recall_at_fpr_0.10": 0.08490566037735849,
+        "brier": 0.27253398463466777,
+        "ece": 0.12165449861929101,
+        "n_total": 212,
+        "n_pos": 106
+      },
+      "dilated_span_max_metrics": {
+        "auc": 0.3316126735493058,
+        "auc_ci_lo": 0.2583664159141796,
+        "auc_ci_hi": 0.40531770198082856,
+        "accuracy": 0.37735849056603776,
+        "precision": 0.3522727272727273,
+        "recall": 0.29245283018867924,
+        "f1": 0.31958762886597936,
+        "recall_at_fpr_0.05": 0.02830188679245283,
+        "recall_at_fpr_0.10": 0.03773584905660377,
+        "brier": 0.34086921768783246,
+        "ece": 0.2598990294730888,
+        "n_total": 212,
+        "n_pos": 106,
+        "proximity_window": 10
+      },
+      "proximal_all_metrics": {
+        "auc": 0.7136664069208226,
+        "auc_ci_lo": 0.7041866890676302,
+        "auc_ci_hi": 0.7222690259932762,
+        "accuracy": 0.9771334231550183,
+        "precision": 0.1806282722513089,
+        "recall": 0.02046870364876891,
+        "f1": 0.03677058353317346,
+        "recall_at_fpr_0.05": 0.22337585286265202,
+        "recall_at_fpr_0.10": 0.3334322159596559,
+        "brier": 0.023956609306420483,
+        "ece": 0.03435417832613474,
+        "n_total": 158091,
+        "n_pos": 3371,
+        "proximity_window": 10
+      },
+      "baseline_aucs": {
+        "random": {
+          "all_auc": 0.5532525608395614,
+          "span_auc": NaN,
+          "span_max_auc": 0.5152189391242434,
+          "dilated_span_max_auc": 0.5152189391242434
+        },
+        "length": {
+          "all_auc": 0.22039345147968697,
+          "span_auc": NaN,
+          "span_max_auc": 0.48980954076183697,
+          "dilated_span_max_auc": 0.48980954076183697
+        },
+        "regex": {
+          "all_auc": 0.4984065170393396,
+          "span_auc": NaN,
+          "span_max_auc": 0.5,
+          "dilated_span_max_auc": 0.5
+        }
+      }
+    },
+    {
+      "split_name": "heldout_cwe::CWE-078",
+      "n_train_examples": 1164,
+      "n_test_examples": 210,
+      "note": "train all-except / test only",
+      "all_metrics": {
+        "auc": 0.8806561008172106,
+        "auc_ci_lo": 0.8732302371360536,
+        "auc_ci_hi": 0.8881984278259569,
+        "accuracy": 0.9531300706216872,
+        "precision": 0.33959537572254334,
+        "recall": 0.4382284382284382,
+        "f1": 0.3826582536128638,
+        "recall_at_fpr_0.05": 0.5491841491841492,
+        "recall_at_fpr_0.10": 0.6815850815850816,
+        "brier": 0.04340411615050454,
+        "ece": 0.09511051965281303,
+        "n_total": 64711,
+        "n_pos": 2145
+      },
+      "span_metrics": {
+        "auc": NaN,
+        "auc_ci_lo": NaN,
+        "auc_ci_hi": NaN,
+        "accuracy": 0.4382284382284382,
+        "precision": 1.0,
+        "recall": 0.4382284382284382,
+        "f1": 0.6094003241491086,
+        "recall_at_fpr_0.05": NaN,
+        "recall_at_fpr_0.10": NaN,
+        "brier": NaN,
+        "ece": NaN,
+        "n_total": 2145,
+        "n_pos": 2145
+      },
+      "span_max_metrics": {
+        "auc": 0.7237188208616779,
+        "auc_ci_lo": 0.655356362217018,
+        "auc_ci_hi": 0.7894332155401932,
+        "accuracy": 0.5,
+        "precision": 0.5,
+        "recall": 0.9428571428571428,
+        "f1": 0.6534653465346535,
+        "recall_at_fpr_0.05": 0.29523809523809524,
+        "recall_at_fpr_0.10": 0.4095238095238095,
+        "brier": 0.3177656646060927,
+        "ece": 0.31002071797847747,
+        "n_total": 210,
+        "n_pos": 105
+      },
+      "dilated_span_max_metrics": {
+        "auc": 0.68562358276644,
+        "auc_ci_lo": 0.6128582266563215,
+        "auc_ci_hi": 0.7578702027920778,
+        "accuracy": 0.48095238095238096,
+        "precision": 0.4896907216494845,
+        "recall": 0.9047619047619048,
+        "f1": 0.6354515050167224,
+        "recall_at_fpr_0.05": 0.2857142857142857,
+        "recall_at_fpr_0.10": 0.4,
+        "brier": 0.3282622972153013,
+        "ece": 0.326252941716285,
+        "n_total": 210,
+        "n_pos": 105,
+        "proximity_window": 10
+      },
+      "proximal_all_metrics": {
+        "auc": 0.7481883106070897,
+        "auc_ci_lo": 0.7409915038840356,
+        "auc_ci_hi": 0.7549370505304503,
+        "accuracy": 0.903277649858602,
+        "precision": 0.516257225433526,
+        "recall": 0.22507481493148526,
+        "f1": 0.3134803115059778,
+        "recall_at_fpr_0.05": 0.3274531422271224,
+        "recall_at_fpr_0.10": 0.43455662309025045,
+        "brier": 0.08019005481290785,
+        "ece": 0.04103688254981811,
+        "n_total": 64711,
+        "n_pos": 6349,
+        "proximity_window": 10
+      },
+      "baseline_aucs": {
+        "random": {
+          "all_auc": 0.5305832118206251,
+          "span_auc": NaN,
+          "span_max_auc": 0.493968253968254,
+          "dilated_span_max_auc": 0.493968253968254
+        },
+        "length": {
+          "all_auc": 0.3753055775432146,
+          "span_auc": NaN,
+          "span_max_auc": 0.4901133786848073,
+          "dilated_span_max_auc": 0.4901133786848073
+        },
+        "regex": {
+          "all_auc": 0.5266721456361196,
+          "span_auc": NaN,
+          "span_max_auc": 0.5289795918367347,
+          "dilated_span_max_auc": 0.5289795918367347
+        }
+      }
+    },
+    {
+      "split_name": "heldout_cwe::CWE-476",
+      "n_train_examples": 1260,
+      "n_test_examples": 114,
+      "note": "train all-except / test only",
+      "all_metrics": {
+        "auc": 0.7939888064061341,
+        "auc_ci_lo": 0.7794760302874192,
+        "auc_ci_hi": 0.8075391673240753,
+        "accuracy": 0.9868738036018908,
+        "precision": 0.14545454545454545,
+        "recall": 0.026936026936026935,
+        "f1": 0.045454545454545456,
+        "recall_at_fpr_0.05": 0.2828282828282828,
+        "recall_at_fpr_0.10": 0.4208754208754209,
+        "brier": 0.016325248841468413,
+        "ece": 0.04692358595426882,
+        "n_total": 76793,
+        "n_pos": 891
+      },
+      "span_metrics": {
+        "auc": NaN,
+        "auc_ci_lo": NaN,
+        "auc_ci_hi": NaN,
+        "accuracy": 0.026936026936026935,
+        "precision": 1.0,
+        "recall": 0.026936026936026935,
+        "f1": 0.05245901639344262,
+        "recall_at_fpr_0.05": NaN,
+        "recall_at_fpr_0.10": NaN,
+        "brier": NaN,
+        "ece": NaN,
+        "n_total": 891,
+        "n_pos": 891
+      },
+      "span_max_metrics": {
+        "auc": 0.4966143428747307,
+        "auc_ci_lo": 0.3856055579994279,
+        "auc_ci_hi": 0.58937441643324,
+        "accuracy": 0.49122807017543857,
+        "precision": 0.49206349206349204,
+        "recall": 0.543859649122807,
+        "f1": 0.5166666666666667,
+        "recall_at_fpr_0.05": 0.07017543859649122,
+        "recall_at_fpr_0.10": 0.08771929824561403,
+        "brier": 0.27859400652564364,
+        "ece": 0.15835563851553094,
+        "n_total": 114,
+        "n_pos": 57
+      },
+      "dilated_span_max_metrics": {
+        "auc": 0.2848568790397045,
+        "auc_ci_lo": 0.19579436414488,
+        "auc_ci_hi": 0.38846064814814807,
+        "accuracy": 0.3508771929824561,
+        "precision": 0.3191489361702128,
+        "recall": 0.2631578947368421,
+        "f1": 0.28846153846153844,
+        "recall_at_fpr_0.05": 0.05263157894736842,
+        "recall_at_fpr_0.10": 0.05263157894736842,
+        "brier": 0.3555935600091588,
+        "ece": 0.3152651175725878,
+        "n_total": 114,
+        "n_pos": 57,
+        "proximity_window": 10
+      },
+      "proximal_all_metrics": {
+        "auc": 0.706185769813078,
+        "auc_ci_lo": 0.6943569467425676,
+        "auc_ci_hi": 0.7169026660329656,
+        "accuracy": 0.9724323831599234,
+        "precision": 0.15151515151515152,
+        "recall": 0.012487512487512488,
+        "f1": 0.023073373327180433,
+        "recall_at_fpr_0.05": 0.1813186813186813,
+        "recall_at_fpr_0.10": 0.2882117882117882,
+        "brier": 0.028357432026591037,
+        "ece": 0.03245612147182901,
+        "n_total": 76793,
+        "n_pos": 2002,
+        "proximity_window": 10
+      },
+      "baseline_aucs": {
+        "random": {
+          "all_auc": 0.48005646479994973,
+          "span_auc": NaN,
+          "span_max_auc": 0.41674361341951377,
+          "dilated_span_max_auc": 0.41674361341951377
+        },
+        "length": {
+          "all_auc": 0.19789939126715494,
+          "span_auc": NaN,
+          "span_max_auc": 0.4786088027085257,
+          "dilated_span_max_auc": 0.4786088027085257
+        },
+        "regex": {
+          "all_auc": 0.47244215109796167,
+          "span_auc": NaN,
+          "span_max_auc": 0.4913819636811326,
+          "dilated_span_max_auc": 0.4913819636811326
+        }
+      }
+    },
+    {
+      "split_name": "heldout_cwe::CWE-079",
+      "n_train_examples": 1274,
+      "n_test_examples": 100,
+      "note": "train all-except / test only",
+      "all_metrics": {
+        "auc": 0.7961208721597384,
+        "auc_ci_lo": 0.7834227354812109,
+        "auc_ci_hi": 0.8079288282043132,
+        "accuracy": 0.9401972521521008,
+        "precision": 0.23905943827563683,
+        "recall": 0.2817551963048499,
+        "f1": 0.25865724381625443,
+        "recall_at_fpr_0.05": 0.3279445727482679,
+        "recall_at_fpr_0.10": 0.45496535796766746,
+        "brier": 0.054103121847219764,
+        "ece": 0.10238288758422436,
+        "n_total": 35082,
+        "n_pos": 1299
+      },
+      "span_metrics": {
+        "auc": NaN,
+        "auc_ci_lo": NaN,
+        "auc_ci_hi": NaN,
+        "accuracy": 0.2817551963048499,
+        "precision": 1.0,
+        "recall": 0.2817551963048499,
+        "f1": 0.43963963963963965,
+        "recall_at_fpr_0.05": NaN,
+        "recall_at_fpr_0.10": NaN,
+        "brier": NaN,
+        "ece": NaN,
+        "n_total": 1299,
+        "n_pos": 1299
+      },
+      "span_max_metrics": {
+        "auc": 0.4918,
+        "auc_ci_lo": 0.3726952144494161,
+        "auc_ci_hi": 0.6134128576014678,
+        "accuracy": 0.51,
+        "precision": 0.5051546391752577,
+        "recall": 0.98,
+        "f1": 0.6666666666666666,
+        "recall_at_fpr_0.05": 0.04,
+        "recall_at_fpr_0.10": 0.1,
+        "brier": 0.3307761475541274,
+        "ece": 0.2843451005220413,
+        "n_total": 100,
+        "n_pos": 50
+      },
+      "dilated_span_max_metrics": {
+        "auc": 0.299,
+        "auc_ci_lo": 0.1935237873558514,
+        "auc_ci_hi": 0.40926758175399475,
+        "accuracy": 0.43,
+        "precision": 0.4606741573033708,
+        "recall": 0.82,
+        "f1": 0.5899280575539568,
+        "recall_at_fpr_0.05": 0.04,
+        "recall_at_fpr_0.10": 0.06,
+        "brier": 0.3714245059320629,
+        "ece": 0.36479666531085975,
+        "n_total": 100,
+        "n_pos": 50,
+        "proximity_window": 10
+      },
+      "proximal_all_metrics": {
+        "auc": 0.6773031324740452,
+        "auc_ci_lo": 0.6661556086234467,
+        "auc_ci_hi": 0.68743133479512,
+        "accuracy": 0.9017444843509492,
+        "precision": 0.2756368386675376,
+        "recall": 0.15289855072463768,
+        "f1": 0.19669074807737125,
+        "recall_at_fpr_0.05": 0.18442028985507247,
+        "recall_at_fpr_0.10": 0.2793478260869565,
+        "brier": 0.08297419726841418,
+        "ece": 0.07019728033199757,
+        "n_total": 35082,
+        "n_pos": 2760,
+        "proximity_window": 10
+      },
+      "baseline_aucs": {
+        "random": {
+          "all_auc": 0.4392386202051189,
+          "span_auc": NaN,
+          "span_max_auc": 0.5764,
+          "dilated_span_max_auc": 0.5764
+        },
+        "length": {
+          "all_auc": 0.3242799667132416,
+          "span_auc": NaN,
+          "span_max_auc": 0.48319999999999996,
+          "dilated_span_max_auc": 0.48319999999999996
+        },
+        "regex": {
+          "all_auc": 0.4780828448707308,
+          "span_auc": NaN,
+          "span_max_auc": 0.5,
+          "dilated_span_max_auc": 0.5
+        }
+      }
+    },
+    {
+      "split_name": "heldout_lang::test=c",
+      "n_train_examples": 836,
+      "n_test_examples": 538,
+      "note": "",
+      "all_metrics": {
+        "auc": 0.7911050077895547,
+        "auc_ci_lo": 0.784401990961081,
+        "auc_ci_hi": 0.7980342177561983,
+        "accuracy": 0.9880962045869698,
+        "precision": 0.11932938856015779,
+        "recall": 0.03086734693877551,
+        "f1": 0.04904742602351034,
+        "recall_at_fpr_0.05": 0.3066326530612245,
+        "recall_at_fpr_0.10": 0.44005102040816324,
+        "brier": 0.015085343476182487,
+        "ece": 0.04787585559604258,
+        "n_total": 394160,
+        "n_pos": 3920
+      },
+      "span_metrics": {
+        "auc": NaN,
+        "auc_ci_lo": NaN,
+        "auc_ci_hi": NaN,
+        "accuracy": 0.03086734693877551,
+        "precision": 1.0,
+        "recall": 0.03086734693877551,
+        "f1": 0.05988616679039842,
+        "recall_at_fpr_0.05": NaN,
+        "recall_at_fpr_0.10": NaN,
+        "brier": NaN,
+        "ece": NaN,
+        "n_total": 3920,
+        "n_pos": 3920
+      },
+      "span_max_metrics": {
+        "auc": 0.4969665980293252,
+        "auc_ci_lo": 0.44710162920261004,
+        "auc_ci_hi": 0.5466605562296141,
+        "accuracy": 0.49070631970260226,
+        "precision": 0.49146757679180886,
+        "recall": 0.5353159851301115,
+        "f1": 0.5124555160142349,
+        "recall_at_fpr_0.05": 0.055762081784386616,
+        "recall_at_fpr_0.10": 0.10037174721189591,
+        "brier": 0.27575060967859194,
+        "ece": 0.12760535849182136,
+        "n_total": 538,
+        "n_pos": 269
+      },
+      "dilated_span_max_metrics": {
+        "auc": 0.2916902751482152,
+        "auc_ci_lo": 0.25058978718317054,
+        "auc_ci_hi": 0.33706220409042725,
+        "accuracy": 0.36617100371747213,
+        "precision": 0.3407079646017699,
+        "recall": 0.2862453531598513,
+        "f1": 0.3111111111111111,
+        "recall_at_fpr_0.05": 0.022304832713754646,
+        "recall_at_fpr_0.10": 0.040892193308550186,
+        "brier": 0.3524953840563277,
+        "ece": 0.30263958890752723,
+        "n_total": 538,
+        "n_pos": 269,
+        "proximity_window": 10
+      },
+      "proximal_all_metrics": {
+        "auc": 0.6973686239207751,
+        "auc_ci_lo": 0.6919642581918721,
+        "auc_ci_hi": 0.7022865497673625,
+        "accuracy": 0.9729678303227116,
+        "precision": 0.15285996055226825,
+        "recall": 0.01557632398753894,
+        "f1": 0.028271773825809393,
+        "recall_at_fpr_0.05": 0.1885237664556326,
+        "recall_at_fpr_0.10": 0.2950457240478344,
+        "brier": 0.027745495450514347,
+        "ece": 0.032574962557682524,
+        "n_total": 394160,
+        "n_pos": 9951,
+        "proximity_window": 10
+      },
+      "baseline_aucs": {
+        "random": {
+          "all_auc": 0.5502099885157015,
+          "span_auc": NaN,
+          "span_max_auc": 0.4988875222841033,
+          "dilated_span_max_auc": 0.4988875222841033
+        },
+        "length": {
+          "all_auc": 0.2651731607080101,
+          "span_auc": NaN,
+          "span_max_auc": 0.4902433631376017,
+          "dilated_span_max_auc": 0.4902433631376017
+        },
+        "regex": {
+          "all_auc": 0.46748228327308783,
+          "span_auc": NaN,
+          "span_max_auc": 0.4999861804010447,
+          "dilated_span_max_auc": 0.4999861804010447
+        }
+      }
+    },
+    {
+      "split_name": "heldout_lang::test=cpp",
+      "n_train_examples": 1298,
+      "n_test_examples": 76,
+      "note": "",
+      "all_metrics": {
+        "auc": 0.7777879380980315,
+        "auc_ci_lo": 0.7601892716131746,
+        "auc_ci_hi": 0.7947345157008497,
+        "accuracy": 0.9676576188424529,
+        "precision": 0.08492201039861352,
+        "recall": 0.07573415765069552,
+        "f1": 0.08006535947712418,
+        "recall_at_fpr_0.05": 0.23802163833075735,
+        "recall_at_fpr_0.10": 0.4034003091190108,
+        "brier": 0.03341856381913668,
+        "ece": 0.08117346346393438,
+        "n_total": 34815,
+        "n_pos": 647
+      },
+      "span_metrics": {
+        "auc": NaN,
+        "auc_ci_lo": NaN,
+        "auc_ci_hi": NaN,
+        "accuracy": 0.07573415765069552,
+        "precision": 1.0,
+        "recall": 0.07573415765069552,
+        "f1": 0.14080459770114942,
+        "recall_at_fpr_0.05": NaN,
+        "recall_at_fpr_0.10": NaN,
+        "brier": NaN,
+        "ece": NaN,
+        "n_total": 647,
+        "n_pos": 647
+      },
+      "span_max_metrics": {
+        "auc": 0.4941135734072022,
+        "auc_ci_lo": 0.36159545494334094,
+        "auc_ci_hi": 0.6232801542776998,
+        "accuracy": 0.5263157894736842,
+        "precision": 0.515625,
+        "recall": 0.868421052631579,
+        "f1": 0.6470588235294118,
+        "recall_at_fpr_0.05": 0.07894736842105263,
+        "recall_at_fpr_0.10": 0.10526315789473684,
+        "brier": 0.30399750452125057,
+        "ece": 0.23132538560189697,
+        "n_total": 76,
+        "n_pos": 38
+      },
+      "dilated_span_max_metrics": {
+        "auc": 0.3071329639889197,
+        "auc_ci_lo": 0.19361253966517122,
+        "auc_ci_hi": 0.4286743622180054,
+        "accuracy": 0.34210526315789475,
+        "precision": 0.38,
+        "recall": 0.5,
+        "f1": 0.4318181818181818,
+        "recall_at_fpr_0.05": 0.02631578947368421,
+        "recall_at_fpr_0.10": 0.05263157894736842,
+        "brier": 0.37104036682973723,
+        "ece": 0.35786413077853235,
+        "n_total": 76,
+        "n_pos": 38,
+        "proximity_window": 10
+      },
+      "proximal_all_metrics": {
+        "auc": 0.6960642540533657,
+        "auc_ci_lo": 0.6842093805351493,
+        "auc_ci_hi": 0.7067377525880977,
+        "accuracy": 0.9403992531954617,
+        "precision": 0.14904679376083188,
+        "recall": 0.05149700598802395,
+        "f1": 0.07654650645304852,
+        "recall_at_fpr_0.05": 0.1592814371257485,
+        "recall_at_fpr_0.10": 0.2868263473053892,
+        "brier": 0.05406197040359917,
+        "ece": 0.051789577208010217,
+        "n_total": 34815,
+        "n_pos": 1670,
+        "proximity_window": 10
+      },
+      "baseline_aucs": {
+        "random": {
+          "all_auc": 0.6549734750050392,
+          "span_auc": NaN,
+          "span_max_auc": 0.6301939058171745,
+          "dilated_span_max_auc": 0.6301939058171745
+        },
+        "length": {
+          "all_auc": 0.24949782183642458,
+          "span_auc": NaN,
+          "span_max_auc": 0.45186980609418287,
+          "dilated_span_max_auc": 0.45186980609418287
+        },
+        "regex": {
+          "all_auc": 0.48104397871124654,
+          "span_auc": NaN,
+          "span_max_auc": 0.5,
+          "dilated_span_max_auc": 0.5
+        }
+      }
+    },
+    {
+      "split_name": "heldout_lang::test=python",
+      "n_train_examples": 614,
+      "n_test_examples": 760,
+      "note": "",
+      "all_metrics": {
+        "auc": 0.8963776320042015,
+        "auc_ci_lo": 0.89249534545576,
+        "auc_ci_hi": 0.900160418168086,
+        "accuracy": 0.9570646410872341,
+        "precision": 0.3768267223382046,
+        "recall": 0.4775132275132275,
+        "f1": 0.42123687281213534,
+        "recall_at_fpr_0.05": 0.5927579365079365,
+        "recall_at_fpr_0.10": 0.7182539682539683,
+        "brier": 0.03954492457438365,
+        "ece": 0.08604482911127222,
+        "n_total": 184836,
+        "n_pos": 6048
+      },
+      "span_metrics": {
+        "auc": NaN,
+        "auc_ci_lo": NaN,
+        "auc_ci_hi": NaN,
+        "accuracy": 0.4775132275132275,
+        "precision": 1.0,
+        "recall": 0.4775132275132275,
+        "f1": 0.6463742166517458,
+        "recall_at_fpr_0.05": NaN,
+        "recall_at_fpr_0.10": NaN,
+        "brier": NaN,
+        "ece": NaN,
+        "n_total": 6048,
+        "n_pos": 6048
+      },
+      "span_max_metrics": {
+        "auc": 0.8238885041551247,
+        "auc_ci_lo": 0.7959389350766638,
+        "auc_ci_hi": 0.8522920367358753,
+        "accuracy": 0.5618421052631579,
+        "precision": 0.5335235378031383,
+        "recall": 0.9842105263157894,
+        "f1": 0.6919518963922294,
+        "recall_at_fpr_0.05": 0.41842105263157897,
+        "recall_at_fpr_0.10": 0.5473684210526316,
+        "brier": 0.2700570808779009,
+        "ece": 0.2839001406866469,
+        "n_total": 760,
+        "n_pos": 380
+      },
+      "dilated_span_max_metrics": {
+        "auc": 0.7807513850415513,
+        "auc_ci_lo": 0.7476400355690892,
+        "auc_ci_hi": 0.8127500223965881,
+        "accuracy": 0.5368421052631579,
+        "precision": 0.5205278592375366,
+        "recall": 0.9342105263157895,
+        "f1": 0.6685499058380414,
+        "recall_at_fpr_0.05": 0.4052631578947368,
+        "recall_at_fpr_0.10": 0.531578947368421,
+        "brier": 0.2820408965538119,
+        "ece": 0.2791484716789503,
+        "n_total": 760,
+        "n_pos": 380,
+        "proximity_window": 10
+      },
+      "proximal_all_metrics": {
+        "auc": 0.7109213914387531,
+        "auc_ci_lo": 0.7066137944258232,
+        "auc_ci_hi": 0.7150740565012443,
+        "accuracy": 0.8961349520656149,
+        "precision": 0.5284446764091858,
+        "recall": 0.20627482937761027,
+        "f1": 0.2967250348010843,
+        "recall_at_fpr_0.05": 0.2993276968523989,
+        "recall_at_fpr_0.10": 0.3945197107059183,
+        "brier": 0.08691457174926079,
+        "ece": 0.04380154398187362,
+        "n_total": 184836,
+        "n_pos": 19634,
+        "proximity_window": 10
+      },
+      "baseline_aucs": {
+        "random": {
+          "all_auc": 0.48545671309835436,
+          "span_auc": NaN,
+          "span_max_auc": 0.4949584487534626,
+          "dilated_span_max_auc": 0.4949584487534626
+        },
+        "length": {
+          "all_auc": 0.3855969544950698,
+          "span_auc": NaN,
+          "span_max_auc": 0.49246883656509693,
+          "dilated_span_max_auc": 0.49246883656509693
+        },
+        "regex": {
+          "all_auc": 0.4966525260201465,
+          "span_auc": NaN,
+          "span_max_auc": 0.5174030470914127,
+          "dilated_span_max_auc": 0.5174030470914127
+        }
+      }
+    },
+    {
+      "split_name": "heldout_source::SVEN-after",
+      "n_train_examples": 687,
+      "n_test_examples": 687,
+      "note": "single-class test (0/687 pos); skipped",
+      "all_metrics": {
+        "auc": NaN,
+        "auc_ci_lo": NaN,
+        "auc_ci_hi": NaN,
+        "accuracy": 0.9890883162387923,
+        "precision": 0.0,
+        "recall": 0.0,
+        "f1": 0.0,
+        "recall_at_fpr_0.05": NaN,
+        "recall_at_fpr_0.10": NaN,
+        "brier": NaN,
+        "ece": NaN,
+        "n_total": 309943,
+        "n_pos": 0
+      },
+      "span_metrics": {
+        "n_total": 0,
+        "n_pos": 0
+      },
+      "span_max_metrics": {
+        "auc": NaN,
+        "auc_ci_lo": NaN,
+        "auc_ci_hi": NaN,
+        "accuracy": 0.26200873362445415,
+        "precision": 0.0,
+        "recall": 0.0,
+        "f1": 0.0,
+        "recall_at_fpr_0.05": NaN,
+        "recall_at_fpr_0.10": NaN,
+        "brier": NaN,
+        "ece": NaN,
+        "n_total": 687,
+        "n_pos": 0
+      },
+      "dilated_span_max_metrics": {
+        "auc": NaN,
+        "auc_ci_lo": NaN,
+        "auc_ci_hi": NaN,
+        "accuracy": 0.26200873362445415,
+        "precision": 0.0,
+        "recall": 0.0,
+        "f1": 0.0,
+        "recall_at_fpr_0.05": NaN,
+        "recall_at_fpr_0.10": NaN,
+        "brier": NaN,
+        "ece": NaN,
+        "n_total": 687,
+        "n_pos": 0,
+        "proximity_window": 10
+      },
+      "proximal_all_metrics": {
+        "auc": NaN,
+        "auc_ci_lo": NaN,
+        "auc_ci_hi": NaN,
+        "accuracy": 0.9890883162387923,
+        "precision": 0.0,
+        "recall": 0.0,
+        "f1": 0.0,
+        "recall_at_fpr_0.05": NaN,
+        "recall_at_fpr_0.10": NaN,
+        "brier": NaN,
+        "ece": NaN,
+        "n_total": 309943,
+        "n_pos": 0,
+        "proximity_window": 10
+      },
+      "baseline_aucs": {
+        "random": {
+          "all_auc": NaN,
+          "span_auc": NaN,
+          "span_max_auc": NaN,
+          "dilated_span_max_auc": NaN
+        },
+        "length": {
+          "all_auc": NaN,
+          "span_auc": NaN,
+          "span_max_auc": NaN,
+          "dilated_span_max_auc": NaN
+        },
+        "regex": {
+          "all_auc": NaN,
+          "span_auc": NaN,
+          "span_max_auc": NaN,
+          "dilated_span_max_auc": NaN
+        }
+      }
+    },
+    {
+      "split_name": "heldout_source::SVEN-before",
+      "n_train_examples": 687,
+      "n_test_examples": 687,
+      "note": "single-class test (687/687 pos); skipped",
+      "all_metrics": {
+        "auc": 0.8609468174926895,
+        "auc_ci_lo": 0.8568614825828509,
+        "auc_ci_hi": 0.8645309035849856,
+        "accuracy": 0.9658667579343663,
+        "precision": 0.5206878937510642,
+        "recall": 0.28808290155440414,
+        "f1": 0.37093643862202813,
+        "recall_at_fpr_0.05": 0.5130475741874706,
+        "recall_at_fpr_0.10": 0.6304286387187942,
+        "brier": 0.03139360386477098,
+        "ece": 0.04647482378408015,
+        "n_total": 303868,
+        "n_pos": 10615
+      },
+      "span_metrics": {
+        "auc": NaN,
+        "auc_ci_lo": NaN,
+        "auc_ci_hi": NaN,
+        "accuracy": 0.28808290155440414,
+        "precision": 1.0,
+        "recall": 0.28808290155440414,
+        "f1": 0.4473049074818986,
+        "recall_at_fpr_0.05": NaN,
+        "recall_at_fpr_0.10": NaN,
+        "brier": NaN,
+        "ece": NaN,
+        "n_total": 10615,
+        "n_pos": 10615
+      },
+      "span_max_metrics": {
+        "auc": NaN,
+        "auc_ci_lo": NaN,
+        "auc_ci_hi": NaN,
+        "accuracy": 0.8020378457059679,
+        "precision": 1.0,
+        "recall": 0.8020378457059679,
+        "f1": 0.8901453957996769,
+        "recall_at_fpr_0.05": NaN,
+        "recall_at_fpr_0.10": NaN,
+        "brier": NaN,
+        "ece": NaN,
+        "n_total": 687,
+        "n_pos": 687
+      },
+      "dilated_span_max_metrics": {
+        "auc": NaN,
+        "auc_ci_lo": NaN,
+        "auc_ci_hi": NaN,
+        "accuracy": 0.6564774381368268,
+        "precision": 1.0,
+        "recall": 0.6564774381368268,
+        "f1": 0.7926186291739895,
+        "recall_at_fpr_0.05": NaN,
+        "recall_at_fpr_0.10": NaN,
+        "brier": NaN,
+        "ece": NaN,
+        "n_total": 687,
+        "n_pos": 687,
+        "proximity_window": 10
+      },
+      "proximal_all_metrics": {
+        "auc": 0.7406192669959343,
+        "auc_ci_lo": 0.737677813720132,
+        "auc_ci_hi": 0.7436982269891117,
+        "accuracy": 0.9060578935590454,
+        "precision": 0.7306317044100119,
+        "recall": 0.13729003359462486,
+        "f1": 0.23114630467571645,
+        "recall_at_fpr_0.05": 0.32721164613661813,
+        "recall_at_fpr_0.10": 0.42857142857142855,
+        "brier": 0.07899467343332146,
+        "ece": 0.02144940647379499,
+        "n_total": 303868,
+        "n_pos": 31255,
+        "proximity_window": 10
+      },
+      "baseline_aucs": {
+        "random": {
+          "all_auc": 0.5080352442815109,
+          "span_auc": NaN,
+          "span_max_auc": NaN,
+          "dilated_span_max_auc": NaN
+        },
+        "length": {
+          "all_auc": 0.2634446423409954,
+          "span_auc": NaN,
+          "span_max_auc": NaN,
+          "dilated_span_max_auc": NaN
+        },
+        "regex": {
+          "all_auc": 0.4747351790086892,
+          "span_auc": NaN,
+          "span_max_auc": NaN,
+          "dilated_span_max_auc": NaN
+        }
+      }
+    }
+  ]
+}

token_report_31b.md

@@ -0,0 +1,108 @@
+# Token-level probe evaluation report
+
+Dataset: `data/dataset.jsonl` (N=1374, pos=687)
+
+## Per-split metrics (W=10 for dilated_span_max)
+
+| split | level | AUC (95% CI) | F1 | R@10%FPR | Brier | n_total | n_pos |
+|---|---|---|---|---|---|---:|---:|
+| `random_stratified` | all | 0.879 (0.872-0.887) | 0.363 | 0.677 | 0.023 | 121499 | 2053 |
+| `random_stratified` | proximal_all | 0.729 (0.722-0.736) | 0.244 | 0.413 | 0.045 | 121499 | 5862 |
+| `random_stratified` | span | nan (nan-nan) | 0.526 | nan | nan | 2053 | 2053 |
+| `random_stratified` | span_max | 0.669 (0.603-0.726) | 0.649 | 0.355 | 0.267 | 275 | 138 |
+| `random_stratified` | dilated_span_max | 0.565 (0.493-0.630) | 0.576 | 0.333 | 0.311 | 275 | 138 |
+| `group_repo` | all | 0.812 (0.804-0.821) | 0.247 | 0.520 | 0.024 | 128710 | 2275 |
+| `group_repo` | proximal_all | 0.714 (0.708-0.720) | 0.172 | 0.375 | 0.047 | 128710 | 6323 |
+| `group_repo` | span | nan (nan-nan) | 0.346 | nan | nan | 2275 | 2275 |
+| `group_repo` | span_max | 0.623 (0.550-0.689) | 0.624 | 0.336 | 0.277 | 274 | 137 |
+| `group_repo` | dilated_span_max | 0.495 (0.421-0.569) | 0.515 | 0.307 | 0.323 | 274 | 137 |
+| `heldout_cwe::CWE-089` | all | 0.960 (0.956-0.964) | 0.546 | 0.894 | 0.028 | 79251 | 2204 |
+| `heldout_cwe::CWE-089` | proximal_all | 0.713 (0.707-0.720) | 0.319 | 0.410 | 0.090 | 79251 | 9348 |
+| `heldout_cwe::CWE-089` | span | nan (nan-nan) | 0.786 | nan | nan | 2204 | 2204 |
+| `heldout_cwe::CWE-089` | span_max | 0.944 (0.923-0.964) | 0.721 | 0.779 | 0.229 | 408 | 204 |
+| `heldout_cwe::CWE-089` | dilated_span_max | 0.941 (0.919-0.962) | 0.719 | 0.770 | 0.230 | 408 | 204 |
+| `heldout_cwe::CWE-125` | all | 0.813 (0.801-0.824) | 0.057 | 0.509 | 0.013 | 158091 | 1335 |
+| `heldout_cwe::CWE-125` | proximal_all | 0.714 (0.704-0.722) | 0.037 | 0.333 | 0.024 | 158091 | 3371 |
+| `heldout_cwe::CWE-125` | span | nan (nan-nan) | 0.071 | nan | nan | 1335 | 1335 |
+| `heldout_cwe::CWE-125` | span_max | 0.512 (0.434-0.585) | 0.518 | 0.085 | 0.273 | 212 | 106 |
+| `heldout_cwe::CWE-125` | dilated_span_max | 0.332 (0.258-0.405) | 0.320 | 0.038 | 0.341 | 212 | 106 |
+| `heldout_cwe::CWE-078` | all | 0.881 (0.873-0.888) | 0.383 | 0.682 | 0.043 | 64711 | 2145 |
+| `heldout_cwe::CWE-078` | proximal_all | 0.748 (0.741-0.755) | 0.313 | 0.435 | 0.080 | 64711 | 6349 |
+| `heldout_cwe::CWE-078` | span | nan (nan-nan) | 0.609 | nan | nan | 2145 | 2145 |
+| `heldout_cwe::CWE-078` | span_max | 0.724 (0.655-0.789) | 0.653 | 0.410 | 0.318 | 210 | 105 |
+| `heldout_cwe::CWE-078` | dilated_span_max | 0.686 (0.613-0.758) | 0.635 | 0.400 | 0.328 | 210 | 105 |
+| `heldout_cwe::CWE-476` | all | 0.794 (0.779-0.808) | 0.045 | 0.421 | 0.016 | 76793 | 891 |
+| `heldout_cwe::CWE-476` | proximal_all | 0.706 (0.694-0.717) | 0.023 | 0.288 | 0.028 | 76793 | 2002 |
+| `heldout_cwe::CWE-476` | span | nan (nan-nan) | 0.052 | nan | nan | 891 | 891 |
+| `heldout_cwe::CWE-476` | span_max | 0.497 (0.386-0.589) | 0.517 | 0.088 | 0.279 | 114 | 57 |
+| `heldout_cwe::CWE-476` | dilated_span_max | 0.285 (0.196-0.388) | 0.288 | 0.053 | 0.356 | 114 | 57 |
+| `heldout_cwe::CWE-079` | all | 0.796 (0.783-0.808) | 0.259 | 0.455 | 0.054 | 35082 | 1299 |
+| `heldout_cwe::CWE-079` | proximal_all | 0.677 (0.666-0.687) | 0.197 | 0.279 | 0.083 | 35082 | 2760 |
+| `heldout_cwe::CWE-079` | span | nan (nan-nan) | 0.440 | nan | nan | 1299 | 1299 |
+| `heldout_cwe::CWE-079` | span_max | 0.492 (0.373-0.613) | 0.667 | 0.100 | 0.331 | 100 | 50 |
+| `heldout_cwe::CWE-079` | dilated_span_max | 0.299 (0.194-0.409) | 0.590 | 0.060 | 0.371 | 100 | 50 |
+| `heldout_lang::test=c` | all | 0.791 (0.784-0.798) | 0.049 | 0.440 | 0.015 | 394160 | 3920 |
+| `heldout_lang::test=c` | proximal_all | 0.697 (0.692-0.702) | 0.028 | 0.295 | 0.028 | 394160 | 9951 |
+| `heldout_lang::test=c` | span | nan (nan-nan) | 0.060 | nan | nan | 3920 | 3920 |
+| `heldout_lang::test=c` | span_max | 0.497 (0.447-0.547) | 0.512 | 0.100 | 0.276 | 538 | 269 |
+| `heldout_lang::test=c` | dilated_span_max | 0.292 (0.251-0.337) | 0.311 | 0.041 | 0.352 | 538 | 269 |
+| `heldout_lang::test=cpp` | all | 0.778 (0.760-0.795) | 0.080 | 0.403 | 0.033 | 34815 | 647 |
+| `heldout_lang::test=cpp` | proximal_all | 0.696 (0.684-0.707) | 0.077 | 0.287 | 0.054 | 34815 | 1670 |
+| `heldout_lang::test=cpp` | span | nan (nan-nan) | 0.141 | nan | nan | 647 | 647 |
+| `heldout_lang::test=cpp` | span_max | 0.494 (0.362-0.623) | 0.647 | 0.105 | 0.304 | 76 | 38 |
+| `heldout_lang::test=cpp` | dilated_span_max | 0.307 (0.194-0.429) | 0.432 | 0.053 | 0.371 | 76 | 38 |
+| `heldout_lang::test=python` | all | 0.896 (0.892-0.900) | 0.421 | 0.718 | 0.040 | 184836 | 6048 |
+| `heldout_lang::test=python` | proximal_all | 0.711 (0.707-0.715) | 0.297 | 0.395 | 0.087 | 184836 | 19634 |
+| `heldout_lang::test=python` | span | nan (nan-nan) | 0.646 | nan | nan | 6048 | 6048 |
+| `heldout_lang::test=python` | span_max | 0.824 (0.796-0.852) | 0.692 | 0.547 | 0.270 | 760 | 380 |
+| `heldout_lang::test=python` | dilated_span_max | 0.781 (0.748-0.813) | 0.669 | 0.532 | 0.282 | 760 | 380 |
+| `heldout_source::SVEN-after` | all | nan (nan-nan) | 0.000 | nan | nan | 309943 | 0 |
+| `heldout_source::SVEN-after` | proximal_all | nan (nan-nan) | 0.000 | nan | nan | 309943 | 0 |
+| `heldout_source::SVEN-after` | span_max | nan (nan-nan) | 0.000 | nan | nan | 687 | 0 |
+| `heldout_source::SVEN-after` | dilated_span_max | nan (nan-nan) | 0.000 | nan | nan | 687 | 0 |
+| `heldout_source::SVEN-before` | all | 0.861 (0.857-0.865) | 0.371 | 0.630 | 0.031 | 303868 | 10615 |
+| `heldout_source::SVEN-before` | proximal_all | 0.741 (0.738-0.744) | 0.231 | 0.429 | 0.079 | 303868 | 31255 |
+| `heldout_source::SVEN-before` | span | nan (nan-nan) | 0.447 | nan | nan | 10615 | 10615 |
+| `heldout_source::SVEN-before` | span_max | nan (nan-nan) | 0.890 | nan | nan | 687 | 687 |
+| `heldout_source::SVEN-before` | dilated_span_max | nan (nan-nan) | 0.793 | nan | nan | 687 | 687 |
+
+## Baselines (AUC at each level)
+
+| split | baseline | all | span | span_max | dilated_span_max |
+|---|---|---:|---:|---:|---:|
+| `random_stratified` | random | 0.406 | nan | 0.515 | 0.515 |
+| `random_stratified` | length | 0.243 | nan | 0.472 | 0.472 |
+| `random_stratified` | regex | 0.497 | nan | 0.483 | 0.483 |
+| `group_repo` | random | 0.468 | nan | 0.471 | 0.471 |
+| `group_repo` | length | 0.301 | nan | 0.489 | 0.489 |
+| `group_repo` | regex | 0.459 | nan | 0.504 | 0.504 |
+| `heldout_cwe::CWE-089` | random | 0.501 | nan | 0.504 | 0.504 |
+| `heldout_cwe::CWE-089` | length | 0.369 | nan | 0.499 | 0.499 |
+| `heldout_cwe::CWE-089` | regex | 0.504 | nan | 0.518 | 0.518 |
+| `heldout_cwe::CWE-125` | random | 0.553 | nan | 0.515 | 0.515 |
+| `heldout_cwe::CWE-125` | length | 0.220 | nan | 0.490 | 0.490 |
+| `heldout_cwe::CWE-125` | regex | 0.498 | nan | 0.500 | 0.500 |
+| `heldout_cwe::CWE-078` | random | 0.531 | nan | 0.494 | 0.494 |
+| `heldout_cwe::CWE-078` | length | 0.375 | nan | 0.490 | 0.490 |
+| `heldout_cwe::CWE-078` | regex | 0.527 | nan | 0.529 | 0.529 |
+| `heldout_cwe::CWE-476` | random | 0.480 | nan | 0.417 | 0.417 |
+| `heldout_cwe::CWE-476` | length | 0.198 | nan | 0.479 | 0.479 |
+| `heldout_cwe::CWE-476` | regex | 0.472 | nan | 0.491 | 0.491 |
+| `heldout_cwe::CWE-079` | random | 0.439 | nan | 0.576 | 0.576 |
+| `heldout_cwe::CWE-079` | length | 0.324 | nan | 0.483 | 0.483 |
+| `heldout_cwe::CWE-079` | regex | 0.478 | nan | 0.500 | 0.500 |
+| `heldout_lang::test=c` | random | 0.550 | nan | 0.499 | 0.499 |
+| `heldout_lang::test=c` | length | 0.265 | nan | 0.490 | 0.490 |
+| `heldout_lang::test=c` | regex | 0.467 | nan | 0.500 | 0.500 |
+| `heldout_lang::test=cpp` | random | 0.655 | nan | 0.630 | 0.630 |
+| `heldout_lang::test=cpp` | length | 0.249 | nan | 0.452 | 0.452 |
+| `heldout_lang::test=cpp` | regex | 0.481 | nan | 0.500 | 0.500 |
+| `heldout_lang::test=python` | random | 0.485 | nan | 0.495 | 0.495 |
+| `heldout_lang::test=python` | length | 0.386 | nan | 0.492 | 0.492 |
+| `heldout_lang::test=python` | regex | 0.497 | nan | 0.517 | 0.517 |
+| `heldout_source::SVEN-after` | random | nan | nan | nan | nan |
+| `heldout_source::SVEN-after` | length | nan | nan | nan | nan |
+| `heldout_source::SVEN-after` | regex | nan | nan | nan | nan |
+| `heldout_source::SVEN-before` | random | 0.508 | nan | nan | nan |
+| `heldout_source::SVEN-before` | length | 0.263 | nan | nan | nan |
+| `heldout_source::SVEN-before` | regex | 0.475 | nan | nan | nan |