Upload folder using huggingface_hub

sieve_bench/LEADERBOARD.md

@@ -12,30 +12,29 @@ Mean `score_vs_baseline` over the SAME set of CORE readout tasks that EVERY eval
 
 **All-audit-pass intersection (5): t01_lexical_bag, t05_position_order, t08_length_generalization, t13_sae_monosemanticity, t19_crosslingual_readout**
 
-**LIVE intersection — genuinely discriminating, cross-encoder svb std≥0.02 (4): t01_lexical_bag, t05_position_order, t13_sae_monosemanticity, t19_crosslingual_readout**
+**LIVE intersection — genuinely discriminating, cross-encoder svb std≥0.02 (5): t01_lexical_bag, t05_position_order, t08_length_generalization, t13_sae_monosemanticity, t19_crosslingual_readout**
 
 CONSTANT / saturated / non-comparative tasks (≈1.0 for every encoder, or a fixed external grid) are kept in the all-pass column for continuity but are **excluded from the LIVE-task Δ** because they carry no ranking signal. The LIVE-task Δ is the honest comparative headline.
 
 | Encoder | Δ (LIVE tasks, ↑) | Δ (all-pass) | effective_n LIVE / all |
 |---|---|---|---|
-| sonar | 0.770 | 0.816 | 4 / 5 |
-| st-LaBSE-mean | 0.706 | 0.765 | 4 / 5 |
-| st-all-mpnet-base-v2-mean | 0.704 | 0.763 | 4 / 5 |
-| st-bert-base-uncased-mean | 0.671 | 0.737 | 4 / 5 |
-| st-e5-base-v2-mean | 0.723 | 0.778 | 4 / 5 |
-| st-e5-large-v2-mean | 0.684 | 0.747 | 4 / 5 |
-| st-e5-small-v2-mean | 0.677 | 0.741 | 4 / 5 |
-| st-gte-large-mean | 0.724 | 0.779 | 4 / 5 |
-| static-average_word_embeddings_glove.6B.300d | 0.416 | 0.530 | 4 / 5 |
+| sonar | 0.762 | 0.762 | 5 / 5 |
+| st-LaBSE-mean | 0.671 | 0.671 | 5 / 5 |
+| st-all-mpnet-base-v2-mean | 0.704 | 0.704 | 5 / 5 |
+| st-bert-base-uncased-mean | 0.565 | 0.565 | 5 / 5 |
+| st-e5-base-v2-mean | 0.690 | 0.690 | 5 / 5 |
+| st-e5-large-v2-mean | 0.644 | 0.644 | 5 / 5 |
+| st-e5-small-v2-mean | 0.644 | 0.644 | 5 / 5 |
+| st-gte-large-mean | 0.719 | 0.719 | 5 / 5 |
+| static-average_word_embeddings_glove.6B.300d | 0.346 | 0.346 | 5 / 5 |
 
-**LIVE-task ranking:** sonar 0.770 > gte-large 0.724 > e5-base-v2 0.723 > LaBSE 0.706 > all-mpnet-base-v2 0.704 > e5-large-v2 0.684 > e5-small-v2 0.677 > bert-base-uncased 0.671 > static-average_word_embeddings_glove.6B.300d 0.416
+**LIVE-task ranking:** sonar 0.762 > gte-large 0.719 > all-mpnet-base-v2 0.704 > e5-base-v2 0.690 > LaBSE 0.671 > e5-large-v2 0.644 > e5-small-v2 0.644 > bert-base-uncased 0.565 > static-average_word_embeddings_glove.6B.300d 0.346
 
-**LIVE-task spread (max−min) = 0.354.** Lead of top over 2nd = 0.046 (INSIDE the spread).
+**LIVE-task spread (max−min) = 0.416.** Lead of top over 2nd = 0.043 (INSIDE the spread).
 
 ### Tasks in the all-pass set but EXCLUDED from the LIVE Δ (constants / non-comparative)
 
 - **t07_meaning_coverage**: flagged non_comparative (fixed external grid / not encoder-specific) — no cross-encoder ranking signal
-- **t08_length_generalization**: cross-encoder svb std=0.005 < 0.02 (saturated/constant — no ranking signal)
 - **t21_encoder_pooling_generality**: flagged non_comparative (fixed external grid / not encoder-specific) — no cross-encoder ranking signal
 
 ### Tasks excluded from the intersection (not all-encoder / not all-pass)
@@ -74,7 +73,7 @@ Cell = score, with an audit flag: `✓` audit-PASS, `✗` audit-FAIL, `·` skipp
 | t04_negation_scope | core | 1.000✗ | 1.000✗ | 1.000✗ | 1.000✗ | 1.000✗ | 1.000✗ | 1.000✗ | 1.000✗ | 1.000✗ |
 | t05_position_order * | core | 0.840✓ | 0.667✓ | 0.660✓ | 0.798✓ | 0.694✓ | 0.531✓ | 0.512✓ | 0.628✓ | 0.091✓ |
 | t07_meaning_coverage | core | 0.014✗ | · | · | · | · | · | · | · | · |
-| t08_length_generalization * | core | 1.000✓ | 1.000✓ | 1.000✓ | 1.000✓ | 1.000✓ | 1.000✓ | 1.000✓ | 1.000✓ | 0.992✓ |
+| t08_length_generalization * | core | 0.866✓ | 0.771✓ | 0.857✓ | 0.580✓ | 0.785✓ | 0.749✓ | 0.762✓ | 0.853✓ | 0.542✓ |
 | t11_position_unrotation | core | 0.000✗ | 0.000✗ | 0.000✗ | 0.000✓ | 0.000✓ | 0.000✗ | 0.000✓ | 0.000✗ | 0.000✓ |
 | t13_sae_monosemanticity * | core | 0.844✓ | 0.683✓ | 0.960✓ | 0.918✓ | 0.859✓ | 0.820✓ | 0.818✓ | 0.963✓ | 0.767✓ |
 | t14_capacity_law | core | 0.416✓(probe) | 0.000✗(probe) | -0.615✗(probe) | 0.000✗(probe) | 0.000✗(probe) | 0.000✗(probe) | 0.000✗(probe) | 0.000✗(probe) | 0.000✗(probe) |
@@ -177,7 +176,6 @@ Raw bag_cos OVERSTATES additivity (a #-matched RANDOM-word bag already explains
 | t04_negation_scope | A | core | sonar | 1.000 | 0.748 | 1.000 | 1.000 | 1.000 | 1.000 | FAIL(NOT-z-specific; NOT-z-specific) | 0.950 |
 | t05_position_order | A | core | sonar | 0.840 | -0.014 | - | - | 1.000 | 0.842 | PASS | - |
 | t07_meaning_coverage | A | core | sonar | 0.014 | 0.000 | 0.000 | - | 1.000 | 0.014 | FAIL(NOT-z-specific; DEGENERATE) | - |
-| t08_length_generalization | A | core | sonar | 1.000 | 0.500 | 0.000 | 0.000 | 1.000 | 1.000 | PASS | - |
 | t11_position_unrotation | B | core | sonar | 0.000 | 0.000 | - | - | 1.000 | 0.000 | FAIL(DEGENERATE) | - |
 | t13_sae_monosemanticity | B | core | sonar | 0.844 | 0.073 | - | - | 1.000 | 0.832 | PASS | - |
 | t14_capacity_law | B | core | sonar | 0.901 | 0.000 | - | - | 1.000 | 0.901 | PASS | 0.930 |
@@ -190,12 +188,12 @@ Raw bag_cos OVERSTATES additivity (a #-matched RANDOM-word bag already explains
 | t22_word_edit | D | gene | sonar | 1.000 | 1.000 | - | - | 1.000 | 0.000 | FAIL(DEGENERATE) | - |
 | t23_edit_sentence_2of3 | D | gene | sonar | 0.619 | 0.469 | - | - | 1.000 | 0.282 | PASS | - |
 | t25_concept_injection_recovery | D | gene | sonar | 0.744 | 0.787 | - | - | 1.000 | 0.000 | FAIL(DEGENERATE) | - |
+| t08_length_generalization | A | core | sonar | 0.866 | 0.510 | 0.000 | 0.000 | 1.000 | 0.727 | PASS | - |
 | t01_lexical_bag | A | core | st-LaBSE-mean | 0.502 | 0.000 | 1.000 | 0.182 | 1.000 | 0.502 | PASS | - |
 | t02_number_exact | A | core | st-LaBSE-mean | 0.000 | 0.000 | 0.635 | 0.285 | 1.000 | 0.000 | FAIL(NOT-z-specific; NOT-z-specific; DEGENERATE) | 0.990 |
 | t03_entity_presence | A | core | st-LaBSE-mean | 1.000 | 0.500 | 1.000 | 0.876 | 1.000 | 1.000 | FAIL(NOT-z-specific) | 1.000 |
 | t04_negation_scope | A | core | st-LaBSE-mean | 1.000 | 0.748 | 1.000 | 1.000 | 1.000 | 1.000 | FAIL(NOT-z-specific; NOT-z-specific) | 0.950 |
 | t05_position_order | A | core | st-LaBSE-mean | 0.667 | 0.006 | - | - | 1.000 | 0.665 | PASS | - |
-| t08_length_generalization | A | core | st-LaBSE-mean | 1.000 | 0.500 | 0.000 | 0.000 | 1.000 | 1.000 | PASS | - |
 | t11_position_unrotation | B | core | st-LaBSE-mean | 0.000 | 0.000 | - | - | 1.000 | 0.000 | FAIL(DEGENERATE) | - |
 | t13_sae_monosemanticity | B | core | st-LaBSE-mean | 0.683 | 0.076 | - | - | 1.000 | 0.657 | PASS | - |
 | t14_capacity_law | B | core | st-LaBSE-mean | 0.000 | 0.000 | - | - | 1.000 | 0.000 | FAIL(DEGENERATE) | 0.930 |
@@ -203,12 +201,12 @@ Raw bag_cos OVERSTATES additivity (a #-matched RANDOM-word bag already explains
 | t19_crosslingual_readout | E | core | st-LaBSE-mean | 1.000 | 0.500 | - | - | 1.000 | 1.000 | PASS | 1.000 |
 | t21_encoder_pooling_generality | E | core | st-LaBSE-mean | 0.500 | 0.333 | - | - | 1.000 | 0.250 | PASS | 1.000 |
 | t25_concept_injection_recovery | D | gene | st-LaBSE-mean | 0.846 | 0.706 | - | - | 1.000 | 0.477 | PASS | - |
+| t08_length_generalization | A | core | st-LaBSE-mean | 0.771 | 0.510 | 0.000 | 0.000 | 1.000 | 0.533 | PASS | - |
 | t01_lexical_bag | A | core | st-all-mpnet-base-v2-mean | 0.263 | 0.000 | 1.000 | 0.182 | 1.000 | 0.263 | PASS | - |
 | t02_number_exact | A | core | st-all-mpnet-base-v2-mean | 0.000 | 0.000 | 0.635 | 0.285 | 1.000 | 0.000 | FAIL(NOT-z-specific; NOT-z-specific; DEGENERATE) | 0.990 |
 | t03_entity_presence | A | core | st-all-mpnet-base-v2-mean | 1.000 | 0.500 | 1.000 | 0.876 | 1.000 | 1.000 | FAIL(NOT-z-specific) | 1.000 |
 | t04_negation_scope | A | core | st-all-mpnet-base-v2-mean | 1.000 | 0.748 | 1.000 | 1.000 | 1.000 | 1.000 | FAIL(NOT-z-specific; NOT-z-specific) | 0.950 |
 | t05_position_order | A | core | st-all-mpnet-base-v2-mean | 0.660 | -0.005 | - | - | 1.000 | 0.661 | PASS | - |
-| t08_length_generalization | A | core | st-all-mpnet-base-v2-mean | 1.000 | 0.500 | 0.000 | 0.000 | 1.000 | 1.000 | PASS | - |
 | t11_position_unrotation | B | core | st-all-mpnet-base-v2-mean | 0.000 | 0.000 | - | - | 1.000 | 0.000 | FAIL(DEGENERATE) | - |
 | t13_sae_monosemanticity | B | core | st-all-mpnet-base-v2-mean | 0.960 | 0.081 | - | - | 1.000 | 0.957 | PASS | - |
 | t14_capacity_law | B | core | st-all-mpnet-base-v2-mean | -0.615 | 0.000 | - | - | 1.000 | 0.000 | FAIL(DEGENERATE) | 0.930 |
@@ -216,34 +214,34 @@ Raw bag_cos OVERSTATES additivity (a #-matched RANDOM-word bag already explains
 | t19_crosslingual_readout | E | core | st-all-mpnet-base-v2-mean | 0.966 | 0.500 | - | - | 1.000 | 0.933 | PASS | 1.000 |
 | t21_encoder_pooling_generality | E | core | st-all-mpnet-base-v2-mean | 0.400 | 0.333 | - | - | 1.000 | 0.100 | PASS | 1.000 |
 | t25_concept_injection_recovery | D | gene | st-all-mpnet-base-v2-mean | 0.800 | 0.817 | - | - | 1.000 | 0.000 | FAIL(DEGENERATE) | - |
+| t08_length_generalization | A | core | st-all-mpnet-base-v2-mean | 0.857 | 0.510 | 0.000 | 0.000 | 1.000 | 0.707 | PASS | - |
 | t01_lexical_bag | A | core | st-bert-base-uncased-mean | 0.364 | 0.000 | 1.000 | 0.226 | 1.000 | 0.364 | PASS | - |
 | t02_number_exact | A | core | st-bert-base-uncased-mean | 0.000 | 0.000 | 0.635 | 0.285 | 1.000 | 0.000 | FAIL(NOT-z-specific; NOT-z-specific; DEGENERATE) | 0.990 |
 | t03_entity_presence | A | core | st-bert-base-uncased-mean | 0.990 | 0.500 | 1.000 | 0.876 | 1.000 | 0.980 | FAIL(NOT-z-specific) | 1.000 |
 | t04_negation_scope | A | core | st-bert-base-uncased-mean | 1.000 | 0.748 | 1.000 | 1.000 | 1.000 | 1.000 | FAIL(NOT-z-specific; NOT-z-specific) | 0.950 |
 | t05_position_order | A | core | st-bert-base-uncased-mean | 0.798 | -0.024 | - | - | 1.000 | 0.803 | PASS | - |
-| t08_length_generalization | A | core | st-bert-base-uncased-mean | 1.000 | 0.500 | 0.000 | 0.000 | 1.000 | 1.000 | PASS | - |
 | t13_sae_monosemanticity | B | core | st-bert-base-uncased-mean | 0.918 | 0.080 | - | - | 1.000 | 0.911 | PASS | - |
 | t14_capacity_law | B | core | st-bert-base-uncased-mean | 0.000 | 0.000 | - | - | 1.000 | 0.000 | FAIL(DEGENERATE) | 0.930 |
 | t19_crosslingual_readout | E | core | st-bert-base-uncased-mean | 0.803 | 0.500 | - | - | 1.000 | 0.607 | PASS | 1.000 |
 | t21_encoder_pooling_generality | E | core | st-bert-base-uncased-mean | 1.000 | 0.333 | - | - | 1.000 | 1.000 | PASS | 1.000 |
 | t25_concept_injection_recovery | D | gene | st-bert-base-uncased-mean | 0.817 | 0.589 | - | - | 1.000 | 0.555 | PASS | - |
+| t08_length_generalization | A | core | st-bert-base-uncased-mean | 0.580 | 0.510 | 0.000 | 0.000 | 1.000 | 0.142 | PASS | - |
 | t01_lexical_bag | A | core | st-e5-base-v2-mean | 0.381 | 0.000 | 1.000 | 0.226 | 1.000 | 0.381 | PASS | - |
 | t02_number_exact | A | core | st-e5-base-v2-mean | 0.000 | 0.000 | 0.635 | 0.285 | 1.000 | 0.000 | FAIL(NOT-z-specific; NOT-z-specific; DEGENERATE) | 0.990 |
 | t03_entity_presence | A | core | st-e5-base-v2-mean | 1.000 | 0.500 | 1.000 | 0.876 | 1.000 | 1.000 | FAIL(NOT-z-specific) | 1.000 |
 | t04_negation_scope | A | core | st-e5-base-v2-mean | 1.000 | 0.748 | 1.000 | 1.000 | 1.000 | 1.000 | FAIL(NOT-z-specific; NOT-z-specific) | 0.950 |
 | t05_position_order | A | core | st-e5-base-v2-mean | 0.694 | -0.024 | - | - | 1.000 | 0.701 | PASS | - |
-| t08_length_generalization | A | core | st-e5-base-v2-mean | 1.000 | 0.500 | 0.000 | 0.000 | 1.000 | 1.000 | PASS | - |
 | t13_sae_monosemanticity | B | core | st-e5-base-v2-mean | 0.859 | 0.079 | - | - | 1.000 | 0.846 | PASS | - |
 | t14_capacity_law | B | core | st-e5-base-v2-mean | 0.000 | 0.000 | - | - | 1.000 | 0.000 | FAIL(DEGENERATE) | 0.930 |
 | t19_crosslingual_readout | E | core | st-e5-base-v2-mean | 0.981 | 0.500 | - | - | 1.000 | 0.962 | PASS | 1.000 |
 | t21_encoder_pooling_generality | E | core | st-e5-base-v2-mean | 1.000 | 0.333 | - | - | 1.000 | 1.000 | PASS | 1.000 |
 | t25_concept_injection_recovery | D | gene | st-e5-base-v2-mean | 0.884 | 0.593 | - | - | 1.000 | 0.714 | PASS | - |
+| t08_length_generalization | A | core | st-e5-base-v2-mean | 0.785 | 0.510 | 0.000 | 0.000 | 1.000 | 0.561 | PASS | - |
 | t01_lexical_bag | A | core | st-e5-large-v2-mean | 0.403 | 0.000 | 1.000 | 0.182 | 1.000 | 0.403 | PASS | - |
 | t02_number_exact | A | core | st-e5-large-v2-mean | 0.113 | 0.000 | 0.635 | 0.285 | 1.000 | 0.113 | FAIL(NOT-z-specific; NOT-z-specific) | 0.990 |
 | t03_entity_presence | A | core | st-e5-large-v2-mean | 1.000 | 0.500 | 1.000 | 0.876 | 1.000 | 1.000 | FAIL(NOT-z-specific) | 1.000 |
 | t04_negation_scope | A | core | st-e5-large-v2-mean | 1.000 | 0.748 | 1.000 | 1.000 | 1.000 | 1.000 | FAIL(NOT-z-specific; NOT-z-specific) | 0.950 |
 | t05_position_order | A | core | st-e5-large-v2-mean | 0.531 | -0.005 | - | - | 1.000 | 0.533 | PASS | - |
-| t08_length_generalization | A | core | st-e5-large-v2-mean | 1.000 | 0.500 | 0.000 | 0.000 | 1.000 | 1.000 | PASS | - |
 | t11_position_unrotation | B | core | st-e5-large-v2-mean | 0.000 | 0.000 | - | - | 1.000 | 0.000 | FAIL(DEGENERATE) | - |
 | t13_sae_monosemanticity | B | core | st-e5-large-v2-mean | 0.820 | 0.080 | - | - | 1.000 | 0.804 | PASS | - |
 | t14_capacity_law | B | core | st-e5-large-v2-mean | 0.000 | 0.000 | - | - | 1.000 | 0.000 | FAIL(DEGENERATE) | 0.930 |
@@ -251,23 +249,23 @@ Raw bag_cos OVERSTATES additivity (a #-matched RANDOM-word bag already explains
 | t19_crosslingual_readout | E | core | st-e5-large-v2-mean | 0.997 | 0.500 | - | - | 1.000 | 0.994 | PASS | 1.000 |
 | t21_encoder_pooling_generality | E | core | st-e5-large-v2-mean | 0.500 | 0.333 | - | - | 1.000 | 0.250 | PASS | 1.000 |
 | t25_concept_injection_recovery | D | gene | st-e5-large-v2-mean | 0.946 | 0.641 | - | - | 1.000 | 0.851 | PASS | - |
+| t08_length_generalization | A | core | st-e5-large-v2-mean | 0.749 | 0.510 | 0.000 | 0.000 | 1.000 | 0.487 | PASS | - |
 | t01_lexical_bag | A | core | st-e5-small-v2-mean | 0.401 | 0.000 | 1.000 | 0.226 | 1.000 | 0.401 | PASS | - |
 | t02_number_exact | A | core | st-e5-small-v2-mean | 0.000 | 0.000 | 0.635 | 0.285 | 1.000 | 0.000 | FAIL(NOT-z-specific; NOT-z-specific; DEGENERATE) | 0.990 |
 | t03_entity_presence | A | core | st-e5-small-v2-mean | 1.000 | 0.500 | 1.000 | 0.876 | 1.000 | 1.000 | FAIL(NOT-z-specific) | 1.000 |
 | t04_negation_scope | A | core | st-e5-small-v2-mean | 1.000 | 0.748 | 1.000 | 1.000 | 1.000 | 1.000 | FAIL(NOT-z-specific; NOT-z-specific) | 0.950 |
 | t05_position_order | A | core | st-e5-small-v2-mean | 0.512 | -0.024 | - | - | 1.000 | 0.523 | PASS | - |
-| t08_length_generalization | A | core | st-e5-small-v2-mean | 1.000 | 0.500 | 0.000 | 0.000 | 1.000 | 1.000 | PASS | - |
 | t13_sae_monosemanticity | B | core | st-e5-small-v2-mean | 0.818 | 0.080 | - | - | 1.000 | 0.802 | PASS | - |
 | t14_capacity_law | B | core | st-e5-small-v2-mean | 0.000 | 0.000 | - | - | 1.000 | 0.000 | FAIL(DEGENERATE) | 0.930 |
 | t19_crosslingual_readout | E | core | st-e5-small-v2-mean | 0.990 | 0.500 | - | - | 1.000 | 0.980 | PASS | 1.000 |
 | t21_encoder_pooling_generality | E | core | st-e5-small-v2-mean | 1.000 | 0.333 | - | - | 1.000 | 1.000 | PASS | 1.000 |
 | t25_concept_injection_recovery | D | gene | st-e5-small-v2-mean | 0.871 | 0.520 | - | - | 1.000 | 0.730 | PASS | - |
+| t08_length_generalization | A | core | st-e5-small-v2-mean | 0.762 | 0.510 | 0.000 | 0.000 | 1.000 | 0.513 | PASS | - |
 | t01_lexical_bag | A | core | st-gte-large-mean | 0.306 | 0.000 | 1.000 | 0.182 | 1.000 | 0.306 | PASS | - |
 | t02_number_exact | A | core | st-gte-large-mean | 0.340 | 0.000 | 0.635 | 0.285 | 1.000 | 0.340 | FAIL(NOT-z-specific) | 0.990 |
 | t03_entity_presence | A | core | st-gte-large-mean | 1.000 | 0.500 | 1.000 | 0.876 | 1.000 | 1.000 | FAIL(NOT-z-specific) | 1.000 |
 | t04_negation_scope | A | core | st-gte-large-mean | 1.000 | 0.748 | 1.000 | 1.000 | 1.000 | 1.000 | FAIL(NOT-z-specific; NOT-z-specific) | 0.950 |
 | t05_position_order | A | core | st-gte-large-mean | 0.628 | -0.005 | - | - | 1.000 | 0.629 | PASS | - |
-| t08_length_generalization | A | core | st-gte-large-mean | 1.000 | 0.500 | 0.000 | 0.000 | 1.000 | 1.000 | PASS | - |
 | t11_position_unrotation | B | core | st-gte-large-mean | 0.000 | 0.000 | - | - | 1.000 | 0.000 | FAIL(DEGENERATE) | - |
 | t13_sae_monosemanticity | B | core | st-gte-large-mean | 0.963 | 0.080 | - | - | 1.000 | 0.960 | PASS | - |
 | t14_capacity_law | B | core | st-gte-large-mean | 0.000 | 0.000 | - | - | 1.000 | 0.000 | FAIL(DEGENERATE) | 0.930 |
@@ -275,17 +273,18 @@ Raw bag_cos OVERSTATES additivity (a #-matched RANDOM-word bag already explains
 | t19_crosslingual_readout | E | core | st-gte-large-mean | 1.000 | 0.500 | - | - | 1.000 | 1.000 | PASS | 1.000 |
 | t21_encoder_pooling_generality | E | core | st-gte-large-mean | 0.400 | 0.333 | - | - | 1.000 | 0.100 | PASS | 1.000 |
 | t25_concept_injection_recovery | D | gene | st-gte-large-mean | 0.894 | 0.673 | - | - | 1.000 | 0.677 | PASS | - |
+| t08_length_generalization | A | core | st-gte-large-mean | 0.853 | 0.510 | 0.000 | 0.000 | 1.000 | 0.700 | PASS | - |
 | t01_lexical_bag | A | core | static-average_word_embeddings_glove.6B.300d | 0.326 | 0.000 | 1.000 | 0.226 | 1.000 | 0.326 | PASS | - |
 | t02_number_exact | A | core | static-average_word_embeddings_glove.6B.300d | 0.304 | 0.000 | 0.635 | 0.285 | 1.000 | 0.304 | FAIL(NOT-z-specific; NOT-z-specific) | 0.990 |
 | t03_entity_presence | A | core | static-average_word_embeddings_glove.6B.300d | 1.000 | 0.500 | 1.000 | 0.876 | 1.000 | 1.000 | FAIL(NOT-z-specific) | 1.000 |
 | t04_negation_scope | A | core | static-average_word_embeddings_glove.6B.300d | 1.000 | 0.748 | 1.000 | 1.000 | 1.000 | 1.000 | FAIL(NOT-z-specific; NOT-z-specific) | 0.950 |
 | t05_position_order | A | core | static-average_word_embeddings_glove.6B.300d | 0.091 | 0.004 | - | - | 1.000 | 0.088 | PASS | - |
-| t08_length_generalization | A | core | static-average_word_embeddings_glove.6B.300d | 0.992 | 0.500 | 0.000 | 0.000 | 1.000 | 0.984 | PASS | - |
 | t13_sae_monosemanticity | B | core | static-average_word_embeddings_glove.6B.300d | 0.767 | 0.086 | - | - | 1.000 | 0.745 | PASS | - |
 | t14_capacity_law | B | core | static-average_word_embeddings_glove.6B.300d | 0.000 | 0.000 | - | - | 1.000 | 0.000 | FAIL(DEGENERATE) | 0.930 |
 | t19_crosslingual_readout | E | core | static-average_word_embeddings_glove.6B.300d | 0.690 | 0.500 | - | - | 0.875 | 0.506 | PASS | 1.000 |
 | t21_encoder_pooling_generality | E | core | static-average_word_embeddings_glove.6B.300d | 1.000 | 0.333 | - | - | 1.000 | 1.000 | PASS | 1.000 |
 | t25_concept_injection_recovery | D | gene | static-average_word_embeddings_glove.6B.300d | 0.652 | 0.510 | - | - | 1.000 | 0.291 | PASS | - |
+| t08_length_generalization | A | core | static-average_word_embeddings_glove.6B.300d | 0.542 | 0.510 | 0.000 | 0.000 | 1.000 | 0.064 | PASS | - |
 
 ## Non-OK tasks (skipped / error)
 
@@ -365,4 +364,4 @@ Raw bag_cos OVERSTATES additivity (a #-matched RANDOM-word bag already explains
 | t26_causal_identifiability | static-average_word_embeddings_glove.6B.300d | skipped | missing caps {'decode'} |
 
 ## Key finding: interpretability is MULTI-DIMENSIONAL (not a single score)
-Across the 9 encoders, the LIVE discriminating tasks rank encoders almost **independently** — mean pairwise Spearman rank-correlation = **+0.10**. Lexical-bag readout (t01) and SAE-monosemanticity (t13) are even **anti-correlated (−0.67)**: encoders best at reading words out (LaBSE) are worst at having monosemantic features (mpnet/gte top t13), while SONAR leads order + cross-lingual but is mid on lexical. **There is no coherent single "interpretability axis"** — which is why SIEVE reports a per-axis profile + heatmap and refuses a single winner column. See `figs/sieve_multidim_interpretability.png`.
+Across the 9 encoders, the LIVE discriminating tasks rank encoders largely **independently**. With the hardened **t08 length-generalization** now a 5th LIVE comparative axis, the mean pairwise Spearman rank-correlation over the 5 LIVE tasks (t01 lexical-bag, t05 order, t08 length-gen, t13 SAE-monosemanticity, t19 cross-lingual) is **+0.21** — up from **+0.10** over the prior 4 LIVE tasks, but still far below a single coherent axis. Lexical-bag readout (t01) and SAE-monosemanticity (t13) remain **anti-correlated (−0.67)**: encoders best at reading words out (LaBSE) are worst at monosemantic features (mpnet/gte top t13). **t08 leans toward the binding/compositional cluster** — it correlates with cross-lingual t19 (+0.53), order t05 (+0.50) and SAE t13 (+0.47) — yet stays **orthogonal to raw lexical-bag readout t01 (+0.00)**: an encoder reading words out of a bag is no better at carrying that readout across length. The static GloVe mean-pool bag is the clear length-generalization **floor (0.542 ≈ the surface null)** and the MLM-only bert control is next (0.580), while the contrastive/retrieval encoders (sonar 0.866, mpnet 0.857, gte 0.853) and SONAR's fixed-width bottleneck lead — exactly the pre-registered prediction that compositional binding, not bag-readout, drives length-generalization. **There is still no coherent single "interpretability axis"** — SIEVE keeps the per-axis profile + heatmap and refuses a single winner column. See `figs/sieve_multidim_interpretability.png`.

sieve_bench/common/data.py

@@ -744,71 +744,112 @@ def three_sentence_items(seed: int = 0, n: int = 200, smoke: bool = False):
 # serves both tracks.
 # ---------------------------------------------------------------------------
 
-# A8 length-generalization (v0.2 paraphrase-robust rework).
+# A8 length-generalization (v0.3 natural-corpus binding-beyond-bag rework).
 #
-# The old version used a single LITERAL target token ("Amazon"). A bag-of-words
-# probe reads that one token perfectly at ANY length, so score==bag==1.0 and the
-# task was vacuous (it measured token presence, not binding-across-length). The
-# rework makes the CONTENT TARGET a SEMANTIC CONCEPT (financial/monetary content)
-# expressed through MANY paraphrastic trigger words, with the positive trigger
-# vocab DISJOINT between the SHORT and LONG bins. So:
-#   - a literal/bag reader trained on SHORT triggers cannot saturate on LONG
-#     (LONG uses held-out trigger words it never saw) -> bag short->long is weak;
-#   - a surface-position reader is weak (concept can appear at any slot);
-#   - only an encoder that BINDS the concept semantically, robust to length AND
-#     paraphrase, transfers short->long. That transfer-beyond-bag is the signal.
+# HISTORY. v0.1 used one LITERAL token ("Amazon") -> a bag read it at any length
+# so score==bag==1.0 (vacuous). v0.2 used a money-vs-nature CONCEPT with disjoint
+# short/long trigger vocab in 4 fixed TEMPLATES; the bag floor dropped to ~0 (good)
+# but the concept was so cleanly separable in EVERY encoder that the short->long z
+# AUC hit the within-long ceiling for all 9 encoders -> normalized score==1.0 for
+# all (cross-encoder std .005), so the task still carried NO ranking signal and was
+# excluded from the LIVE intersection.
 #
-# Each item: {text, label(1=concept present), length_bin('short'|'long')}.
+# v0.3 keeps the binding-beyond-bag idea but makes the concept LIVE in REAL,
+# lexically-varied corpus context so that LENGTH genuinely DILUTES the concept
+# signal and the short-trained readout direction must GENERALIZE across length:
+#   - target = a paraphrase-robust SEMANTIC concept (FINANCE/MONEY topic), labelled
+#     by a money lexicon, with the POSITIVE trigger vocab DISJOINT short vs long
+#     (short-train words never appear in the long-test bin) -> a bag CANNOT
+#     short->long transfer (held-out words) so the bag floor is ~0.5;
+#   - the concept clause is EMBEDDED inside REAL corpus sentences drawn from the
+#     v2 corpus: SHORT items are a single short corpus sentence carrying (or not)
+#     a short money word; LONG items prepend/append 2-3 unrelated long corpus
+#     sentences so the money word is ONE content token among ~25-40 -> 1/N
+#     dilution. An encoder that BINDS the concept (keeps it salient + in a
+#     length-stable direction) transfers short->long; a length-fragile / pure
+#     averaging encoder sees the direction rotate/dilute and DEGRADES.
+#   - score = short-trained -> LONG z AUC, normalized into [surface-position null,
+#     within-long ceiling]: it measures GENERALIZATION-across-length, not raw
+#     readability. The within-long ceiling factors OUT how hard the diluted long
+#     read is in absolute terms, so the headline is pure length-transfer.
+#
+# Each item: {text, label(1=money concept present), length_bin('short'|'long'),
+#             trigger(the money/other word injected), n_words}.
 
-# concept = MONEY/FINANCE, trigger vocab split so short-train words != long-test.
-_LG_MONEY_SHORT = ["dollars", "salary", "budget", "loan", "profit", "tax"]
+# concept = MONEY/FINANCE. Trigger vocab split so short-train words != long-test
+# -> a literal bag trained on SHORT money words cannot fire on LONG money words.
+_LG_MONEY_SHORT = ["dollars", "salary", "budget", "loan", "profit", "tax",
+                   "wages", "savings"]
 _LG_MONEY_LONG = ["revenue", "investment", "dividend", "mortgage", "earnings",
-                  "funding", "payroll", "subsidy"]
-# negative (non-money) content words, also split short/long.
-_LG_OTHER_SHORT = ["garden", "mountain", "river", "forest", "harbor", "meadow"]
-_LG_OTHER_LONG = ["glacier", "orchard", "canyon", "tundra", "lagoon", "savanna",
-                  "plateau", "estuary"]
-
-_LG_SHORT_TMPL = [
-    "The report noted the {W} clearly.",
-    "Officials discussed the {W} today.",
-    "Analysts mentioned a large {W}.",
-    "The team reviewed the {W} again.",
-]
-_LG_LONG_TMPL = [
-    "After a long and detailed session among the senior staff, the committee "
-    "carefully examined the {W} before any further discussion could continue.",
-    "Despite repeated concerns raised by several independent observers, the "
-    "panel returned once more to the question of the {W} during the meeting.",
-    "While markets remained volatile across most major global regions, the "
-    "delegates spent considerable time deliberating over the {W} in detail.",
-    "Following months of careful negotiation behind closed doors, the working "
-    "group finally turned its attention to the matter of the {W} at length.",
+                  "funding", "payroll", "subsidy", "capital", "pension"]
+# negative (non-money) injected content words, also split short/long, so the
+# NEGATIVE class is ALSO a held-out content word (not just "no money word"):
+# this keeps the bag from solving negatives by the absence of a known word.
+_LG_OTHER_SHORT = ["garden", "mountain", "river", "forest", "harbor", "meadow",
+                   "orchard", "valley"]
+_LG_OTHER_LONG = ["glacier", "canyon", "tundra", "lagoon", "savanna",
+                  "plateau", "estuary", "prairie", "fjord", "marsh"]
+
+# Carrier clauses (short, neutral) into which the trigger word is injected at a
+# varied content slot. Real corpus sentences supply the LONG distractor context.
+_LG_CARRIERS = [
+    "the {W} was mentioned in the notes",
+    "they finally discussed the {W} today",
+    "a report described the {W} in brief",
+    "everyone kept asking about the {W}",
+    "the {W} came up during the review",
+    "she wrote a short line about the {W}",
 ]
 
 
 def length_generalization_set(seed: int = 0, n: int = 2000, smoke: bool = False):
+    """v0.3: money-concept presence embedded in REAL corpus context, binned
+    SHORT vs LONG, with DISJOINT short/long trigger vocab. Probe trains short,
+    tests long; the score measures whether the concept readout GENERALIZES across
+    length (binding-beyond-bag), normalized into the within-long ceiling."""
     fp = _data_path(f"length_gen_{n}_{seed}.json")
     if os.path.exists(fp):
         with open(fp) as f:
             return json.load(f)
     rng = random.Random(seed)
+    # Real corpus distractor sentences for the LONG bin (lexically varied).
+    try:
+        pool = load_corpus(n=max(n, 1500), seed=seed, min_words=8, max_words=22)
+    except Exception:
+        pool = []
+    if len(pool) < 50:
+        # offline fallback: synthetic neutral distractors (still lexically varied)
+        pool = [c.format(W=w) for c in _LG_CARRIERS
+                for w in ("meeting", "weather", "schedule", "building", "journey",
+                          "festival", "machine", "lecture", "harvest", "voyage")]
+    rng.shuffle(pool)
+
+    def _make(label, length_bin):
+        money = (label == 1)
+        if length_bin == "short":
+            words = _LG_MONEY_SHORT if money else _LG_OTHER_SHORT
+        else:
+            words = _LG_MONEY_LONG if money else _LG_OTHER_LONG
+        w = rng.choice(words)
+        clause = rng.choice(_LG_CARRIERS).format(W=w)
+        if length_bin == "short":
+            text = clause[0].upper() + clause[1:] + "."
+        else:
+            # embed the concept clause among 2-3 real corpus distractor sentences
+            k = rng.randint(2, 3)
+            distractors = [pool[rng.randrange(len(pool))] for _ in range(k)]
+            parts = distractors[:]
+            insert = rng.randint(0, len(parts))  # varied position of the clause
+            parts.insert(insert, clause[0].upper() + clause[1:])
+            text = ". ".join(p.rstrip(".") for p in parts) + "."
+        return {"text": text, "label": label, "length_bin": length_bin,
+                "trigger": w, "n_words": len(text.split())}
+
     items = []
     while len(items) < n:
-        # SHORT bin: short triggers + short templates
-        for label, pool, tmplpool in [
-                (1, _LG_MONEY_SHORT, _LG_SHORT_TMPL),
-                (0, _LG_OTHER_SHORT, _LG_SHORT_TMPL)]:
-            w = rng.choice(pool)
-            items.append({"text": rng.choice(tmplpool).format(W=w),
-                          "label": label, "length_bin": "short"})
-        # LONG bin: HELD-OUT triggers (disjoint vocab) + long templates
-        for label, pool, tmplpool in [
-                (1, _LG_MONEY_LONG, _LG_LONG_TMPL),
-                (0, _LG_OTHER_LONG, _LG_LONG_TMPL)]:
-            w = rng.choice(pool)
-            items.append({"text": rng.choice(tmplpool).format(W=w),
-                          "label": label, "length_bin": "long"})
+        for length_bin in ("short", "long"):
+            items.append(_make(1, length_bin))
+            items.append(_make(0, length_bin))
     rng.shuffle(items)
     items = items[:n]
     with open(fp, "w") as f:

sieve_bench/results/sonar.json

@@ -251,49 +251,6 @@
     "task_id": "t07",
     "family": "A"
   },
-  {
-    "task": "t08_length_generalization",
-    "encoder": "sonar",
-    "status": "ok",
-    "tier": "core",
-    "is_diagnostic": false,
-    "caps": [
-      "encode"
-    ],
-    "score": 0.9999999980000001,
-    "baseline": 0.5,
-    "ceiling": 1.0,
-    "control": 0.0,
-    "score_vs_baseline": 0.9999999940000002,
-    "score_over_ceiling": 0.999999997,
-    "target": 0.5,
-    "falsifier": 0.2,
-    "passed": true,
-    "falsified": false,
-    "ref_sonar": null,
-    "interp": "Semantic-concept probe trained on SHORT, tested on LONG with HELD-OUT trigger vocab. score = transfer beyond surface-position, normalized into the within-long ceiling. Bag CANNOT saturate (disjoint vocab); audit FAILS it if bag/surface null still solves it or no headroom.",
-    "raw": {
-      "n_items": 2000,
-      "n_short": 1000,
-      "n_long": 1000,
-      "z_short_to_long_auc": 1.0,
-      "bag_short_to_long_auc": 0.48605769230769236,
-      "surface_pos_short_to_long_auc": 0.5,
-      "within_long_auc": 0.9999999999999999,
-      "within_short_auc": 1.0,
-      "shuffled_control_auc": 0.12256410256410256,
-      "gap": -1.1102230246251565e-16,
-      "note": "score = (z short->long AUC - surface-pos)/(within-long - surface-pos). bag/surf in normalized space; gap = within-long - short->long. Disjoint short/long trigger vocab => bag cannot saturate."
-    },
-    "surface_position_baseline": 0.0,
-    "bag_baseline": 0.0,
-    "audit": {
-      "status": "PASS",
-      "reasons": []
-    },
-    "task_id": "t08",
-    "family": "A"
-  },
   {
     "task": "t09_coreference",
     "encoder": "sonar",
@@ -1362,5 +1319,48 @@
     },
     "task_id": "t06",
     "family": "A"
+  },
+  {
+    "task": "t08_length_generalization",
+    "encoder": "sonar",
+    "status": "ok",
+    "tier": "core",
+    "is_diagnostic": false,
+    "caps": [
+      "encode"
+    ],
+    "score": 0.8664829782390072,
+    "baseline": 0.51035349090269,
+    "ceiling": 1.0,
+    "control": 0.0,
+    "score_vs_baseline": 0.7273195662428835,
+    "score_over_ceiling": 0.8664829773725242,
+    "target": 0.5,
+    "falsifier": 0.2,
+    "passed": true,
+    "falsified": false,
+    "ref_sonar": null,
+    "interp": "Money-concept probe trained on SHORT, tested on LONG (concept embedded in real corpus context; DISJOINT short/long trigger vocab). score = short->long transfer beyond surface-position, normalized into the within-long ceiling -> pure length-GENERALIZATION. Bag CANNOT saturate (held-out long vocab); audit FAILS it if bag/surface solves it / no headroom.",
+    "raw": {
+      "n_items": 2000,
+      "n_short": 1000,
+      "n_long": 1000,
+      "z_short_to_long_auc": 0.8726856667360101,
+      "bag_short_to_long_auc": 0.4861340032964747,
+      "surface_pos_short_to_long_auc": 0.51035349090269,
+      "within_long_auc": 0.9285177065497432,
+      "within_short_auc": 1.0,
+      "shuffled_control_auc": 0.4220447744475204,
+      "gap": 0.05583203981373308,
+      "note": "score = (z short->long AUC - surface-pos)/(within-long - surface-pos). bag/surf in normalized space; gap = within-long - short->long. Concept embedded in real corpus context + disjoint short/long trigger vocab => bag cannot saturate."
+    },
+    "surface_position_baseline": 0.0,
+    "bag_baseline": 0.0,
+    "audit": {
+      "status": "PASS",
+      "reasons": []
+    },
+    "task_id": "t08",
+    "family": "A"
   }
 ]

sieve_bench/results/st-LaBSE-mean.json

@@ -219,49 +219,6 @@
     "task_id": "t07",
     "family": "A"
   },
-  {
-    "task": "t08_length_generalization",
-    "encoder": "st-LaBSE-mean",
-    "status": "ok",
-    "tier": "core",
-    "is_diagnostic": false,
-    "caps": [
-      "encode"
-    ],
-    "score": 0.9999999980000001,
-    "baseline": 0.5,
-    "ceiling": 1.0,
-    "control": 0.08048076906980765,
-    "score_vs_baseline": 0.9999999940000002,
-    "score_over_ceiling": 0.999999997,
-    "target": 0.5,
-    "falsifier": 0.2,
-    "passed": true,
-    "falsified": false,
-    "ref_sonar": null,
-    "interp": "Semantic-concept probe trained on SHORT, tested on LONG with HELD-OUT trigger vocab. score = transfer beyond surface-position, normalized into the within-long ceiling. Bag CANNOT saturate (disjoint vocab); audit FAILS it if bag/surface null still solves it or no headroom.",
-    "raw": {
-      "n_items": 2000,
-      "n_short": 1000,
-      "n_long": 1000,
-      "z_short_to_long_auc": 1.0,
-      "bag_short_to_long_auc": 0.48605769230769236,
-      "surface_pos_short_to_long_auc": 0.5,
-      "within_long_auc": 1.0,
-      "within_short_auc": 1.0,
-      "shuffled_control_auc": 0.5402403846153846,
-      "gap": 0.0,
-      "note": "score = (z short->long AUC - surface-pos)/(within-long - surface-pos). bag/surf in normalized space; gap = within-long - short->long. Disjoint short/long trigger vocab => bag cannot saturate."
-    },
-    "surface_position_baseline": 0.0,
-    "bag_baseline": 0.0,
-    "audit": {
-      "status": "PASS",
-      "reasons": []
-    },
-    "task_id": "t08",
-    "family": "A"
-  },
   {
     "task": "t09_coreference",
     "encoder": "st-LaBSE-mean",
@@ -957,5 +914,48 @@
     },
     "task_id": "t06",
     "family": "A"
+  },
+  {
+    "task": "t08_length_generalization",
+    "encoder": "st-LaBSE-mean",
+    "status": "ok",
+    "tier": "core",
+    "is_diagnostic": false,
+    "caps": [
+      "encode"
+    ],
+    "score": 0.7712365369933855,
+    "baseline": 0.51035349090269,
+    "ceiling": 1.0,
+    "control": 0.0,
+    "score_vs_baseline": 0.5327987450351659,
+    "score_over_ceiling": 0.7712365362221488,
+    "target": 0.5,
+    "falsifier": 0.2,
+    "passed": true,
+    "falsified": false,
+    "ref_sonar": null,
+    "interp": "Money-concept probe trained on SHORT, tested on LONG (concept embedded in real corpus context; DISJOINT short/long trigger vocab). score = short->long transfer beyond surface-position, normalized into the within-long ceiling -> pure length-GENERALIZATION. Bag CANNOT saturate (held-out long vocab); audit FAILS it if bag/surface solves it / no headroom.",
+    "raw": {
+      "n_items": 2000,
+      "n_short": 1000,
+      "n_long": 1000,
+      "z_short_to_long_auc": 0.7619177161511259,
+      "bag_short_to_long_auc": 0.4861340032964747,
+      "surface_pos_short_to_long_auc": 0.51035349090269,
+      "within_long_auc": 0.8365364612504199,
+      "within_short_auc": 1.0,
+      "shuffled_control_auc": 0.4939591301147365,
+      "gap": 0.074618745099294,
+      "note": "score = (z short->long AUC - surface-pos)/(within-long - surface-pos). bag/surf in normalized space; gap = within-long - short->long. Concept embedded in real corpus context + disjoint short/long trigger vocab => bag cannot saturate."
+    },
+    "surface_position_baseline": 0.0,
+    "bag_baseline": 0.0,
+    "audit": {
+      "status": "PASS",
+      "reasons": []
+    },
+    "task_id": "t08",
+    "family": "A"
   }
 ]

sieve_bench/results/st-all-mpnet-base-v2-mean.json

@@ -219,49 +219,6 @@
     "task_id": "t07",
     "family": "A"
   },
-  {
-    "task": "t08_length_generalization",
-    "encoder": "st-all-mpnet-base-v2-mean",
-    "status": "ok",
-    "tier": "core",
-    "is_diagnostic": false,
-    "caps": [
-      "encode"
-    ],
-    "score": 0.9999999980000001,
-    "baseline": 0.5,
-    "ceiling": 1.0,
-    "control": 0.3760897428375642,
-    "score_vs_baseline": 0.9999999940000002,
-    "score_over_ceiling": 0.999999997,
-    "target": 0.5,
-    "falsifier": 0.2,
-    "passed": true,
-    "falsified": false,
-    "ref_sonar": null,
-    "interp": "Semantic-concept probe trained on SHORT, tested on LONG with HELD-OUT trigger vocab. score = transfer beyond surface-position, normalized into the within-long ceiling. Bag CANNOT saturate (disjoint vocab); audit FAILS it if bag/surface null still solves it or no headroom.",
-    "raw": {
-      "n_items": 2000,
-      "n_short": 1000,
-      "n_long": 1000,
-      "z_short_to_long_auc": 1.0,
-      "bag_short_to_long_auc": 0.48605769230769236,
-      "surface_pos_short_to_long_auc": 0.5,
-      "within_long_auc": 1.0,
-      "within_short_auc": 1.0,
-      "shuffled_control_auc": 0.6880448717948718,
-      "gap": 0.0,
-      "note": "score = (z short->long AUC - surface-pos)/(within-long - surface-pos). bag/surf in normalized space; gap = within-long - short->long. Disjoint short/long trigger vocab => bag cannot saturate."
-    },
-    "surface_position_baseline": 0.0,
-    "bag_baseline": 0.0,
-    "audit": {
-      "status": "PASS",
-      "reasons": []
-    },
-    "task_id": "t08",
-    "family": "A"
-  },
   {
     "task": "t09_coreference",
     "encoder": "st-all-mpnet-base-v2-mean",
@@ -942,5 +899,48 @@
     },
     "task_id": "t06",
     "family": "A"
+  },
+  {
+    "task": "t08_length_generalization",
+    "encoder": "st-all-mpnet-base-v2-mean",
+    "status": "ok",
+    "tier": "core",
+    "is_diagnostic": false,
+    "caps": [
+      "encode"
+    ],
+    "score": 0.8565105172004784,
+    "baseline": 0.51035349090269,
+    "ceiling": 1.0,
+    "control": 0.0,
+    "score_vs_baseline": 0.7069529122733762,
+    "score_over_ceiling": 0.8565105163439678,
+    "target": 0.5,
+    "falsifier": 0.2,
+    "passed": true,
+    "falsified": false,
+    "ref_sonar": null,
+    "interp": "Money-concept probe trained on SHORT, tested on LONG (concept embedded in real corpus context; DISJOINT short/long trigger vocab). score = short->long transfer beyond surface-position, normalized into the within-long ceiling -> pure length-GENERALIZATION. Bag CANNOT saturate (held-out long vocab); audit FAILS it if bag/surface solves it / no headroom.",
+    "raw": {
+      "n_items": 2000,
+      "n_short": 1000,
+      "n_long": 1000,
+      "z_short_to_long_auc": 0.9113472340016963,
+      "bag_short_to_long_auc": 0.4861340032964747,
+      "surface_pos_short_to_long_auc": 0.51035349090269,
+      "within_long_auc": 0.9785249075866925,
+      "within_short_auc": 1.0,
+      "shuffled_control_auc": 0.49741562785041044,
+      "gap": 0.06717767358499627,
+      "note": "score = (z short->long AUC - surface-pos)/(within-long - surface-pos). bag/surf in normalized space; gap = within-long - short->long. Concept embedded in real corpus context + disjoint short/long trigger vocab => bag cannot saturate."
+    },
+    "surface_position_baseline": 0.0,
+    "bag_baseline": 0.0,
+    "audit": {
+      "status": "PASS",
+      "reasons": []
+    },
+    "task_id": "t08",
+    "family": "A"
   }
 ]

sieve_bench/results/st-bert-base-uncased-mean.json

@@ -219,49 +219,6 @@
     "task_id": "t07",
     "family": "A"
   },
-  {
-    "task": "t08_length_generalization",
-    "encoder": "st-bert-base-uncased-mean",
-    "status": "ok",
-    "tier": "core",
-    "is_diagnostic": false,
-    "caps": [
-      "encode"
-    ],
-    "score": 0.9999999980000001,
-    "baseline": 0.5,
-    "ceiling": 1.0,
-    "control": 0.08391025624243578,
-    "score_vs_baseline": 0.9999999940000002,
-    "score_over_ceiling": 0.999999997,
-    "target": 0.5,
-    "falsifier": 0.2,
-    "passed": true,
-    "falsified": false,
-    "ref_sonar": null,
-    "interp": "Semantic-concept probe trained on SHORT, tested on LONG with HELD-OUT trigger vocab. score = transfer beyond surface-position, normalized into the within-long ceiling. Bag CANNOT saturate (disjoint vocab); audit FAILS it if bag/surface null still solves it or no headroom.",
-    "raw": {
-      "n_items": 2000,
-      "n_short": 1000,
-      "n_long": 1000,
-      "z_short_to_long_auc": 1.0,
-      "bag_short_to_long_auc": 0.48605769230769236,
-      "surface_pos_short_to_long_auc": 0.5,
-      "within_long_auc": 1.0,
-      "within_short_auc": 1.0,
-      "shuffled_control_auc": 0.5419551282051281,
-      "gap": 0.0,
-      "note": "score = (z short->long AUC - surface-pos)/(within-long - surface-pos). bag/surf in normalized space; gap = within-long - short->long. Disjoint short/long trigger vocab => bag cannot saturate."
-    },
-    "surface_position_baseline": 0.0,
-    "bag_baseline": 0.0,
-    "audit": {
-      "status": "PASS",
-      "reasons": []
-    },
-    "task_id": "t08",
-    "family": "A"
-  },
   {
     "task": "t09_coreference",
     "encoder": "st-bert-base-uncased-mean",
@@ -990,5 +947,48 @@
     },
     "task_id": "t06",
     "family": "A"
+  },
+  {
+    "task": "t08_length_generalization",
+    "encoder": "st-bert-base-uncased-mean",
+    "status": "ok",
+    "tier": "core",
+    "is_diagnostic": false,
+    "caps": [
+      "encode"
+    ],
+    "score": 0.5797783361402702,
+    "baseline": 0.51035349090269,
+    "ceiling": 1.0,
+    "control": 0.0,
+    "score_vs_baseline": 0.14178564291979334,
+    "score_over_ceiling": 0.5797783355604919,
+    "target": 0.5,
+    "falsifier": 0.2,
+    "passed": true,
+    "falsified": false,
+    "ref_sonar": null,
+    "interp": "Money-concept probe trained on SHORT, tested on LONG (concept embedded in real corpus context; DISJOINT short/long trigger vocab). score = short->long transfer beyond surface-position, normalized into the within-long ceiling -> pure length-GENERALIZATION. Bag CANNOT saturate (held-out long vocab); audit FAILS it if bag/surface solves it / no headroom.",
+    "raw": {
+      "n_items": 2000,
+      "n_short": 1000,
+      "n_long": 1000,
+      "z_short_to_long_auc": 0.7183754460642332,
+      "bag_short_to_long_auc": 0.4861340032964747,
+      "surface_pos_short_to_long_auc": 0.51035349090269,
+      "within_long_auc": 0.8691491574786769,
+      "within_short_auc": 1.0,
+      "shuffled_control_auc": 0.47822886495655376,
+      "gap": 0.15077371141444373,
+      "note": "score = (z short->long AUC - surface-pos)/(within-long - surface-pos). bag/surf in normalized space; gap = within-long - short->long. Concept embedded in real corpus context + disjoint short/long trigger vocab => bag cannot saturate."
+    },
+    "surface_position_baseline": 0.0,
+    "bag_baseline": 0.0,
+    "audit": {
+      "status": "PASS",
+      "reasons": []
+    },
+    "task_id": "t08",
+    "family": "A"
   }
 ]

sieve_bench/results/st-e5-base-v2-mean.json

@@ -219,49 +219,6 @@
     "task_id": "t07",
     "family": "A"
   },
-  {
-    "task": "t08_length_generalization",
-    "encoder": "st-e5-base-v2-mean",
-    "status": "ok",
-    "tier": "core",
-    "is_diagnostic": false,
-    "caps": [
-      "encode"
-    ],
-    "score": 0.9999999980000001,
-    "baseline": 0.5,
-    "ceiling": 1.0,
-    "control": 0.1681410253047436,
-    "score_vs_baseline": 0.9999999940000002,
-    "score_over_ceiling": 0.999999997,
-    "target": 0.5,
-    "falsifier": 0.2,
-    "passed": true,
-    "falsified": false,
-    "ref_sonar": null,
-    "interp": "Semantic-concept probe trained on SHORT, tested on LONG with HELD-OUT trigger vocab. score = transfer beyond surface-position, normalized into the within-long ceiling. Bag CANNOT saturate (disjoint vocab); audit FAILS it if bag/surface null still solves it or no headroom.",
-    "raw": {
-      "n_items": 2000,
-      "n_short": 1000,
-      "n_long": 1000,
-      "z_short_to_long_auc": 1.0,
-      "bag_short_to_long_auc": 0.48605769230769236,
-      "surface_pos_short_to_long_auc": 0.5,
-      "within_long_auc": 0.9999999999999999,
-      "within_short_auc": 1.0,
-      "shuffled_control_auc": 0.5840705128205128,
-      "gap": -1.1102230246251565e-16,
-      "note": "score = (z short->long AUC - surface-pos)/(within-long - surface-pos). bag/surf in normalized space; gap = within-long - short->long. Disjoint short/long trigger vocab => bag cannot saturate."
-    },
-    "surface_position_baseline": 0.0,
-    "bag_baseline": 0.0,
-    "audit": {
-      "status": "PASS",
-      "reasons": []
-    },
-    "task_id": "t08",
-    "family": "A"
-  },
   {
     "task": "t09_coreference",
     "encoder": "st-e5-base-v2-mean",
@@ -990,5 +947,48 @@
     },
     "task_id": "t06",
     "family": "A"
+  },
+  {
+    "task": "t08_length_generalization",
+    "encoder": "st-e5-base-v2-mean",
+    "status": "ok",
+    "tier": "core",
+    "is_diagnostic": false,
+    "caps": [
+      "encode"
+    ],
+    "score": 0.7848129272647348,
+    "baseline": 0.51035349090269,
+    "ceiling": 1.0,
+    "control": 0.1703819754211495,
+    "score_vs_baseline": 0.5605256663781798,
+    "score_over_ceiling": 0.7848129264799218,
+    "target": 0.5,
+    "falsifier": 0.2,
+    "passed": true,
+    "falsified": false,
+    "ref_sonar": null,
+    "interp": "Money-concept probe trained on SHORT, tested on LONG (concept embedded in real corpus context; DISJOINT short/long trigger vocab). score = short->long transfer beyond surface-position, normalized into the within-long ceiling -> pure length-GENERALIZATION. Bag CANNOT saturate (held-out long vocab); audit FAILS it if bag/surface solves it / no headroom.",
+    "raw": {
+      "n_items": 2000,
+      "n_short": 1000,
+      "n_long": 1000,
+      "z_short_to_long_auc": 0.8790865884687394,
+      "bag_short_to_long_auc": 0.4861340032964747,
+      "surface_pos_short_to_long_auc": 0.51035349090269,
+      "within_long_auc": 0.9801891472372022,
+      "within_short_auc": 1.0,
+      "shuffled_control_auc": 0.5904050183226385,
+      "gap": 0.10110255876846275,
+      "note": "score = (z short->long AUC - surface-pos)/(within-long - surface-pos). bag/surf in normalized space; gap = within-long - short->long. Concept embedded in real corpus context + disjoint short/long trigger vocab => bag cannot saturate."
+    },
+    "surface_position_baseline": 0.0,
+    "bag_baseline": 0.0,
+    "audit": {
+      "status": "PASS",
+      "reasons": []
+    },
+    "task_id": "t08",
+    "family": "A"
   }
 ]

sieve_bench/results/st-e5-large-v2-mean.json

@@ -218,49 +218,6 @@
     "task_id": "t07",
     "family": "A"
   },
-  {
-    "task": "t08_length_generalization",
-    "encoder": "st-e5-large-v2-mean",
-    "status": "ok",
-    "tier": "core",
-    "is_diagnostic": false,
-    "caps": [
-      "encode"
-    ],
-    "score": 0.9999999979999998,
-    "baseline": 0.5,
-    "ceiling": 1.0,
-    "control": 0.009871794852051204,
-    "score_vs_baseline": 0.9999999939999997,
-    "score_over_ceiling": 0.9999999969999998,
-    "target": 0.5,
-    "falsifier": 0.2,
-    "passed": true,
-    "falsified": false,
-    "ref_sonar": null,
-    "interp": "Semantic-concept probe trained on SHORT, tested on LONG with HELD-OUT trigger vocab. score = transfer beyond surface-position, normalized into the within-long ceiling. Bag CANNOT saturate (disjoint vocab); audit FAILS it if bag/surface null still solves it or no headroom.",
-    "raw": {
-      "n_items": 2000,
-      "n_short": 1000,
-      "n_long": 1000,
-      "z_short_to_long_auc": 0.9999999999999999,
-      "bag_short_to_long_auc": 0.48605769230769236,
-      "surface_pos_short_to_long_auc": 0.5,
-      "within_long_auc": 1.0,
-      "within_short_auc": 1.0,
-      "shuffled_control_auc": 0.5049358974358974,
-      "gap": 1.1102230246251565e-16,
-      "note": "score = (z short->long AUC - surface-pos)/(within-long - surface-pos). bag/surf in normalized space; gap = within-long - short->long. Disjoint short/long trigger vocab => bag cannot saturate."
-    },
-    "surface_position_baseline": 0.0,
-    "bag_baseline": 0.0,
-    "audit": {
-      "status": "PASS",
-      "reasons": []
-    },
-    "task_id": "t08",
-    "family": "A"
-  },
   {
     "task": "t09_coreference",
     "encoder": "st-e5-large-v2-mean",
@@ -956,5 +913,48 @@
     },
     "task_id": "t06",
     "family": "A"
+  },
+  {
+    "task": "t08_length_generalization",
+    "encoder": "st-e5-large-v2-mean",
+    "status": "ok",
+    "tier": "core",
+    "is_diagnostic": false,
+    "caps": [
+      "encode"
+    ],
+    "score": 0.7487559235913531,
+    "baseline": 0.51035349090269,
+    "ceiling": 1.0,
+    "control": 0.0,
+    "score_vs_baseline": 0.4868868209461641,
+    "score_over_ceiling": 0.7487559228425972,
+    "target": 0.5,
+    "falsifier": 0.2,
+    "passed": true,
+    "falsified": false,
+    "ref_sonar": null,
+    "interp": "Money-concept probe trained on SHORT, tested on LONG (concept embedded in real corpus context; DISJOINT short/long trigger vocab). score = short->long transfer beyond surface-position, normalized into the within-long ceiling -> pure length-GENERALIZATION. Bag CANNOT saturate (held-out long vocab); audit FAILS it if bag/surface solves it / no headroom.",
+    "raw": {
+      "n_items": 2000,
+      "n_short": 1000,
+      "n_long": 1000,
+      "z_short_to_long_auc": 0.8655006320910212,
+      "bag_short_to_long_auc": 0.4861340032964747,
+      "surface_pos_short_to_long_auc": 0.51035349090269,
+      "within_long_auc": 0.9846697924501128,
+      "within_short_auc": 1.0,
+      "shuffled_control_auc": 0.5006480933254388,
+      "gap": 0.11916916035909153,
+      "note": "score = (z short->long AUC - surface-pos)/(within-long - surface-pos). bag/surf in normalized space; gap = within-long - short->long. Concept embedded in real corpus context + disjoint short/long trigger vocab => bag cannot saturate."
+    },
+    "surface_position_baseline": 0.0,
+    "bag_baseline": 0.0,
+    "audit": {
+      "status": "PASS",
+      "reasons": []
+    },
+    "task_id": "t08",
+    "family": "A"
   }
 ]

sieve_bench/results/st-e5-small-v2-mean.json

@@ -219,49 +219,6 @@
     "task_id": "t07",
     "family": "A"
   },
-  {
-    "task": "t08_length_generalization",
-    "encoder": "st-e5-small-v2-mean",
-    "status": "ok",
-    "tier": "core",
-    "is_diagnostic": false,
-    "caps": [
-      "encode"
-    ],
-    "score": 0.9999999980000001,
-    "baseline": 0.5,
-    "ceiling": 1.0,
-    "control": 0.0,
-    "score_vs_baseline": 0.9999999940000002,
-    "score_over_ceiling": 0.999999997,
-    "target": 0.5,
-    "falsifier": 0.2,
-    "passed": true,
-    "falsified": false,
-    "ref_sonar": null,
-    "interp": "Semantic-concept probe trained on SHORT, tested on LONG with HELD-OUT trigger vocab. score = transfer beyond surface-position, normalized into the within-long ceiling. Bag CANNOT saturate (disjoint vocab); audit FAILS it if bag/surface null still solves it or no headroom.",
-    "raw": {
-      "n_items": 2000,
-      "n_short": 1000,
-      "n_long": 1000,
-      "z_short_to_long_auc": 1.0,
-      "bag_short_to_long_auc": 0.48605769230769236,
-      "surface_pos_short_to_long_auc": 0.5,
-      "within_long_auc": 0.9999999999999999,
-      "within_short_auc": 1.0,
-      "shuffled_control_auc": 0.4963942307692308,
-      "gap": -1.1102230246251565e-16,
-      "note": "score = (z short->long AUC - surface-pos)/(within-long - surface-pos). bag/surf in normalized space; gap = within-long - short->long. Disjoint short/long trigger vocab => bag cannot saturate."
-    },
-    "surface_position_baseline": 0.0,
-    "bag_baseline": 0.0,
-    "audit": {
-      "status": "PASS",
-      "reasons": []
-    },
-    "task_id": "t08",
-    "family": "A"
-  },
   {
     "task": "t09_coreference",
     "encoder": "st-e5-small-v2-mean",
@@ -990,5 +947,48 @@
     },
     "task_id": "t06",
     "family": "A"
+  },
+  {
+    "task": "t08_length_generalization",
+    "encoder": "st-e5-small-v2-mean",
+    "status": "ok",
+    "tier": "core",
+    "is_diagnostic": false,
+    "caps": [
+      "encode"
+    ],
+    "score": 0.7616208535594768,
+    "baseline": 0.51035349090269,
+    "ceiling": 1.0,
+    "control": 0.0,
+    "score_vs_baseline": 0.5131607342751225,
+    "score_over_ceiling": 0.761620852797856,
+    "target": 0.5,
+    "falsifier": 0.2,
+    "passed": true,
+    "falsified": false,
+    "ref_sonar": null,
+    "interp": "Money-concept probe trained on SHORT, tested on LONG (concept embedded in real corpus context; DISJOINT short/long trigger vocab). score = short->long transfer beyond surface-position, normalized into the within-long ceiling -> pure length-GENERALIZATION. Bag CANNOT saturate (held-out long vocab); audit FAILS it if bag/surface solves it / no headroom.",
+    "raw": {
+      "n_items": 2000,
+      "n_short": 1000,
+      "n_long": 1000,
+      "z_short_to_long_auc": 0.8670688579155398,
+      "bag_short_to_long_auc": 0.4861340032964747,
+      "surface_pos_short_to_long_auc": 0.51035349090269,
+      "within_long_auc": 0.9787169352386743,
+      "within_short_auc": 1.0,
+      "shuffled_control_auc": 0.4911747291609992,
+      "gap": 0.11164807732313453,
+      "note": "score = (z short->long AUC - surface-pos)/(within-long - surface-pos). bag/surf in normalized space; gap = within-long - short->long. Concept embedded in real corpus context + disjoint short/long trigger vocab => bag cannot saturate."
+    },
+    "surface_position_baseline": 0.0,
+    "bag_baseline": 0.0,
+    "audit": {
+      "status": "PASS",
+      "reasons": []
+    },
+    "task_id": "t08",
+    "family": "A"
   }
 ]

sieve_bench/results/st-gte-large-mean.json

@@ -217,49 +217,6 @@
     "task_id": "t07",
     "family": "A"
   },
-  {
-    "task": "t08_length_generalization",
-    "encoder": "st-gte-large-mean",
-    "status": "ok",
-    "tier": "core",
-    "is_diagnostic": false,
-    "caps": [
-      "encode"
-    ],
-    "score": 0.9999999980000001,
-    "baseline": 0.5,
-    "ceiling": 1.0,
-    "control": 0.17596153810961543,
-    "score_vs_baseline": 0.9999999940000002,
-    "score_over_ceiling": 0.999999997,
-    "target": 0.5,
-    "falsifier": 0.2,
-    "passed": true,
-    "falsified": false,
-    "ref_sonar": null,
-    "interp": "Semantic-concept probe trained on SHORT, tested on LONG with HELD-OUT trigger vocab. score = transfer beyond surface-position, normalized into the within-long ceiling. Bag CANNOT saturate (disjoint vocab); audit FAILS it if bag/surface null still solves it or no headroom.",
-    "raw": {
-      "n_items": 2000,
-      "n_short": 1000,
-      "n_long": 1000,
-      "z_short_to_long_auc": 1.0,
-      "bag_short_to_long_auc": 0.48605769230769236,
-      "surface_pos_short_to_long_auc": 0.5,
-      "within_long_auc": 1.0,
-      "within_short_auc": 0.9999999999999999,
-      "shuffled_control_auc": 0.5879807692307693,
-      "gap": 0.0,
-      "note": "score = (z short->long AUC - surface-pos)/(within-long - surface-pos). bag/surf in normalized space; gap = within-long - short->long. Disjoint short/long trigger vocab => bag cannot saturate."
-    },
-    "surface_position_baseline": 0.0,
-    "bag_baseline": 0.0,
-    "audit": {
-      "status": "PASS",
-      "reasons": []
-    },
-    "task_id": "t08",
-    "family": "A"
-  },
   {
     "task": "t09_coreference",
     "encoder": "st-gte-large-mean",
@@ -940,5 +897,48 @@
     },
     "task_id": "t06",
     "family": "A"
+  },
+  {
+    "task": "t08_length_generalization",
+    "encoder": "st-gte-large-mean",
+    "status": "ok",
+    "tier": "core",
+    "is_diagnostic": false,
+    "caps": [
+      "encode"
+    ],
+    "score": 0.8528773511431307,
+    "baseline": 0.51035349090269,
+    "ceiling": 1.0,
+    "control": 0.23700102562583694,
+    "score_vs_baseline": 0.6995329348357229,
+    "score_over_ceiling": 0.8528773502902532,
+    "target": 0.5,
+    "falsifier": 0.2,
+    "passed": true,
+    "falsified": false,
+    "ref_sonar": null,
+    "interp": "Money-concept probe trained on SHORT, tested on LONG (concept embedded in real corpus context; DISJOINT short/long trigger vocab). score = short->long transfer beyond surface-position, normalized into the within-long ceiling -> pure length-GENERALIZATION. Bag CANNOT saturate (held-out long vocab); audit FAILS it if bag/surface solves it / no headroom.",
+    "raw": {
+      "n_items": 2000,
+      "n_short": 1000,
+      "n_long": 1000,
+      "z_short_to_long_auc": 0.9160199068665889,
+      "bag_short_to_long_auc": 0.4861340032964747,
+      "surface_pos_short_to_long_auc": 0.51035349090269,
+      "within_long_auc": 0.9859979837096542,
+      "within_short_auc": 0.9999999999999999,
+      "shuffled_control_auc": 0.6230817237682226,
+      "gap": 0.06997807684306534,
+      "note": "score = (z short->long AUC - surface-pos)/(within-long - surface-pos). bag/surf in normalized space; gap = within-long - short->long. Concept embedded in real corpus context + disjoint short/long trigger vocab => bag cannot saturate."
+    },
+    "surface_position_baseline": 0.0,
+    "bag_baseline": 0.0,
+    "audit": {
+      "status": "PASS",
+      "reasons": []
+    },
+    "task_id": "t08",
+    "family": "A"
   }
 ]

sieve_bench/results/static-average_word_embeddings_glove.6B.300d.json

@@ -218,49 +218,6 @@
     "task_id": "t07",
     "family": "A"
   },
-  {
-    "task": "t08_length_generalization",
-    "encoder": "static-average_word_embeddings_glove.6B.300d",
-    "status": "ok",
-    "tier": "core",
-    "is_diagnostic": false,
-    "caps": [
-      "encode"
-    ],
-    "score": 0.992147433913141,
-    "baseline": 0.5,
-    "ceiling": 1.0,
-    "control": 0.0,
-    "score_vs_baseline": 0.9842948658576923,
-    "score_over_ceiling": 0.9921474329209935,
-    "target": 0.5,
-    "falsifier": 0.2,
-    "passed": true,
-    "falsified": false,
-    "ref_sonar": null,
-    "interp": "Semantic-concept probe trained on SHORT, tested on LONG with HELD-OUT trigger vocab. score = transfer beyond surface-position, normalized into the within-long ceiling. Bag CANNOT saturate (disjoint vocab); audit FAILS it if bag/surface null still solves it or no headroom.",
-    "raw": {
-      "n_items": 2000,
-      "n_short": 1000,
-      "n_long": 1000,
-      "z_short_to_long_auc": 0.9960737179487179,
-      "bag_short_to_long_auc": 0.48605769230769236,
-      "surface_pos_short_to_long_auc": 0.5,
-      "within_long_auc": 1.0,
-      "within_short_auc": 1.0,
-      "shuffled_control_auc": 0.3524038461538461,
-      "gap": 0.003926282051282093,
-      "note": "score = (z short->long AUC - surface-pos)/(within-long - surface-pos). bag/surf in normalized space; gap = within-long - short->long. Disjoint short/long trigger vocab => bag cannot saturate."
-    },
-    "surface_position_baseline": 0.0,
-    "bag_baseline": 0.0,
-    "audit": {
-      "status": "PASS",
-      "reasons": []
-    },
-    "task_id": "t08",
-    "family": "A"
-  },
   {
     "task": "t09_coreference",
     "encoder": "static-average_word_embeddings_glove.6B.300d",
@@ -989,5 +946,48 @@
     },
     "task_id": "t06",
     "family": "A"
+  },
+  {
+    "task": "t08_length_generalization",
+    "encoder": "static-average_word_embeddings_glove.6B.300d",
+    "status": "ok",
+    "tier": "core",
+    "is_diagnostic": false,
+    "caps": [
+      "encode"
+    ],
+    "score": 0.5418366037309732,
+    "baseline": 0.51035349090269,
+    "ceiling": 1.0,
+    "control": 0.0,
+    "score_vs_baseline": 0.06429763549632243,
+    "score_over_ceiling": 0.5418366031891365,
+    "target": 0.5,
+    "falsifier": 0.2,
+    "passed": true,
+    "falsified": false,
+    "ref_sonar": null,
+    "interp": "Money-concept probe trained on SHORT, tested on LONG (concept embedded in real corpus context; DISJOINT short/long trigger vocab). score = short->long transfer beyond surface-position, normalized into the within-long ceiling -> pure length-GENERALIZATION. Bag CANNOT saturate (held-out long vocab); audit FAILS it if bag/surface solves it / no headroom.",
+    "raw": {
+      "n_items": 2000,
+      "n_short": 1000,
+      "n_long": 1000,
+      "z_short_to_long_auc": 0.6814901345793793,
+      "bag_short_to_long_auc": 0.4861340032964747,
+      "surface_pos_short_to_long_auc": 0.51035349090269,
+      "within_long_auc": 0.8261989726520619,
+      "within_short_auc": 1.0,
+      "shuffled_control_auc": 0.49619945272119187,
+      "gap": 0.14470883807268253,
+      "note": "score = (z short->long AUC - surface-pos)/(within-long - surface-pos). bag/surf in normalized space; gap = within-long - short->long. Concept embedded in real corpus context + disjoint short/long trigger vocab => bag cannot saturate."
+    },
+    "surface_position_baseline": 0.0,
+    "bag_baseline": 0.0,
+    "audit": {
+      "status": "PASS",
+      "reasons": []
+    },
+    "task_id": "t08",
+    "family": "A"
   }
 ]

sieve_bench/tasks/t08_length_generalization.py

@@ -1,28 +1,50 @@
-"""A8 length-generalization — core readout, T0 (v0.2 paraphrase-robust rework).
-
-Train a SEMANTIC-concept probe (money/finance present) on SHORT sentences, test
-on LONG sentences whose trigger vocab is HELD OUT (disjoint from the short
-triggers). Does the readout BIND the concept beyond a literal bag AND across
-length (compositional / dilution-robust)?
-
-The old version used one literal token ("Amazon") so a bag read it perfectly at
-any length -> score==baseline==1.0, vacuous. The rework (a) makes the target a
-paraphrase-robust concept with DISJOINT short/long trigger vocab so a literal
-bag CANNOT saturate, and (b) sets the baseline to the SURFACE-POSITION null
-(~0.5) and scores the transfer-beyond-surface, normalized into the within-long
-ceiling. We also report the length-generalization GAP = within-long − short→long.
+"""A8 length-generalization — core readout, T0 (v0.3 natural-corpus rework).
+
+Train a MONEY/FINANCE concept probe on SHORT sentences (a short corpus-style
+clause carrying, or not, a money word), test on LONG sentences (the same concept
+clause EMBEDDED among 2-3 real, unrelated corpus sentences so the money token is
+ONE content word among ~25-40 -> 1/N dilution). The positive trigger vocab is
+DISJOINT short vs long, so a literal bag trained on the SHORT money words CANNOT
+fire on the held-out LONG money words. The question: does the readout BIND the
+concept beyond a literal bag AND GENERALIZE the readout direction across length?
+
+Why the old versions saturated:
+  v0.1 used one literal token ("Amazon") -> a bag read it at any length, score==1.
+  v0.2 used a money-vs-nature concept in 4 fixed TEMPLATES -> the concept was so
+       cleanly separable in every encoder that short->long z AUC hit the
+       within-long ceiling for ALL 9 (normalized score==1.0, cross-enc std .005),
+       so it carried no ranking signal and was excluded from the LIVE intersection.
+  v0.3 embeds the concept in REAL, lexically-varied corpus context so LENGTH
+       genuinely DILUTES the signal: a binding encoder transfers short->long; a
+       length-fragile / pure-averaging one sees the direction rotate/dilute and
+       DEGRADES -> the score now SPREADS across encoders.
 
   score    = short-trained -> LONG z AUC, normalized into
-             [surface_position_baseline, within-long ceiling] (binding beyond
-             surface, robust to length+paraphrase). Raw AUC in raw.
+             [surface-position null, within-long ceiling]. Measures
+             GENERALIZATION across length (binding-beyond-bag), NOT raw
+             readability: the within-long ceiling factors out absolute difficulty
+             so the headline is pure length-transfer. Raw AUCs in raw.
   baseline = surface-position short->long AUC (the null we must beat)
-  ceiling  = WITHIN-LONG z AUC (probe trained AND tested on long)
-  control  = shuffled-label null (short-train -> long) AUC      -> ~0.5
-  bag_baseline               = bag-of-words short->long AUC (held-out vocab ->
-                               weak; if it saturates, the audit FAILS the task)
-  surface_position_baseline  = fixed-slot word features short->long AUC
-  raw.gap                    = within-long ceiling − short→long z AUC
-  raw.*_auc                  = the raw AUCs behind the normalized score
+  ceiling  = WITHIN-LONG z AUC (probe trained AND tested on long)  -> headroom
+  control  = shuffled-label null (short-train -> long) AUC          -> ~0.5
+  bag_baseline               = bag-of-words short->long AUC. Held-out LONG money
+                               vocab => the bag CANNOT transfer -> ~0.5 (floor).
+  surface_position_baseline  = fixed-slot word features short->long AUC -> ~0.5
+  raw.gap                    = within-long ceiling - short->long z AUC (the
+                               length-generalization GAP; 0 = perfect transfer)
+
+PRE-REGISTRATION.
+  Prediction: encoders trained with sentence-level contrastive / retrieval
+    objectives on varied-length data (gte, e5, mpnet, LaBSE, SONAR) GENERALIZE
+    the concept readout across length better than a static mean-of-GloVe bag,
+    which has NO compositional binding and dilutes the concept ~1/N, so it should
+    be the WORST length-generalizer. SONAR (fixed-width bottleneck, length-robust
+    by design) should be at or near the top.
+  Falsifier: if the task is STILL saturated (cross-encoder std < 0.02) or the
+    bag/surface null still solves it, the task is STILL BROKEN -> report so
+    honestly rather than forcing a spread. If instead every encoder generalizes
+    EQUALLY well at high score (std < 0.02 but scores high), the honest finding
+    is "length-generalization of this concept is UNIVERSAL".
 
 A literal-entity bag can no longer saturate this; the audit rejects it if the
 bag/surface null still solves it or if there is no headroom.
@@ -41,10 +63,11 @@ IS_DIAGNOSTIC = False
 TARGET = 0.5
 FALSIFIER = 0.2
 REF_SONAR = None
-INTERP = ("Semantic-concept probe trained on SHORT, tested on LONG with HELD-OUT "
-          "trigger vocab. score = transfer beyond surface-position, normalized "
-          "into the within-long ceiling. Bag CANNOT saturate (disjoint vocab); "
-          "audit FAILS it if bag/surface null still solves it or no headroom.")
+INTERP = ("Money-concept probe trained on SHORT, tested on LONG (concept embedded "
+          "in real corpus context; DISJOINT short/long trigger vocab). score = "
+          "short->long transfer beyond surface-position, normalized into the "
+          "within-long ceiling -> pure length-GENERALIZATION. Bag CANNOT saturate "
+          "(held-out long vocab); audit FAILS it if bag/surface solves it / no headroom.")
 
 
 def build_data(seed: int = 0, n: int = None, smoke: bool = False):
@@ -67,13 +90,15 @@ def run(encoder, data):
     short_idx = np.where(is_short)[0]
     long_idx = np.where(~is_short)[0]
 
-    # within-long ceiling: split long into train/test
+    # within-long ceiling: split long into train/test; the short->long probe is
+    # evaluated on the SAME held-out long test set (l_te) so ceiling and transfer
+    # are measured on identical items.
     rng = np.random.RandomState(seed)
     perm_l = rng.permutation(long_idx)
     cut = int(len(perm_l) * 0.5)
     l_tr, l_te = perm_l[:cut], perm_l[cut:]
 
-    # short-trained -> evaluate on the SAME held-out long test set (l_te)
+    # short-trained -> evaluate on the held-out long test set (l_te)
     z_auc = _probe.logistic_auc(Z[short_idx], y[short_idx], Z[l_te], y[l_te])
     bag_b = _probe.logistic_auc(Xbag[short_idx], y[short_idx], Xbag[l_te], y[l_te])
     pos_b = _probe.logistic_auc(Xpos[short_idx], y[short_idx], Xpos[l_te], y[l_te])
@@ -82,15 +107,15 @@ def run(encoder, data):
 
     within_long = _probe.logistic_auc(Z[l_tr], y[l_tr], Z[l_te], y[l_te])
 
-    # sanity: within-short AUC
+    # sanity: within-short AUC (probe trained AND tested on short)
     perm_s = rng.permutation(short_idx)
     cs = int(len(perm_s) * 0.6)
     within_short = _probe.logistic_auc(Z[perm_s[:cs]], y[perm_s[:cs]],
                                        Z[perm_s[cs:]], y[perm_s[cs:]])
 
     # SCORE = transfer beyond the surface-position null, normalized into the
-    # within-long ceiling. baseline = surface-position (NOT bag), so the
-    # headline measures binding-beyond-surface that survives length+paraphrase.
+    # within-long ceiling. baseline = surface-position (NOT bag), so the headline
+    # measures length-GENERALIZATION (binding-beyond-surface that survives length).
     base = pos_b
     ceil = max(within_long, z_auc, base + 1e-3)
     score = float(np.clip((z_auc - base) / (ceil - base + 1e-9), 0.0, 1.0))
@@ -115,7 +140,7 @@ def run(encoder, data):
             "gap": float(within_long - z_auc),
             "note": ("score = (z short->long AUC - surface-pos)/(within-long - "
                      "surface-pos). bag/surf in normalized space; gap = within-long "
-                     "- short->long. Disjoint short/long trigger vocab => bag cannot "
-                     "saturate."),
+                     "- short->long. Concept embedded in real corpus context + "
+                     "disjoint short/long trigger vocab => bag cannot saturate."),
         },
     }