Update model card: align with published paper (arXiv:2602.07670), add selection strategy results, fix metrics

README.md

@@ -23,17 +23,22 @@ model-index:
           name: KernelBench L1
           type: ScalingIntelligence/KernelBench
         metrics:
-          - name: fast_1
+          - name: task_success_rate (K=64, 20 tasks)
             type: custom
-            value: 30.6
-          - name: correctness
+            value: 90.0
+          - name: fast_1 (K=1, per-sample)
+            type: custom
+            value: 53.3
+          - name: correctness (training dist.)
             type: accuracy
-            value: 91.5
+            value: 98.4
 ---
 
 # KernelBench-RLVR-120b
 
-A 120B parameter LLM fine-tuned with GRPO (Group Relative Policy Optimization) for GPU kernel generation, evaluated on KernelBench L1.
+A 120B-parameter model fine-tuned with GRPO (Group Relative Policy Optimization) for GPU kernel generation. This model was used to study compute-optimal test-time strategies in [Surprisal-Guided Selection](http://arxiv.org/abs/2602.07670), where we find that Best-of-N search with surprisal-guided selection recovers oracle performance at zero additional cost.
+
+**Paper**: [arXiv:2602.07670](http://arxiv.org/abs/2602.07670) | **Code**: [GitHub](https://github.com/jbarnes850/test-time-training)
 
 ## Quick Start
 
@@ -75,25 +80,39 @@ This model was trained using an execution-grounded RL framework where:
 - Correctness: 98.4%
 - Mean Speedup: 0.87x on training distribution
 
-**Test-Time Evaluation (3 seeds, 10 tasks):**
+**Best-of-N Search (Full L1 Eval, 20 tasks):**
+- 18/20 tasks (90%) achieve fast_1 = 1 at K=64
+- Performance saturates at K=16 (99.9% on 5-task subsets)
+
+**Selection Strategy Comparison (Subset 1, 5 tasks x 2 seeds):**
+
+| Strategy | fast_1 | std | Mean Speedup |
+|----------|--------|-----|--------------|
+| best-correct (Oracle) | 100% | 0% | 226.9x |
+| **surprisal-guided-top3** | **100%** | **0%** | **139.0x** |
+| **surprisal-guided** | **80%** | **0%** | **41.2x** |
+| random-correct | 59.2% | 2.7% | 30.0x |
+| confidence-guided | 50% | 14.1% | 11.6x |
+
+**Test-Time Training Comparison (Subset 1, 3 seeds):**
 
-| Method | fast_1 | std | Correctness | Rollouts |
-|--------|--------|-----|-------------|----------|
-| **Batch-TTT BoA** | **30.6%** | 11.3% | 91.5% | 960 |
-| **SDPO Prompt-Only** | **30.4%** | 7.6% | 91.9% | 320 |
-| Best-of-N (K=64) | 30.9% | - | 87.2% | 320 |
+| Method | fast_1 | std | Rollouts |
+|--------|--------|-----|----------|
+| Best-of-N (K=64) | 100% | 0% | 320 |
+| Batch-TTT BoA | 30.6% | 11.3% | 960 |
+| SDPO Prompt-Only | 30.4% | 7.6% | 320 |
 
 **Note:** fast_1 = fraction of samples that are both correct AND achieve speedup > 1x.
 
 ## Key Findings
 
-This model was developed as part of research on test-time training efficiency for verifiable execution-grounded tasks. Three key findings emerged:
+This model was developed as part of research on compute-optimal test-time strategies for verifiable execution-grounded (VEG) tasks. Three findings:
 
-1. **Efficiency Frontier**: Performance peaks after 1-2 adaptation steps then regresses, indicating that checkpoint selection (Best-of-Adaptation) rather than extended training drives the benefit.
+1. **Surprisal-guided selection recovers oracle performance.** Selecting the highest-surprisal (lowest log-probability) correct sample achieves 80% fast_1 vs. 50% for confidence-guided (+30pp, Cohen's h = 0.64). Extending to surprisal-guided-top3 matches oracle at 100%. The model's probability distribution maps frequency, not quality. Rare, hardware-optimized kernels occupy the Expert Tail that surprisal recovers at zero cost.
 
-2. **Feedback Redundancy**: Rich tokenized execution feedback provides no lift over prompt-only self-distillation (+4.2% advantage for prompt-only across 3 seeds). When the world provides dense continuous rewards, teacher-based interpretation becomes redundant.
+2. **Search outperforms adaptation.** Best-of-N at K=64 achieves 90% task success (18/20 L1 tasks). TTT's Best-of-Adaptation reaches 30.6% (3-seed mean), with "equivalent K" below 1 -- worse than single-sample inference. The failure mode is over-sharpening: gradient updates collapse diversity toward mediocre solutions.
 
-3. **Regime Dependence**: Adaptation helps when base policy achieves >30% coverage, but hurts performance below that threshold. Search maintains a diversity advantage in hard regimes.
+3. **Feedback redundancy.** SDPO with execution feedback (26.3%) underperforms prompt-only self-distillation (30.4%). When the world provides dense continuous rewards, teacher-based interpretation becomes redundant.
 
 ## Hardware Requirements
 
@@ -134,6 +153,7 @@ Write an optimized CUDA kernel that computes the same result.
 - Hardware-specific optimizations (A100)
 - Extended test-time adaptation may cause regression (use BoA selection with early stopping)
 - Single model size evaluated (120B)
+- Surprisal-guided selection requires sufficient intra-task logprob variance; on 11/20 L1 tasks with near-identical logprobs, all selection strategies perform equivalently
 
 ## Citation