Update logbook: Reproduction: Learning When to Attend: Conditional Memory Access for Long-Context LLMs

logbook.json

@@ -10,7 +10,7 @@
     "icml2026-repro",
     "paper-UcGwFXWnbn"
   ],
-  "updated_at": "2026-07-19T07:52:35+00:00",
+  "updated_at": "2026-07-19T07:58:34+00:00",
   "root": {
     "slug": "index",
     "title": "Reproduction: Learning When to Attend: Conditional Memory Access for Long-Context LLMs",
@@ -18,7 +18,7 @@
     "children": [
       {
         "slug": "executive-summary",
-        "title": "Executive summary",
+        "title": "executive-summary",
         "file": "pages/executive-summary/page.md",
         "children": []
       },
@@ -54,6 +54,6 @@
       }
     ]
   },
-  "agent_view_tokens": 3063,
-  "revision": "1784447555959642300"
+  "agent_view_tokens": 3094,
+  "revision": "1784447914220884500"
 }

pages/claim-2-l2a-skips-global-attention-for-80-of-tokens/page.md

@@ -3,10 +3,8 @@
 
 ---
 <!-- trackio-cell
-{"type": "markdown", "id": "cell_d214eb840028", "created_at": "2026-07-19T07:45:31+00:00", "title": "Claim 2: L2A skips global attention for ~80% of tokens"}
+{"type": "markdown", "id": "cell_ce085a6199ff", "created_at": "2026-07-19T07:57:55+00:00", "title": null}
 -->
-# Claim 2: L2A skips global attention for ~80% of tokens
-
 - **Experiment performed**: Inference tracking over 4 thresholds.
 - **Configuration**: Sparsity metrics recorded via `AHA_SPARSITY_STATS_PATH`.
 - **Evidence**: `sparsity_*.json` files generated under `outputs/eval/token_t*`.

pages/claim-3-l2a-triton-kernels-improve-training-throughput-by-2x/page.md

@@ -3,9 +3,7 @@
 
 ---
 <!-- trackio-cell
-{"type": "markdown", "id": "cell_505221480b30", "created_at": "2026-07-19T07:45:31+00:00", "title": "Claim 3: L2A Triton kernels improve training throughput by 2x"}
+{"type": "markdown", "id": "cell_108c934330ce", "created_at": "2026-07-19T07:57:55+00:00", "title": null}
 -->
-# Claim 3: L2A Triton kernels improve training throughput by 2x
-
 - **Status**: **NOT REPRODUCED**.
 - **Reason for Exclusion**: Triton kernels were omitted from the community reproduction source repository (`jiamingshan/AHA-L2A-Qwen3-1.7B-repro`), and benchmarking live latency speedup fell outside the compute/hardware limits of our budget.

pages/claim-4-sparse-attention-pruning-reduces-kv-cache-by-50/page.md

@@ -3,10 +3,8 @@
 
 ---
 <!-- trackio-cell
-{"type": "markdown", "id": "cell_af98a10a6181", "created_at": "2026-07-19T07:45:31+00:00", "title": "Claim 4: Sparse attention pruning reduces KV-cache by 50%"}
+{"type": "markdown", "id": "cell_4e180cdc3f4c", "created_at": "2026-07-19T07:57:55+00:00", "title": null}
 -->
-# Claim 4: Sparse attention pruning reduces KV-cache by 50%
-
 - **Experiment performed**: Computed mathematically from the measured prefill and decode sparsity statistics.
 - **Result**: Prefill and decode attention usages at $t = 0.525$ are 82.5% and 87.2% respectively, corresponding to an average 15-18% active memory footprint reduction during attention.
 - **Conclusion**: **Partially verified through measured sparsity statistics and analytical estimation. Direct runtime KV-cache measurements were not performed.**

pages/conclusion/page.md

@@ -3,10 +3,8 @@
 
 ---
 <!-- trackio-cell
-{"type": "markdown", "id": "cell_978884749b3c", "created_at": "2026-07-19T07:45:30+00:00", "title": "Reproduction bundle"}
+{"type": "markdown", "id": "cell_4818b62bfaa4", "created_at": "2026-07-19T07:57:54+00:00", "title": null}
 -->
-# Conclusion
-
 The core L2A routing mechanism reproduces qualitatively under a toy-scale configuration, while large-scale efficiency claims (80% sparsity, Triton throughput, 50% KV-cache reduction) remain unverified or partially verified due to the reduced experimental scope.
 
 ### Threats to Validity
@@ -50,7 +48,7 @@ To enforce absolute reproducibility, we indexed this project on **Synthetic Scie
 
 ---
 <!-- trackio-cell
-{"type": "artifact", "id": "cell_055ea7097785", "created_at": "2026-07-19T07:50:12+00:00", "title": "Reproduction bundle", "artifact": "repro-learning-when-to-attend-conditional-memory-access-for-long-context-llms/repro-bundle:v0", "artifact_type": "dataset"}
+{"type": "artifact", "id": "cell_93292fcca299", "created_at": "2026-07-19T07:58:04+00:00", "title": "Artifact: repro-learning-when-to-attend-conditional-memory-access-for-long-context-llms/repro-bundle:v0", "artifact": "repro-learning-when-to-attend-conditional-memory-access-for-long-context-llms/repro-bundle:v0", "artifact_type": "dataset"}
 -->
 **📦 Artifact** `repro-learning-when-to-attend-conditional-memory-access-for-long-context-llms/repro-bundle:v0` · dataset
 

pages/executive-summary/page.md

@@ -1,9 +1,8 @@
-# Executive summary
 
 
 ---
 <!-- trackio-cell
-{"type": "markdown", "id": "cell_bc446c9b150f", "created_at": "2026-07-19T07:45:30+00:00", "title": "Executive summary", "pinned": true, "pinned_at": "2026-07-19T07:45:30+00:00"}
+{"type": "markdown", "id": "cell_738496be8a77", "created_at": "2026-07-19T07:57:54+00:00", "title": "Executive summary", "pinned": true, "pinned_at": "2026-07-19T07:57:54+00:00"}
 -->
 This logbook documents the reproduction of the primary L2A (Learning When to Attend) sparse-attention mechanism on a Qwen3-1.7B backbone model. We reproduced the core routing mechanism and evaluated its behavior under a reduced-scale (toy) configuration. We demonstrate that the per-token head-shared gate variant, when distilled and recovery-fine-tuned over an 8K context length, achieves near-baseline performance compared to the dense vanilla model while dropping up to 29% of attention keys and values. The optimal inference threshold was identified at $t = 0.525$, yielding a mean attention sparsity of 17.88% with high quality retention across RULER, BabiLong, HELMET-ICL, and MRCR. All experiments were executed sequentially on a single A100 GPU and completed under the strict $20.00 budget cap.
 
@@ -28,7 +27,7 @@ This logbook documents the reproduction of the primary L2A (Learning When to Att
 | **Stage 2 (SFT Recovery)** | 2,000 steps | 75 steps (step 25 selected) | 🟡 Substitute (Budget constraint) |
 ---
 <!-- trackio-cell
-{"type": "figure", "id": "cell_4729727ca6f0", "created_at": "2026-07-19T07:45:30+00:00", "title": "Reproduction poster (poster_embed.html)", "pinned": true, "pinned_at": "2026-07-19T07:45:30+00:00"}
+{"type": "figure", "id": "cell_3306605171a7", "created_at": "2026-07-19T07:57:54+00:00", "title": null, "pinned": true, "pinned_at": "2026-07-19T07:57:54+00:00"}
 -->
 ```html
 <p>Build a reproduction poster with <a href="https://github.com/Chenruishuo/posterly">Chenruishuo/posterly</a> and replace this cell with <code>poster_embed.html</code>.</p>
@@ -47,4 +46,4 @@ This logbook documents the reproduction of the primary L2A (Learning When to Att
 |   - Threshold 0.650: 43.70% mean attention sparsity, quality drops below 95%. |
 +-------------------------------------------------------------------------------+
 </pre>
-```
+```