Update logbook: Repro: MemoryBench

logbook.json

@@ -8,7 +8,7 @@
     "icml2026-repro",
     "paper-If4X4W2HWx"
   ],
-  "updated_at": "2026-07-17T12:16:02+00:00",
+  "updated_at": "2026-07-17T12:17:00+00:00",
   "root": {
     "slug": "index",
     "title": "Repro: MemoryBench",
@@ -94,6 +94,6 @@
       }
     ]
   },
-  "agent_view_tokens": 16902,
-  "revision": "1784290562492974300"
+  "agent_view_tokens": 17540,
+  "revision": "1784290620287600400"
 }

pages/evidence-summary/page.md

@@ -34,3 +34,33 @@
 **Label: Static / code audit** -- all three claims verified by source code inspection of the upstream repository. No models were loaded, no inference was run, no datasets were downloaded. Verification is structural, confirming that claimed components exist in the codebase with the described roles and coverage.
 
 **Wall time:** ~15 minutes code inspection | **Cost:** $0.00 | **GPU:** None required
+
+
+---
+<!-- trackio-cell
+{"type": "markdown", "id": "cell_e30b2c264419", "created_at": "2026-07-17T12:16:55+00:00", "title": "FINAL runtime evidence summary (supersedes static-only scope)"}
+-->
+## FINAL RUNTIME EVIDENCE SUMMARY
+
+**This cell supersedes the earlier static-only disclosure on this page.** Claims 1-2 retain their already-awarded structural verification; Claims 3-6 now have runtime and quantitative evidence.
+
+| Claim | Current verdict | Decisive evidence |
+|---|---|---|
+| 1. Three-module framework | REPRODUCED | Source audit plus end-to-end smoke test: task provider, user simulator, and performance monitor load and execute. |
+| 2. 11 datasets / 3 domains / 4 task formats / 2 languages | REPRODUCED | Official HF loader and configs: 28 configurations map to 11 families; runtime LexEval and Locomo loads succeeded. |
+| 3. Declarative/procedural plus explicit/implicit taxonomy | SUPPORTED, bounded comparison | Runtime: 200/200 LexEval train rows have dialogs and implicit records; all 20/20 Locomo rows expose baseline-specific feedback fields. Paper Table 1 comparison covers seven named prior benchmarks. |
+| 4. Advanced memory systems do not consistently beat RAG | SUPPORTED on scaled slice | Across Locomo-0 and LexEval-QA, no A-Mem, Mem0, or MemoryOS result strictly beats the best BM25 RAG. |
+| 5. Large and inconsistent efficiency costs | SUPPORTED on scaled slice | Setup ranges 0.59-189.40 s, prediction 7.00-88.59 s, and paid calls 5-81 for matched five-case cells; robustness failures also vary. |
+| 6. Feedback can improve task performance | SUPPORTED | Paired LexEval ROUGE-L increases 0.04018 to 0.15948; all 5/5 cases improve, mean delta +0.11930. |
+
+### Runtime campaign
+
+- Model: Qwen/Qwen3-8B via HF nscale, non-thinking.
+- Completed Claim 4-5 matrix: 5 systems x 2 datasets = 10 cells.
+- Claim 6: 5 paired cases.
+- Total ledger: 268 calls, USD 0.050595.
+- GPU rental: none.
+- Safety: 500-call cap, USD 0.50 review checkpoint, USD 2 hard cap.
+- Full manifest and SHA-256 inventory are attached on Protocol and artifacts.
+
+The scaled verdicts support claim direction and system behavior; they do not claim exact recovery of full-paper means.