A useful stress test would be conflicting updates: after a schedule changes, does the assistant use the newer version consistently across a context reset? That separates retaining a fact from using it correctly. I'm a co-author of Harness RSI; our survey discusses how to evaluate whether retained experience improves later behavior: https://huggingface.co/spaces/lucas1026/harness-rsi-atlas . Have you tried a before/after-reset test with deliberately outdated facts?
Xin lucas-Yu PRO
lucas1026
AI & ML interests
None yet
Recent Activity
updated a Space 2 days ago
lucas1026/harness-rsi-atlas updated a dataset 2 days ago
lucas1026/harness-rsi-evidence published a dataset 4 days ago
lucas1026/mlsbench-agent-trajectoriesOrganizations
replied to JohnVeds15's post 15 days ago
replied to boxiangwanglm's post 15 days ago
I'd prioritize evaluation: a higher final score can hide extra retries or regressions on earlier tasks. For memory, retained-versus-reset experience under the same search budget is a useful test of whether history helps later adaptation. I'm a co-author of Harness RSI; we discuss these distinctions in our survey and source-linked atlas: https://huggingface.co/spaces/lucas1026/harness-rsi-atlas . Which recurring failure is hardest to evaluate in your workflows?
posted an update 15 days ago
Post
109
Does a successful harness edit make the next improvement easier?
Harness RSI studies reusable changes to agent tools, context, memory and execution control. The continual-RSI question is whether retained experience improves later proposals, evaluation efficiency and adaptation—not only the final task score.
Three distinctions matter:
• Retained vs. reset memory: does history help at the same search budget?
• Artifact vs. optimizer transfer: does an unchanged harness help another model, or must optimization restart?
• Held-out transfer vs. target adaptation: which target feedback influenced selection?
Track the evidence chain:
failure → modification → comparison → retained lesson → limits of reuse.
Keep regressions and search costs alongside gains. A better warm start alone does not establish faster subsequent improvement.
Updated November 4: 13 verified papers added; KARMA v2 rechecked. Dataset v0.4.1 contains 145 records across 113 works, 60 AI-assisted primary-source checks and 45 scoped signals. 85 inherited records and zero independent human adjudications remain explicit. Records are not independent effects. The original 95-record domain atlas stays frozen; these are literature observations, not our own experimental results.
The v0.4.1 correction patch fixes benchmark identities and evidence scope; counts are unchanged.
Dataset: lucas1026/harness-rsi-evidence
Paper: https://lucasxinyu.github.io/harness-rsi/paper/Harness_RSI_Preprint_2026-11-04_corrected.pdf
Corrected November 4 author reading version; not peer-reviewed; separate from posted Preprints.org v2.
Atlas: lucas1026/harness-rsi-atlas
Studies and counterexamples: https://huggingface.co/blog/lucas1026/beyond-model-weights-what-can-we-actually-improve
🎬 Attached: 90-second silent overview.
Which retained experience changed your next improvement decision?
#AIAgents #HarnessOptimization #RecursiveSelfImprovement
Harness RSI studies reusable changes to agent tools, context, memory and execution control. The continual-RSI question is whether retained experience improves later proposals, evaluation efficiency and adaptation—not only the final task score.
Three distinctions matter:
• Retained vs. reset memory: does history help at the same search budget?
• Artifact vs. optimizer transfer: does an unchanged harness help another model, or must optimization restart?
• Held-out transfer vs. target adaptation: which target feedback influenced selection?
Track the evidence chain:
failure → modification → comparison → retained lesson → limits of reuse.
Keep regressions and search costs alongside gains. A better warm start alone does not establish faster subsequent improvement.
Updated November 4: 13 verified papers added; KARMA v2 rechecked. Dataset v0.4.1 contains 145 records across 113 works, 60 AI-assisted primary-source checks and 45 scoped signals. 85 inherited records and zero independent human adjudications remain explicit. Records are not independent effects. The original 95-record domain atlas stays frozen; these are literature observations, not our own experimental results.
The v0.4.1 correction patch fixes benchmark identities and evidence scope; counts are unchanged.
Dataset: lucas1026/harness-rsi-evidence
Paper: https://lucasxinyu.github.io/harness-rsi/paper/Harness_RSI_Preprint_2026-11-04_corrected.pdf
Corrected November 4 author reading version; not peer-reviewed; separate from posted Preprints.org v2.
Atlas: lucas1026/harness-rsi-atlas
Studies and counterexamples: https://huggingface.co/blog/lucas1026/beyond-model-weights-what-can-we-actually-improve
🎬 Attached: 90-second silent overview.
Which retained experience changed your next improvement decision?
#AIAgents #HarnessOptimization #RecursiveSelfImprovement