Spaces:
Running
Context Repair Pilot v1: deleting the source was not enough
We tested a narrow question: after an erroneous claim has propagated through several LLM conversation turns, is deleting the original source enough?
The pilot compares five paired context states across four model endpoints, nine task families, and three propagation depths.
Among 72 cases that were correct with clean context and derailed after pollution:
- source deletion repaired 68/72
- dependency-ordered recomputation repaired 71/72
- contaminated-subgraph removal repaired 72/72
The failures were not hidden model memory. The source had been removed, but stale downstream turns still carried its consequences into the next request.
This is not a general model ranking. The tasks are synthetic and objectively scored, with one sample per endpoint at temperature 0 and provider-default reasoning.
Full report:
https://chenxiachan.github.io/thoughtdag/research/context-repair-pilot-v1/
Cases, traces, and scorer:
https://github.com/chenxiachan/thoughtdag/tree/main/benchmark
Next milestone: reproduce the intervention on local models and different quantization levels. Which model and runtime should we test first?