Spaces:
Running
Running
Publish ThoughtDAG benchmark organization card
Browse files
README.md
CHANGED
|
@@ -1,10 +1,51 @@
|
|
| 1 |
---
|
| 2 |
-
title:
|
| 3 |
-
emoji:
|
| 4 |
colorFrom: purple
|
| 5 |
-
colorTo:
|
| 6 |
sdk: static
|
| 7 |
pinned: false
|
|
|
|
| 8 |
---
|
| 9 |
|
| 10 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
title: ThoughtDAG
|
| 3 |
+
emoji: 📊
|
| 4 |
colorFrom: purple
|
| 5 |
+
colorTo: blue
|
| 6 |
sdk: static
|
| 7 |
pinned: false
|
| 8 |
+
short_description: Editable LLM context and intervention benchmarks
|
| 9 |
---
|
| 10 |
|
| 11 |
+
# ThoughtDAG
|
| 12 |
+
|
| 13 |
+
**Visible, editable context for LLM conversations — and reproducible experiments on context intervention.**
|
| 14 |
+
|
| 15 |
+
ThoughtDAG is a local-first, human-in-the-loop research workbench where the graph is not decoration: **wires determine the context sent to the model**. The same representation is used to study what happens when erroneous information enters a conversation, propagates downstream, and is later pruned or recomputed.
|
| 16 |
+
|
| 17 |
+
[Website](https://chenxiachan.github.io/thoughtdag/) · [GitHub](https://github.com/chenxiachan/thoughtdag) · [Interactive demo](https://app.thoughtdag.workers.dev)
|
| 18 |
+
|
| 19 |
+
## Context Intervention Benchmark
|
| 20 |
+
|
| 21 |
+
The benchmark represents each task as a portable ThoughtDAG graph and evaluates five paired conditions:
|
| 22 |
+
|
| 23 |
+
1. **clean** — the correct context chain;
|
| 24 |
+
2. **polluted** — an incorrect claim enters the chain;
|
| 25 |
+
3. **source prune** — the original error is removed while downstream descendants remain;
|
| 26 |
+
4. **subgraph prune** — the source and contaminated descendants are removed;
|
| 27 |
+
5. **recompute descendants** — the source is removed and dependent nodes are regenerated in order.
|
| 28 |
+
|
| 29 |
+
### Context Repair Pilot v1
|
| 30 |
+
|
| 31 |
+
`4 models` · `9 task families` · `3 propagation depths` · `540 conditions`
|
| 32 |
+
|
| 33 |
+
Across 72 cases where clean context was answered correctly and polluted context produced a wrong answer:
|
| 34 |
+
|
| 35 |
+
| Intervention | Repaired cases |
|
| 36 |
+
|---|---:|
|
| 37 |
+
| Remove the erroneous source only | 68 / 72 |
|
| 38 |
+
| Recompute descendants in dependency order | 71 / 72 |
|
| 39 |
+
| Remove the contaminated subgraph | 72 / 72 |
|
| 40 |
+
|
| 41 |
+
This pilot studies **context intervention**, not hidden model memory and not general model quality. Tasks are synthetic and objectively scored; each endpoint was sampled once at temperature zero with provider-default reasoning behavior.
|
| 42 |
+
|
| 43 |
+
[Read the case study](https://chenxiachan.github.io/thoughtdag/stories/context-repair/) · [Full methodology and results](https://chenxiachan.github.io/thoughtdag/research/context-repair-pilot-v1/) · [Browse the benchmark](https://github.com/chenxiachan/thoughtdag/tree/main/benchmark)
|
| 44 |
+
|
| 45 |
+
## What will be published here
|
| 46 |
+
|
| 47 |
+
- **Dataset:** portable graph cases, compiled requests, traces, scores, and provenance.
|
| 48 |
+
- **Space:** an interactive explorer for models, pollution families, propagation depth, and repair strategies.
|
| 49 |
+
- **Updates:** new model runs and new intervention families, versioned rather than folded into an unqualified leaderboard.
|
| 50 |
+
|
| 51 |
+
ThoughtDAG is MIT licensed. Benchmark limitations, model provenance, and scoring rules are published alongside every result.
|