Chatchan commited on
Commit
91aa544
·
verified ·
1 Parent(s): 3cfe29f

Publish ThoughtDAG benchmark organization card

Browse files
Files changed (1) hide show
  1. README.md +45 -4
README.md CHANGED
@@ -1,10 +1,51 @@
1
  ---
2
- title: README
3
- emoji: 🐢
4
  colorFrom: purple
5
- colorTo: gray
6
  sdk: static
7
  pinned: false
 
8
  ---
9
 
10
- Edit this `README.md` markdown file to author your organization card.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ title: ThoughtDAG
3
+ emoji: 📊
4
  colorFrom: purple
5
+ colorTo: blue
6
  sdk: static
7
  pinned: false
8
+ short_description: Editable LLM context and intervention benchmarks
9
  ---
10
 
11
+ # ThoughtDAG
12
+
13
+ **Visible, editable context for LLM conversations — and reproducible experiments on context intervention.**
14
+
15
+ ThoughtDAG is a local-first, human-in-the-loop research workbench where the graph is not decoration: **wires determine the context sent to the model**. The same representation is used to study what happens when erroneous information enters a conversation, propagates downstream, and is later pruned or recomputed.
16
+
17
+ [Website](https://chenxiachan.github.io/thoughtdag/) · [GitHub](https://github.com/chenxiachan/thoughtdag) · [Interactive demo](https://app.thoughtdag.workers.dev)
18
+
19
+ ## Context Intervention Benchmark
20
+
21
+ The benchmark represents each task as a portable ThoughtDAG graph and evaluates five paired conditions:
22
+
23
+ 1. **clean** — the correct context chain;
24
+ 2. **polluted** — an incorrect claim enters the chain;
25
+ 3. **source prune** — the original error is removed while downstream descendants remain;
26
+ 4. **subgraph prune** — the source and contaminated descendants are removed;
27
+ 5. **recompute descendants** — the source is removed and dependent nodes are regenerated in order.
28
+
29
+ ### Context Repair Pilot v1
30
+
31
+ `4 models` · `9 task families` · `3 propagation depths` · `540 conditions`
32
+
33
+ Across 72 cases where clean context was answered correctly and polluted context produced a wrong answer:
34
+
35
+ | Intervention | Repaired cases |
36
+ |---|---:|
37
+ | Remove the erroneous source only | 68 / 72 |
38
+ | Recompute descendants in dependency order | 71 / 72 |
39
+ | Remove the contaminated subgraph | 72 / 72 |
40
+
41
+ This pilot studies **context intervention**, not hidden model memory and not general model quality. Tasks are synthetic and objectively scored; each endpoint was sampled once at temperature zero with provider-default reasoning behavior.
42
+
43
+ [Read the case study](https://chenxiachan.github.io/thoughtdag/stories/context-repair/) · [Full methodology and results](https://chenxiachan.github.io/thoughtdag/research/context-repair-pilot-v1/) · [Browse the benchmark](https://github.com/chenxiachan/thoughtdag/tree/main/benchmark)
44
+
45
+ ## What will be published here
46
+
47
+ - **Dataset:** portable graph cases, compiled requests, traces, scores, and provenance.
48
+ - **Space:** an interactive explorer for models, pollution families, propagation depth, and repair strategies.
49
+ - **Updates:** new model runs and new intervention families, versioned rather than folded into an unqualified leaderboard.
50
+
51
+ ThoughtDAG is MIT licensed. Benchmark limitations, model provenance, and scoring rules are published alongside every result.