anonym-ous commited on
Commit
a1a4f5e
·
verified ·
1 Parent(s): f3c0700

Add files using upload-large-folder tool

Browse files
Files changed (50) hide show
  1. DATASHEET.md +42 -0
  2. PROVENANCE.txt +6 -0
  3. README.md +42 -0
  4. benchmark/benchmark_labelled.jsonl +0 -0
  5. benchmark/labels.tsv +0 -0
  6. eval/bm25-anchor-tfilter.json +0 -0
  7. eval/bm25-anchor.json +0 -0
  8. eval/bm25-rag-qwen3.json +0 -0
  9. eval/bm25-rag-qwen3.judged-1743.json +0 -0
  10. eval/llama31-sft-seed1337.extracted.json +0 -0
  11. eval/llama31-sft-seed1337.json +0 -0
  12. eval/llama31-sft-seed7.extracted.json +0 -0
  13. eval/llama31-sft-seed7.json +0 -0
  14. eval/llama31-sft.extracted.json +0 -0
  15. eval/llama31-sft.json +0 -0
  16. eval/llama31-sft.judged-500.json +0 -0
  17. eval/llama31-sft.judged.json +0 -0
  18. eval/mistral-sft-seed1337.extracted.json +0 -0
  19. eval/mistral-sft-seed1337.json +0 -0
  20. eval/mistral-sft-seed7.extracted.json +0 -0
  21. eval/mistral-sft-seed7.json +0 -0
  22. eval/mistral-sft.extracted.json +0 -0
  23. eval/mistral-sft.json +0 -0
  24. eval/mistral-sft.judged-500.json +0 -0
  25. eval/mistral-sft.judged.json +0 -0
  26. eval/multitq-3hop.json +0 -0
  27. eval/multitq-bm25.json +0 -0
  28. eval/multitq-bm25.judged.json +0 -0
  29. eval/multitq-bm25.judgeready.json +0 -0
  30. eval/multitq-llama-sft.json +0 -0
  31. eval/multitq-mistral-sft.json +0 -0
  32. eval/multitq-noevidence.json +0 -0
  33. eval/multitq-seed1337.json +0 -0
  34. eval/multitq-seed7.json +0 -0
  35. eval/multitq-v3-sft.json +0 -0
  36. eval/multitq-v3-sft.judged.json +0 -0
  37. eval/multitq-v3-sft.judgeready.json +0 -0
  38. eval/v2-grpo-10000.json +0 -0
  39. eval/v2-grpo-10000.judged-1743.json +0 -0
  40. eval/v3-sft-1k.extracted.json +0 -0
  41. eval/v3-sft-1k.json +0 -0
  42. eval/v3-sft-2k.extracted.json +0 -0
  43. eval/v3-sft-2k.json +0 -0
  44. eval/v3-sft-3hop-seed1337.extracted.json +0 -0
  45. eval/v3-sft-3hop-seed1337.json +0 -0
  46. eval/v3-sft-3hop-seed7.extracted.json +0 -0
  47. eval/v3-sft-3hop-seed7.json +0 -0
  48. eval/v3-sft-3hop.extracted.json +0 -0
  49. eval/v3-sft-3hop.json +0 -0
  50. eval/v3-sft-baseline.extracted.json +0 -0
DATASHEET.md ADDED
@@ -0,0 +1,42 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Datasheet — data accompanying the TempGraphRAG methods paper
2
+
3
+ Anonymous double-blind artifact. Covers the data shipped to reproduce the methods
4
+ paper. The benchmark itself is described in concurrent anonymous work
5
+ (`anon-bench`); here it is a read-only input.
6
+
7
+ ## Provenance
8
+ - **Source KG:** `tkgl-smallpedia` from TGB 2.0, derived from **Wikidata**
9
+ (public, **CC0**). 550,376 timestamped edges, 47,433 entities, 283 relations,
10
+ span 1900–2024.
11
+ - **Benchmark questions** (`benchmark_labelled.jsonl`): 8,710 multi-hop
12
+ temporal-KGQA questions, each with a gold supporting subgraph ($S^{*}$),
13
+ same-$(s,r)$ distractors ($S_{\mathrm{dist}}$), and stale-fact set
14
+ ($S_{\mathrm{stale}}$), stratified over a 4×3 operator × complexity matrix.
15
+ 70/10/20 train/dev/test split (6,096 / 871 / 1,743). Generated from a
16
+ TimelineKGQA-derived pipeline (see `anon-bench`).
17
+ - **SFT corpora** (`sft_data_5k_cot.jsonl`, `*_terse_lever.jsonl`,
18
+ `*_multitq_5k.jsonl`): retrieval-conditioned traces derived from the training
19
+ split; used to fine-tune the generator.
20
+ - **Evaluation outputs** (`outputs/eval/*.json`): per-question prediction + gold
21
+ for every reported run (3-seed headline, ablations, judge runs, MultiTQ).
22
+
23
+ ## Composition / splits
24
+ Test results in the paper are on the 1,743-question test split. The train/test
25
+ splits share the source KG; the paper measures and reports contamination (0.40%
26
+ verbatim-question overlap, 5.1% anchor overlap) and treats absolute TempBench EM
27
+ as an in-distribution upper bound, with MultiTQ as the cross-distribution check.
28
+
29
+ ## Intended use
30
+ Research reproduction of retrieval-augmented temporal multi-hop KGQA. Not
31
+ intended as a production correctness oracle. The triple-level grounding metric
32
+ (appendix) is a population-level diagnostic, not a per-instance guarantee.
33
+
34
+ ## Personal / sensitive data
35
+ Questions concern historical and public-figure facts as exposed by Wikidata; no
36
+ personal data beyond Wikidata's public content. No human-subject data in this
37
+ archive. (Human inter-annotator validation is reported with `anon-bench`, not
38
+ here.)
39
+
40
+ ## License
41
+ CC0-derived (Wikidata) via TGB 2.0; redistribution under permissive terms
42
+ consistent with TGB 2.0. Trained LoRA adapters are released for research use.
PROVENANCE.txt ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ The temporal-KGQA benchmark used here (outputs/benchmark/benchmark_labelled.jsonl,
2
+ labels.tsv) is described in concurrent anonymous work cited in the paper as
3
+ [anon-bench]. It is included in this archive as a read-only INPUT required to
4
+ reproduce the methods paper's results, and is NOT a contribution of this paper.
5
+
6
+ Underlying KG: tkgl-smallpedia (TGB 2.0), derived from Wikidata (CC0).
README.md ADDED
@@ -0,0 +1,42 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Temporal-Aware GraphRAG — anonymous data archive
2
+
3
+ Anonymous double-blind artifact accompanying the EMNLP submission *"The Lever Is
4
+ the Prompt: Retrieval-Conditioned Supervision in Temporal-Aware GraphRAG."* This
5
+ is the **data + trained-adapter** half; the **code** is in the companion anonymous
6
+ repository linked from the paper. See `DATASHEET.md` for provenance and license.
7
+
8
+ > No author, affiliation, or identifying information is included. Please do not
9
+ > attempt to de-anonymize.
10
+
11
+ ## Contents
12
+ ```
13
+ adapters/ LoRA adapters (inference-ready: adapter_model.safetensors +
14
+ adapter_config.json + tokenizer + chat_template). 14 policies:
15
+ sft-v3{,-seed1337,-seed7} Qwen3-8B headline (3 seeds)
16
+ sft-llama31{,-seed1337,-seed7} Llama-3.1-8B cross-arch (3 seeds)
17
+ sft-mistral{,-seed1337,-seed7} Mistral-7B cross-arch (3 seeds)
18
+ sft-multitq{,-seed1337,-seed7} MultiTQ cross-benchmark (3 seeds)
19
+ sft-multitq-{llama,mistral} MultiTQ cross-arch
20
+ eval/ Per-question predictions + gold for every reported run (74 JSONs):
21
+ TempBench 3-seed, 3-hop ablation, empty/shuffled-evidence,
22
+ data-scale (1k/2k), full-test LLM-judge runs, MultiTQ, baselines.
23
+ benchmark/ benchmark_labelled.jsonl + labels.tsv (the benchmark of [anon-bench],
24
+ provided as a read-only input; see PROVENANCE.txt).
25
+ sft_data/ Retrieval-conditioned SFT corpora (CoT, terse, MultiTQ).
26
+ logs/ Training / evaluation logs (negative-result evidence).
27
+ ```
28
+
29
+ ## Mapping to the code repository
30
+ Place these so the code repo's scripts find them:
31
+ ```
32
+ adapters/<name>/ -> checkpoints/<name>/final/
33
+ eval/*.json -> outputs/eval/
34
+ benchmark/* -> outputs/benchmark/
35
+ sft_data/*.jsonl -> outputs/
36
+ ```
37
+
38
+ ## Notes
39
+ - Adapters are **inference-only** (training state / optimizer checkpoints removed).
40
+ - Three contaminated judge runs (errored API calls) are **excluded**; all reported
41
+ judge-EM numbers come from the clean full-test runs included here.
42
+ - License: derived from Wikidata (CC0) via TGB 2.0 — permissive (see `DATASHEET.md`).
benchmark/benchmark_labelled.jsonl ADDED
The diff for this file is too large to render. See raw diff
 
benchmark/labels.tsv ADDED
The diff for this file is too large to render. See raw diff
 
eval/bm25-anchor-tfilter.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/bm25-anchor.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/bm25-rag-qwen3.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/bm25-rag-qwen3.judged-1743.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/llama31-sft-seed1337.extracted.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/llama31-sft-seed1337.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/llama31-sft-seed7.extracted.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/llama31-sft-seed7.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/llama31-sft.extracted.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/llama31-sft.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/llama31-sft.judged-500.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/llama31-sft.judged.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/mistral-sft-seed1337.extracted.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/mistral-sft-seed1337.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/mistral-sft-seed7.extracted.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/mistral-sft-seed7.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/mistral-sft.extracted.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/mistral-sft.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/mistral-sft.judged-500.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/mistral-sft.judged.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/multitq-3hop.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/multitq-bm25.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/multitq-bm25.judged.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/multitq-bm25.judgeready.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/multitq-llama-sft.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/multitq-mistral-sft.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/multitq-noevidence.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/multitq-seed1337.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/multitq-seed7.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/multitq-v3-sft.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/multitq-v3-sft.judged.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/multitq-v3-sft.judgeready.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/v2-grpo-10000.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/v2-grpo-10000.judged-1743.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/v3-sft-1k.extracted.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/v3-sft-1k.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/v3-sft-2k.extracted.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/v3-sft-2k.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/v3-sft-3hop-seed1337.extracted.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/v3-sft-3hop-seed1337.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/v3-sft-3hop-seed7.extracted.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/v3-sft-3hop-seed7.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/v3-sft-3hop.extracted.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/v3-sft-3hop.json ADDED
The diff for this file is too large to render. See raw diff
 
eval/v3-sft-baseline.extracted.json ADDED
The diff for this file is too large to render. See raw diff