PEFT
Safetensors
English
temporal-reasoning
knowledge-graph
graphrag
retrieval-augmented-generation
lora
Instructions to use anonym-ous/tempgraphrag-artifacts with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use anonym-ous/tempgraphrag-artifacts with PEFT:
Task type is invalid.
- Notebooks
- Google Colab
- Kaggle
Add files using upload-large-folder tool
Browse files- DATASHEET.md +42 -0
- PROVENANCE.txt +6 -0
- README.md +42 -0
- benchmark/benchmark_labelled.jsonl +0 -0
- benchmark/labels.tsv +0 -0
- eval/bm25-anchor-tfilter.json +0 -0
- eval/bm25-anchor.json +0 -0
- eval/bm25-rag-qwen3.json +0 -0
- eval/bm25-rag-qwen3.judged-1743.json +0 -0
- eval/llama31-sft-seed1337.extracted.json +0 -0
- eval/llama31-sft-seed1337.json +0 -0
- eval/llama31-sft-seed7.extracted.json +0 -0
- eval/llama31-sft-seed7.json +0 -0
- eval/llama31-sft.extracted.json +0 -0
- eval/llama31-sft.json +0 -0
- eval/llama31-sft.judged-500.json +0 -0
- eval/llama31-sft.judged.json +0 -0
- eval/mistral-sft-seed1337.extracted.json +0 -0
- eval/mistral-sft-seed1337.json +0 -0
- eval/mistral-sft-seed7.extracted.json +0 -0
- eval/mistral-sft-seed7.json +0 -0
- eval/mistral-sft.extracted.json +0 -0
- eval/mistral-sft.json +0 -0
- eval/mistral-sft.judged-500.json +0 -0
- eval/mistral-sft.judged.json +0 -0
- eval/multitq-3hop.json +0 -0
- eval/multitq-bm25.json +0 -0
- eval/multitq-bm25.judged.json +0 -0
- eval/multitq-bm25.judgeready.json +0 -0
- eval/multitq-llama-sft.json +0 -0
- eval/multitq-mistral-sft.json +0 -0
- eval/multitq-noevidence.json +0 -0
- eval/multitq-seed1337.json +0 -0
- eval/multitq-seed7.json +0 -0
- eval/multitq-v3-sft.json +0 -0
- eval/multitq-v3-sft.judged.json +0 -0
- eval/multitq-v3-sft.judgeready.json +0 -0
- eval/v2-grpo-10000.json +0 -0
- eval/v2-grpo-10000.judged-1743.json +0 -0
- eval/v3-sft-1k.extracted.json +0 -0
- eval/v3-sft-1k.json +0 -0
- eval/v3-sft-2k.extracted.json +0 -0
- eval/v3-sft-2k.json +0 -0
- eval/v3-sft-3hop-seed1337.extracted.json +0 -0
- eval/v3-sft-3hop-seed1337.json +0 -0
- eval/v3-sft-3hop-seed7.extracted.json +0 -0
- eval/v3-sft-3hop-seed7.json +0 -0
- eval/v3-sft-3hop.extracted.json +0 -0
- eval/v3-sft-3hop.json +0 -0
- eval/v3-sft-baseline.extracted.json +0 -0
DATASHEET.md
ADDED
|
@@ -0,0 +1,42 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Datasheet — data accompanying the TempGraphRAG methods paper
|
| 2 |
+
|
| 3 |
+
Anonymous double-blind artifact. Covers the data shipped to reproduce the methods
|
| 4 |
+
paper. The benchmark itself is described in concurrent anonymous work
|
| 5 |
+
(`anon-bench`); here it is a read-only input.
|
| 6 |
+
|
| 7 |
+
## Provenance
|
| 8 |
+
- **Source KG:** `tkgl-smallpedia` from TGB 2.0, derived from **Wikidata**
|
| 9 |
+
(public, **CC0**). 550,376 timestamped edges, 47,433 entities, 283 relations,
|
| 10 |
+
span 1900–2024.
|
| 11 |
+
- **Benchmark questions** (`benchmark_labelled.jsonl`): 8,710 multi-hop
|
| 12 |
+
temporal-KGQA questions, each with a gold supporting subgraph ($S^{*}$),
|
| 13 |
+
same-$(s,r)$ distractors ($S_{\mathrm{dist}}$), and stale-fact set
|
| 14 |
+
($S_{\mathrm{stale}}$), stratified over a 4×3 operator × complexity matrix.
|
| 15 |
+
70/10/20 train/dev/test split (6,096 / 871 / 1,743). Generated from a
|
| 16 |
+
TimelineKGQA-derived pipeline (see `anon-bench`).
|
| 17 |
+
- **SFT corpora** (`sft_data_5k_cot.jsonl`, `*_terse_lever.jsonl`,
|
| 18 |
+
`*_multitq_5k.jsonl`): retrieval-conditioned traces derived from the training
|
| 19 |
+
split; used to fine-tune the generator.
|
| 20 |
+
- **Evaluation outputs** (`outputs/eval/*.json`): per-question prediction + gold
|
| 21 |
+
for every reported run (3-seed headline, ablations, judge runs, MultiTQ).
|
| 22 |
+
|
| 23 |
+
## Composition / splits
|
| 24 |
+
Test results in the paper are on the 1,743-question test split. The train/test
|
| 25 |
+
splits share the source KG; the paper measures and reports contamination (0.40%
|
| 26 |
+
verbatim-question overlap, 5.1% anchor overlap) and treats absolute TempBench EM
|
| 27 |
+
as an in-distribution upper bound, with MultiTQ as the cross-distribution check.
|
| 28 |
+
|
| 29 |
+
## Intended use
|
| 30 |
+
Research reproduction of retrieval-augmented temporal multi-hop KGQA. Not
|
| 31 |
+
intended as a production correctness oracle. The triple-level grounding metric
|
| 32 |
+
(appendix) is a population-level diagnostic, not a per-instance guarantee.
|
| 33 |
+
|
| 34 |
+
## Personal / sensitive data
|
| 35 |
+
Questions concern historical and public-figure facts as exposed by Wikidata; no
|
| 36 |
+
personal data beyond Wikidata's public content. No human-subject data in this
|
| 37 |
+
archive. (Human inter-annotator validation is reported with `anon-bench`, not
|
| 38 |
+
here.)
|
| 39 |
+
|
| 40 |
+
## License
|
| 41 |
+
CC0-derived (Wikidata) via TGB 2.0; redistribution under permissive terms
|
| 42 |
+
consistent with TGB 2.0. Trained LoRA adapters are released for research use.
|
PROVENANCE.txt
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
The temporal-KGQA benchmark used here (outputs/benchmark/benchmark_labelled.jsonl,
|
| 2 |
+
labels.tsv) is described in concurrent anonymous work cited in the paper as
|
| 3 |
+
[anon-bench]. It is included in this archive as a read-only INPUT required to
|
| 4 |
+
reproduce the methods paper's results, and is NOT a contribution of this paper.
|
| 5 |
+
|
| 6 |
+
Underlying KG: tkgl-smallpedia (TGB 2.0), derived from Wikidata (CC0).
|
README.md
ADDED
|
@@ -0,0 +1,42 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Temporal-Aware GraphRAG — anonymous data archive
|
| 2 |
+
|
| 3 |
+
Anonymous double-blind artifact accompanying the EMNLP submission *"The Lever Is
|
| 4 |
+
the Prompt: Retrieval-Conditioned Supervision in Temporal-Aware GraphRAG."* This
|
| 5 |
+
is the **data + trained-adapter** half; the **code** is in the companion anonymous
|
| 6 |
+
repository linked from the paper. See `DATASHEET.md` for provenance and license.
|
| 7 |
+
|
| 8 |
+
> No author, affiliation, or identifying information is included. Please do not
|
| 9 |
+
> attempt to de-anonymize.
|
| 10 |
+
|
| 11 |
+
## Contents
|
| 12 |
+
```
|
| 13 |
+
adapters/ LoRA adapters (inference-ready: adapter_model.safetensors +
|
| 14 |
+
adapter_config.json + tokenizer + chat_template). 14 policies:
|
| 15 |
+
sft-v3{,-seed1337,-seed7} Qwen3-8B headline (3 seeds)
|
| 16 |
+
sft-llama31{,-seed1337,-seed7} Llama-3.1-8B cross-arch (3 seeds)
|
| 17 |
+
sft-mistral{,-seed1337,-seed7} Mistral-7B cross-arch (3 seeds)
|
| 18 |
+
sft-multitq{,-seed1337,-seed7} MultiTQ cross-benchmark (3 seeds)
|
| 19 |
+
sft-multitq-{llama,mistral} MultiTQ cross-arch
|
| 20 |
+
eval/ Per-question predictions + gold for every reported run (74 JSONs):
|
| 21 |
+
TempBench 3-seed, 3-hop ablation, empty/shuffled-evidence,
|
| 22 |
+
data-scale (1k/2k), full-test LLM-judge runs, MultiTQ, baselines.
|
| 23 |
+
benchmark/ benchmark_labelled.jsonl + labels.tsv (the benchmark of [anon-bench],
|
| 24 |
+
provided as a read-only input; see PROVENANCE.txt).
|
| 25 |
+
sft_data/ Retrieval-conditioned SFT corpora (CoT, terse, MultiTQ).
|
| 26 |
+
logs/ Training / evaluation logs (negative-result evidence).
|
| 27 |
+
```
|
| 28 |
+
|
| 29 |
+
## Mapping to the code repository
|
| 30 |
+
Place these so the code repo's scripts find them:
|
| 31 |
+
```
|
| 32 |
+
adapters/<name>/ -> checkpoints/<name>/final/
|
| 33 |
+
eval/*.json -> outputs/eval/
|
| 34 |
+
benchmark/* -> outputs/benchmark/
|
| 35 |
+
sft_data/*.jsonl -> outputs/
|
| 36 |
+
```
|
| 37 |
+
|
| 38 |
+
## Notes
|
| 39 |
+
- Adapters are **inference-only** (training state / optimizer checkpoints removed).
|
| 40 |
+
- Three contaminated judge runs (errored API calls) are **excluded**; all reported
|
| 41 |
+
judge-EM numbers come from the clean full-test runs included here.
|
| 42 |
+
- License: derived from Wikidata (CC0) via TGB 2.0 — permissive (see `DATASHEET.md`).
|
benchmark/benchmark_labelled.jsonl
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
benchmark/labels.tsv
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/bm25-anchor-tfilter.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/bm25-anchor.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/bm25-rag-qwen3.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/bm25-rag-qwen3.judged-1743.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/llama31-sft-seed1337.extracted.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/llama31-sft-seed1337.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/llama31-sft-seed7.extracted.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/llama31-sft-seed7.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/llama31-sft.extracted.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/llama31-sft.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/llama31-sft.judged-500.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/llama31-sft.judged.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/mistral-sft-seed1337.extracted.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/mistral-sft-seed1337.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/mistral-sft-seed7.extracted.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/mistral-sft-seed7.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/mistral-sft.extracted.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/mistral-sft.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/mistral-sft.judged-500.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/mistral-sft.judged.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/multitq-3hop.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/multitq-bm25.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/multitq-bm25.judged.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/multitq-bm25.judgeready.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/multitq-llama-sft.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/multitq-mistral-sft.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/multitq-noevidence.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/multitq-seed1337.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/multitq-seed7.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/multitq-v3-sft.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/multitq-v3-sft.judged.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/multitq-v3-sft.judgeready.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/v2-grpo-10000.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/v2-grpo-10000.judged-1743.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/v3-sft-1k.extracted.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/v3-sft-1k.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/v3-sft-2k.extracted.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/v3-sft-2k.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/v3-sft-3hop-seed1337.extracted.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/v3-sft-3hop-seed1337.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/v3-sft-3hop-seed7.extracted.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/v3-sft-3hop-seed7.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/v3-sft-3hop.extracted.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/v3-sft-3hop.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
eval/v3-sft-baseline.extracted.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|