Add code: src, benchmarks, data, requirements, run.sh, README

.gitattributes

@@ -33,3 +33,9 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+data/multihop/musique.json filter=lfs diff=lfs merge=lfs -text
+data/text2sql/bird/databases/superhero/superhero.sqlite filter=lfs diff=lfs merge=lfs -text
+data/text2sql/spider2-lite/sqlite/Pagila/Pagila.sqlite filter=lfs diff=lfs merge=lfs -text
+docs/figs/GraphRAG.png filter=lfs diff=lfs merge=lfs -text
+docs/figs/LightRAG.png filter=lfs diff=lfs merge=lfs -text
+docs/figs/QAFD-RAG.png filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,47 @@
+# Knowledge graphs (large, download from https://huggingface.co/datasets/qafd/kg)
+kg/**
+!kg/
+!kg/ultradomain/
+!kg/ultradomain/.gitkeep
+!kg/multihop/
+!kg/multihop/.gitkeep
+!kg/summarization/
+!kg/summarization/.gitkeep
+!kg/text2sql/
+!kg/text2sql/.gitkeep
+
+# Python
+__pycache__/
+*.pyc
+*.pyo
+*.pyd
+*.egg-info/
+dist/
+build/
+
+# Environment
+.env
+*.env
+.venv/
+env/
+
+# Jupyter
+.ipynb_checkpoints/
+
+# IDE
+.vscode/
+.idea/
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Large datasets (downloaded at runtime)
+data/multihop/narrativeqa*
+
+# Run logs and results (regenerable)
+run_all_embeddings.log
+results/
+outputs/
+outputs_ablation/
+experiments/results/

README.md

@@ -0,0 +1,345 @@
+# QAFD-RAG
+
+Official code for ICLR 2026 paper: **[Query-Aware Flow Diffusion for Graph-Based RAG with Retrieval Guarantees](https://openreview.net/forum?id=n28wnc2QTc)**
+
+QAFD-RAG uses **query-aware flow diffusion** to retrieve contextually relevant subgraphs from a knowledge graph. Unlike community-based (GraphRAG) or entity-centric (LightRAG) approaches, QAFD-RAG dynamically re-weights edges based on query relevance and propagates flow through the graph to discover multi-hop context with retrieval guarantees.
+
+<p align="center"><em>Query: "Introduce Steve Jobs's products in Apple."</em></p>
+
+<table align="center"><tr>
+  <td align="center"><img src="docs/figs/GraphRAG.png" width="260" alt="GraphRAG"/><br/><em>GraphRAG</em></td>
+  <td align="center"><img src="docs/figs/LightRAG.png" width="260" alt="LightRAG"/><br/><em>LightRAG</em></td>
+  <td align="center"><img src="docs/figs/QAFD-RAG.png" width="260" alt="QAFD-RAG"/><br/><em>QAFD-RAG</em></td>
+</tr></table>
+<p align="center"><em>QAFD-RAG reweights edges by query relevance, suppressing irrelevant neighborhoods (e.g., Amazon, Apple fruit).</em></p>
+
+## Quick Start
+
+```bash
+# 1. Create conda environment and install dependencies
+conda create -n qafd-rag python=3.10 -y
+conda activate qafd-rag
+pip install -r requirements.txt
+
+# 2. Set your OpenAI API key (required for LLM and answer generation)
+export OPENAI_API_KEY="sk-..."
+
+# 3. Download pre-built KGs (recommended — saves hours of build time + API costs)
+huggingface-cli download qafd/kg --repo-type dataset --include "multihop/*" --local-dir ./kg
+huggingface-cli download qafd/kg --repo-type dataset --include "ultradomain/*" --local-dir ./kg
+
+# 4. Run a benchmark
+#    Note: First run for multihop will auto-download the nvidia/NV-Embed-v2
+#    embedding model (~8GB, one-time, requires GPU with 16GB+ VRAM).
+#    No GPU? See "Rebuild with openai-small" below.
+python benchmarks/run.py --task multihop --dataset musique --questions 10
+python benchmarks/run.py --task ultradomain --dataset mix --questions 10
+```
+
+> **Embeddings:** For best multihop results, download the pre-built KGs which use `nvidia-nv-embed-v2` (requires GPU with 16GB+ VRAM, auto-downloaded on first run). If you don't have a GPU, you can rebuild the KGs from scratch with `openai-small` instead: `python benchmarks/run.py --task multihop --dataset musique --force_build --embedding openai-small`.
+>
+> **Graph Types:** Multihop defaults to **passage-entity** graph (entities + passages + facts as nodes). UltraDomain/text2sql/summarization default to **entity** graph (classic KG). Override with `--graph_type`.
+
+## Two Graph Types
+
+QAFD-RAG supports two knowledge graph representations, both using the same Query-Aware Flow Diffusion algorithm:
+
+| | **Entity Graph** | **Passage-Entity Graph** |
+|---|---|---|
+| **Nodes** | Entities + relationships | Entities + passages + facts ([Gutiérrez et al., 2024](https://arxiv.org/abs/2405.14831)) |
+| **Extraction** | LLM entity/relationship extraction | OpenIE (NER + triple extraction) |
+| **Edges** | Entity-entity relationships | Fact edges + passage edges + synonymy edges |
+| **QAFD traversal** | Flow reaches entities, passages looked up after | Flow reaches passages directly as graph nodes |
+| **Best for** | General QA, text2sql, summarization | Multi-hop reasoning |
+| **Default tasks** | ultradomain, text2sql, summarization | multihop |
+
+Both graph types use query-aware flow diffusion with the same parameters (alpha=2.0, epsilon=0.01, step_size=0.2).
+
+## Project Structure
+
+```
+QAFD-RAG/
+├── benchmarks/
+│   ├── run.py                       # Unified benchmark runner (--graph_type, --task)
+│   ├── multihop/                    # Multi-hop reasoning
+│   ├── ultradomain/                 # General domain QA
+│   ├── text2sql/                    # Natural language to SQL
+│   └── summarization/               # Document summarization
+├── src/
+│   ├── QAFD_RAG.py                  # Entity graph pipeline (main class)
+│   ├── base.py                      # Storage interfaces, QueryParam
+│   ├── llm.py                       # LLM and embedding functions
+│   ├── storage.py                   # KV, vector, and graph storage
+│   ├── operate.py                   # Query operations
+│   ├── evaluation.py                # Answer evaluation
+│   ├── answering/                   # Query processing (entity graph)
+│   │   ├── handler.py               # kg_query entry point
+│   │   ├── context.py               # Context building (local/global/hybrid)
+│   │   ├── clusters.py              # Flow diffusion clustering
+│   │   └── text_units.py            # Text chunk retrieval
+│   ├── retrievers/
+│   │   ├── flow_diffusion.py        # QAFD on NetworkX (entity graph)
+│   │   └── base.py                  # Retriever interface
+│   ├── hipporag_pipeline/           # Passage-entity graph pipeline
+│   │   ├── kg_builder.py            # OpenIE → igraph (entities + passages + facts)
+│   │   ├── graph_adapter.py         # QAFD on igraph (passage-entity graph)
+│   │   ├── retriever.py             # Fact reranking → seed selection → QAFD
+│   │   ├── config.py                # Pipeline configuration
+│   │   ├── openie.py                # NER + triple extraction
+│   │   ├── embedding_store.py       # Parquet-backed vector store
+│   │   ├── reranker.py              # LLM-based fact reranker
+│   │   ├── prompts.py               # Prompt templates
+│   │   ├── benchmark_runner.py      # Standalone runner
+│   │   └── utils.py                 # Helpers
+│   ├── indexing/                     # KG construction (entity graph)
+│   │   ├── chunkers.py              # Token-based chunking
+│   │   ├── extractors.py            # Entity/relationship extraction
+│   │   ├── schema_builder.py        # Database schema KG builder
+│   │   └── build_kg.py              # CLI KG builder
+│   ├── embedding_models/            # Embedding model implementations
+│   ├── prompts/                     # LLM prompt templates
+│   ├── text2sql/                    # Text-to-SQL support
+│   └── utils/                       # Helpers
+├── data/                            # Datasets (auto-downloaded from HuggingFace)
+├── kg/                              # Knowledge graphs (auto-generated or downloaded)
+├── docs/figs/                       # Figures for README
+├── ICLR2026/                        # Paper source (LaTeX)
+└── results/                         # Benchmark results (JSON)
+```
+
+## Benchmarks
+
+| Task | Datasets | Metrics |
+|------|----------|---------|
+| **UltraDomain** | agriculture, biology, cs, finance, legal, math, medicine, mix, music, philosophy, physics | Quality scores (comprehensiveness, diversity, relevance, logicality, coherence) |
+| **Multi-hop QA** | MuSiQue, HotpotQA, 2WikiMultiHopQA | F1, Exact Match |
+| **Text-to-SQL** | Spider2-lite, Bird | Schema retrieval precision/recall |
+| **Summarization** | SQuALITY | BLEU, ROUGE, METEOR, quality scores |
+
+## Usage
+
+### Unified Benchmark Runner
+
+```bash
+python benchmarks/run.py --task <task> --dataset <dataset> [options]
+```
+
+The runner automatically selects the appropriate graph type (passage-entity for multihop, entity for others). Override with `--graph_type`:
+
+```bash
+# Multihop with passage-entity graph (default)
+python benchmarks/run.py --task multihop --dataset musique --questions 100
+
+# Multihop with entity graph (override)
+python benchmarks/run.py --task multihop --dataset musique --graph_type entity
+
+# Ultradomain
+python benchmarks/run.py --task ultradomain --dataset mix --questions 10
+
+# Retrieval only (skip QA)
+python benchmarks/run.py --task multihop --dataset musique --skip_qa
+
+# Build KG only
+python benchmarks/run.py --task multihop --dataset musique --build_only
+```
+
+### Legacy CLI (entity graph only)
+
+All benchmarks can also be run through `./run.sh` (always uses entity graph):
+
+```bash
+./run.sh <task> [options]
+```
+
+### Knowledge Graphs
+
+Pre-built KGs are available at [huggingface.co/datasets/qafd/kg](https://huggingface.co/datasets/qafd/kg). Downloading is **recommended** to avoid hours of build time and API costs.
+
+| Benchmark | Embedding | GPU needed? | Notes |
+|-----------|-----------|-------------|-------|
+| **multihop** | `nvidia-nv-embed-v2` | Yes (16GB+ VRAM) | Auto-downloaded from HuggingFace on first run |
+| **ultradomain** | `openai-small` | No | Uses OpenAI API (`OPENAI_API_KEY`) |
+| **text2sql** | `openai-small` | No | Uses OpenAI API (`OPENAI_API_KEY`) |
+
+```bash
+# Download pre-built KGs (recommended)
+huggingface-cli download qafd/kg --repo-type dataset --include "multihop/*" --local-dir ./kg
+huggingface-cli download qafd/kg --repo-type dataset --include "ultradomain/*" --local-dir ./kg
+
+# Or download everything
+huggingface-cli download qafd/kg --repo-type dataset --local-dir ./kg
+```
+
+To rebuild KGs with a different embedding (e.g., no GPU available):
+
+```bash
+# Rebuild multihop with openai-small (no GPU needed, uses OpenAI API)
+python benchmarks/run.py --task multihop --dataset musique --force_build --embedding openai-small
+```
+
+### Run Benchmarks
+
+```bash
+# Multi-hop QA (passage-entity graph by default)
+python benchmarks/run.py --task multihop --dataset musique --questions 100
+python benchmarks/run.py --task multihop --dataset hotpotqa --questions 100
+python benchmarks/run.py --task multihop --dataset 2wikimultihopqa --questions 100
+
+# UltraDomain (entity graph by default)
+python benchmarks/run.py --task ultradomain --dataset mix --questions 10
+
+# UltraDomain with passage-entity graph
+python benchmarks/run.py --task ultradomain --dataset mix --graph_type passage-entity --questions 10
+
+# Text-to-SQL (entity graph)
+python benchmarks/run.py --task text2sql --dataset spider2-lite
+
+# Summarization (entity graph)
+python benchmarks/run.py --task summarization --dataset squality
+```
+
+### Common Options
+
+| Option | Description |
+|--------|-------------|
+| `--task TASK` | `multihop`, `ultradomain`, `text2sql`, `summarization` |
+| `--dataset NAME` | Dataset name (e.g., `musique`, `mix`, `spider2-lite`) |
+| `--graph_type TYPE` | `passage-entity` or `entity` (auto-selected by task) |
+| `--questions N` | Number of questions to evaluate |
+| `--build_only` | Build KG only, skip benchmark |
+| `--force_build` | Rebuild KG even if it exists |
+| `--skip_qa` | Run retrieval only, skip QA (passage-entity) |
+| `--embedding MODEL` | Embedding model (see table below) |
+| `--llm MODEL` | LLM model (`gpt-4o-mini`, `gpt-4o`, `gpt-5-nano`, `gpt-5-mini`, `gpt-5`, `gpt-oss-120b`) |
+| `--alpha FLOAT` | QAFD alpha parameter (default: 2.0) |
+| `--epsilon FLOAT` | QAFD convergence threshold (default: 0.01) |
+| `--weight_scheme` | Query-aware edge weighting: `original`, `multiply`, `add` |
+
+## Data
+
+Datasets are automatically downloaded from HuggingFace on first run:
+
+| Task | Source | Reference |
+|------|--------|-----------|
+| **UltraDomain** | [TommyChien/UltraDomain](https://huggingface.co/datasets/TommyChien/UltraDomain) | agriculture, biology, cs, finance, legal, math, medicine, mix, music, philosophy, physics |
+| **Multi-hop QA** | [osunlp](https://huggingface.co/osunlp) | MuSiQue, HotpotQA, 2WikiMultiHopQA |
+| **Summarization** | [pszemraj/SQuALITY-v1.3](https://huggingface.co/datasets/pszemraj/SQuALITY-v1.3) | SQuALITY |
+| **Text-to-SQL** | Included (`data/text2sql/`) | Spider2-lite (Pagila) + Bird (superhero) with auto-generated DB summaries |
+
+## Configuration
+
+### Embedding Models
+
+| Key | Model | Dimensions |
+|-----|-------|-----------|
+| `openai-small` | text-embedding-3-small | 1536 |
+| `openai-large` | text-embedding-3-large | 3072 |
+| `jina-v3` | jinaai/jina-embeddings-v3 | 1024 |
+| `gritlm` | GritLM/GritLM-7B | 4096 |
+| `nvidia-nv-embed-v2` | nvidia/NV-Embed-v2 | 4096 |
+
+The default embedding is **`openai-small`** which requires `OPENAI_API_KEY`.
+
+Use `--embedding <key>` to select a model:
+
+```bash
+./run.sh ultradomain --questions 10 --embedding jina-v3
+```
+
+### Local Embedding Models
+
+Local embeddings (`jina-v3`, `gritlm`, `nvidia-nv-embed-v2`) run on your GPU and do not require an API key for embeddings. They are downloaded automatically from HuggingFace on first use.
+
+**Requirements:**
+- CUDA-capable GPU with sufficient VRAM (8GB+ recommended)
+- Models are cached in `~/.cache/huggingface/`
+
+**Usage with local embeddings (no OpenAI API needed for embeddings):**
+
+```bash
+# Use Jina v3 (1024-dim, lightweight, good quality)
+./run.sh ultradomain --questions 10 --embedding jina-v3
+
+# Use GritLM (4096-dim, unified embedding+generation)
+./run.sh multihop --dataset musique --questions 10 --embedding gritlm
+
+# Use NVIDIA NV-Embed-v2 (4096-dim, 32K context, high quality)
+./run.sh ultradomain --questions 10 --embedding nvidia-nv-embed-v2
+```
+
+> **Note:** Even with local embeddings, an LLM API key is still required for entity extraction (KG building) and answer generation. Set `OPENAI_API_KEY` or use `--llm gpt-oss-120b` for a free open-source LLM.
+
+### LLM Models
+
+`gpt-4o-mini` (default), `gpt-4o`, `gpt-5-nano`, `gpt-5-mini`, `gpt-5`, `gpt-oss-120b` (free, local)
+
+## Python API
+
+```python
+from src import QAFD_RAG, QueryParam
+
+# Initialize
+rag = QAFD_RAG(
+    working_dir="./my_kg",
+    llm_model_name="gpt-4o-mini",
+    embedding_model_key="jina-v3",
+)
+
+# Index documents
+rag.insert(["Document text 1...", "Document text 2..."])
+
+# Query
+answer = rag.query(
+    "What is X?",
+    param=QueryParam(mode="hybrid"),
+)
+print(answer)
+```
+
+### QueryParam Options
+
+| Parameter | Default | Description |
+|-----------|---------|-------------|
+| `mode` | `"hybrid"` | Retrieval mode: `local`, `global`, `hybrid` |
+| `top_k` | 40 | Number of entities to retrieve |
+| `max_source_nodes` | 40 | Max source nodes for flow diffusion |
+| `min_flow_threshold` | 0.1 | Minimum flow value to include a node |
+| `enable_query_aware_flow_diffusion` | `True` | Use query-aware edge weighting |
+| `alpha` | 50.0 | Flow diffusion initialization factor |
+| `response_type` | `"Multiple Paragraphs"` | Output format |
+
+## How It Works
+
+QAFD-RAG processes documents through a two-stage pipeline:
+
+**Stage 1 -- Knowledge Graph Construction**
+
+*Entity Graph:*
+1. Documents are split into token-based chunks.
+2. An LLM extracts entities and relationships from each chunk.
+3. Entities become nodes, relationships become weighted edges.
+
+*Passage-Entity Graph:*
+1. Documents are split into passages (chunks).
+2. OpenIE extracts named entities and (subject, predicate, object) triples.
+3. Entities and passages are both graph nodes, connected by fact edges, passage-entity edges, and synonymy edges (entity pairs with cosine similarity > 0.8).
+
+**Stage 2 -- Query-Aware Flow Diffusion**
+
+1. **Seed Selection**: Query is matched to entities/facts via embedding similarity. An LLM reranker filters the most relevant facts.
+2. **Flow Diffusion**: Mass is injected at seed nodes and propagated through the graph via push-relabel. Edge weights are dynamically adjusted based on each node's similarity to the query.
+3. **Passage Ranking**: Nodes accumulate importance scores proportional to flow. In the passage-entity graph, passages are ranked directly. In the entity graph, associated text chunks are retrieved from top-ranked entities.
+4. **LLM Answering**: Top passages are assembled into context and passed to an LLM for answer generation.
+
+The flow diffusion algorithm is the key differentiator: rather than simple graph traversal or vector search alone, it combines graph structure with query relevance to find contextually important information that may be several hops away from the initial match.
+
+## Citation
+
+```bibtex
+@inproceedings{zhou2026qafd,
+  title={Query-Aware Flow Diffusion for Graph-Based RAG with Retrieval Guarantees},
+  author={Zhou, Zhuoping and Ataee Tarzanagh, Davoud and Didari, Sima and Hu, Wenjun and Gutow, Baruch and Verkholyak, Oxana and Faraki, Masoud and Hao, Heng and Moon, Hankyu and Min, Seungjai},
+  booktitle={International Conference on Learning Representations (ICLR)},
+  year={2026}
+}
+```

benchmarks/multihop/benchmark_multihop.py

@@ -0,0 +1,531 @@
+#!/usr/bin/env python3
+"""
+Unified Multi-hop QA Benchmark for QAFD-RAG
+Supports: MuSiQue, HotpotQA, 2WikiMultiHopQA
+"""
+
+import os
+import sys
+import asyncio
+import time
+import json
+import re
+import string
+import logging
+from typing import List, Dict
+from dataclasses import dataclass, asdict
+from datetime import datetime
+from pathlib import Path
+from collections import Counter
+import numpy as np
+
+# Suppress verbose logging
+logging.getLogger("httpx").setLevel(logging.ERROR)
+logging.getLogger("QAFD_RAG").setLevel(logging.ERROR)
+logging.getLogger("nano-vectordb").setLevel(logging.ERROR)
+logging.getLogger("OpenAI").setLevel(logging.ERROR)
+
+# Add QAFD-RAG to path
+QAFD_RAG_HOME = str(Path(__file__).parent.parent.parent)
+sys.path.insert(0, QAFD_RAG_HOME)
+
+import nest_asyncio
+nest_asyncio.apply()
+
+# Dataset configurations
+DATASETS = {
+    "musique": {
+        "name": "MuSiQue",
+        "data_file": "musique.json",
+        "corpus_file": "musique_corpus.json",
+        "kg_dir": "musique",
+    },
+    "hotpotqa": {
+        "name": "HotpotQA",
+        "data_file": "hotpotqa.json",
+        "corpus_file": "hotpotqa_corpus.json",
+        "kg_dir": "hotpotqa",
+    },
+    "2wikimultihopqa": {
+        "name": "2WikiMultiHopQA",
+        "data_file": "2wikimultihopqa.json",
+        "corpus_file": "2wikimultihopqa_corpus.json",
+        "kg_dir": "2wikimultihopqa",
+    },
+}
+
+
+def print_header(title: str, width: int = 70):
+    """Print a formatted header"""
+    print(f"\n{'=' * width}")
+    print(f"  {title}")
+    print(f"{'=' * width}")
+
+
+def print_config(items: Dict[str, str], width: int = 70):
+    """Print configuration items"""
+    print(f"{'─' * width}")
+    for key, value in items.items():
+        print(f"  {key:<20} {value}")
+    print(f"{'─' * width}")
+
+
+def print_progress(current: int, total: int, prefix: str = "", width: int = 40):
+    """Print a progress bar"""
+    percent = current / total
+    filled = int(width * percent)
+    bar = '█' * filled + '░' * (width - filled)
+    print(f"\r  {prefix} [{bar}] {current}/{total} ({percent*100:.1f}%)", end='', flush=True)
+    if current == total:
+        print()
+
+
+def print_metric(name: str, value: float, std: float = None, width: int = 25):
+    """Print a metric with optional std"""
+    if std is not None:
+        print(f"  {name:<{width}} {value:.4f} ± {std:.4f}")
+    else:
+        print(f"  {name:<{width}} {value:.4f}")
+
+
+@dataclass
+class BenchmarkResult:
+    """Benchmark result for multi-hop QA"""
+    model_name: str
+    dataset_name: str
+    total_questions: int
+    success_count: int
+    total_time: float
+    kg_build_time: float
+    query_time: float
+    avg_time_per_question: float
+    f1_score_mean: float
+    f1_score_std: float
+    exact_match_mean: float
+    exact_match_std: float
+    f1_scores: List[float] = None
+    exact_match_scores: List[float] = None
+    responses: List[str] = None
+    questions: List[str] = None
+    gold_answers: List[List[str]] = None
+    error_message: str = ""
+
+
+def normalize_answer(answer: str) -> str:
+    """Normalize answer for comparison"""
+    def remove_articles(text):
+        return re.sub(r"\b(a|an|the)\b", " ", text)
+    def white_space_fix(text):
+        return " ".join(text.split())
+    def remove_punc(text):
+        exclude = set(string.punctuation)
+        return "".join(ch for ch in text if ch not in exclude)
+    def lower(text):
+        return text.lower()
+    return white_space_fix(remove_articles(remove_punc(lower(answer))))
+
+
+def compute_f1(gold: str, predicted: str) -> float:
+    """Compute F1 score between gold and predicted answers"""
+    gold_tokens = normalize_answer(gold).split()
+    predicted_tokens = normalize_answer(predicted).split()
+    common = Counter(predicted_tokens) & Counter(gold_tokens)
+    num_same = sum(common.values())
+    if num_same == 0:
+        return 0.0
+    precision = 1.0 * num_same / len(predicted_tokens) if predicted_tokens else 0.0
+    recall = 1.0 * num_same / len(gold_tokens) if gold_tokens else 0.0
+    if precision + recall == 0:
+        return 0.0
+    return 2 * (precision * recall) / (precision + recall)
+
+
+def compute_exact_match(gold: str, predicted: str) -> float:
+    """Compute exact match score"""
+    return 1.0 if normalize_answer(gold) == normalize_answer(predicted) else 0.0
+
+
+def get_gold_answers(samples):
+    """Extract gold answers from samples"""
+    gold_answers = []
+    for sample in samples:
+        if 'answer' in sample:
+            gold_ans = sample['answer']
+        elif 'reference' in sample:
+            gold_ans = sample['reference']
+        else:
+            gold_ans = "Unknown"
+        if isinstance(gold_ans, str):
+            gold_ans = [gold_ans]
+        elif not isinstance(gold_ans, list):
+            gold_ans = [str(gold_ans)]
+        gold_answers.append(gold_ans)
+    return gold_answers
+
+
+class MultiHopBenchmark:
+    def __init__(self, dataset: str, api_key: str, embedding_model: str = "openai-small",
+                 llm_model: str = "gpt-4o-mini"):
+        if dataset not in DATASETS:
+            raise ValueError(f"Unknown dataset: {dataset}. Choose from: {list(DATASETS.keys())}")
+
+        self.dataset = dataset
+        self.config = DATASETS[dataset]
+        self.api_key = api_key
+        self.embedding_model = embedding_model
+        self.llm_model = llm_model
+        os.environ["OPENAI_API_KEY"] = api_key
+        os.environ["OPENAI_API_BASE"] = "https://api.openai.com/v1"
+
+    def _get_working_dir(self) -> str:
+        return os.path.join(QAFD_RAG_HOME, "kg", "multihop", f"{self.llm_model}_{self.embedding_model}_{self.config['kg_dir']}")
+
+    def _kg_exists(self, working_dir: str) -> bool:
+        kg_files = [
+            os.path.join(working_dir, "vdb_entities.json"),
+            os.path.join(working_dir, "vdb_chunks.json"),
+            os.path.join(working_dir, "kv_store_full_docs.json"),
+        ]
+        return all(os.path.exists(f) for f in kg_files)
+
+    def _get_llm_func(self):
+        from src import llm
+        llm_funcs = {
+            "gpt-4o-mini": llm.gpt_4o_mini_complete,
+            "gpt-4o": llm.gpt_4o_complete,
+            "gpt-oss-120b": llm.gpt_oss_120b_complete,
+            "gpt-5": llm.gpt_5_complete,
+            "gpt-5-mini": llm.gpt_5_mini_complete,
+            "gpt-5-nano": llm.gpt_5_nano_complete,
+        }
+        return llm_funcs.get(self.llm_model, llm.gpt_4o_mini_complete)
+
+    def _ensure_data_file(self, filename: str) -> str:
+        """Return path to data file, downloading from HuggingFace if missing."""
+        data_dir = os.path.join(QAFD_RAG_HOME, "data", "multihop")
+        filepath = os.path.join(data_dir, filename)
+        if not os.path.exists(filepath):
+            print(f"  Downloading {filename} from HuggingFace...", end=" ", flush=True)
+            from huggingface_hub import hf_hub_download
+            os.makedirs(data_dir, exist_ok=True)
+            hf_hub_download(
+                repo_id="osunlp/HippoRAG_2",
+                filename=filename,
+                repo_type="dataset",
+                local_dir=data_dir,
+            )
+            print("done")
+        return filepath
+
+    def _load_dataset(self) -> List[Dict]:
+        dataset_path = self._ensure_data_file(self.config["data_file"])
+        with open(dataset_path, 'r', encoding='utf-8') as f:
+            samples = json.load(f)
+        return samples
+
+    def _load_corpus(self) -> List[str]:
+        corpus_path = self._ensure_data_file(self.config["corpus_file"])
+        with open(corpus_path, 'r', encoding='utf-8') as f:
+            corpus = json.load(f)
+        docs = [f"{doc['title']}\n{doc['text']}" for doc in corpus]
+        return docs
+
+    async def build_kg(self, max_documents: int = None) -> bool:
+        """Build KG only (no benchmark)"""
+        from src.QAFD_RAG import QAFD_RAG
+
+        print_header(f"QAFD-RAG Knowledge Graph Builder")
+        print_config({
+            "Graph Type": "entity",
+            "Dataset": self.config['name'],
+            "Embedding": self.embedding_model,
+            "LLM": self.llm_model,
+            "Working Dir": self._get_working_dir()
+        })
+
+        working_dir = self._get_working_dir()
+        os.makedirs(working_dir, exist_ok=True)
+
+        llm_func = self._get_llm_func()
+        rag = QAFD_RAG(
+            working_dir=working_dir,
+            llm_model_func=llm_func,
+            llm_model_name=self.llm_model,
+            embedding_model_key=self.embedding_model,
+            enable_llm_cache=True,
+        )
+
+        print("\n  Loading corpus...", end=" ", flush=True)
+        docs = self._load_corpus()
+        docs_to_process = min(max_documents, len(docs)) if max_documents else len(docs)
+        print(f"done ({len(docs)} documents available)")
+
+        print(f"\n  Building KG from {docs_to_process} documents...")
+        start_time = time.time()
+
+        for i, doc in enumerate(docs[:docs_to_process]):
+            print_progress(i + 1, docs_to_process, "Progress")
+            await rag.ainsert(doc)
+
+        build_time = time.time() - start_time
+
+        print_header("Build Complete")
+        print(f"  Documents processed:  {docs_to_process}")
+        print(f"  Time elapsed:         {build_time:.2f}s")
+        print(f"  Avg per document:     {build_time/docs_to_process:.2f}s")
+        print(f"  Output directory:     {working_dir}")
+        print()
+
+        return True
+
+    async def run_benchmark(self, question_count: int = 100, force_build: bool = False,
+                           max_documents: int = None, mode: str = "hybrid",
+                           max_source_nodes: int = 20, min_flow_threshold: float = 0.1,
+                           alpha: float = 2.0) -> BenchmarkResult:
+        """Run benchmark"""
+        from src.QAFD_RAG import QAFD_RAG, QueryParam
+
+        print_header(f"QAFD-RAG Multi-hop QA Benchmark")
+        print_config({
+            "Graph Type": "entity",
+            "Dataset": self.config['name'],
+            "Questions": str(question_count),
+            "Embedding": self.embedding_model,
+            "LLM": self.llm_model,
+            "Mode": mode,
+            "Max Nodes": str(max_source_nodes),
+            "Threshold": str(min_flow_threshold),
+            "Alpha": str(alpha)
+        })
+
+        working_dir = self._get_working_dir()
+        os.makedirs(working_dir, exist_ok=True)
+
+        llm_func = self._get_llm_func()
+        rag = QAFD_RAG(
+            working_dir=working_dir,
+            llm_model_func=llm_func,
+            llm_model_name=self.llm_model,
+            embedding_model_key=self.embedding_model,
+            enable_llm_cache=True,
+        )
+
+        # Check if KG exists or needs to be built
+        kg_build_time = 0.0
+        if self._kg_exists(working_dir) and not force_build:
+            print(f"\n  Using existing KG: {working_dir}")
+        else:
+            print("\n  Loading corpus...", end=" ", flush=True)
+            docs = self._load_corpus()
+            docs_to_process = min(max_documents, len(docs)) if max_documents else len(docs)
+            print(f"done ({docs_to_process} documents)")
+
+            print(f"  Building KG...")
+            start_time = time.time()
+            for i, doc in enumerate(docs[:docs_to_process]):
+                print_progress(i + 1, docs_to_process, "Progress")
+                await rag.ainsert(doc)
+            kg_build_time = time.time() - start_time
+            print(f"  KG built in {kg_build_time:.2f}s")
+
+        # Load dataset
+        print("\n  Loading dataset...", end=" ", flush=True)
+        samples = self._load_dataset()
+        samples = samples[:question_count]
+        questions = [s['question'] for s in samples]
+        gold_answers = get_gold_answers(samples)
+        print(f"done ({len(questions)} questions)")
+
+        # Run queries
+        print(f"\n  Running queries...")
+        start_time = time.time()
+        responses = []
+        success_count = 0
+
+        for i, question in enumerate(questions):
+            try:
+                print_progress(i + 1, len(questions), "Progress")
+                query_param = QueryParam(
+                    mode=mode,
+                    max_source_nodes=max_source_nodes,
+                    min_flow_threshold=min_flow_threshold,
+                    alpha=alpha,
+                    response_type="Brief, accurate answer (maximum 14 words)."
+                )
+                response = await rag.aquery(question, query_param)
+                if response and len(response.split()) > 14:
+                    response = " ".join(response.split()[:14])
+                responses.append(response)
+                success_count += 1
+            except Exception as e:
+                responses.append("")
+
+        query_time = time.time() - start_time
+
+        # Calculate metrics
+        f1_scores = []
+        em_scores = []
+        for gold_list, predicted in zip(gold_answers, responses):
+            if not predicted:
+                f1_scores.append(0.0)
+                em_scores.append(0.0)
+                continue
+            f1_scores.append(max(compute_f1(g, predicted) for g in gold_list))
+            em_scores.append(max(compute_exact_match(g, predicted) for g in gold_list))
+
+        result = BenchmarkResult(
+            model_name="QAFD_RAG",
+            dataset_name=self.dataset,
+            total_questions=len(questions),
+            success_count=success_count,
+            total_time=kg_build_time + query_time,
+            kg_build_time=kg_build_time,
+            query_time=query_time,
+            avg_time_per_question=query_time / len(questions) if questions else 0,
+            f1_score_mean=float(np.mean(f1_scores)) if f1_scores else 0,
+            f1_score_std=float(np.std(f1_scores)) if f1_scores else 0,
+            exact_match_mean=float(np.mean(em_scores)) if em_scores else 0,
+            exact_match_std=float(np.std(em_scores)) if em_scores else 0,
+            f1_scores=f1_scores,
+            exact_match_scores=em_scores,
+            responses=responses,
+            questions=questions,
+            gold_answers=gold_answers,
+        )
+
+        self.print_results(result)
+        return result
+
+    def print_results(self, result: BenchmarkResult):
+        """Print benchmark results"""
+        print_header(f"Results: {self.config['name']}")
+
+        print("\n  PERFORMANCE")
+        print(f"  {'─' * 40}")
+        print(f"  {'Questions':<25} {result.total_questions}")
+        print(f"  {'Successful':<25} {result.success_count}/{result.total_questions}")
+        print(f"  {'KG Build Time':<25} {result.kg_build_time:.2f}s")
+        print(f"  {'Query Time':<25} {result.query_time:.2f}s")
+        print(f"  {'Avg per Question':<25} {result.avg_time_per_question:.2f}s")
+
+        print("\n  ACCURACY METRICS")
+        print(f"  {'─' * 40}")
+        print_metric("F1 Score", result.f1_score_mean, result.f1_score_std)
+        print_metric("Exact Match", result.exact_match_mean, result.exact_match_std)
+        print()
+
+    def save_results(self, result: BenchmarkResult):
+        """Save results as two separate files: eval metrics and generated responses"""
+        timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
+        results_dir = os.path.join(QAFD_RAG_HOME, "results", "multihop", self.dataset)
+        os.makedirs(results_dir, exist_ok=True)
+
+        eval_file = os.path.join(results_dir, f"{self.dataset}_{timestamp}_eval.json")
+        output_file = os.path.join(results_dir, f"{self.dataset}_{timestamp}_responses.json")
+
+        # --- Eval file: metrics and timing ---
+        eval_data = {
+            "timestamp": datetime.now().isoformat(),
+            "graph_type": "entity",
+            "model": result.model_name,
+            "llm": self.llm_model,
+            "embedding": self.embedding_model,
+            "dataset": result.dataset_name,
+            "performance": {
+                "total_questions": result.total_questions,
+                "success_count": result.success_count,
+                "kg_build_time": result.kg_build_time,
+                "query_time": result.query_time,
+                "total_time": result.total_time,
+                "avg_time_per_question": result.avg_time_per_question,
+            },
+            "metrics": {
+                "f1_score_mean": result.f1_score_mean,
+                "f1_score_std": result.f1_score_std,
+                "exact_match_mean": result.exact_match_mean,
+                "exact_match_std": result.exact_match_std,
+            },
+            "per_question_f1": result.f1_scores,
+            "per_question_em": result.exact_match_scores,
+            "error": result.error_message,
+        }
+
+        with open(eval_file, 'w', encoding='utf-8') as f:
+            json.dump(eval_data, f, indent=2, ensure_ascii=False)
+
+        # --- Responses file: questions + generated answers + gold answers ---
+        output_entries = []
+        if result.responses:
+            for i, response in enumerate(result.responses):
+                entry = {
+                    "id": i + 1,
+                    "question": result.questions[i] if result.questions else "",
+                    "generated_answer": response,
+                    "gold_answers": result.gold_answers[i] if result.gold_answers else [],
+                }
+                output_entries.append(entry)
+
+        output_data = {
+            "timestamp": datetime.now().isoformat(),
+            "model": result.model_name,
+            "llm": self.llm_model,
+            "embedding": self.embedding_model,
+            "dataset": result.dataset_name,
+            "num_responses": len(output_entries),
+            "responses": output_entries,
+        }
+
+        with open(output_file, 'w', encoding='utf-8') as f:
+            json.dump(output_data, f, indent=2, ensure_ascii=False)
+
+        print(f"  Eval saved:      {eval_file}")
+        print(f"  Responses saved: {output_file}\n")
+
+
+async def main():
+    import argparse
+
+    parser = argparse.ArgumentParser(description="QAFD_RAG Multi-hop QA Benchmark")
+    parser.add_argument("--dataset", type=str, required=True,
+                        choices=["musique", "hotpotqa", "2wikimultihopqa"])
+    parser.add_argument("--questions", type=int, default=100)
+    parser.add_argument("--max-documents", type=int, default=None)
+    parser.add_argument("--build", action="store_true")
+    parser.add_argument("--force-build", action="store_true")
+    parser.add_argument("--embedding", type=str, default="openai-small",
+                        choices=["openai-small", "openai-large", "jina-v3", "gritlm", "nvidia-nv-embed-v2"])
+    parser.add_argument("--llm", type=str, default="gpt-4o-mini",
+                        choices=["gpt-4o-mini", "gpt-4o", "gpt-oss-120b", "gpt-5", "gpt-5-mini", "gpt-5-nano"])
+    parser.add_argument("--mode", type=str, default="hybrid",
+                        choices=["local", "global", "hybrid"])
+    parser.add_argument("--max-source-nodes", type=int, default=20)
+    parser.add_argument("--min-flow-threshold", type=float, default=0.1)
+    parser.add_argument("--alpha", type=float, default=2.0)
+
+    args = parser.parse_args()
+
+    api_key = os.environ.get("OPENAI_API_KEY")
+    if not api_key:
+        print("ERROR: Set OPENAI_API_KEY environment variable")
+        return
+
+    benchmark = MultiHopBenchmark(args.dataset, api_key, args.embedding, args.llm)
+
+    if args.build:
+        await benchmark.build_kg(max_documents=args.max_documents)
+        return
+
+    result = await benchmark.run_benchmark(
+        question_count=args.questions,
+        force_build=args.force_build,
+        max_documents=args.max_documents,
+        mode=args.mode,
+        max_source_nodes=args.max_source_nodes,
+        min_flow_threshold=args.min_flow_threshold,
+        alpha=args.alpha
+    )
+    benchmark.save_results(result)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

benchmarks/run.py

@@ -0,0 +1,371 @@
+#!/usr/bin/env python3
+"""
+Unified Benchmark Runner for QAFD-RAG
+======================================
+
+Supports two graph types (always with Query-Aware Flow Diffusion):
+
+  - **passage-entity**: Entities + passages + facts as nodes, synonymy edges.
+    Flow diffusion reaches passages directly. Default for multihop.
+
+  - **entity**: Classic KG with entity + relationship nodes.
+    Passages are looked up after graph traversal. Default for other tasks.
+
+Usage::
+
+    # Multihop (auto-selects passage-entity graph)
+    python benchmarks/run.py --task multihop --dataset musique
+
+    # Override graph type
+    python benchmarks/run.py --task multihop --dataset musique --graph_type entity
+
+    # Ultradomain (auto-selects entity graph)
+    python benchmarks/run.py --task ultradomain --dataset mix
+
+    # Text2SQL
+    python benchmarks/run.py --task text2sql --dataset spider2-lite
+
+    # Build KG only
+    python benchmarks/run.py --task multihop --dataset musique --build_only
+"""
+
+import argparse
+import asyncio
+import os
+import sys
+
+QAFD_RAG_HOME = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
+sys.path.insert(0, QAFD_RAG_HOME)
+
+# ── Task → dataset mapping ──────────────────────────────────────────────
+
+TASK_DATASETS = {
+    "multihop": ["musique", "hotpotqa", "2wikimultihopqa"],
+    "ultradomain": [
+        "agriculture", "biology", "cs", "finance", "legal",
+        "math", "medicine", "mix", "music", "philosophy",
+        "physics", "psychology",
+    ],
+    "text2sql": ["spider2-lite", "bird"],
+    "summarization": ["squality"],
+}
+
+# ── Default graph type per task ─────────────────────────────────────────
+
+TASK_DEFAULT_GRAPH_TYPE = {
+    "multihop": "passage-entity",
+    "ultradomain": "entity",
+    "text2sql": "entity",
+    "summarization": "entity",
+}
+
+# ── Default QAFD parameters per graph type ──────────────────────────────
+# These come from proven successful runs in each pipeline.
+
+GRAPH_TYPE_DEFAULTS = {
+    "passage-entity": {
+        "alpha": 2.0,
+        "epsilon": 0.01,
+        "max_iterations": 500,
+        "step_size": 0.2,
+        "weight_scheme": "original",
+        "linking_top_k": 5,           # fact seeds
+        "passage_node_weight": 0.05,
+        "retrieval_top_k": 200,
+    },
+    "entity": {
+        "alpha": 2.0,
+        "epsilon": 0.01,
+        "max_iterations": 500,
+        "step_size": 0.2,
+        "weight_scheme": "original",
+        "max_source_nodes": 20,
+        "min_flow_threshold": 0.1,
+    },
+}
+
+
+def run_passage_entity(args):
+    """Run benchmark using passage-entity graph (hipporag_pipeline)."""
+    # Bypass src/__init__.py (heavy AWS deps)
+    import types as _types
+    for _pkg_path in ["src", "src.retrievers", "src.hipporag_pipeline"]:
+        if _pkg_path not in sys.modules:
+            _m = _types.ModuleType(_pkg_path)
+            _m.__path__ = [os.path.join(QAFD_RAG_HOME, *_pkg_path.split("."))]
+            _m.__package__ = _pkg_path
+            sys.modules[_pkg_path] = _m
+
+    import importlib.util as _ilu
+    def _load_mod(fqn, filepath):
+        spec = _ilu.spec_from_file_location(fqn, filepath)
+        mod = _ilu.module_from_spec(spec)
+        sys.modules[fqn] = mod
+        spec.loader.exec_module(mod)
+        return mod
+
+    _src = os.path.join(QAFD_RAG_HOME, "src")
+    _load_mod("src.retrievers.base", os.path.join(_src, "retrievers", "base.py"))
+    _load_mod("src.retrievers.flow_diffusion", os.path.join(_src, "retrievers", "flow_diffusion.py"))
+
+    # Import after module setup
+    from src.hipporag_pipeline.benchmark_runner import main as hippo_main
+
+    # Build sys.argv for the sub-module
+    sub_argv = [
+        "benchmark_runner",
+        "--task", args.task,
+        "--dataset", args.dataset,
+        "--data_dir", os.path.join(QAFD_RAG_HOME, "data", "multihop"),
+        "--embedding_model", args.embedding,
+        "--llm_model", args.llm,
+        "--num_queries", str(args.questions),
+        "--qafd_alpha", str(args.alpha),
+        "--qafd_epsilon", str(args.epsilon),
+        "--qafd_max_iterations", str(args.max_iterations),
+        "--qafd_step_size", str(args.step_size),
+        "--qafd_weight_scheme", str(args.weight_scheme),
+        "--linking_top_k", str(args.linking_top_k),
+        "--passage_node_weight", str(args.passage_node_weight),
+        "--retrieval_top_k", str(args.retrieval_top_k),
+    ]
+    if args.skip_qa:
+        sub_argv.append("--skip_qa")
+    if getattr(args, 'batch_push', False):
+        sub_argv.append("--batch_push")
+    if args.force_build:
+        sub_argv.append("--force_index")
+        sub_argv.append("--force_openie")
+
+    old_argv = sys.argv
+    sys.argv = sub_argv
+    try:
+        hippo_main()
+    finally:
+        sys.argv = old_argv
+
+
+def run_entity(args):
+    """Run benchmark using entity graph (original QAFD-RAG pipeline)."""
+    import nest_asyncio
+    nest_asyncio.apply()
+
+    task = args.task
+
+    if task == "multihop":
+        from benchmarks.multihop.benchmark_multihop import MultiHopBenchmark
+
+        api_key = os.environ.get("OPENAI_API_KEY")
+        if not api_key:
+            print("ERROR: Set OPENAI_API_KEY environment variable")
+            return
+
+        benchmark = MultiHopBenchmark(args.dataset, api_key, args.embedding, args.llm)
+
+        if args.build_only:
+            asyncio.run(benchmark.build_kg(max_documents=args.max_documents))
+            return
+
+        result = asyncio.run(benchmark.run_benchmark(
+            question_count=args.questions,
+            force_build=args.force_build,
+            max_documents=args.max_documents,
+            mode="hybrid",
+            max_source_nodes=args.max_source_nodes,
+            min_flow_threshold=args.min_flow_threshold,
+            alpha=args.alpha,
+        ))
+        benchmark.save_results(result)
+
+    elif task == "ultradomain":
+        # Delegate to ultradomain's own argparse
+        # Ultradomain expects dataset as "mix.jsonl" format
+        ud_dataset = args.dataset if args.dataset.endswith(".jsonl") else f"{args.dataset}.jsonl"
+        sub_argv = [
+            "benchmark_ultradomain",
+            "--dataset", ud_dataset,
+            "--questions", str(args.questions),
+            "--embedding", args.embedding,
+            "--llm", args.llm,
+        ]
+        if args.force_build:
+            sub_argv.append("--force-build")
+        if args.build_only:
+            sub_argv.append("--build")
+        if args.max_documents:
+            sub_argv.extend(["--max-documents", str(args.max_documents)])
+
+        old_argv = sys.argv
+        sys.argv = sub_argv
+        try:
+            from benchmarks.ultradomain.benchmark_ultradomain import main as ultra_main
+            asyncio.run(ultra_main())
+        finally:
+            sys.argv = old_argv
+
+    elif task == "text2sql":
+        sub_argv = ["benchmark_text2sql"]
+        if args.max_documents:
+            sub_argv.extend(["--max-documents", str(args.max_documents)])
+
+        old_argv = sys.argv
+        sys.argv = sub_argv
+        try:
+            from benchmarks.text2sql.benchmark_text2sql import main as text2sql_main
+            text2sql_main()
+        finally:
+            sys.argv = old_argv
+
+    elif task == "summarization":
+        sub_argv = [
+            "benchmark_summarization",
+            "--dataset", args.dataset,
+            "--questions", str(args.questions),
+            "--embedding", args.embedding,
+            "--llm", args.llm,
+        ]
+        if args.force_build:
+            sub_argv.append("--force-build")
+        if args.build_only:
+            sub_argv.append("--build")
+        if args.max_documents:
+            sub_argv.extend(["--max-documents", str(args.max_documents)])
+
+        old_argv = sys.argv
+        sys.argv = sub_argv
+        try:
+            from benchmarks.summarization.benchmark_summarization import main as summ_main
+            asyncio.run(summ_main())
+        finally:
+            sys.argv = old_argv
+
+    else:
+        print(f"ERROR: Unknown task '{task}'")
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="QAFD-RAG Unified Benchmark Runner",
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+        epilog="""
+Examples:
+  # Multihop with passage-entity graph (default)
+  python benchmarks/run.py --task multihop --dataset musique
+
+  # Multihop with entity graph (override)
+  python benchmarks/run.py --task multihop --dataset musique --graph_type entity
+
+  # Ultradomain with entity graph (default)
+  python benchmarks/run.py --task ultradomain --dataset mix
+
+  # Retrieval only (skip QA)
+  python benchmarks/run.py --task multihop --dataset musique --skip_qa
+
+  # Build KG only
+  python benchmarks/run.py --task multihop --dataset musique --build_only
+""",
+    )
+
+    # ── Required ────────────────────────────────────────────────────────
+    parser.add_argument("--task", type=str, required=True,
+                        choices=["multihop", "ultradomain", "text2sql", "summarization"])
+    parser.add_argument("--dataset", type=str, required=True,
+                        help="Dataset name (e.g. musique, hotpotqa, mix, spider2-lite)")
+
+    # ── Graph type ──────────────────────────────────────────────────────
+    parser.add_argument("--graph_type", type=str, default=None,
+                        choices=["passage-entity", "entity"],
+                        help="Graph type (default: passage-entity for multihop, entity for others)")
+
+    # ── Model ──────────────────────────���────────────────────────────────
+    parser.add_argument("--llm", type=str, default="gpt-4o-mini")
+    parser.add_argument("--embedding", type=str, default=None,
+                        help="Embedding model (auto-selected per task: "
+                             "nvidia-nv-embed-v2 for multihop, openai-small for others)")
+
+    # ── Run control ─────────────────────────────────────────────────────
+    parser.add_argument("--questions", type=int, default=100)
+    parser.add_argument("--max_documents", type=int, default=None)
+    parser.add_argument("--build_only", action="store_true",
+                        help="Build KG only, skip benchmark")
+    parser.add_argument("--force_build", action="store_true",
+                        help="Rebuild KG even if it exists")
+    parser.add_argument("--skip_qa", action="store_true",
+                        help="Run retrieval only, skip QA (passage-entity only)")
+    parser.add_argument("--batch_push", action="store_true",
+                        help="Batch push-relabel (process all excess nodes per iter)")
+
+    # ── QAFD parameters (shared) ────────────────────────────────────────
+    parser.add_argument("--alpha", type=float, default=None,
+                        help="QAFD alpha (default: 2.0)")
+    parser.add_argument("--epsilon", type=float, default=None)
+    parser.add_argument("--max_iterations", type=int, default=None)
+    parser.add_argument("--step_size", type=float, default=None)
+    parser.add_argument("--weight_scheme", type=str, default=None,
+                        choices=["original", "multiply", "add"])
+
+    # ── Passage-entity specific ─────────────────────────────────────────
+    parser.add_argument("--linking_top_k", type=int, default=None,
+                        help="Number of fact seeds (passage-entity only)")
+    parser.add_argument("--passage_node_weight", type=float, default=None,
+                        help="Passage node weight in seed computation (passage-entity only)")
+    parser.add_argument("--retrieval_top_k", type=int, default=None,
+                        help="Number of passages to retrieve (passage-entity only)")
+
+    # ── Entity graph specific ───────────────────────────────────────────
+    parser.add_argument("--max_source_nodes", type=int, default=None,
+                        help="Max source nodes for flow diffusion (entity only)")
+    parser.add_argument("--min_flow_threshold", type=float, default=None,
+                        help="Min flow threshold for clusters (entity only)")
+
+    args = parser.parse_args()
+
+    # ── Resolve graph type ──────────────────────────────────────────────
+    if args.graph_type is None:
+        args.graph_type = TASK_DEFAULT_GRAPH_TYPE[args.task]
+
+    # ── Resolve embedding (must match pre-built KGs on HuggingFace) ────
+    if args.embedding is None:
+        _task_embeddings = {
+            "multihop": "nvidia-nv-embed-v2",
+            "ultradomain": "openai-small",
+            "text2sql": "openai-small",
+            "summarization": "openai-small",
+        }
+        args.embedding = _task_embeddings[args.task]
+
+    # ── Validate dataset ────────────────────────────────────────────────
+    valid = TASK_DATASETS.get(args.task, [])
+    if args.dataset not in valid and args.dataset != "all":
+        print(f"ERROR: Unknown dataset '{args.dataset}' for task '{args.task}'")
+        print(f"  Valid: {valid}")
+        return
+
+    # ── Apply graph-type defaults for unset params ──────────────────────
+    defaults = GRAPH_TYPE_DEFAULTS[args.graph_type]
+    for key, default_val in defaults.items():
+        if getattr(args, key, None) is None:
+            setattr(args, key, default_val)
+
+    # ── Print config ────────────────────────────────────────────────────
+    print(f"\n{'=' * 70}")
+    print(f"  QAFD-RAG Benchmark")
+    print(f"{'=' * 70}")
+    print(f"  Task:         {args.task}")
+    print(f"  Dataset:      {args.dataset}")
+    print(f"  Graph type:   {args.graph_type}")
+    print(f"  LLM:          {args.llm}")
+    print(f"  Embedding:    {args.embedding}")
+    print(f"  QAFD alpha:   {args.alpha}")
+    print(f"  Questions:    {args.questions}")
+    print(f"{'=' * 70}\n")
+
+    # ── Dispatch ──────────────────���─────────────────────────────────────
+    if args.graph_type == "passage-entity":
+        run_passage_entity(args)
+    else:
+        run_entity(args)
+
+
+if __name__ == "__main__":
+    main()

benchmarks/summarization/benchmark_summarization.py

@@ -0,0 +1,673 @@
+#!/usr/bin/env python3
+"""
+SQuALITY Benchmark for QAFD_RAG
+Multi-reference BLEU/ROUGE/METEOR metrics + LLM quality evaluation
+"""
+
+import os
+import sys
+import asyncio
+import time
+import json
+import logging
+import nest_asyncio
+from typing import Dict, List
+from dataclasses import dataclass
+from datetime import datetime
+from pathlib import Path
+
+# Suppress verbose logging
+logging.getLogger("httpx").setLevel(logging.ERROR)
+logging.getLogger("QAFD_RAG").setLevel(logging.ERROR)
+logging.getLogger("nano-vectordb").setLevel(logging.ERROR)
+logging.getLogger("OpenAI").setLevel(logging.ERROR)
+logging.getLogger("absl").setLevel(logging.ERROR)
+
+QAFD_RAG_HOME = str(Path(__file__).parent.parent.parent)
+sys.path.insert(0, QAFD_RAG_HOME)
+
+nest_asyncio.apply()
+
+
+def print_header(title: str, width: int = 70):
+    """Print a formatted header"""
+    print(f"\n{'=' * width}")
+    print(f"  {title}")
+    print(f"{'=' * width}")
+
+
+def print_config(items: Dict[str, str], width: int = 70):
+    """Print configuration items"""
+    print(f"{'─' * width}")
+    for key, value in items.items():
+        print(f"  {key:<20} {value}")
+    print(f"{'─' * width}")
+
+
+def print_progress(current: int, total: int, prefix: str = "", width: int = 40):
+    """Print a progress bar"""
+    percent = current / total
+    filled = int(width * percent)
+    bar = '█' * filled + '░' * (width - filled)
+    print(f"\r  {prefix} [{bar}] {current}/{total} ({percent*100:.1f}%)", end='', flush=True)
+    if current == total:
+        print()
+
+@dataclass
+class BenchmarkResult:
+    """Benchmark results"""
+    model_name: str
+    dataset_name: str
+    success: bool = False
+    total_time: float = 0.0
+    kg_build_time: float = 0.0
+    query_time: float = 0.0
+    num_questions: int = 0
+    metrics: Dict[str, float] = None  # BLEU, ROUGE, METEOR
+    quality_scores: Dict[str, List[float]] = None  # Comprehensiveness, Diversity, etc.
+    error_message: str = ""
+    responses: List[str] = None  # Generated summaries
+    questions: List[str] = None  # Original questions
+    reference_answers: List[List[str]] = None  # Reference answers per question
+
+class SQuALITYBenchmark:
+    def __init__(self, api_key: str, embedding_model: str = "openai-small", llm_model: str = "gpt-4o-mini"):
+        self.api_key = api_key
+        self.embedding_model = embedding_model
+        self.llm_model = llm_model
+        os.environ["OPENAI_API_KEY"] = api_key
+        os.environ["OPENAI_API_BASE"] = "https://api.openai.com/v1"
+
+    def _get_llm_func(self):
+        """Get LLM function based on model name"""
+        from src import llm
+        llm_funcs = {
+            "gpt-4o-mini": llm.gpt_4o_mini_complete,
+            "gpt-4o": llm.gpt_4o_complete,
+            "gpt-oss-120b": llm.gpt_oss_120b_complete,
+            "gpt-5": llm.gpt_5_complete,
+            "gpt-5-mini": llm.gpt_5_mini_complete,
+            "gpt-5-nano": llm.gpt_5_nano_complete,
+        }
+        return llm_funcs.get(self.llm_model, llm.gpt_4o_mini_complete)
+
+    def _get_working_dir(self, dataset_name: str) -> str:
+        """Get working directory for dataset"""
+        return os.path.join(QAFD_RAG_HOME, "kg", "summarization", f"{self.llm_model}_{self.embedding_model}_{dataset_name}")
+
+    def _kg_exists(self, working_dir: str) -> bool:
+        """Check if KG already exists"""
+        # Check for key files that indicate a built KG
+        kg_files = [
+            os.path.join(working_dir, "vdb_entities.json"),
+            os.path.join(working_dir, "vdb_chunks.json"),
+            os.path.join(working_dir, "kv_store_full_docs.json"),
+        ]
+        return all(os.path.exists(f) for f in kg_files)
+
+    async def _benchmark_qafd(self, dataset: List[Dict], dataset_name: str,
+                            question_count: int, max_documents: int = None,
+                            mode: str = "hybrid", max_source_nodes: int = 40,
+                            min_flow_threshold: float = 0.01, alpha: float = 5.0,
+                            evaluator: str = "mini", force_build: bool = False) -> BenchmarkResult:
+        """Benchmark QAFD_RAG on SQuALITY"""
+        try:
+            from src.QAFD_RAG import QAFD_RAG, QueryParam
+
+            working_dir = self._get_working_dir(dataset_name)
+
+            llm_func = self._get_llm_func()
+            rag = QAFD_RAG(
+                working_dir=working_dir,
+                llm_model_func=llm_func,
+                llm_model_name=self.llm_model,
+                embedding_model_key=self.embedding_model,
+                enable_llm_cache=True,
+            )
+
+            # Get unique documents and selected questions
+            unique_docs = {}
+            selected_questions = []
+
+            for item in dataset:
+                passage_id = item["passage_id"]
+
+                if passage_id not in unique_docs:
+                    if max_documents and len(unique_docs) >= max_documents:
+                        continue
+                    unique_docs[passage_id] = item["document"]
+
+                if passage_id in unique_docs:
+                    selected_questions.append(item)
+                    if len(selected_questions) >= question_count:
+                        break
+
+            dataset = selected_questions
+
+            # Check if KG exists or needs to be built
+            kg_build_time = 0.0
+            if self._kg_exists(working_dir) and not force_build:
+                print(f"\n  Using existing KG: {working_dir}")
+            else:
+                print(f"\n  Building KG from {len(unique_docs)} documents...")
+                start_time = time.time()
+
+                for i, (passage_id, document) in enumerate(unique_docs.items()):
+                    print_progress(i + 1, len(unique_docs), "Progress")
+                    await rag.ainsert(document)
+
+                kg_build_time = time.time() - start_time
+                print(f"  KG built in {kg_build_time:.2f}s")
+
+            # Query phase
+            print(f"\n  Running queries...")
+            start_time = time.time()
+            responses = []
+
+            for i in range(min(question_count, len(dataset))):
+                print_progress(i + 1, min(question_count, len(dataset)), "Progress")
+                question = dataset[i]["question"]
+
+                query_param = QueryParam(
+                    mode=mode,
+                    max_source_nodes=max_source_nodes,
+                    min_flow_threshold=min_flow_threshold,
+                    alpha=alpha
+                )
+                response = await rag.aquery(question, query_param)
+                responses.append(response)
+
+            query_time = time.time() - start_time
+
+            # Calculate metrics with MULTIPLE REFERENCES
+            print(f"\n  Calculating reference metrics...", end=" ", flush=True)
+            metrics = self._calculate_metrics(responses, dataset[:len(responses)])
+            print("done")
+
+            # Calculate quality scores
+            print(f"  Evaluating quality ({len(responses)} responses)...")
+            quality_scores = await self._evaluate_quality(dataset, responses, evaluator)
+            
+            # Collect questions and reference answers for output
+            questions_list = [dataset[i]["question"] for i in range(len(responses))]
+            references_list = [dataset[i].get("all_answers", [dataset[i]["answer"]]) for i in range(len(responses))]
+
+            return BenchmarkResult(
+                model_name="QAFD_RAG",
+                dataset_name=dataset_name,
+                success=True,
+                total_time=kg_build_time + query_time,
+                kg_build_time=kg_build_time,
+                query_time=query_time,
+                num_questions=len(responses),
+                metrics=metrics,
+                quality_scores=quality_scores,
+                responses=responses,
+                questions=questions_list,
+                reference_answers=references_list,
+            )
+
+        except Exception as e:
+            print(f"  ERROR: {str(e)}")
+            import traceback
+            traceback.print_exc()
+            return BenchmarkResult(
+                model_name="QAFD_RAG",
+                dataset_name=dataset_name,
+                success=False,
+                error_message=str(e)
+            )
+    
+    def _calculate_metrics(self, responses: List[str], dataset: List[Dict]) -> Dict[str, float]:
+        """Calculate BLEU, ROUGE, METEOR metrics with MULTIPLE REFERENCES"""
+        try:
+            from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction
+            from nltk.translate.meteor_score import meteor_score
+            from nltk.tokenize import word_tokenize
+            from rouge_score import rouge_scorer
+            import nltk
+            
+            nltk.download('wordnet', quiet=True)
+            nltk.download('omw-1.4', quiet=True)
+            nltk.download('punkt', quiet=True)
+            
+            scorer = rouge_scorer.RougeScorer(['rouge1', 'rouge2'], use_stemmer=True)
+            smoothing = SmoothingFunction().method1
+            
+            bleu1_scores = []
+            bleu2_scores = []
+            rouge1_scores = []
+            rouge2_scores = []
+            meteor_scores = []
+            
+            for i, response in enumerate(responses):
+                if not response:
+                    continue
+                
+                # Get ALL reference answers for this question
+                all_refs = dataset[i].get("all_answers", [dataset[i]["answer"]])
+                
+                # Tokenize all references
+                try:
+                    ref_tokens_list = [word_tokenize(ref.lower()) for ref in all_refs]
+                    resp_tokens = word_tokenize(response.lower())
+                except:
+                    ref_tokens_list = [ref.lower().split() for ref in all_refs]
+                    resp_tokens = response.lower().split()
+                
+                # BLEU with multiple references
+                bleu1 = sentence_bleu(ref_tokens_list, resp_tokens, weights=(1,0,0,0), smoothing_function=smoothing)
+                bleu2 = sentence_bleu(ref_tokens_list, resp_tokens, weights=(0.5,0.5,0,0), smoothing_function=smoothing)
+                bleu1_scores.append(bleu1)
+                bleu2_scores.append(bleu2)
+                
+                # ROUGE uses first reference (standard)
+                primary_ref = all_refs[0]
+                rouge_result = scorer.score(primary_ref, response)
+                rouge1_scores.append(rouge_result['rouge1'].fmeasure)
+                rouge2_scores.append(rouge_result['rouge2'].fmeasure)
+                
+                # METEOR with multiple references
+                try:
+                    meteor = meteor_score(ref_tokens_list, resp_tokens)
+                    meteor_scores.append(meteor)
+                except:
+                    meteor_scores.append(0.0)
+            
+            return {
+                "bleu_1": sum(bleu1_scores) / len(bleu1_scores) if bleu1_scores else 0,
+                "bleu_2": sum(bleu2_scores) / len(bleu2_scores) if bleu2_scores else 0,
+                "rouge_1_f1": sum(rouge1_scores) / len(rouge1_scores) if rouge1_scores else 0,
+                "rouge_2_f1": sum(rouge2_scores) / len(rouge2_scores) if rouge2_scores else 0,
+                "meteor": sum(meteor_scores) / len(meteor_scores) if meteor_scores else 0,
+            }
+            
+        except Exception as e:
+            print(f"  Warning: Metrics calculation failed: {e}")
+            import traceback
+            traceback.print_exc()
+            return {}
+    
+    async def _evaluate_quality(self, dataset: List[Dict], responses: List[str],
+                               evaluator: str = "mini") -> Dict[str, List[float]]:
+        """Evaluate response quality using GPT-4o or GPT-4o-mini"""
+        if not responses:
+            return {}
+
+        try:
+            if evaluator == "4o":
+                from src.llm import gpt_4o_complete as eval_complete
+            else:
+                from src.llm import gpt_4o_mini_complete as eval_complete
+
+            criteria = ["comprehensiveness", "diversity", "logicality", "relevance", "coherence"]
+            response_scores = {criterion: [] for criterion in criteria}
+
+            for i in range(len(responses)):
+                print_progress(i + 1, len(responses), "Evaluating")
+                
+                question = dataset[i]["question"]
+                response = responses[i]
+                
+                if not response:
+                    for criterion in criteria:
+                        response_scores[criterion].append(0.0)
+                    continue
+                
+                response_criterion_scores = {criterion: [] for criterion in criteria}
+                for _ in range(5):
+                    prompt = f"""Evaluate the following response based on five criteria. Rate each from 0-100.
+
+Question: {question}
+Response: {response}
+
+Please evaluate based on these criteria:
+- Comprehensiveness: How much detail does the answer provide to cover all aspects and details of the question?
+- Diversity: How varied and rich is the answer in providing different perspectives and insights on the question?
+- Logicality: How logically does the answer respond to all parts of the question?
+- Relevance: How relevant is the answer to the question, staying focused and addressing the intended topic or issue?
+- Coherence: How well does the answer maintain internal logical connections between its parts, ensuring a smooth and consistent structure?
+
+JSON format:
+{{
+    "comprehensiveness": [score],
+    "diversity": [score],
+    "logicality": [score],
+    "relevance": [score],
+    "coherence": [score]
+}}"""
+
+                    evaluation_result = await eval_complete(prompt, max_tokens=200)
+                    
+                    import re
+                    json_match = re.search(r'\{.*\}', evaluation_result, re.DOTALL)
+                    if json_match:
+                        try:
+                            scores = json.loads(json_match.group())
+                            for criterion in criteria:
+                                if criterion in scores:
+                                    score_value = float(scores[criterion])
+                                    if 0 <= score_value <= 100:
+                                        response_criterion_scores[criterion].append(score_value)
+                        except:
+                            pass
+                
+                for criterion in criteria:
+                    if response_criterion_scores[criterion]:
+                        avg = sum(response_criterion_scores[criterion]) / len(response_criterion_scores[criterion])
+                        response_scores[criterion].append(avg)
+                    else:
+                        response_scores[criterion].append(0.0)
+            
+            return response_scores
+        except Exception as e:
+            print(f"  Warning: Quality evaluation failed: {e}")
+            return {}
+    
+    def _download_dataset(self) -> List[Dict]:
+        """Download SQuALITY dataset - COLLECTS ALL REFERENCE ANSWERS"""
+        try:
+            from datasets import load_dataset
+            import warnings
+            warnings.filterwarnings("ignore", category=FutureWarning)
+            logging.getLogger("datasets").setLevel(logging.WARNING)
+        except ImportError:
+            print("ERROR: Install datasets with: pip install datasets")
+            sys.exit(1)
+
+        dataset = load_dataset("pszemraj/SQuALITY-v1.3", split="train")
+        
+        prepared_data = []
+        for i, item in enumerate(dataset):
+            document = item.get("document", "")
+            questions = item.get("questions", [])
+            metadata = item.get("metadata", {})
+            passage_id = metadata.get("passage_id", f"passage_{i}") if isinstance(metadata, dict) else f"passage_{i}"
+            
+            if not document or not questions:
+                continue
+            
+            for q in questions:
+                question_text = q.get("question_text", "")
+                responses = q.get("responses", [])
+                
+                if responses and len(responses) > 0:
+                    # COLLECT ALL REFERENCE ANSWERS (KEY CHANGE!)
+                    answer_texts = [r.get("response_text", "") for r in responses if r.get("response_text", "")]
+                    
+                    if question_text and answer_texts:
+                        prepared_data.append({
+                            "document": document,
+                            "question": question_text,
+                            "answer": answer_texts[0],  # Primary answer
+                            "all_answers": answer_texts,  # All references for BLEU
+                            "passage_id": passage_id
+                        })
+        
+        return prepared_data
+    
+    async def build_kg(self, dataset_name: str = "squality", max_documents: int = None) -> bool:
+        """Build KG only (no benchmark)"""
+        print_header("QAFD-RAG Knowledge Graph Builder")
+        print_config({
+            "Dataset": "SQuALITY",
+            "Embedding": self.embedding_model,
+            "LLM": self.llm_model,
+            "Max Documents": str(max_documents) if max_documents else "all"
+        })
+
+        print("\n  Downloading dataset...", end=" ", flush=True)
+        dataset = self._download_dataset()
+        print("done")
+
+        try:
+            from src.QAFD_RAG import QAFD_RAG
+
+            working_dir = self._get_working_dir(dataset_name)
+            os.makedirs(working_dir, exist_ok=True)
+
+            llm_func = self._get_llm_func()
+            rag = QAFD_RAG(
+                working_dir=working_dir,
+                llm_model_func=llm_func,
+                llm_model_name=self.llm_model,
+                embedding_model_key=self.embedding_model,
+                enable_llm_cache=True,
+            )
+
+            # Get unique documents
+            unique_docs = {}
+            for item in dataset:
+                passage_id = item["passage_id"]
+                if passage_id not in unique_docs:
+                    if max_documents and len(unique_docs) >= max_documents:
+                        break
+                    unique_docs[passage_id] = item["document"]
+
+            print(f"\n  Building KG from {len(unique_docs)} documents...")
+            start_time = time.time()
+
+            for i, (passage_id, document) in enumerate(unique_docs.items()):
+                print_progress(i + 1, len(unique_docs), "Progress")
+                await rag.ainsert(document)
+
+            build_time = time.time() - start_time
+
+            print_header("Build Complete")
+            print(f"  Documents processed:  {len(unique_docs)}")
+            print(f"  Time elapsed:         {build_time:.2f}s")
+            print(f"  Avg per document:     {build_time/len(unique_docs):.2f}s")
+            print(f"  Output directory:     {working_dir}")
+            print()
+            return True
+
+        except Exception as e:
+            print(f"\n  ERROR: {e}")
+            return False
+
+    async def run_benchmark(self, dataset_name: str = "squality", question_count: int = 250, max_documents: int = None,
+                           mode: str = "hybrid", max_source_nodes: int = 40,
+                           min_flow_threshold: float = 0.01, alpha: float = 10.0,
+                           evaluator: str = "mini", force_build: bool = False) -> BenchmarkResult:
+        """Run benchmark"""
+        print_header("QAFD-RAG SQuALITY Benchmark")
+        print_config({
+            "Questions": str(question_count),
+            "Embedding": self.embedding_model,
+            "LLM": self.llm_model,
+            "Mode": mode,
+            "Max Nodes": str(max_source_nodes),
+            "Threshold": str(min_flow_threshold),
+            "Alpha": str(alpha),
+            "Evaluator": "GPT-4o" if evaluator == "4o" else "GPT-4o-mini"
+        })
+
+        print("\n  Downloading dataset...", end=" ", flush=True)
+        dataset = self._download_dataset()
+        print("done")
+
+        if len(dataset) < question_count:
+            question_count = len(dataset)
+
+        result = await self._benchmark_qafd(dataset, dataset_name, question_count, max_documents,
+                                           mode, max_source_nodes, min_flow_threshold, alpha,
+                                           evaluator, force_build)
+        return result
+    
+    def print_results(self, result: BenchmarkResult):
+        """Print results"""
+        print_header("Results: SQuALITY")
+
+        if not result.success:
+            print(f"\n  Status: FAILED")
+            print(f"  Error: {result.error_message}")
+            return
+
+        print("\n  PERFORMANCE")
+        print(f"  {'─' * 40}")
+        print(f"  {'Questions':<25} {result.num_questions}")
+        print(f"  {'KG Build Time':<25} {result.kg_build_time:.2f}s")
+        print(f"  {'Query Time':<25} {result.query_time:.2f}s")
+        print(f"  {'Total Time':<25} {result.total_time:.2f}s")
+        if result.num_questions > 0:
+            print(f"  {'Avg per Question':<25} {result.query_time/result.num_questions:.2f}s")
+
+        if result.metrics:
+            print("\n  REFERENCE METRICS")
+            print(f"  {'─' * 40}")
+            for k, v in result.metrics.items():
+                print(f"  {k:<25} {v*100:.2f}%")
+
+        if result.quality_scores:
+            print("\n  QUALITY METRICS")
+            print(f"  {'─' * 40}")
+            import statistics
+            for k, v in result.quality_scores.items():
+                if v:
+                    avg = sum(v) / len(v)
+                    std = statistics.stdev(v) if len(v) > 1 else 0
+                    print(f"  {k:<25} {avg:.2f} ± {std:.2f}")
+
+            all_scores = []
+            for scores in result.quality_scores.values():
+                all_scores.extend(scores)
+            if all_scores:
+                overall = sum(all_scores) / len(all_scores)
+                print(f"\n  {'Overall Average':<25} {overall:.2f}")
+        print()
+    
+    def save_results(self, result: BenchmarkResult, dataset_name: str = "squality"):
+        """Save results as two separate files: eval metrics and generated responses"""
+        timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
+        results_dir = os.path.join(QAFD_RAG_HOME, "results", "summarization", dataset_name)
+        os.makedirs(results_dir, exist_ok=True)
+
+        eval_file = os.path.join(results_dir, f"{dataset_name}_{timestamp}_eval.json")
+        output_file = os.path.join(results_dir, f"{dataset_name}_{timestamp}_responses.json")
+
+        # --- Eval file: metrics, quality scores, timing ---
+        quality_averages = {}
+        quality_stds = {}
+        if result.quality_scores:
+            import statistics
+            for criterion, scores in result.quality_scores.items():
+                if scores:
+                    quality_averages[criterion] = sum(scores) / len(scores)
+                    quality_stds[criterion] = statistics.stdev(scores) if len(scores) > 1 else 0.0
+
+        eval_data = {
+            "timestamp": datetime.now().isoformat(),
+            "model": result.model_name,
+            "llm": self.llm_model,
+            "embedding": self.embedding_model,
+            "success": result.success,
+            "performance": {
+                "num_questions": result.num_questions,
+                "kg_build_time": result.kg_build_time,
+                "query_time": result.query_time,
+                "total_time": result.total_time,
+                "avg_per_question": result.query_time / result.num_questions if result.num_questions > 0 else 0
+            },
+            "reference_metrics": result.metrics,
+            "quality_scores": quality_averages,
+            "quality_stds": quality_stds,
+            "quality_raw": result.quality_scores,
+            "error": result.error_message
+        }
+
+        with open(eval_file, 'w') as f:
+            json.dump(eval_data, f, indent=2)
+
+        # --- Responses file: questions + generated summaries + references ---
+        output_entries = []
+        if result.responses:
+            for i, response in enumerate(result.responses):
+                entry = {
+                    "id": i + 1,
+                    "question": result.questions[i] if result.questions else "",
+                    "generated_summary": response,
+                    "reference_answers": result.reference_answers[i] if result.reference_answers else [],
+                }
+                output_entries.append(entry)
+
+        output_data = {
+            "timestamp": datetime.now().isoformat(),
+            "model": result.model_name,
+            "llm": self.llm_model,
+            "embedding": self.embedding_model,
+            "num_responses": len(output_entries),
+            "responses": output_entries,
+        }
+
+        with open(output_file, 'w') as f:
+            json.dump(output_data, f, indent=2, ensure_ascii=False)
+
+        print(f"\nEval saved:      {eval_file}")
+        print(f"Responses saved: {output_file}")
+
+async def main():
+    """Main entry point"""
+    import argparse
+    
+    parser = argparse.ArgumentParser(description="SQuALITY Benchmark for QAFD_RAG")
+    parser.add_argument("--questions", type=int, default=250,
+                       help="Number of questions (default: 250)")
+    parser.add_argument("--max-documents", type=int, default=None,
+                       help="Max documents (default: unlimited)")
+    parser.add_argument("--mode", type=str, default="hybrid",
+                       choices=["local", "global", "hybrid"],
+                       help="Query mode (default: hybrid)")
+    parser.add_argument("--max-source-nodes", type=int, default=40,
+                       help="Max source nodes (default: 40)")
+    parser.add_argument("--min-flow-threshold", type=float, default=0.01,
+                       help="Min flow threshold (default: 0.01)")
+    parser.add_argument("--alpha", type=float, default=10.0,
+                       help="Alpha (default: 10.0)")
+    parser.add_argument("--evaluator", type=str, default="mini",
+                       choices=["mini", "4o"],
+                       help="Evaluator model: mini (gpt-4o-mini) or 4o (gpt-4o) (default: mini)")
+    parser.add_argument("--api-key", type=str,
+                       help="OpenAI API key")
+    parser.add_argument("--build", action="store_true",
+                       help="Build KG only (no benchmark)")
+    parser.add_argument("--force-build", action="store_true",
+                       help="Force rebuild KG even if it exists")
+    parser.add_argument("--embedding", type=str, default="openai-small",
+                       choices=["openai-small", "openai-large", "jina-v3", "gritlm", "nvidia-nv-embed-v2"],
+                       help="Embedding model (default: openai-small)")
+    parser.add_argument("--llm", type=str, default="gpt-4o-mini",
+                       choices=["gpt-4o-mini", "gpt-4o", "gpt-oss-120b", "gpt-5", "gpt-5-mini", "gpt-5-nano"],
+                       help="LLM model (default: gpt-4o-mini)")
+    parser.add_argument("--dataset", type=str, default="squality",
+                       help="Dataset name (default: squality)")
+
+    args = parser.parse_args()
+
+    api_key = args.api_key or os.environ.get("OPENAI_API_KEY")
+    if not api_key:
+        print("ERROR: Set OPENAI_API_KEY")
+        return
+
+    benchmark = SQuALITYBenchmark(api_key, args.embedding, args.llm)
+
+    # Build-only mode
+    if args.build:
+        await benchmark.build_kg(dataset_name=args.dataset, max_documents=args.max_documents)
+        return
+
+    result = await benchmark.run_benchmark(
+        dataset_name=args.dataset,
+        question_count=args.questions,
+        max_documents=args.max_documents,
+        mode=args.mode,
+        max_source_nodes=args.max_source_nodes,
+        min_flow_threshold=args.min_flow_threshold,
+        alpha=args.alpha,
+        evaluator=args.evaluator,
+        force_build=args.force_build
+    )
+    benchmark.print_results(result)
+    benchmark.save_results(result, dataset_name=args.dataset)
+
+if __name__ == "__main__":
+    asyncio.run(main())

benchmarks/text2sql/benchmark_text2sql.py

@@ -0,0 +1,385 @@
+#!/usr/bin/env python3
+"""
+Text2SQL Benchmark using QAFD-RAG for schema context retrieval.
+Auto-builds KGs from database summaries.
+"""
+
+import os
+import sys
+import asyncio
+import json
+import logging
+from datetime import datetime
+from typing import Dict, List, Any
+from dataclasses import dataclass, asdict
+from pathlib import Path
+
+# Suppress verbose logging
+logging.getLogger("httpx").setLevel(logging.ERROR)
+logging.getLogger("QAFD_RAG").setLevel(logging.ERROR)
+logging.getLogger("nano-vectordb").setLevel(logging.ERROR)
+logging.getLogger("OpenAI").setLevel(logging.ERROR)
+
+# Add QAFD-RAG to path
+QAFD_RAG_HOME = Path(__file__).parent.parent.parent
+sys.path.insert(0, str(QAFD_RAG_HOME))
+
+import nest_asyncio
+nest_asyncio.apply()
+
+
+def print_header(title: str, width: int = 70):
+    """Print a formatted header"""
+    print(f"\n{'=' * width}")
+    print(f"  {title}")
+    print(f"{'=' * width}")
+
+
+def print_config(items: Dict[str, str], width: int = 70):
+    """Print configuration items"""
+    print(f"{'─' * width}")
+    for key, value in items.items():
+        print(f"  {key:<20} {value}")
+    print(f"{'─' * width}")
+
+
+def print_progress(current: int, total: int, prefix: str = "", width: int = 40):
+    """Print a progress bar"""
+    percent = current / total
+    filled = int(width * percent)
+    bar = '█' * filled + '░' * (width - filled)
+    print(f"\r  {prefix} [{bar}] {current}/{total} ({percent*100:.1f}%)", end='', flush=True)
+    if current == total:
+        print()
+
+
+@dataclass
+class Text2SQLResult:
+    instance_id: str
+    db: str
+    question: str
+    create_table: str
+    success: bool = True
+    error_message: str = ""
+    retrieved_tables: List[str] = None
+    retrieved_columns: List[str] = None
+
+
+def parse_schema_from_create_table(create_table: str) -> tuple:
+    """Extract table names and table.column pairs from CREATE TABLE output."""
+    import re
+    tables = []
+    columns = []
+    current_table = None
+    for line in create_table.split('\n'):
+        m = re.match(r'^CREATE TABLE\s+`?(\w+)`?\s*\(', line)
+        if m:
+            current_table = m.group(1)
+            tables.append(current_table)
+            continue
+        if current_table and line.strip().startswith('`'):
+            cm = re.match(r'\s*`(\w+)`', line)
+            if cm:
+                columns.append(f"{current_table}.{cm.group(1)}")
+        elif current_table and line.strip().startswith('"'):
+            cm = re.match(r'\s*"(\w+)"', line)
+            if cm:
+                columns.append(f"{current_table}.{cm.group(1)}")
+    return tables, columns
+
+
+def compute_schema_metrics(results: List[Text2SQLResult], golden_path: str) -> dict:
+    """Compute table and column recall/precision/F1 against golden annotations."""
+    if not os.path.exists(golden_path):
+        return {}
+
+    with open(golden_path, 'r') as f:
+        golden = json.load(f)
+
+    table_metrics = []
+    column_metrics = []
+
+    for r in results:
+        if not r.success or r.instance_id not in golden:
+            continue
+        g = golden[r.instance_id]
+        if 'schema_extraction' not in g:
+            continue
+
+        golden_tables = set(t.lower() for t in g['schema_extraction'].get('tables', []))
+        golden_cols = set(c.lower() for c in g['schema_extraction'].get('columns', []))
+
+        retrieved_tables = set(t.lower() for t in (r.retrieved_tables or []))
+        retrieved_cols = set(c.lower() for c in (r.retrieved_columns or []))
+
+        # Table metrics
+        if golden_tables:
+            tp = len(retrieved_tables & golden_tables)
+            precision = tp / len(retrieved_tables) if retrieved_tables else 0
+            recall = tp / len(golden_tables)
+            f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
+            table_metrics.append({"precision": precision, "recall": recall, "f1": f1})
+
+        # Column metrics
+        if golden_cols:
+            tp = len(retrieved_cols & golden_cols)
+            precision = tp / len(retrieved_cols) if retrieved_cols else 0
+            recall = tp / len(golden_cols)
+            f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
+            column_metrics.append({"precision": precision, "recall": recall, "f1": f1})
+
+    if not table_metrics:
+        return {}
+
+    avg = lambda lst, key: sum(m[key] for m in lst) / len(lst)
+    return {
+        "table_recall": round(avg(table_metrics, "recall"), 4),
+        "table_precision": round(avg(table_metrics, "precision"), 4),
+        "table_f1": round(avg(table_metrics, "f1"), 4),
+        "column_recall": round(avg(column_metrics, "recall"), 4),
+        "column_precision": round(avg(column_metrics, "precision"), 4),
+        "column_f1": round(avg(column_metrics, "f1"), 4),
+        "num_evaluated": len(table_metrics),
+    }
+
+
+class Text2SQLBenchmark:
+    def __init__(self, api_key: str, embedding_model: str = "openai-small", llm_model: str = "gpt-4o-mini"):
+        self.api_key = api_key
+        self.embedding_model = embedding_model
+        self.llm_model = llm_model
+        os.environ["OPENAI_API_KEY"] = api_key
+        os.environ["OPENAI_API_BASE"] = "https://api.openai.com/v1"
+
+    def load_questions(self, data_path: str, db_filter: List[str] = None) -> List[Dict]:
+        """Load questions from spider2-lite.jsonl"""
+        questions = []
+        with open(data_path, 'r') as f:
+            for line in f:
+                item = json.loads(line)
+                if db_filter is None or item.get("db") in db_filter:
+                    questions.append(item)
+        return questions
+
+    async def run_benchmark(self, questions: List[Dict], question_count: int = 5) -> List[Text2SQLResult]:
+        """Run text2sql benchmark using the generic runner."""
+        from src.text2sql.runner import kg_exists, get_schema_path, ensure_kg, query_kg, get_kg_dir
+        from src.text2sql.prompt_parser import parse_qafd_clusters
+
+        results = []
+        total = min(question_count, len(questions))
+
+        print(f"\n  Running queries...")
+
+        for i, item in enumerate(questions[:question_count]):
+            instance_id = item.get("instance_id", f"q{i}")
+            db = item.get("db", "unknown")
+            question = item.get("question", "")
+
+            print_progress(i + 1, total, "Progress")
+
+            # Auto-build KG if schema summary exists (generates from .sqlite if needed)
+            if not kg_exists(db):
+                from src.text2sql.runner import ensure_db_summary
+                schema_path = ensure_db_summary(db) or get_schema_path(db)
+                if not schema_path:
+                    results.append(Text2SQLResult(
+                        instance_id=instance_id, db=db, question=question,
+                        create_table="",
+                        success=False, error_message=f"No DB summary or .sqlite file for {db}"
+                    ))
+                    continue
+
+                try:
+                    await ensure_kg(db, schema_path,
+                                    embedding_model=self.embedding_model,
+                                    llm_model=self.llm_model)
+                except Exception as e:
+                    results.append(Text2SQLResult(
+                        instance_id=instance_id, db=db, question=question,
+                        create_table="",
+                        success=False, error_message=f"KG build failed: {e}"
+                    ))
+                    continue
+
+            working_dir = get_kg_dir(db, llm_model=self.llm_model, embedding_model=self.embedding_model)
+
+            try:
+                # Get raw clusters (same as CoFD-M pipeline)
+                clusters = await query_kg(
+                    question, working_dir,
+                    embedding_model=self.embedding_model,
+                    llm_model=self.llm_model,
+                    return_raw=True,
+                )
+
+                # Load db_summary for types, PK/FK, sample rows
+                schema_data = None
+                schema_path = get_schema_path(db)
+                if schema_path:
+                    with open(schema_path, 'r') as sf:
+                        schema_data = json.load(sf)
+
+                # Format as CREATE TABLE (with types, PK/FK, constraints, sample rows)
+                if isinstance(clusters, list):
+                    create_table_str = parse_qafd_clusters(
+                        clusters,
+                        add_sample_rows=True,
+                        schema_data=schema_data,
+                        format_type="create_table",
+                    )
+                else:
+                    create_table_str = str(clusters)
+
+                tables, columns = parse_schema_from_create_table(create_table_str)
+                results.append(Text2SQLResult(
+                    instance_id=instance_id, db=db, question=question,
+                    create_table=create_table_str,
+                    success=True,
+                    retrieved_tables=tables,
+                    retrieved_columns=columns,
+                ))
+
+            except Exception as e:
+                results.append(Text2SQLResult(
+                    instance_id=instance_id, db=db, question=question,
+                    create_table="",
+                    success=False, error_message=str(e)
+                ))
+
+        return results
+
+    def save_results(self, results: List[Text2SQLResult], db_name: str = None,
+                     benchmark: str = "spider2-lite"):
+        """Save results as two separate files: eval metrics and generated responses"""
+        timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
+        if db_name:
+            results_dir = QAFD_RAG_HOME / "results" / "text2sql" / db_name
+        else:
+            results_dir = QAFD_RAG_HOME / "results" / "text2sql"
+        results_dir.mkdir(parents=True, exist_ok=True)
+
+        eval_file = str(results_dir / f"text2sql_{timestamp}_eval.json")
+
+        success_count = sum(1 for r in results if r.success)
+
+        # --- Schema accuracy metrics (if golden file exists) ---
+        golden_paths = {
+            "spider2-lite": QAFD_RAG_HOME / "data" / "text2sql" / "spider2-lite" / "golden_lite_spider_total.json",
+        }
+        golden_path = str(golden_paths.get(benchmark, ""))
+        schema_metrics = compute_schema_metrics(results, golden_path)
+
+        # --- Eval file: success/fail stats + schema metrics ---
+        eval_data = {
+            "timestamp": datetime.now().isoformat(),
+            "llm": self.llm_model,
+            "embedding": self.embedding_model,
+            "total_questions": len(results),
+            "success_count": success_count,
+            "fail_count": len(results) - success_count,
+            "failed_instances": [
+                {"instance_id": r.instance_id, "db": r.db, "error": r.error_message}
+                for r in results if not r.success
+            ],
+        }
+        if schema_metrics:
+            eval_data["schema_metrics"] = schema_metrics
+
+        with open(eval_file, 'w', encoding='utf-8') as f:
+            json.dump(eval_data, f, indent=2, ensure_ascii=False)
+
+        # --- Responses: one prompts txt per instance ---
+        for r in results:
+            if not r.success or not r.create_table:
+                continue
+            prompt_file = results_dir / f"{r.instance_id}_prompts.txt"
+            with open(prompt_file, 'w', encoding='utf-8') as f:
+                f.write(r.create_table)
+
+        print(f"  Eval saved:      {eval_file}")
+        print(f"  Prompts saved:   {results_dir}/<instance_id>_prompts.txt\n")
+
+
+async def main():
+    import argparse
+
+    parser = argparse.ArgumentParser(description="Text2SQL Benchmark using QAFD_RAG")
+    parser.add_argument("--questions", type=int, default=5, help="Number of questions")
+    parser.add_argument("--embedding", type=str, default="openai-small",
+                        choices=["openai-small", "openai-large", "jina-v3", "gritlm", "nvidia-nv-embed-v2"])
+    parser.add_argument("--llm", type=str, default="gpt-4o-mini",
+                        choices=["gpt-4o-mini", "gpt-4o", "gpt-oss-120b", "gpt-5", "gpt-5-mini", "gpt-5-nano"])
+    parser.add_argument("--db", type=str, default=None,
+                        help="Filter by database name (e.g. Pagila, superhero)")
+    parser.add_argument("--benchmark", type=str, default="spider2-lite",
+                        choices=["spider2-lite", "bird"],
+                        help="Benchmark dataset (default: spider2-lite)")
+
+    args = parser.parse_args()
+
+    api_key = os.environ.get("OPENAI_API_KEY")
+    if not api_key:
+        print("ERROR: Set OPENAI_API_KEY environment variable")
+        return
+
+    benchmark = Text2SQLBenchmark(api_key, args.embedding, args.llm)
+
+    jsonl_files = {
+        "spider2-lite": QAFD_RAG_HOME / "data" / "text2sql" / "spider2-lite" / "spider2-lite.jsonl",
+        "bird": QAFD_RAG_HOME / "data" / "text2sql" / "bird" / "bird.jsonl",
+    }
+    data_path = str(jsonl_files[args.benchmark])
+    db_filter = [args.db] if args.db else None
+    questions = benchmark.load_questions(data_path, db_filter)
+
+    print_header("QAFD-RAG Text2SQL Benchmark")
+    print_config({
+        "Benchmark": args.benchmark,
+        "Questions": f"{min(args.questions, len(questions))} / {len(questions)}",
+        "DB Filter": str(db_filter[0]) if db_filter else "all",
+        "Embedding": args.embedding,
+        "LLM": args.llm
+    })
+
+    results = await benchmark.run_benchmark(questions, args.questions)
+
+    success_count = sum(1 for r in results if r.success)
+
+    print_header("Results: Text2SQL")
+    print(f"\n  SUMMARY")
+    print(f"  {'─' * 40}")
+    print(f"  {'Successful':<25} {success_count}/{len(results)}")
+    print(f"  {'Failed':<25} {len(results) - success_count}")
+
+    # Schema accuracy metrics
+    golden_paths = {
+        "spider2-lite": QAFD_RAG_HOME / "data" / "text2sql" / "spider2-lite" / "golden_lite_spider_total.json",
+    }
+    golden_path = str(golden_paths.get(args.benchmark, ""))
+    schema_metrics = compute_schema_metrics(results, golden_path)
+    if schema_metrics:
+        print(f"\n  SCHEMA RETRIEVAL ACCURACY")
+        print(f"  {'─' * 40}")
+        print(f"  {'Table Recall':<25} {schema_metrics['table_recall']*100:.1f}%")
+        print(f"  {'Table Precision':<25} {schema_metrics['table_precision']*100:.1f}%")
+        print(f"  {'Table F1':<25} {schema_metrics['table_f1']*100:.1f}%")
+        print(f"  {'Column Recall':<25} {schema_metrics['column_recall']*100:.1f}%")
+        print(f"  {'Column Precision':<25} {schema_metrics['column_precision']*100:.1f}%")
+        print(f"  {'Column F1':<25} {schema_metrics['column_f1']*100:.1f}%")
+
+    # Show failed instances
+    failed = [r for r in results if not r.success]
+    if failed:
+        print(f"\n  FAILED INSTANCES")
+        print(f"  {'─' * 40}")
+        for r in failed[:5]:
+            print(f"  {r.instance_id:<20} {r.error_message[:40]}")
+        if len(failed) > 5:
+            print(f"  ... and {len(failed) - 5} more")
+    print()
+
+    benchmark.save_results(results, db_name=args.db, benchmark=args.benchmark)
+
+if __name__ == "__main__":
+    asyncio.run(main())

benchmarks/text2sql/evaluate_schema.py

@@ -0,0 +1,213 @@
+#!/usr/bin/env python3
+"""
+Schema Retrieval Precision/Recall Evaluator for QAFD-RAG Text2SQL.
+
+Compares retrieved tables/columns against gold standard.
+
+Usage:
+    python benchmarks/text2sql/evaluate_schema.py --results <benchmark_results.json>
+    python benchmarks/text2sql/evaluate_schema.py --results <results.json> --gold <gold.json>
+"""
+
+import os
+import sys
+import json
+import re
+import argparse
+from pathlib import Path
+from typing import Dict, Set, Tuple
+
+QAFD_RAG_HOME = Path(__file__).parent.parent.parent
+DEFAULT_GOLD = QAFD_RAG_HOME / "data" / "text2sql" / "spider2-lite" / "golden_lite_spider_total.json"
+
+
+def load_json_file(file_path: str) -> Dict:
+    with open(file_path, 'r', encoding='utf-8') as f:
+        return json.load(f)
+
+
+def extract_columns_from_gold(data: Dict) -> Dict[str, Set[str]]:
+    """Extract columns from gold standard (CoFD-M format)."""
+    columns_by_instance = {}
+    for instance_id, instance_data in data.items():
+        if isinstance(instance_data, dict) and 'schema_extraction' in instance_data:
+            schema = instance_data['schema_extraction']
+            columns_by_instance[instance_id] = set(schema.get('columns', []))
+        else:
+            columns_by_instance[instance_id] = set()
+    return columns_by_instance
+
+
+def _parse_create_table(create_table_text: str) -> Tuple[Set[str], Set[str]]:
+    """Parse CREATE TABLE statements to extract table and column names."""
+    tables = set()
+    columns = set()
+    current_table = None
+    for line in create_table_text.splitlines():
+        stripped = line.strip()
+        table_match = re.match(r'CREATE\s+TABLE\s+`?(\w+)`?\s*\(', stripped, re.IGNORECASE)
+        if table_match:
+            current_table = table_match.group(1)
+            tables.add(current_table)
+            continue
+        if current_table and stripped.startswith('`'):
+            col_match = re.match(r'`(\w+)`\s+\w+', stripped)
+            if col_match:
+                columns.add(f"{current_table}.{col_match.group(1)}")
+        if stripped.startswith(');'):
+            current_table = None
+    return tables, columns
+
+
+def extract_columns_from_results(results_data: Dict) -> Dict[str, Set[str]]:
+    """Extract columns from QAFD-RAG benchmark results by parsing create_table field."""
+    columns_by_instance = {}
+    for result in results_data.get('results', []):
+        instance_id = result.get('instance_id', '')
+        create_table = result.get('create_table', '')
+        if create_table:
+            _, cols = _parse_create_table(create_table)
+            columns_by_instance[instance_id] = cols
+        else:
+            columns_by_instance[instance_id] = set()
+    return columns_by_instance
+
+
+def extract_tables_from_gold(data: Dict) -> Dict[str, Set[str]]:
+    """Extract tables from gold standard."""
+    tables_by_instance = {}
+    for instance_id, instance_data in data.items():
+        if isinstance(instance_data, dict) and 'schema_extraction' in instance_data:
+            schema = instance_data['schema_extraction']
+            tables_by_instance[instance_id] = set(schema.get('tables', []))
+        else:
+            tables_by_instance[instance_id] = set()
+    return tables_by_instance
+
+
+def extract_tables_from_results(results_data: Dict) -> Dict[str, Set[str]]:
+    """Extract tables from QAFD-RAG benchmark results by parsing create_table field."""
+    tables_by_instance = {}
+    for result in results_data.get('results', []):
+        instance_id = result.get('instance_id', '')
+        create_table = result.get('create_table', '')
+        if create_table:
+            tbls, _ = _parse_create_table(create_table)
+            tables_by_instance[instance_id] = tbls
+        else:
+            tables_by_instance[instance_id] = set()
+    return tables_by_instance
+
+
+def calculate_precision_recall(predicted: Set[str], actual: Set[str]) -> Tuple[float, float]:
+    """Calculate precision and recall."""
+    if not predicted and not actual:
+        return 1.0, 1.0
+    if not predicted:
+        return 0.0, 0.0
+    if not actual:
+        return 0.0, 0.0
+
+    tp = len(predicted & actual)
+    precision = tp / len(predicted) if predicted else 0.0
+    recall = tp / len(actual) if actual else 0.0
+    return precision, recall
+
+
+def evaluate(gold: Dict[str, Set[str]], predicted: Dict[str, Set[str]]) -> Dict:
+    """Evaluate predicted against gold for all common instances."""
+    common = set(gold.keys()) & set(predicted.keys())
+    if not common:
+        return {'precision': 0.0, 'recall': 0.0, 'f1': 0.0, 'instances': 0, 'details': {}}
+
+    precisions, recalls = [], []
+    details = {}
+
+    for iid in sorted(common):
+        p, r = calculate_precision_recall(predicted[iid], gold[iid])
+        precisions.append(p)
+        recalls.append(r)
+        details[iid] = {
+            'precision': round(p, 4),
+            'recall': round(r, 4),
+            'predicted': sorted(predicted[iid]),
+            'gold': sorted(gold[iid]),
+            'missing': sorted(gold[iid] - predicted[iid]),
+            'extra': sorted(predicted[iid] - gold[iid]),
+        }
+
+    avg_p = sum(precisions) / len(precisions)
+    avg_r = sum(recalls) / len(recalls)
+    f1 = 2 * avg_p * avg_r / (avg_p + avg_r) if (avg_p + avg_r) > 0 else 0.0
+
+    return {
+        'precision': round(avg_p, 4),
+        'recall': round(avg_r, 4),
+        'f1': round(f1, 4),
+        'instances': len(common),
+        'details': details,
+    }
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Evaluate text2sql schema retrieval")
+    parser.add_argument("--results", required=True, help="QAFD-RAG benchmark results JSON")
+    parser.add_argument("--gold", default=str(DEFAULT_GOLD), help="Gold standard JSON")
+    parser.add_argument("--output", default=None, help="Save detailed results to JSON")
+    args = parser.parse_args()
+
+    print("Loading files...")
+    results_data = load_json_file(args.results)
+    gold_data = load_json_file(args.gold)
+
+    # Extract columns and tables
+    gold_cols = extract_columns_from_gold(gold_data)
+    pred_cols = extract_columns_from_results(results_data)
+    gold_tbls = extract_tables_from_gold(gold_data)
+    pred_tbls = extract_tables_from_results(results_data)
+
+    # Evaluate
+    col_eval = evaluate(gold_cols, pred_cols)
+    tbl_eval = evaluate(gold_tbls, pred_tbls)
+
+    # Print results
+    print(f"\n{'=' * 50}")
+    print("SCHEMA RETRIEVAL EVALUATION")
+    print(f"{'=' * 50}")
+    print(f"  Instances evaluated: {col_eval['instances']}")
+
+    print(f"\n  COLUMN RETRIEVAL")
+    print(f"  {'─' * 40}")
+    print(f"  Precision:  {col_eval['precision']:.4f}")
+    print(f"  Recall:     {col_eval['recall']:.4f}")
+    print(f"  F1:         {col_eval['f1']:.4f}")
+
+    print(f"\n  TABLE RETRIEVAL")
+    print(f"  {'─' * 40}")
+    print(f"  Precision:  {tbl_eval['precision']:.4f}")
+    print(f"  Recall:     {tbl_eval['recall']:.4f}")
+    print(f"  F1:         {tbl_eval['f1']:.4f}")
+
+    # Per-instance details
+    print(f"\n  PER-INSTANCE COLUMN DETAILS")
+    print(f"  {'─' * 40}")
+    for iid, d in col_eval['details'].items():
+        status = "OK" if d['recall'] == 1.0 else f"missing: {d['missing']}"
+        print(f"  {iid:<15} P={d['precision']:.2f} R={d['recall']:.2f}  {status}")
+
+    # Save
+    output_path = args.output
+    if not output_path:
+        output_path = args.results.replace('.json', '_eval.json')
+
+    output = {
+        'column_evaluation': col_eval,
+        'table_evaluation': tbl_eval,
+    }
+    with open(output_path, 'w') as f:
+        json.dump(output, f, indent=2)
+    print(f"\nDetailed results saved to: {output_path}")
+
+
+if __name__ == "__main__":
+    main()

benchmarks/ultradomain/benchmark_ultradomain.py

@@ -0,0 +1,548 @@
+#!/usr/bin/env python3
+"""
+UltraDomain Benchmark for QAFD-RAG
+Quality evaluation using GPT-4o as judge
+"""
+
+import os
+import sys
+import asyncio
+import time
+import json
+import logging
+import nest_asyncio
+from pathlib import Path
+from typing import Dict, List, Any, Optional
+from dataclasses import dataclass, asdict
+from datetime import datetime
+
+# Suppress verbose logging
+logging.getLogger("httpx").setLevel(logging.ERROR)
+logging.getLogger("QAFD_RAG").setLevel(logging.ERROR)
+logging.getLogger("nano-vectordb").setLevel(logging.ERROR)
+logging.getLogger("OpenAI").setLevel(logging.ERROR)
+logging.getLogger("datasets").setLevel(logging.ERROR)
+
+# Add QAFD-RAG to path (so 'src' becomes the package)
+QAFD_RAG_HOME = str(Path(__file__).parent.parent.parent)
+sys.path.insert(0, QAFD_RAG_HOME)
+
+nest_asyncio.apply()
+
+# Define datasets
+DATASETS = ["mix.jsonl"]
+
+
+def print_header(title: str, width: int = 70):
+    """Print a formatted header"""
+    print(f"\n{'=' * width}")
+    print(f"  {title}")
+    print(f"{'=' * width}")
+
+
+def print_config(items: Dict[str, str], width: int = 70):
+    """Print configuration items"""
+    print(f"{'─' * width}")
+    for key, value in items.items():
+        print(f"  {key:<20} {value}")
+    print(f"{'─' * width}")
+
+
+def print_progress(current: int, total: int, prefix: str = "", width: int = 40):
+    """Print a progress bar"""
+    percent = current / total
+    filled = int(width * percent)
+    bar = '█' * filled + '░' * (width - filled)
+    print(f"\r  {prefix} [{bar}] {current}/{total} ({percent*100:.1f}%)", end='', flush=True)
+    if current == total:
+        print()
+
+@dataclass
+class BenchmarkResult:
+    """Combined performance and quality metrics"""
+    model_name: str
+    dataset_name: str
+    success: bool = False
+    total_time: float = 0.0
+    quality_scores: Dict[str, List[float]] = None
+    responses: List[str] = None
+    questions: List[str] = None
+    error_message: str = ""
+
+class RAGBenchmark:
+    def __init__(self, api_key: str, embedding_model: str = "openai-small", llm_model: str = "gpt-4o-mini"):
+        self.api_key = api_key
+        self.embedding_model = embedding_model
+        self.llm_model = llm_model
+        os.environ["OPENAI_API_KEY"] = api_key
+        os.environ["OPENAI_API_BASE"] = "https://api.openai.com/v1"
+
+        # Initialize LLM for evaluation
+        try:
+            from src.llm import gpt_4o_mini_complete
+            self.llm_func = gpt_4o_mini_complete
+        except ImportError:
+            print("Warning: Could not import LLM function for evaluation")
+            self.llm_func = None
+
+
+    def _get_working_dir(self, dataset_name: str) -> str:
+        """Get working directory for a dataset"""
+        return os.path.join(QAFD_RAG_HOME, "kg", "ultradomain", f"{self.llm_model}_{self.embedding_model}_{dataset_name}")
+
+    def _kg_exists(self, working_dir: str) -> bool:
+        """Check if KG already exists"""
+        kg_files = [
+            os.path.join(working_dir, "vdb_entities.json"),
+            os.path.join(working_dir, "vdb_chunks.json"),
+            os.path.join(working_dir, "kv_store_full_docs.json"),
+        ]
+        return all(os.path.exists(f) for f in kg_files)
+
+    def _get_llm_func(self):
+        """Get LLM function based on model name"""
+        from src import llm
+        llm_funcs = {
+            "gpt-4o-mini": llm.gpt_4o_mini_complete,
+            "gpt-4o": llm.gpt_4o_complete,
+            "gpt-oss-120b": llm.gpt_oss_120b_complete,
+            "gpt-5": llm.gpt_5_complete,
+            "gpt-5-mini": llm.gpt_5_mini_complete,
+            "gpt-5-nano": llm.gpt_5_nano_complete,
+        }
+        return llm_funcs.get(self.llm_model, llm.gpt_4o_mini_complete)
+
+    async def _benchmark_qafd(self, dataset, dataset_name: str, question_count: int,
+                              force_build: bool = False) -> BenchmarkResult:
+        """Benchmark QAFD_RAG"""
+        try:
+            from src import QAFD_RAG, QueryParam
+
+            working_dir = self._get_working_dir(dataset_name)
+            os.makedirs(working_dir, exist_ok=True)
+
+            # Get LLM function
+            llm_func = self._get_llm_func()
+
+            rag = QAFD_RAG(
+                working_dir=working_dir,
+                llm_model_func=llm_func,
+                llm_model_name=self.llm_model,
+                embedding_model_key=self.embedding_model,
+                enable_llm_cache=True,
+            )
+
+            # Check if KG exists or needs to be built
+            insertion_time = 0.0
+            if self._kg_exists(working_dir) and not force_build:
+                print(f"\n  Using existing KG: {working_dir}")
+            else:
+                print(f"\n  Building KG from {len(dataset)} documents...")
+                start_time = time.time()
+                for i, item in enumerate(dataset):
+                    print_progress(i + 1, len(dataset), "Progress")
+                    context = item.get("context", "")
+                    if context:
+                        await rag.ainsert(context)
+                insertion_time = time.time() - start_time
+                print(f"  KG built in {insertion_time:.2f}s")
+
+            # Query phase
+            print(f"\n  Running queries...")
+            start_time = time.time()
+            responses = []
+            questions_list = []
+            total_questions = min(question_count, len(dataset))
+            for i in range(total_questions):
+                print_progress(i + 1, total_questions, "Progress")
+
+                question = dataset[i]["input"]
+                questions_list.append(question)
+                query_param = QueryParam(
+                    mode="hybrid",
+                    max_source_nodes=40,
+                    min_flow_threshold=0.01
+                )
+                response = await rag.aquery(question, query_param)
+                responses.append(response)
+
+            query_time = time.time() - start_time
+
+            return BenchmarkResult(
+                model_name="QAFD_RAG",
+                dataset_name=dataset_name,
+                success=True,
+                total_time=insertion_time + query_time,
+                responses=responses,
+                questions=questions_list,
+            )
+
+        except Exception as e:
+            print(f"  ERROR: {str(e)}")
+            import traceback
+            traceback.print_exc()
+            return BenchmarkResult(
+                model_name="QAFD_RAG",
+                dataset_name=dataset_name,
+                success=False,
+                error_message=str(e)
+            )
+
+    async def _evaluate_quality(self, dataset, responses: List[str], model_name: str, dataset_name: str, question_count: int) -> Dict[str, List[float]]:
+        """Evaluate response quality using GPT-4o"""
+        if not responses:
+            return {}
+
+        try:
+            # Import gpt_4o_complete specifically for evaluation
+            from src.llm import gpt_4o_complete
+
+            criteria = ["comprehensiveness", "diversity", "logicality", "relevance", "coherence"]
+            response_scores = {criterion: [] for criterion in criteria}
+
+            eval_count = min(question_count, len(dataset), len(responses))
+
+            for i in range(eval_count):
+                print_progress(i + 1, eval_count, "Evaluating")
+
+                query = dataset[i]["input"]
+                response = responses[i]
+
+                # Evaluate this response 5 times
+                response_criterion_scores = {criterion: [] for criterion in criteria}
+                for eval_round in range(5):
+                    prompt = f"""Evaluate the following response to a question based on five criteria. Rate each criterion from 0-100.
+
+Question: {query}
+Response: {response}
+
+Please evaluate based on these criteria:
+- Comprehensiveness: How much detail does the answer provide to cover all aspects and details of the question?
+- Diversity: How varied and rich is the answer in providing different perspectives and insights on the question?
+- Logicality: How logically does the answer respond to all parts of the question?
+- Relevance: How relevant is the answer to the question, staying focused and addressing the intended topic or issue?
+- Coherence: How well does the answer maintain internal logical connections between its parts, ensuring a smooth and consistent structure?
+
+Provide scores in JSON format:
+{{
+    "comprehensiveness": [score],
+    "diversity": [score],
+    "logicality": [score],
+    "relevance": [score],
+    "coherence": [score]
+}}"""
+
+                    evaluation_result = await gpt_4o_complete(prompt, max_tokens=200)
+
+                    import re
+                    json_match = re.search(r'\{.*\}', evaluation_result, re.DOTALL)
+                    if json_match:
+                        try:
+                            scores = json.loads(json_match.group())
+                            for criterion in criteria:
+                                if criterion in scores:
+                                    try:
+                                        score_value = float(scores[criterion])
+                                        if 0 <= score_value <= 100:  # Validate score range
+                                            response_criterion_scores[criterion].append(score_value)
+                                    except (ValueError, TypeError):
+                                        continue
+                        except json.JSONDecodeError:
+                            continue
+
+                # Calculate average for this response and add to the list
+                for criterion in criteria:
+                    if response_criterion_scores[criterion]:
+                        avg_score = sum(response_criterion_scores[criterion]) / len(response_criterion_scores[criterion])
+                        response_scores[criterion].append(avg_score)
+                    else:
+                        response_scores[criterion].append(0.0)
+
+            return response_scores
+
+        except Exception as e:
+            print(f"Quality evaluation failed: {e}")
+            import traceback
+            traceback.print_exc()
+            return {}
+
+    async def build_kg(self, max_documents: int = None) -> bool:
+        """Build KG only (no benchmark)"""
+        try:
+            from datasets import load_dataset
+            from src import QAFD_RAG
+        except ImportError as e:
+            print(f"ERROR: Missing dependency: {e}")
+            return False
+
+        for dataset_file in DATASETS:
+            dataset_name = dataset_file.replace('.jsonl', '')
+
+            print_header("QAFD-RAG Knowledge Graph Builder")
+            print_config({
+                "Graph Type": "entity",
+                "Dataset": f"UltraDomain ({dataset_name})",
+                "Embedding": self.embedding_model,
+                "LLM": self.llm_model,
+                "Max Documents": str(max_documents) if max_documents else "all"
+            })
+
+            try:
+                print("\n  Loading dataset...", end=" ", flush=True)
+                dataset = load_dataset("TommyChien/UltraDomain", data_files=dataset_file, split="train")
+                print(f"done ({len(dataset)} samples)")
+            except Exception as e:
+                print(f"failed: {e}")
+                return False
+
+            working_dir = self._get_working_dir(dataset_name)
+            os.makedirs(working_dir, exist_ok=True)
+
+            llm_func = self._get_llm_func()
+            rag = QAFD_RAG(
+                working_dir=working_dir,
+                llm_model_func=llm_func,
+                llm_model_name=self.llm_model,
+                embedding_model_key=self.embedding_model,
+                enable_llm_cache=True,
+            )
+
+            # Determine how many documents to process
+            docs_to_process = len(dataset)
+            if max_documents:
+                docs_to_process = min(max_documents, len(dataset))
+
+            print(f"\n  Building KG from {docs_to_process} documents...")
+            start_time = time.time()
+
+            for i, item in enumerate(dataset):
+                if i >= docs_to_process:
+                    break
+                print_progress(i + 1, docs_to_process, "Progress")
+                context = item.get("context", "")
+                if context:
+                    await rag.ainsert(context)
+
+            build_time = time.time() - start_time
+
+            print_header("Build Complete")
+            print(f"  Documents processed:  {docs_to_process}")
+            print(f"  Time elapsed:         {build_time:.2f}s")
+            print(f"  Avg per document:     {build_time/docs_to_process:.2f}s")
+            print(f"  Output directory:     {working_dir}")
+            print()
+
+        return True
+
+    async def run_benchmark(self, question_count: int = 10, force_build: bool = False) -> Dict[str, List[BenchmarkResult]]:
+        """Run benchmark on QAFD_RAG for all datasets"""
+        try:
+            from datasets import load_dataset
+        except ImportError:
+            print("ERROR: 'datasets' library not found. Install with: pip install datasets")
+            return {}
+
+        all_results = {}
+        models = [("QAFD_RAG", self._benchmark_qafd)]
+
+        for dataset_file in DATASETS:
+            dataset_name = dataset_file.replace('.jsonl', '')
+
+            print_header("QAFD-RAG UltraDomain Benchmark")
+            print_config({
+                "Graph Type": "entity",
+                "Dataset": dataset_name,
+                "Questions": str(question_count),
+                "Embedding": self.embedding_model,
+                "LLM": self.llm_model
+            })
+
+            try:
+                print("\n  Loading dataset...", end=" ", flush=True)
+                dataset = load_dataset("TommyChien/UltraDomain", data_files=dataset_file, split="train")
+                print(f"done ({len(dataset)} samples)")
+            except Exception as e:
+                print(f"failed: {e}")
+                continue
+
+            results = []
+
+            for model_name, benchmark_func in models:
+                result = await benchmark_func(dataset, dataset_name, question_count, force_build)
+
+                # Add quality evaluation if successful
+                if result.success:
+                    print(f"\n  Evaluating quality ({len(result.responses)} responses)...")
+                    result.quality_scores = await self._evaluate_quality(
+                        dataset, result.responses, model_name, dataset_name, question_count
+                    )
+
+                results.append(result)
+
+            all_results[dataset_name] = results
+            self.print_results({dataset_name: results})
+
+        return all_results
+
+    def print_results(self, all_results: Dict[str, List[BenchmarkResult]]):
+        """Print benchmark results"""
+        import statistics
+
+        for dataset_name, results in all_results.items():
+            print_header(f"Results: UltraDomain ({dataset_name})")
+
+            for result in results:
+                if not result.success:
+                    print(f"\n  Status: FAILED")
+                    print(f"  Error: {result.error_message}")
+                    continue
+
+                print("\n  PERFORMANCE")
+                print(f"  {'─' * 40}")
+                print(f"  {'Total Time':<25} {result.total_time:.2f}s")
+
+                if result.quality_scores:
+                    scores = result.quality_scores
+
+                    def safe_avg_std(score_list):
+                        if not score_list:
+                            return 0.0, 0.0
+                        avg = sum(score_list) / len(score_list)
+                        try:
+                            std = statistics.stdev(score_list) if len(score_list) > 1 else 0.0
+                        except statistics.StatisticsError:
+                            std = 0.0
+                        return avg, std
+
+                    print("\n  QUALITY METRICS")
+                    print(f"  {'─' * 40}")
+
+                    compreh_avg, compreh_std = safe_avg_std(scores.get('comprehensiveness', []))
+                    diversity_avg, diversity_std = safe_avg_std(scores.get('diversity', []))
+                    logical_avg, logical_std = safe_avg_std(scores.get('logicality', []))
+                    relevance_avg, relevance_std = safe_avg_std(scores.get('relevance', []))
+                    coherence_avg, coherence_std = safe_avg_std(scores.get('coherence', []))
+
+                    print(f"  {'Comprehensiveness':<25} {compreh_avg:.2f} ± {compreh_std:.2f}")
+                    print(f"  {'Diversity':<25} {diversity_avg:.2f} ± {diversity_std:.2f}")
+                    print(f"  {'Logicality':<25} {logical_avg:.2f} ± {logical_std:.2f}")
+                    print(f"  {'Relevance':<25} {relevance_avg:.2f} ± {relevance_std:.2f}")
+                    print(f"  {'Coherence':<25} {coherence_avg:.2f} ± {coherence_std:.2f}")
+
+                    overall_avg = (compreh_avg + diversity_avg + logical_avg + relevance_avg + coherence_avg) / 5
+                    print(f"\n  {'Overall Average':<25} {overall_avg:.2f}")
+            print()
+
+    def save_results(self, all_results: Dict[str, List[BenchmarkResult]]):
+        """Save results as two separate files: eval metrics and generated responses"""
+        import statistics
+
+        timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
+        results_dir = os.path.join(QAFD_RAG_HOME, "results", "ultradomain")
+        os.makedirs(results_dir, exist_ok=True)
+
+        for dataset_name, results in all_results.items():
+            eval_file = os.path.join(results_dir, f"{dataset_name}_{timestamp}_eval.json")
+            output_file = os.path.join(results_dir, f"{dataset_name}_{timestamp}_responses.json")
+
+            for result in results:
+                # --- Eval file: metrics and timing ---
+                quality_summary = {}
+                if result.quality_scores:
+                    for criterion, scores in result.quality_scores.items():
+                        if scores:
+                            avg = sum(scores) / len(scores)
+                            std = statistics.stdev(scores) if len(scores) > 1 else 0.0
+                            quality_summary[criterion] = {"mean": avg, "std": std}
+
+                eval_data = {
+                    "timestamp": datetime.now().isoformat(),
+                    "graph_type": "entity",
+                    "model": result.model_name,
+                    "llm": self.llm_model,
+                    "embedding": self.embedding_model,
+                    "dataset": dataset_name,
+                    "success": result.success,
+                    "performance": {
+                        "total_time": result.total_time,
+                    },
+                    "quality_scores": quality_summary,
+                    "quality_raw": result.quality_scores,
+                    "error": result.error_message,
+                }
+
+                with open(eval_file, 'w', encoding='utf-8') as f:
+                    json.dump(eval_data, f, indent=2, ensure_ascii=False)
+
+                # --- Responses file: questions + generated answers ---
+                output_entries = []
+                if result.responses:
+                    for i, response in enumerate(result.responses):
+                        entry = {
+                            "id": i + 1,
+                            "question": result.questions[i] if result.questions else "",
+                            "generated_answer": response,
+                        }
+                        output_entries.append(entry)
+
+                output_data = {
+                    "timestamp": datetime.now().isoformat(),
+                    "model": result.model_name,
+                    "llm": self.llm_model,
+                    "embedding": self.embedding_model,
+                    "dataset": dataset_name,
+                    "num_responses": len(output_entries),
+                    "responses": output_entries,
+                }
+
+                with open(output_file, 'w', encoding='utf-8') as f:
+                    json.dump(output_data, f, indent=2, ensure_ascii=False)
+
+            print(f"  Eval saved:      {eval_file}")
+            print(f"  Responses saved: {output_file}\n")
+
+async def main():
+    """Main entry point"""
+    import argparse
+
+    parser = argparse.ArgumentParser(description="QAFD_RAG UltraDomain Benchmark")
+    parser.add_argument("--dataset", type=str, default="mix.jsonl", help="Dataset file (default: mix.jsonl)")
+    parser.add_argument("--questions", type=int, default=10, help="Number of questions to benchmark (default: 10)")
+    parser.add_argument("--api-key", type=str, help="OpenAI API key (or set OPENAI_API_KEY env var)")
+    parser.add_argument("--embedding", type=str, default="openai-small",
+                        choices=["openai-small", "openai-large", "jina-v3", "gritlm", "nvidia-nv-embed-v2"],
+                        help="Embedding model (default: openai-small)")
+    parser.add_argument("--llm", type=str, default="gpt-4o-mini",
+                        choices=["gpt-4o-mini", "gpt-4o", "gpt-oss-120b", "gpt-5", "gpt-5-mini", "gpt-5-nano"],
+                        help="LLM model for response generation (default: gpt-4o-mini)")
+    parser.add_argument("--build", action="store_true",
+                        help="Build KG only (no benchmark)")
+    parser.add_argument("--force-build", action="store_true",
+                        help="Force rebuild KG even if exists")
+    parser.add_argument("--max-documents", type=int, default=None,
+                        help="Max documents for KG building (default: all)")
+
+    args = parser.parse_args()
+
+    # Get API key from argument or environment
+    api_key = args.api_key or os.environ.get("OPENAI_API_KEY")
+    if not api_key:
+        print("ERROR: OpenAI API key not provided. Set OPENAI_API_KEY environment variable or use --api-key")
+        return
+
+    # Update dataset if provided
+    if args.dataset:
+        DATASETS[0] = args.dataset
+
+    benchmark = RAGBenchmark(api_key, embedding_model=args.embedding, llm_model=args.llm)
+
+    # Build-only mode
+    if args.build:
+        await benchmark.build_kg(max_documents=args.max_documents)
+        return
+
+    all_results = await benchmark.run_benchmark(question_count=args.questions, force_build=args.force_build)
+    benchmark.save_results(all_results)
+
+if __name__ == "__main__":
+    asyncio.run(main())

data/multihop/README.md

@@ -0,0 +1,23 @@
+# Multi-hop QA Data
+
+Datasets: MuSiQue, HotpotQA, 2WikiMultiHopQA (included in this directory).
+
+**Graph type:** passage-entity (default) or entity (`--graph_type entity`)
+
+## Pre-built KGs
+
+Pre-built KGs use `nvidia-nv-embed-v2` (requires GPU, 16GB+ VRAM, auto-downloaded):
+
+```bash
+huggingface-cli download qafd/kg --repo-type dataset --include "multihop/*" --local-dir ./kg
+```
+
+## Usage
+
+```bash
+# Uses downloaded KG, or builds from scratch if not found
+python benchmarks/run.py --task multihop --dataset musique --questions 100
+
+# Rebuild with openai-small (no GPU needed)
+python benchmarks/run.py --task multihop --dataset musique --force_build --embedding openai-small
+```

data/multihop/musique.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:98ed4e21d3076532f6388d42320fb809599c63a0d8dffca8ece5e41922be6b46
+size 12543629

data/summarization/README.md

@@ -0,0 +1,26 @@
+# Summarization Data
+
+The summarization benchmark (SQuALITY) data is loaded automatically from HuggingFace at runtime:
+
+```
+Dataset: pszemraj/SQuALITY-v1.3
+Split:   train
+```
+
+No manual download required. The benchmark will fetch the data on first run via the `datasets` library.
+
+## Format
+
+Each item contains:
+- `document`: The full source document text
+- `questions`: List of question objects, each with:
+  - `question_text`: The question
+  - `responses`: List of response objects with `response_text`
+- `metadata.passage_id`: Unique passage identifier
+
+## Usage
+
+```bash
+./run.sh summarization --questions 50
+./run.sh summarization --build --max-documents 10
+```

data/text2sql/README.md

@@ -0,0 +1,70 @@
+# Text-to-SQL Data
+
+This directory contains the data for the Text-to-SQL benchmark, following the
+[Spider2-lite](https://github.com/xlang-ai/Spider2) directory layout.
+
+## Structure
+
+```
+data/text2sql/
+├── spider2-lite.jsonl              # Benchmark questions (instance_id, db, question)
+├── README.md
+└── databases/
+    ├── sqlite/                     # Local SQLite databases
+    │   └── <DB_Name>/
+    │       ├── <DB_Name>.sqlite            # SQLite database file
+    │       └── <DB_Name>_db_summary.json   # Auto-generated schema summary
+    │
+    ├── bigquery/                   # BigQuery dataset schemas
+    │   └── <dataset_group>/
+    │       └── <dataset_group>_bigquery_summary.json   # Schema summary
+    │
+    └── snowflake/                  # Snowflake database schemas
+        └── <DATABASE>/
+            └── <DATABASE>_db_summary.json              # Schema summary
+```
+
+## Sample Databases
+
+One example db_summary is included per backend type:
+
+| Backend   | Database | Summary file                              | Description                  |
+|-----------|----------|-------------------------------------------|------------------------------|
+| sqlite    | Pagila   | `Pagila/Pagila_db_summary.json`           | DVD rental store (16 tables) |
+| bigquery  | austin   | `austin/austin_bigquery_summary.json`     | Austin 311 service requests  |
+| snowflake | AUSTIN   | `AUSTIN/AUSTIN_db_summary.json`           | Austin 311 service requests  |
+
+## Adding a New SQLite Database
+
+1. Place the `.sqlite` file:
+   ```
+   data/text2sql/databases/sqlite/MyDB/MyDB.sqlite
+   ```
+2. The schema summary is auto-generated on first run, or generate manually:
+   ```bash
+   python -m src.indexing.extract_db_summary \
+       --db-path data/text2sql/databases/sqlite/MyDB/MyDB.sqlite
+   ```
+
+## Adding a BigQuery Database
+
+1. Generate the summary using the BigQuery extractor:
+   ```bash
+   python -m src.indexing.extract_db_summary_bigquery \
+       --datasets project_id.dataset_id \
+       --output-dir data/text2sql/databases/bigquery/my_dataset/
+   ```
+
+## Adding a Snowflake Database
+
+1. Generate the summary using the Snowflake extractor:
+   ```bash
+   python -m src.indexing.extract_db_summary_snowflake \
+       --databases MY_DATABASE \
+       --output-dir data/text2sql/databases/snowflake/MY_DATABASE/
+   ```
+
+## Full Spider2-lite Data
+
+To run the full benchmark, clone [Spider2](https://github.com/xlang-ai/Spider2)
+and symlink or copy the databases into `databases/`.

data/text2sql/bird/bird.jsonl

@@ -0,0 +1,129 @@
+{"instance_id": "bird_717", "db": "superhero", "question": "Please list all the superpowers of 3-D Man.", "external_knowledge": "superhero.md", "evidence": "3-D Man refers to superhero_name = '3-D Man'; superpowers refers to power_name", "SQL": "SELECT T3.power_name FROM superhero AS T1 INNER JOIN hero_power AS T2 ON T1.id = T2.hero_id INNER JOIN superpower AS T3 ON T2.power_id = T3.id WHERE T1.superhero_name = '3-D Man'", "difficulty": "simple"}
+{"instance_id": "bird_718", "db": "superhero", "question": "How many superheroes have the super power of \"Super Strength\"?", "external_knowledge": "superhero.md", "evidence": "super power of \"Super Strength\" refers to power_name = 'Super Strength'", "SQL": "SELECT COUNT(T1.hero_id) FROM hero_power AS T1 INNER JOIN superpower AS T2 ON T1.power_id = T2.id WHERE T2.power_name = 'Super Strength'", "difficulty": "simple"}
+{"instance_id": "bird_719", "db": "superhero", "question": "Among the superheroes with the super power of \"Super Strength\", how many of them have a height of over 200cm?", "external_knowledge": "superhero.md", "evidence": "super power of \"Super Strength\" refers to power_name = 'Super Strength'; a height of over 200cm refers to height_cm > 200", "SQL": "SELECT COUNT(T1.id) FROM superhero AS T1 INNER JOIN hero_power AS T2 ON T1.id = T2.hero_id INNER JOIN superpower AS T3 ON T2.power_id = T3.id WHERE T3.power_name = 'Super Strength' AND T1.height_cm > 200", "difficulty": "moderate"}
+{"instance_id": "bird_720", "db": "superhero", "question": "Please list the full names of all the superheroes with over 15 super powers.", "external_knowledge": "superhero.md", "evidence": "15 super powers refers to COUNT(full_name) > 15", "SQL": "SELECT DISTINCT T1.full_name FROM superhero AS T1 INNER JOIN hero_power AS T2 ON T1.id = T2.hero_id GROUP BY T1.full_name HAVING COUNT(T2.power_id) > 15", "difficulty": "simple"}
+{"instance_id": "bird_721", "db": "superhero", "question": "How many superheroes have blue eyes?", "external_knowledge": "superhero.md", "evidence": "blue eyes refers to colour = 'Blue' and eye_colour_id = colour.id;", "SQL": "SELECT COUNT(T1.id) FROM superhero AS T1 INNER JOIN colour AS T2 ON T1.eye_colour_id = T2.id WHERE T2.colour = 'Blue'", "difficulty": "simple"}
+{"instance_id": "bird_722", "db": "superhero", "question": "What is the colour of Apocalypse's skin?", "external_knowledge": "superhero.md", "evidence": "Apocalypse refers to superhero_name = 'Apocalypse'; colour of skin refers to colour where skin_colour_id = colour.id", "SQL": "SELECT T2.colour FROM superhero AS T1 INNER JOIN colour AS T2 ON T1.skin_colour_id = T2.id WHERE T1.superhero_name = 'Apocalypse'", "difficulty": "simple"}
+{"instance_id": "bird_723", "db": "superhero", "question": "Among the superheroes with blue eyes, how many of them have the super power of \"Agility\"?", "external_knowledge": "superhero.md", "evidence": "blue eyes refers to colour = 'Blue' and eye_colour_id = colour.id; super power of \"Agility\" refers to power_name = 'Agility'", "SQL": "SELECT COUNT(T1.id) FROM superhero AS T1 INNER JOIN hero_power AS T2 ON T1.id = T2.hero_id INNER JOIN superpower AS T3 ON T2.power_id = T3.id INNER JOIN colour AS T4 ON T1.eye_colour_id = T4.id WHERE T3.power_name = 'Agility' AND T4.colour = 'Blue'", "difficulty": "moderate"}
+{"instance_id": "bird_724", "db": "superhero", "question": "Please list the superhero names of all the superheroes that have blue eyes and blond hair.", "external_knowledge": "superhero.md", "evidence": "blue eyes refers to colour = 'Blue' and eye_colour_id = colour.id; blond hair refers to colour = 'Blond' and hair_colour_id = colour.id; super power of \"Agility\" refers to power_name = 'Agility'", "SQL": "SELECT T1.superhero_name FROM superhero AS T1 INNER JOIN colour AS T2 ON T1.eye_colour_id = T2.id INNER JOIN colour AS T3 ON T1.hair_colour_id = T3.id WHERE T2.colour = 'Blue' AND T3.colour = 'Blond'", "difficulty": "challenging"}
+{"instance_id": "bird_725", "db": "superhero", "question": "How many superheroes are published by Marvel Comics?", "external_knowledge": "superhero.md", "evidence": "published by Marvel Comics refers to publisher_name = 'Marvel Comics'", "SQL": "SELECT COUNT(T1.id) FROM superhero AS T1 INNER JOIN publisher AS T2 ON T1.publisher_id = T2.id WHERE T2.publisher_name = 'Marvel Comics'", "difficulty": "simple"}
+{"instance_id": "bird_726", "db": "superhero", "question": "Rank heroes published by Marvel Comics by their height in descending order.", "external_knowledge": "superhero.md", "evidence": "name refers to superhero_name; the tallest hero refers to MAX(height_cm); published by Marvel Comics refers to publisher_name = 'Marvel Comics'", "SQL": "SELECT superhero_name, height_cm, RANK() OVER (ORDER BY height_cm DESC) AS HeightRank FROM superhero INNER JOIN publisher ON superhero.publisher_id = publisher.id WHERE publisher.publisher_name = 'Marvel Comics'", "difficulty": "moderate"}
+{"instance_id": "bird_727", "db": "superhero", "question": "Who is the publisher of Sauron?", "external_knowledge": "superhero.md", "evidence": "the publisher refers to publisher_name; Sauron refers to superhero_name = 'Sauron'", "SQL": "SELECT T2.publisher_name FROM superhero AS T1 INNER JOIN publisher AS T2 ON T1.publisher_id = T2.id WHERE T1.superhero_name = 'Sauron'", "difficulty": "simple"}
+{"instance_id": "bird_728", "db": "superhero", "question": "Rank superheroes from Marvel Comics by their eye color popularity, starting with the most common color.", "external_knowledge": "superhero.md", "evidence": "the superheroes from Marvel Comics refers to publisher_name = 'Marvel Comics'; most common color refers to COUNT(superhero.id) DESC;", "SQL": "SELECT colour.colour AS EyeColor, COUNT(superhero.id) AS Count, RANK() OVER (ORDER BY COUNT(superhero.id) DESC) AS PopularityRank FROM superhero INNER JOIN colour ON superhero.eye_colour_id = colour.id INNER JOIN publisher ON superhero.publisher_id = publisher.id WHERE publisher.publisher_name = 'Marvel Comics' GROUP BY colour.colour", "difficulty": "moderate"}
+{"instance_id": "bird_729", "db": "superhero", "question": "What is the average height of the superheroes from Marvel Comics?", "external_knowledge": "superhero.md", "evidence": "superheroes from Marvel Comics refers to publisher_name = 'Marvel Comics'; average height of the superheroes refers to AVG(height_cm)", "SQL": "SELECT AVG(T1.height_cm) FROM superhero AS T1 INNER JOIN publisher AS T2 ON T1.publisher_id = T2.id WHERE T2.publisher_name = 'Marvel Comics'", "difficulty": "simple"}
+{"instance_id": "bird_730", "db": "superhero", "question": "List the superheroes from Marvel Comics who have the super power of 'Super Strength'.", "external_knowledge": "superhero.md", "evidence": "the superheroes from Marvel Comics refers to publisher_name = 'Marvel Comics'; super power of \"Super Strength\" refers to power_name = 'Super Strength';", "SQL": "SELECT superhero_name FROM superhero AS T1 WHERE EXISTS (SELECT 1 FROM hero_power AS T2 INNER JOIN superpower AS T3 ON T2.power_id = T3.id WHERE T3.power_name = 'Super Strength' AND T1.id = T2.hero_id)AND EXISTS (SELECT 1 FROM publisher AS T4 WHERE T4.publisher_name = 'Marvel Comics' AND T1.publisher_id = T4.id)", "difficulty": "challenging"}
+{"instance_id": "bird_731", "db": "superhero", "question": "How many superheroes did DC Comics publish?", "external_knowledge": "superhero.md", "evidence": "superheroes that DC Comics published refers to publisher_name = 'DC Comics'", "SQL": "SELECT COUNT(T1.id) FROM superhero AS T1 INNER JOIN publisher AS T2 ON T1.publisher_id = T2.id WHERE T2.publisher_name = 'DC Comics'", "difficulty": "simple"}
+{"instance_id": "bird_732", "db": "superhero", "question": "Which publisher published the slowest superhero?", "external_knowledge": "superhero.md", "evidence": "the slowest superhero refers to attribute_name = 'Speed' where MIN(attribute_value); publisher refers to publisher_name", "SQL": "SELECT T2.publisher_name FROM superhero AS T1 INNER JOIN publisher AS T2 ON T1.publisher_id = T2.id INNER JOIN hero_attribute AS T3 ON T1.id = T3.hero_id INNER JOIN attribute AS T4 ON T3.attribute_id = T4.id WHERE T4.attribute_name = 'Speed' ORDER BY T3.attribute_value LIMIT 1", "difficulty": "moderate"}
+{"instance_id": "bird_733", "db": "superhero", "question": "How many gold-eyed superheroes did Marvel Comics publish?", "external_knowledge": "superhero.md", "evidence": "gold-eyed refers to colour = 'Gold' where eye_colour_id = colour.id; superheroes that Marvel Comics published refers to publisher_name = 'Marvel Comics'", "SQL": "SELECT COUNT(T1.id) FROM superhero AS T1 INNER JOIN publisher AS T2 ON T1.publisher_id = T2.id INNER JOIN colour AS T3 ON T1.eye_colour_id = T3.id WHERE T2.publisher_name = 'Marvel Comics' AND T3.colour = 'Gold'", "difficulty": "moderate"}
+{"instance_id": "bird_734", "db": "superhero", "question": "What is the publisher's name of Blue Beetle II?", "external_knowledge": "superhero.md", "evidence": "Blue Beetle II refers to superhero_name = 'Blue Beetle II'", "SQL": "SELECT T2.publisher_name FROM superhero AS T1 INNER JOIN publisher AS T2 ON T1.publisher_id = T2.id WHERE T1.superhero_name = 'Blue Beetle II'", "difficulty": "simple"}
+{"instance_id": "bird_735", "db": "superhero", "question": "How many superheroes with blonde hair are there?", "external_knowledge": "superhero.md", "evidence": "superheroes with blonde hair refers to colour = 'Blond' where hair_colour_id = colour.id", "SQL": "SELECT COUNT(T1.id) FROM superhero AS T1 INNER JOIN colour AS T2 ON T1.hair_colour_id = T2.id WHERE T2.colour = 'Blond'", "difficulty": "simple"}
+{"instance_id": "bird_736", "db": "superhero", "question": "Who is the dumbest superhero?", "external_knowledge": "superhero.md", "evidence": "the dumbest superhero refers to MIN(attribute_value) where attribute_name = 'Intelligence'", "SQL": "SELECT T1.superhero_name FROM superhero AS T1 INNER JOIN hero_attribute AS T2 ON T1.id = T2.hero_id INNER JOIN attribute AS T3 ON T2.attribute_id = T3.id WHERE T3.attribute_name = 'Intelligence' ORDER BY T2.attribute_value LIMIT 1", "difficulty": "moderate"}
+{"instance_id": "bird_737", "db": "superhero", "question": "What is Copycat's race?", "external_knowledge": "superhero.md", "evidence": "Copycat is the superhero_name;", "SQL": "SELECT T2.race FROM superhero AS T1 INNER JOIN race AS T2 ON T1.race_id = T2.id WHERE T1.superhero_name = 'Copycat'", "difficulty": "simple"}
+{"instance_id": "bird_738", "db": "superhero", "question": "Which superheroes have a durability attribute value of less than 50?", "external_knowledge": "superhero.md", "evidence": "durability of less than 50 refers to attribute_name = 'Durability' AND attribute_value < 50", "SQL": "SELECT superhero_name FROM superhero AS T1 WHERE EXISTS (SELECT 1 FROM hero_attribute AS T2 INNER JOIN attribute AS T3 ON T2.attribute_id = T3.id WHERE T3.attribute_name = 'Durability' AND T2.attribute_value < 50 AND T1.id = T2.hero_id)", "difficulty": "simple"}
+{"instance_id": "bird_739", "db": "superhero", "question": "What are the names of the superheroes with the power of death touch?", "external_knowledge": "superhero.md", "evidence": "name of superheroes refers to refers to superhero_name; the power of death touch refers to power_name = 'Death Touch'", "SQL": "SELECT T1.superhero_name FROM superhero AS T1 INNER JOIN hero_power AS T2 ON T1.id = T2.hero_id INNER JOIN superpower AS T3 ON T2.power_id = T3.id WHERE T3.power_name = 'Death Touch'", "difficulty": "moderate"}
+{"instance_id": "bird_740", "db": "superhero", "question": "How many female superheroes have a strength value of 100?", "external_knowledge": "superhero.md", "evidence": "female refers to gender = 'Female'; strength value of 100 refers to attribute_name = 'Strength' AND attribute_value = 100", "SQL": "SELECT COUNT(T1.id) FROM superhero AS T1 INNER JOIN hero_attribute AS T2 ON T1.id = T2.hero_id INNER JOIN attribute AS T3 ON T2.attribute_id = T3.id INNER JOIN gender AS T4 ON T1.gender_id = T4.id WHERE T3.attribute_name = 'Strength' AND T2.attribute_value = 100 AND T4.gender = 'Female'", "difficulty": "moderate"}
+{"instance_id": "bird_741", "db": "superhero", "question": "What is the name of the superhero that has the most powers?", "external_knowledge": "superhero.md", "evidence": "name of the superhero refers to superhero_name; superhero that has the most powers refers to MAX(COUNT(superhero_name))", "SQL": "SELECT T1.superhero_name FROM superhero AS T1 INNER JOIN hero_power AS T2 ON T1.id = T2.hero_id GROUP BY T1.superhero_name ORDER BY COUNT(T2.hero_id) DESC LIMIT 1", "difficulty": "simple"}
+{"instance_id": "bird_742", "db": "superhero", "question": "How many vampire superheroes are there?", "external_knowledge": "superhero.md", "evidence": "vampire superheroes refers to race = 'Vampire'", "SQL": "SELECT COUNT(T1.superhero_name) FROM superhero AS T1 INNER JOIN race AS T2 ON T1.race_id = T2.id WHERE T2.race = 'Vampire'", "difficulty": "simple"}
+{"instance_id": "bird_743", "db": "superhero", "question": "What is the percentage of superheroes who act in their own self-interest or make decisions based on their own moral code? Indicate how many of the said superheroes were published by Marvel Comics.", "external_knowledge": "superhero.md", "evidence": "published by Marvel Comics refers to publisher_name = 'Marvel Comics'; superheroes who act in their own self-interest or make decisions based on their own moral code refers to alignment = 'Bad'; calculation = MULTIPLY(DIVIDE(SUM(alignment = 'Bad); count(id)), 100)", "SQL": "SELECT (CAST(COUNT(*) AS REAL) * 100 / (SELECT COUNT(*) FROM superhero)), CAST(SUM(CASE WHEN T2.publisher_name = 'Marvel Comics' THEN 1 ELSE 0 END) AS REAL) FROM superhero AS T1 INNER JOIN publisher AS T2 ON T1.publisher_id = T2.id INNER JOIN alignment AS T3 ON T3.id = T1.alignment_id WHERE T3.alignment = 'Bad'", "difficulty": "challenging"}
+{"instance_id": "bird_744", "db": "superhero", "question": "Between DC and Marvel Comics, which publisher has published more superheroes? Find the difference in the number of superheroes they have published.", "external_knowledge": "superhero.md", "evidence": "DC refers to publisher_name = 'DC Comics'; Marvel Comics refers to publisher_name = 'Marvel Comics'; calculation = SUBTRACT(SUM(publisher_name = 'Marvel Comics'), SUM(publisher_name = 'DC Comics'))", "SQL": "SELECT SUM(CASE WHEN T2.publisher_name = 'Marvel Comics' THEN 1 ELSE 0 END) - SUM(CASE WHEN T2.publisher_name = 'DC Comics' THEN 1 ELSE 0 END) FROM superhero AS T1 INNER JOIN publisher AS T2 ON T1.publisher_id = T2.id", "difficulty": "challenging"}
+{"instance_id": "bird_745", "db": "superhero", "question": "Give the publisher ID of Star Trek.", "external_knowledge": "superhero.md", "evidence": "Star Trek is the publisher_name;", "SQL": "SELECT id FROM publisher WHERE publisher_name = 'Star Trek'", "difficulty": "simple"}
+{"instance_id": "bird_746", "db": "superhero", "question": "Calculate the average attribute value of all superheroes.", "external_knowledge": "superhero.md", "evidence": "average attribute value of all superheroes refers to AVG(attribute_value)", "SQL": "SELECT AVG(attribute_value) FROM hero_attribute", "difficulty": "simple"}
+{"instance_id": "bird_747", "db": "superhero", "question": "What is the total number of superheroes without full name?", "external_knowledge": "superhero.md", "evidence": "superheroes without full name refers to full_name IS NULL", "SQL": "SELECT COUNT(id) FROM superhero WHERE full_name IS NULL", "difficulty": "simple"}
+{"instance_id": "bird_748", "db": "superhero", "question": "What is the eye colour of superhero with superhero ID 75?", "external_knowledge": "superhero.md", "evidence": "eye colour refers to colour where eye_colour_id = colour.id;", "SQL": "SELECT T2.colour FROM superhero AS T1 INNER JOIN colour AS T2 ON T1.eye_colour_id = T2.id WHERE T1.id = 75", "difficulty": "simple"}
+{"instance_id": "bird_749", "db": "superhero", "question": "Provide the superpowers of the superhero called Deathlok.", "external_knowledge": "superhero.md", "evidence": "superpowers refers to power_name; Deathlok refers to superhero_name = 'Deathlok'", "SQL": "SELECT T3.power_name FROM superhero AS T1 INNER JOIN hero_power AS T2 ON T1.id = T2.hero_id INNER JOIN superpower AS T3 ON T2.power_id = T3.id WHERE T1.superhero_name = 'Deathlok'", "difficulty": "simple"}
+{"instance_id": "bird_750", "db": "superhero", "question": "What is the average weight of all female superheroes?", "external_knowledge": "superhero.md", "evidence": "female refers to gender = 'Female'; average weight refers to AVG(weight_kg)", "SQL": "SELECT AVG(T1.weight_kg) FROM superhero AS T1 INNER JOIN gender AS T2 ON T1.gender_id = T2.id WHERE T2.gender = 'Female'", "difficulty": "simple"}
+{"instance_id": "bird_751", "db": "superhero", "question": "List down at least five superpowers of male superheroes.", "external_knowledge": "superhero.md", "evidence": "male refers to gender = 'Male'; superpowers refers to power_name;", "SQL": "SELECT T3.power_name FROM superhero AS T1 INNER JOIN hero_power AS T2 ON T1.id = T2.hero_id INNER JOIN superpower AS T3 ON T3.id = T2.power_id INNER JOIN gender AS T4 ON T4.id = T1.gender_id WHERE T4.gender = 'Male' LIMIT 5", "difficulty": "moderate"}
+{"instance_id": "bird_752", "db": "superhero", "question": "Give the name of the alien superheroes.", "external_knowledge": "superhero.md", "evidence": "alien superheroes refers to race = 'Alien'; name of superhero refers to superhero_name;", "SQL": "SELECT T1.superhero_name FROM superhero AS T1 INNER JOIN race AS T2 ON T1.race_id = T2.id WHERE T2.race = 'Alien'", "difficulty": "simple"}
+{"instance_id": "bird_753", "db": "superhero", "question": "Among the superheroes with height from 170 to 190, list the names of the superheroes with no eye color.", "external_knowledge": "superhero.md", "evidence": "height from 170 to 190 refers to height_cm BETWEEN 170 AND 190; no eye color refers to colour = 'No Colour'", "SQL": "SELECT DISTINCT T1.superhero_name FROM superhero AS T1 INNER JOIN colour AS T2 ON T1.eye_colour_id = T2.id WHERE T1.height_cm BETWEEN 170 AND 190 AND T2.colour = 'No Colour'", "difficulty": "moderate"}
+{"instance_id": "bird_754", "db": "superhero", "question": "What is the superpower of hero ID 56?", "external_knowledge": "superhero.md", "evidence": "superpower refers to hero_power", "SQL": "SELECT T2.power_name FROM hero_power AS T1 INNER JOIN superpower AS T2 ON T1.power_id = T2.id WHERE T1.hero_id = 56", "difficulty": "simple"}
+{"instance_id": "bird_755", "db": "superhero", "question": "List down at least five full name of Demi-God superheroes.", "external_knowledge": "superhero.md", "evidence": "Demi-God superheroes refers to race = 'Demi-God'", "SQL": "SELECT T1.full_name FROM superhero AS T1 INNER JOIN race AS T2 ON T1.race_id = T2.id WHERE T2.race = 'Demi-God'", "difficulty": "simple"}
+{"instance_id": "bird_756", "db": "superhero", "question": "How many bad superheroes are there?", "external_knowledge": "superhero.md", "evidence": "bad superheroes refers to alignment_id = Bad", "SQL": "SELECT COUNT(T1.id) FROM superhero AS T1 INNER JOIN alignment AS T2 ON T1.alignment_id = T2.id WHERE T2.alignment = 'Bad'", "difficulty": "simple"}
+{"instance_id": "bird_757", "db": "superhero", "question": "Identify the race of the superhero who weighed 169 kg.", "external_knowledge": "superhero.md", "evidence": "weighed 169 kg refers to weight_kg = 169", "SQL": "SELECT T2.race FROM superhero AS T1 INNER JOIN race AS T2 ON T1.race_id = T2.id WHERE T1.weight_kg = 169", "difficulty": "simple"}
+{"instance_id": "bird_758", "db": "superhero", "question": "Provide the hair colour of the human superhero who is 185 cm tall.", "external_knowledge": "superhero.md", "evidence": "185 cm tall refers to height_cm = 185; human superhero refers to race = 'human'; hair colour refers to colour where hair_colour_id = colour.id;", "SQL": "SELECT DISTINCT T3.colour FROM superhero AS T1 INNER JOIN race AS T2 ON T1.race_id = T2.id INNER JOIN colour AS T3 ON T1.hair_colour_id = T3.id WHERE T1.height_cm = 185 AND T2.race = 'Human'", "difficulty": "moderate"}
+{"instance_id": "bird_759", "db": "superhero", "question": "What is the eye clolour of the heaviest superhero?", "external_knowledge": "superhero.md", "evidence": "the heaviest superhero refers to MAX(weight_kg); eye colour refers to colour where eye_colour_id = colour.id;", "SQL": "SELECT T2.colour FROM superhero AS T1 INNER JOIN colour AS T2 ON T1.eye_colour_id = T2.id ORDER BY T1.weight_kg DESC LIMIT 1", "difficulty": "simple"}
+{"instance_id": "bird_760", "db": "superhero", "question": "In superheroes with height between 150 to 180, what is the percentage of heroes published by Marvel Comics?", "external_knowledge": "superhero.md", "evidence": "height between 150 to 180 refers to height_cm BETWEEN 150 AND 180; heroes published by Marvel Comics refers to publisher_name = 'Marvel Comics'; calculation = MULTIPLY(DIVIDE(SUM(publisher.id = 13)), COUNT(publisher.id), 100)", "SQL": "SELECT CAST(COUNT(CASE WHEN T2.publisher_name = 'Marvel Comics' THEN 1 ELSE NULL END) AS REAL) * 100 / COUNT(T1.id) FROM superhero AS T1 INNER JOIN publisher AS T2 ON T1.publisher_id = T2.id WHERE T1.height_cm BETWEEN 150 AND 180", "difficulty": "challenging"}
+{"instance_id": "bird_761", "db": "superhero", "question": "Among the male superheroes, list the super hero names of superheroes with weight greater than the 79% average weight of all superheroes.", "external_knowledge": "superhero.md", "evidence": "super hero names refers to superhero_name;male superheros refers to gender = 'Male';Calculation = weight_kg > MULTIPLY(AVG(weight_kg), 0.79)", "SQL": "SELECT T1.superhero_name FROM superhero AS T1 INNER JOIN gender AS T2 ON T1.gender_id = T2.id WHERE T2.gender = 'Male' AND T1.weight_kg * 100 > ( SELECT AVG(weight_kg) FROM superhero ) * 79", "difficulty": "moderate"}
+{"instance_id": "bird_762", "db": "superhero", "question": "Which power do superheroes have the most of?", "external_knowledge": "superhero.md", "evidence": "power that superheroes have the most refers to MAX(COUNT(power_name))", "SQL": "SELECT T2.power_name FROM hero_power AS T1 INNER JOIN superpower AS T2 ON T1.power_id = T2.id GROUP BY T2.power_name ORDER BY COUNT(T1.hero_id) DESC LIMIT 1", "difficulty": "simple"}
+{"instance_id": "bird_763", "db": "superhero", "question": "Indicate the attribute value of superhero Abomination.", "external_knowledge": "superhero.md", "evidence": "Abomination refers to superhero_name = 'Abomination';", "SQL": "SELECT T2.attribute_value FROM superhero AS T1 INNER JOIN hero_attribute AS T2 ON T1.id = T2.hero_id WHERE T1.superhero_name = 'Abomination'", "difficulty": "simple"}
+{"instance_id": "bird_764", "db": "superhero", "question": "What are the superpowers of heroes with ID 1?", "external_knowledge": "superhero.md", "evidence": "superpowers refers to power_name; heroes with ID 1 refers to hero_id = 1;", "SQL": "SELECT DISTINCT T2.power_name FROM hero_power AS T1 INNER JOIN superpower AS T2 ON T1.power_id = T2.id WHERE T1.hero_id = 1", "difficulty": "simple"}
+{"instance_id": "bird_765", "db": "superhero", "question": "How many heroes have stealth power?", "external_knowledge": "superhero.md", "evidence": "stealth power refers to power_name = 'stealth';", "SQL": "SELECT COUNT(T1.hero_id) FROM hero_power AS T1 INNER JOIN superpower AS T2 ON T1.power_id = T2.id WHERE T2.power_name = 'Stealth'", "difficulty": "simple"}
+{"instance_id": "bird_766", "db": "superhero", "question": "What is the hero's full name with the highest attribute in strength?", "external_knowledge": "superhero.md", "evidence": "highest attribute in strength refers to MAX(attribute_value) WHERE attribute_name = 'strength';", "SQL": "SELECT T1.full_name FROM superhero AS T1 INNER JOIN hero_attribute AS T2 ON T1.id = T2.hero_id INNER JOIN attribute AS T3 ON T2.attribute_id = T3.id WHERE T3.attribute_name = 'Strength' ORDER BY T2.attribute_value DESC LIMIT 1", "difficulty": "moderate"}
+{"instance_id": "bird_767", "db": "superhero", "question": "What is the average of superheroes with no skin colour?", "external_knowledge": "superhero.md", "evidence": "average = DIVIDE(COUNT(superhero.id), SUM(skin_colour_id = 1)); no skin colour refers to skin_colour_id WHERE colour.id = 1;", "SQL": "SELECT CAST(COUNT(*) AS REAL) / SUM(CASE WHEN T2.id = 1 THEN 1 ELSE 0 END) FROM superhero AS T1 INNER JOIN colour AS T2 ON T1.skin_colour_id = T2.id", "difficulty": "simple"}
+{"instance_id": "bird_768", "db": "superhero", "question": "How many superheroes were published by Dark Horse Comics?", "external_knowledge": "superhero.md", "evidence": "published by Dark Horse Comics refers to publisher_name = 'Dark Horse Comics';", "SQL": "SELECT COUNT(T1.id) FROM superhero AS T1 INNER JOIN publisher AS T2 ON T1.publisher_id = T2.id WHERE T2.publisher_name = 'Dark Horse Comics'", "difficulty": "simple"}
+{"instance_id": "bird_769", "db": "superhero", "question": "Which superhero has the most durability published by Dark Horse Comics?", "external_knowledge": "superhero.md", "evidence": "which superhero refers to superhero_name; most durability refers to MAX(attribute_value) WHERE attribute_name = 'durability'; published by Dark Horse Comics refers to publisher_name = 'Dark Horse Comics';", "SQL": "SELECT T1.superhero_name FROM superhero AS T1 INNER JOIN hero_attribute AS T2 ON T1.id = T2.hero_id INNER JOIN attribute AS T3 ON T3.id = T2.attribute_id INNER JOIN publisher AS T4 ON T4.id = T1.publisher_id WHERE T4.publisher_name = 'Dark Horse Comics' AND T3.attribute_name = 'Durability' ORDER BY T2.attribute_value DESC LIMIT 1", "difficulty": "challenging"}
+{"instance_id": "bird_770", "db": "superhero", "question": "What is the eyes colour of Abraham Sapien?", "external_knowledge": "superhero.md", "evidence": "eye colour refers to colour.colour where eye_colour_id = colour.id; Abraham Sapien is the full name of superhero;", "SQL": "SELECT T2.colour FROM superhero AS T1 INNER JOIN colour AS T2 ON T1.eye_colour_id = T2.id WHERE T1.full_name = 'Abraham Sapien'", "difficulty": "simple"}
+{"instance_id": "bird_771", "db": "superhero", "question": "List the name of superheroes with flight power.", "external_knowledge": "superhero.md", "evidence": "name of superheroes refers to superhero_name; flight power refers to power_name = 'Flight';", "SQL": "SELECT T1.superhero_name FROM superhero AS T1 INNER JOIN hero_power AS T2 ON T1.id = T2.hero_id INNER JOIN superpower AS T3 ON T2.power_id = T3.id WHERE T3.power_name = 'Flight'", "difficulty": "simple"}
+{"instance_id": "bird_772", "db": "superhero", "question": "List the eyes, hair and skin colour of all female superheroes published by Dark Horse Comics.", "external_knowledge": "superhero.md", "evidence": "eyes refers to eye_colour_id; hair refers to hair_colour_id; skin colour refers to skin_colour_id; female superheroes refers to gender = 'Female'; published by Dark Horse Comics refers to publisher_name = 'Dark Horse Comics';", "SQL": "SELECT T1.eye_colour_id, T1.hair_colour_id, T1.skin_colour_id FROM superhero AS T1 INNER JOIN publisher AS T2 ON T2.id = T1.publisher_id INNER JOIN gender AS T3 ON T3.id = T1.gender_id WHERE T2.publisher_name = 'Dark Horse Comics' AND T3.gender = 'Female'", "difficulty": "challenging"}
+{"instance_id": "bird_773", "db": "superhero", "question": "Which superhero has the same eyes, hair and skin colour? Indicate the publisher of the superhero.", "external_knowledge": "superhero.md", "evidence": "which superhero refers to superhero_name; the same eyes, hair and skin colour refers to hair_colour_id = skin_colour_id AND hair_colour_id = eye_colour_id; publisher refers to publisher_name;", "SQL": "SELECT T1.superhero_name, T2.publisher_name FROM superhero AS T1 INNER JOIN publisher AS T2 ON T1.publisher_id = T2.id WHERE T1.eye_colour_id = T1.hair_colour_id AND T1.eye_colour_id = T1.skin_colour_id", "difficulty": "challenging"}
+{"instance_id": "bird_774", "db": "superhero", "question": "Which group does superhero A-Bomb belong to?", "external_knowledge": "superhero.md", "evidence": "group refers to race; A-Bomb refers to superhero_name = 'A-Bomb';", "SQL": "SELECT T2.race FROM superhero AS T1 INNER JOIN race AS T2 ON T1.race_id = T2.id WHERE T1.superhero_name = 'A-Bomb'", "difficulty": "simple"}
+{"instance_id": "bird_775", "db": "superhero", "question": "What is the percentage of blue female superheroes among all female superheroes?", "external_knowledge": "superhero.md", "evidence": "percentage = MULTIPLY(DIVIDE(SUM(colour = 'Blue' WHERE gender = 'Female'), COUNT(gender = 'Female')), 100); blue refers to the color = 'Blue' WHERE skin_colour_id = colour.id; female refers to gender = 'Female';", "SQL": "SELECT CAST(COUNT(CASE WHEN T3.colour = 'Blue' THEN T1.id ELSE NULL END) AS REAL) * 100 / COUNT(T1.id) FROM superhero AS T1 INNER JOIN gender AS T2 ON T1.gender_id = T2.id INNER JOIN colour AS T3 ON T1.skin_colour_id = T3.id WHERE T2.gender = 'Female'", "difficulty": "challenging"}
+{"instance_id": "bird_776", "db": "superhero", "question": "Provide the hero name and race of Charles Chandler.", "external_knowledge": "superhero.md", "evidence": "hero name refers to superhero_name; Charles Chandler is the full name of superhero;", "SQL": "SELECT T1.superhero_name, T2.race FROM superhero AS T1 INNER JOIN race AS T2 ON T1.race_id = T2.id WHERE T1.full_name = 'Charles Chandler'", "difficulty": "simple"}
+{"instance_id": "bird_777", "db": "superhero", "question": "What is the gender of Agent 13 hero?", "external_knowledge": "superhero.md", "evidence": "Agent 13 hero refers to superhero_name = 'Agent 13';", "SQL": "SELECT T2.gender FROM superhero AS T1 INNER JOIN gender AS T2 ON T1.gender_id = T2.id WHERE T1.superhero_name = 'Agent 13'", "difficulty": "simple"}
+{"instance_id": "bird_778", "db": "superhero", "question": "Provide superheroes' names who have the adaptation power.", "external_knowledge": "superhero.md", "evidence": "adaptation power refers to power_name = 'Adaptation';", "SQL": "SELECT T1.superhero_name FROM superhero AS T1 INNER JOIN hero_power AS T2 ON T1.id = T2.hero_id INNER JOIN superpower AS T3 ON T2.power_id = T3.id WHERE T3.power_name = 'Adaptation'", "difficulty": "simple"}
+{"instance_id": "bird_779", "db": "superhero", "question": "How many powers does Amazo hero have?", "external_knowledge": "superhero.md", "evidence": "Amazo hero refers to superhero_name = 'Amazo';", "SQL": "SELECT COUNT(T1.power_id) FROM hero_power AS T1 INNER JOIN superhero AS T2 ON T1.hero_id = T2.id WHERE T2.superhero_name = 'Amazo'", "difficulty": "simple"}
+{"instance_id": "bird_780", "db": "superhero", "question": "List the powers of Hunter Zolomon.", "external_knowledge": "superhero.md", "evidence": "Hunter Zolomon is the full name of superhero; list the powers refers to power_name;", "SQL": "SELECT T3.power_name FROM superhero AS T1 INNER JOIN hero_power AS T2 ON T1.id = T2.hero_id INNER JOIN superpower AS T3 ON T2.power_id = T3.id WHERE T1.full_name = 'Hunter Zolomon'", "difficulty": "simple"}
+{"instance_id": "bird_781", "db": "superhero", "question": "Provide the heights of the heroes whose eye colours are amber.", "external_knowledge": "superhero.md", "evidence": "heights of the heroes refers to height_cm; eye colours are amber refers to colour.colour = 'Amber' WHERE eye_colour_id = colour.id;", "SQL": "SELECT T1.height_cm FROM superhero AS T1 INNER JOIN colour AS T2 ON T1.eye_colour_id = T2.id WHERE T2.colour = 'Amber'", "difficulty": "simple"}
+{"instance_id": "bird_782", "db": "superhero", "question": "List the heroes' names whose eyes and hair colours are both black.", "external_knowledge": "superhero.md", "evidence": "heroes' names refers to superhero_name; eyes and hair colours are both black refers to eye_colour_id AND hair_colour_id WHERE colour.colour = 'Black';", "SQL": "SELECT T1.superhero_name FROM superhero AS T1 INNER JOIN colour AS T2 ON T1.eye_colour_id = T2.id AND T1.hair_colour_id = T2.id WHERE T2.colour = 'Black'", "difficulty": "moderate"}
+{"instance_id": "bird_783", "db": "superhero", "question": "Provide the eye colours of the heroes whose skin colours are gold.", "external_knowledge": "superhero.md", "evidence": "skin colours are gold refers to colour.colour = 'Gold' WHERE skin_colour_id = colour.id;", "SQL": "SELECT T2.colour FROM superhero AS T1 INNER JOIN colour AS T2 ON T1.eye_colour_id = T2.id INNER JOIN colour AS T3 ON T1.skin_colour_id = T3.id WHERE T3.colour = 'Gold'", "difficulty": "simple"}
+{"instance_id": "bird_784", "db": "superhero", "question": "Provide the full names of vampire heroes.", "external_knowledge": "superhero.md", "evidence": "vampire heroes refers to race = 'Vampire';", "SQL": "SELECT T1.full_name FROM superhero AS T1 INNER JOIN race AS T2 ON T1.race_id = T2.id WHERE T2.race = 'Vampire'", "difficulty": "simple"}
+{"instance_id": "bird_785", "db": "superhero", "question": "Describe the names of neutral alignment superheroes.", "external_knowledge": "superhero.md", "evidence": "names of superheroes refers to superhero_name; neutral alignment refers to alignment = 'Neutral';", "SQL": "SELECT T1.superhero_name FROM superhero AS T1 INNER JOIN alignment AS T2 ON T1.alignment_id = T2.id WHERE T2.alignment = 'Neutral'", "difficulty": "simple"}
+{"instance_id": "bird_786", "db": "superhero", "question": "How many heroes have the highest attribute value in strength?", "external_knowledge": "superhero.md", "evidence": "highest attribute value in strength refers to MAX(attribute_value) WHERE attribute_name = 'Strength';", "SQL": "SELECT COUNT(T1.hero_id) FROM hero_attribute AS T1 INNER JOIN attribute AS T2 ON T1.attribute_id = T2.id WHERE T2.attribute_name = 'Strength' AND T1.attribute_value = ( SELECT MAX(attribute_value) FROM hero_attribute )", "difficulty": "moderate"}
+{"instance_id": "bird_787", "db": "superhero", "question": "What are the race and alignment of Cameron Hicks?", "external_knowledge": "superhero.md", "evidence": "Cameron Hicks refers to superhero_name = 'Cameron Hicks';", "SQL": "SELECT T2.race, T3.alignment FROM superhero AS T1 INNER JOIN race AS T2 ON T1.race_id = T2.id INNER JOIN alignment AS T3 ON T1.alignment_id = T3.id WHERE T1.superhero_name = 'Cameron Hicks'", "difficulty": "simple"}
+{"instance_id": "bird_788", "db": "superhero", "question": "How many percent of female heroes were published by Marvel Comics?", "external_knowledge": "superhero.md", "evidence": "percent = MULTIPLY(DIVIDE(SUM(gender = 'Female' WHERE publisher_name = 'Marvel Comics'), COUNT(publisher_name = 'Marvel Comics')), 100); female heroes refers to gender = 'Female'; Marvel Comics refers to publisher_name = 'Marvel Comics';", "SQL": "SELECT CAST(COUNT(CASE WHEN T2.publisher_name = 'Marvel Comics' THEN 1 ELSE NULL END) AS REAL) * 100 / COUNT(T1.id) FROM superhero AS T1 INNER JOIN publisher AS T2 ON T1.publisher_id = T2.id INNER JOIN gender AS T3 ON T1.gender_id = T3.id WHERE T3.gender = 'Female'", "difficulty": "challenging"}
+{"instance_id": "bird_789", "db": "superhero", "question": "Find the average weight of the heroes who are aliens.", "external_knowledge": "superhero.md", "evidence": "average = AVG(weight_kg); aliens refers to race = 'Alien';", "SQL": "SELECT CAST(SUM(T1.weight_kg) AS REAL) / COUNT(T1.id) FROM superhero AS T1 INNER JOIN race AS T2 ON T1.race_id = T2.id WHERE T2.race = 'Alien'", "difficulty": "simple"}
+{"instance_id": "bird_790", "db": "superhero", "question": "Calculate the difference between Emil Blonsky's weight and Charles Chandler's weight.", "external_knowledge": "superhero.md", "evidence": "difference = SUBTRACT(SUM(weight_kg WHERE full_name = 'Emil Blonsky'), SUM(weight_kg WHERE full_name = 'Charles Chandler')); Emil Blonsky is the full name of superhero; Charles Chandler is the full name of superhero;", "SQL": "SELECT ( SELECT weight_kg FROM superhero WHERE full_name LIKE 'Emil Blonsky' ) - ( SELECT weight_kg FROM superhero WHERE full_name LIKE 'Charles Chandler' ) AS CALCULATE", "difficulty": "moderate"}
+{"instance_id": "bird_791", "db": "superhero", "question": "Calculate the average height for each superhero.", "external_knowledge": "superhero.md", "evidence": "average = DIVIDE(SUM(height_cm), COUNT(all heros));", "SQL": "SELECT CAST(SUM(height_cm) AS REAL) / COUNT(id) FROM superhero", "difficulty": "simple"}
+{"instance_id": "bird_792", "db": "superhero", "question": "What is Abomination's superpower?", "external_knowledge": "superhero.md", "evidence": "Abomination refers to superhero_name = 'Abomination'; superpower refers to power_name;", "SQL": "SELECT T3.power_name FROM superhero AS T1 INNER JOIN hero_power AS T2 ON T1.id = T2.hero_id INNER JOIN superpower AS T3 ON T2.power_id = T3.id WHERE T1.superhero_name = 'Abomination'", "difficulty": "simple"}
+{"instance_id": "bird_793", "db": "superhero", "question": "Among the superheroes with the race of god/eternal, how many of them are male", "external_knowledge": "superhero.md", "evidence": "race \"god/eternal\" refers to race_id = 21; male refers to gender.id = 1", "SQL": "SELECT COUNT(*) FROM superhero AS T1 INNER JOIN race AS T2 ON T1.race_id = T2.id INNER JOIN gender AS T3 ON T3.id = T1.gender_id WHERE T1.race_id = 21 AND T1.gender_id = 1", "difficulty": "simple"}
+{"instance_id": "bird_794", "db": "superhero", "question": "Which hero was the fastest?", "external_knowledge": "superhero.md", "evidence": "which hero refers to superhero_name; fastest refers to MAX(attribute_value) WHERE attribute_name = 'Speed';", "SQL": "SELECT T1.superhero_name FROM superhero AS T1 INNER JOIN hero_attribute AS T2 ON T1.id = T2.hero_id INNER JOIN attribute AS T3 ON T2.attribute_id = T3.id WHERE T3.attribute_name = 'Speed' ORDER BY T2.attribute_value DESC LIMIT 1", "difficulty": "moderate"}
+{"instance_id": "bird_795", "db": "superhero", "question": "How many superheroes have a neutral alignment?", "external_knowledge": "superhero.md", "evidence": "neutral alignment refers to alignment_id = 3;", "SQL": "SELECT COUNT(T1.id) FROM superhero AS T1 INNER JOIN alignment AS T2 ON T1.alignment_id = T2.id WHERE T2.alignment = 'Neutral'", "difficulty": "simple"}
+{"instance_id": "bird_796", "db": "superhero", "question": "State all of 3-D Man's attributes along with their values.", "external_knowledge": "superhero.md", "evidence": "3-D Man is the superhero_name. attributes refers to attribute_name; values refers to attribute_value;", "SQL": "SELECT T3.attribute_name, T2.attribute_value FROM superhero AS T1 INNER JOIN hero_attribute AS T2 ON T1.id = T2.hero_id INNER JOIN attribute AS T3 ON T2.attribute_id = T3.id WHERE T1.superhero_name = '3-D Man'", "difficulty": "moderate"}
+{"instance_id": "bird_797", "db": "superhero", "question": "Which superheroes have blue eyes with brown hair?", "external_knowledge": "superhero.md", "evidence": "which superheroes refers to superhero_name; blue eyes refers to color = 'Blue' and color.id = eye_colour_id; brown hair refers to color = 'Brown' and color.id = hair_colour_id;", "SQL": "SELECT T1.superhero_name FROM superhero AS T1 INNER JOIN colour AS T2 ON T1.eye_colour_id = T2.id INNER JOIN colour AS T3 ON T1.hair_colour_id = T3.id WHERE T2.colour = 'Blue' AND T3.colour = 'Brown'", "difficulty": "moderate"}
+{"instance_id": "bird_798", "db": "superhero", "question": "What is the publisher for Hawkman, Karate Kid and Speedy?", "external_knowledge": "superhero.md", "evidence": "publisher refers to publisher_name; Hawkman refers to superhero_name = 'Hawkman'; Karate Kid refers to superhero_name = 'Karate Kid'; Speedy refers to superhero_name = 'Speedy';", "SQL": "SELECT T2.publisher_name FROM superhero AS T1 INNER JOIN publisher AS T2 ON T1.publisher_id = T2.id WHERE T1.superhero_name IN ('Hawkman', 'Karate Kid', 'Speedy')", "difficulty": "moderate"}
+{"instance_id": "bird_799", "db": "superhero", "question": "How many superheroes didn't have any publisher?", "external_knowledge": "superhero.md", "evidence": "didn't have any publisher refers to publisher.id = 1;", "SQL": "SELECT COUNT(T1.id) FROM superhero AS T1 INNER JOIN publisher AS T2 ON T1.publisher_id = T2.id WHERE T2.id = 1", "difficulty": "simple"}
+{"instance_id": "bird_800", "db": "superhero", "question": "Calculate the percentage of superheroes with blue eyes.", "external_knowledge": "superhero.md", "evidence": "percentage = MULTIPLY(DIVIDE(SUM(superhero_name WHERE color = 'Blue'), COUNT(superhero_name)), 100.0); blue eyes refers to color = 'Blue' and color.id =  eye_colour_id = 7;", "SQL": "SELECT CAST(COUNT(CASE WHEN T2.colour = 'Blue' THEN 1 ELSE NULL END) AS REAL) * 100 / COUNT(T1.id) FROM superhero AS T1 INNER JOIN colour AS T2 ON T1.eye_colour_id = T2.id", "difficulty": "moderate"}
+{"instance_id": "bird_801", "db": "superhero", "question": "Find the ratio between male superheroes and female superheroes.", "external_knowledge": "superhero.md", "evidence": "ratio = DIVIDE(SUM(gender_id = 1) / SUM(gender_id = 2)); male superheroes refers to gender = 'Female'; female superheroes refers to gender = 'Male';", "SQL": "SELECT CAST(COUNT(CASE WHEN T2.gender = 'Male' THEN T1.id ELSE NULL END) AS REAL) / COUNT(CASE WHEN T2.gender = 'Female' THEN T1.id ELSE NULL END) FROM superhero AS T1 INNER JOIN gender AS T2 ON T1.gender_id = T2.id", "difficulty": "moderate"}
+{"instance_id": "bird_802", "db": "superhero", "question": "Who is the tallest superhero?", "external_knowledge": "superhero.md", "evidence": "who refers to superhero_name; tallest superhero refers to MAX(height_cm);", "SQL": "SELECT superhero_name FROM superhero ORDER BY height_cm DESC LIMIT 1", "difficulty": "simple"}
+{"instance_id": "bird_803", "db": "superhero", "question": "What is the power ID of cryokinesis?", "external_knowledge": "superhero.md", "evidence": "power ID refers to superpower.id; cryokinesis refers to power_name = 'cryokinesis';", "SQL": "SELECT id FROM superpower WHERE power_name = 'Cryokinesis'", "difficulty": "simple"}
+{"instance_id": "bird_804", "db": "superhero", "question": "Provide the name of superhero with superhero ID 294.", "external_knowledge": "superhero.md", "evidence": "name of superhero refers to superhero_name; superhero ID 294 refers to superhero.id = 294;", "SQL": "SELECT superhero_name FROM superhero WHERE id = 294", "difficulty": "simple"}
+{"instance_id": "bird_805", "db": "superhero", "question": "List the full names of superheroes with missing weight.", "external_knowledge": "superhero.md", "evidence": "missing weight refers to weight_kg = 0 OR weight_kg = NULL;", "SQL": "SELECT DISTINCT full_name FROM superhero WHERE full_name IS NOT NULL AND (weight_kg IS NULL OR weight_kg = 0)", "difficulty": "simple"}
+{"instance_id": "bird_806", "db": "superhero", "question": "Provide the eye colour of the superhero who has Karen Beecher-Duncan as their full name.", "external_knowledge": "superhero.md", "evidence": "eye colour refers to colour.colour where eye_colour_id = colour.id; Karen Beecher-Duncan is the full name of superhero;", "SQL": "SELECT T2.colour FROM superhero AS T1 INNER JOIN colour AS T2 ON T1.eye_colour_id = T2.id WHERE T1.full_name = 'Karen Beecher-Duncan'", "difficulty": "simple"}
+{"instance_id": "bird_807", "db": "superhero", "question": "What is the superpowers of the superhero has Helen Parr as their full name?", "external_knowledge": "superhero.md", "evidence": "superpowers refers to power_name; Helen Parr is the full name of superhero;", "SQL": "SELECT T3.power_name FROM superhero AS T1 INNER JOIN hero_power AS T2 ON T1.id = T2.hero_id INNER JOIN superpower AS T3 ON T2.power_id = T3.id WHERE T1.full_name = 'Helen Parr'", "difficulty": "simple"}
+{"instance_id": "bird_808", "db": "superhero", "question": "Find the race of the superhero who weighs 108kg and is 188cm tall.", "external_knowledge": "superhero.md", "evidence": "weighs 108kg refers to weight_kg = 108; 188cm tall refers to height_cm = 188;", "SQL": "SELECT DISTINCT T2.race FROM superhero AS T1 INNER JOIN race AS T2 ON T1.race_id = T2.id WHERE T1.weight_kg = 108 AND T1.height_cm = 188", "difficulty": "simple"}
+{"instance_id": "bird_809", "db": "superhero", "question": "What is the publisher name of the superhero ID 38?", "external_knowledge": "superhero.md", "evidence": "superhero ID 38 refers to superhero.id = 38;", "SQL": "SELECT T2.publisher_name FROM superhero AS T1 INNER JOIN publisher AS T2 ON T1.publisher_id = T2.id WHERE T1.id = 38", "difficulty": "simple"}
+{"instance_id": "bird_810", "db": "superhero", "question": "What is the race of the superhero with maximum attribute value?", "external_knowledge": "superhero.md", "evidence": "maximum attribute value refers to MAX(attribute_value);", "SQL": "SELECT T3.race FROM superhero AS T1 INNER JOIN hero_attribute AS T2 ON T1.id = T2.hero_id INNER JOIN race AS T3 ON T1.race_id = T3.id ORDER BY T2.attribute_value DESC LIMIT 1", "difficulty": "simple"}
+{"instance_id": "bird_811", "db": "superhero", "question": "Give the alignment and superpowers of the superhero named Atom IV.", "external_knowledge": "superhero.md", "evidence": "superpowers refers to power_name;", "SQL": "SELECT T4.alignment, T3.power_name FROM superhero AS T1 INNER JOIN hero_power AS T2 ON T1.id = T2.hero_id INNER JOIN superpower AS T3 ON T3.id = T2.power_id INNER JOIN alignment AS T4 ON T1.alignment_id = T4.id WHERE T1.superhero_name = 'Atom IV'", "difficulty": "simple"}
+{"instance_id": "bird_812", "db": "superhero", "question": "List down at least five full names of superheroes with blue eyes.", "external_knowledge": "superhero.md", "evidence": "blue eyes refers to colour.colour = 'Blue' WHERE eye_colour_id = colour.id; Name of superheroes refers to superhero_name;", "SQL": "SELECT T1.superhero_name FROM superhero AS T1 INNER JOIN colour AS T2 ON T1.eye_colour_id = T2.id WHERE T2.colour = 'Blue' LIMIT 5", "difficulty": "simple"}
+{"instance_id": "bird_813", "db": "superhero", "question": "Calculate the average attribute value of all neutral superheroes.", "external_knowledge": "superhero.md", "evidence": "average = AVG(attribute_value); neutral superheroes refers to alignment_id = 3;", "SQL": "SELECT AVG(T1.attribute_value) FROM hero_attribute AS T1 INNER JOIN superhero AS T2 ON T1.hero_id = T2.id INNER JOIN alignment AS T3 ON T2.alignment_id = T3.id WHERE T3.alignment = 'Neutral'", "difficulty": "simple"}
+{"instance_id": "bird_814", "db": "superhero", "question": "List the skin colour of the superheroes with 100 attribute value.", "external_knowledge": "superhero.md", "evidence": "skin colour refers to colour.colour where skin_colour_id = colour.id; 100 attribute value refers to attribute_value = 100;", "SQL": "SELECT DISTINCT T2.colour FROM superhero AS T1 INNER JOIN colour AS T2 ON T1.skin_colour_id = T2.id INNER JOIN hero_attribute AS T3 ON T1.id = T3.hero_id WHERE T3.attribute_value = 100", "difficulty": "moderate"}
+{"instance_id": "bird_815", "db": "superhero", "question": "Count the good female superheroes.", "external_knowledge": "superhero.md", "evidence": "good refers to alignment.id = 1; female refers to gender.id = 2;", "SQL": "SELECT COUNT(T1.id) FROM superhero AS T1 INNER JOIN alignment AS T2 ON T1.alignment_id = T2.id INNER JOIN gender AS T3 ON T1.gender_id = T3.id WHERE T2.alignment = 'Good' AND T3.gender = 'Female'", "difficulty": "simple"}
+{"instance_id": "bird_816", "db": "superhero", "question": "Provide the names of superheroes with attribute value between 75 to 80.", "external_knowledge": "superhero.md", "evidence": "names of superheroes refers to superhero_name; attribute value between 75 to 80 refers to attribute_value BETWEEN 75 AND 80;", "SQL": "SELECT T1.superhero_name FROM superhero AS T1 INNER JOIN hero_attribute AS T2 ON T1.id = T2.hero_id WHERE T2.attribute_value BETWEEN 75 AND 80", "difficulty": "simple"}
+{"instance_id": "bird_817", "db": "superhero", "question": "Give the race of the blue-haired male superhero.", "external_knowledge": "superhero.md", "evidence": "blue-haired refers to colour.colour = 'blue' WHERE hair_colour_id = colour.id; male refers to gender = 'male';", "SQL": "SELECT T3.race FROM superhero AS T1 INNER JOIN colour AS T2 ON T1.hair_colour_id = T2.id INNER JOIN race AS T3 ON T1.race_id = T3.id INNER JOIN gender AS T4 ON T1.gender_id = T4.id WHERE T2.colour = 'Blue' AND T4.gender = 'Male'", "difficulty": "moderate"}
+{"instance_id": "bird_818", "db": "superhero", "question": "Among the bad superheroes, what is the percentage of female superheroes?", "external_knowledge": "superhero.md", "evidence": "bad superheroes refers to alignment.id = 2; percentage = MULTIPLY(DIVIDE(SUM(gender.id = 2 WHERE alignment.id = 2), COUNT(alignment.id = 2)), 100.0); female refers to gender.id = 2;", "SQL": "SELECT CAST(COUNT(CASE WHEN T3.gender = 'Female' THEN T1.id ELSE NULL END) AS REAL) * 100 / COUNT(T1.id) FROM superhero AS T1 INNER JOIN alignment AS T2 ON T1.alignment_id = T2.id INNER JOIN gender AS T3 ON T1.gender_id = T3.id WHERE T2.alignment = 'Bad'", "difficulty": "challenging"}
+{"instance_id": "bird_819", "db": "superhero", "question": "In superheroes with missing weight data, calculate the difference between the number of superheroes with blue eyes and no eye color.", "external_knowledge": "superhero.md", "evidence": "missing weight data refers to weight_kg = 0 OR T1.weight_kg = NULL; difference = SUBTRACT(SUM(colour.id = 7), SUM(colour.id = 1)); blue eyes refers to eye_colour_id WHERE colour.id = 7; no eye color refers to eye_colour_id WHERE colour.id = 1;", "SQL": "SELECT SUM(CASE WHEN T2.id = 7 THEN 1 ELSE 0 END) - SUM(CASE WHEN T2.id = 1 THEN 1 ELSE 0 END) FROM superhero AS T1 INNER JOIN colour AS T2 ON T1.eye_colour_id = T2.id WHERE T1.weight_kg = 0 OR T1.weight_kg is NULL", "difficulty": "challenging"}
+{"instance_id": "bird_820", "db": "superhero", "question": "How strong is the Hulk?", "external_knowledge": "superhero.md", "evidence": "how strong refers to attribute_value WHERE attribute_name = 'Strength'; the Hulk refers to superhero_name = 'Hulk';", "SQL": "SELECT T2.attribute_value FROM superhero AS T1 INNER JOIN hero_attribute AS T2 ON T1.id = T2.hero_id INNER JOIN attribute AS T3 ON T2.attribute_id = T3.id WHERE T1.superhero_name = 'Hulk' AND T3.attribute_name = 'Strength'", "difficulty": "moderate"}
+{"instance_id": "bird_821", "db": "superhero", "question": "List down Ajax's superpowers.", "external_knowledge": "superhero.md", "evidence": "Ajax refers to superhero_name = 'Ajax'; superpowers refers to power_name;", "SQL": "SELECT T3.power_name FROM superhero AS T1 INNER JOIN hero_power AS T2 ON T1.id = T2.hero_id INNER JOIN superpower AS T3 ON T2.power_id = T3.id WHERE T1.superhero_name = 'Ajax'", "difficulty": "simple"}
+{"instance_id": "bird_822", "db": "superhero", "question": "How many green-skinned villains are there in the superhero universe?", "external_knowledge": "superhero.md", "evidence": "green-skinned refers to colour.colour = 'Green' WHERE skin_colour_id = colour.id; villains refers to alignment = 'Bad';", "SQL": "SELECT COUNT(T1.id) FROM superhero AS T1 INNER JOIN alignment AS T2 ON T1.alignment_id = T2.id INNER JOIN colour AS T3 ON T1.skin_colour_id = T3.id WHERE T2.alignment = 'Bad' AND T3.colour = 'Green'", "difficulty": "moderate"}
+{"instance_id": "bird_823", "db": "superhero", "question": "How many female superheroes are in Marvel Comics?", "external_knowledge": "superhero.md", "evidence": "female refers to gender = 'Female'; Marvel Comics refers to publisher_name = 'Marvel Comics';", "SQL": "SELECT COUNT(T1.id) FROM superhero AS T1 INNER JOIN publisher AS T2 ON T1.publisher_id = T2.id INNER JOIN gender AS T3 ON T1.gender_id = T3.id WHERE T2.publisher_name = 'Marvel Comics' AND T3.gender = 'Female'", "difficulty": "moderate"}
+{"instance_id": "bird_824", "db": "superhero", "question": "Identify superheroes who can control wind and list their names in alphabetical order.", "external_knowledge": "superhero.md", "evidence": "superheroes refers to superhero_name; can control wind refers to power_name = 'Wind Control';", "SQL": "SELECT T1.superhero_name FROM superhero AS T1 INNER JOIN hero_power AS T2 ON T1.id = T2.hero_id INNER JOIN superpower AS T3 ON T2.power_id = T3.id WHERE T3.power_name = 'Wind Control' ORDER BY T1.superhero_name", "difficulty": "moderate"}
+{"instance_id": "bird_825", "db": "superhero", "question": "Identify the gender of the superhero who has the ability of Phoenix Force.", "external_knowledge": "superhero.md", "evidence": "ability of Phoenix Force refers to power_name = 'Phoenix Force';", "SQL": "SELECT T4.gender FROM superhero AS T1 INNER JOIN hero_power AS T2 ON T1.id = T2.hero_id INNER JOIN superpower AS T3 ON T2.power_id = T3.id INNER JOIN gender AS T4 ON T1.gender_id = T4.id WHERE T3.power_name = 'Phoenix Force'", "difficulty": "moderate"}
+{"instance_id": "bird_826", "db": "superhero", "question": "Identify the heaviest superhero in DC Comics.", "external_knowledge": "superhero.md", "evidence": "heaviest refers to MAX(weight_kg); DC Comics refers to publisher_name = 'DC Comics'; superhero refers to superhero_name;", "SQL": "SELECT T1.superhero_name FROM superhero AS T1 INNER JOIN publisher AS T2 ON T1.publisher_id = T2.id WHERE T2.publisher_name = 'DC Comics' ORDER BY T1.weight_kg DESC LIMIT 1", "difficulty": "simple"}
+{"instance_id": "bird_827", "db": "superhero", "question": "What is the average height of a non-human superhero in Dark Horse Comics?", "external_knowledge": "superhero.md", "evidence": "average = AVG(height_cm); non-human superhero refers to race <> 'Human'; Dark Horse Comics refers to publisher_name = 'Dark Horse Comics';", "SQL": "SELECT AVG(T1.height_cm) FROM superhero AS T1 INNER JOIN publisher AS T2 ON T1.publisher_id = T2.id INNER JOIN race AS T3 ON T1.race_id = T3.id WHERE T2.publisher_name = 'Dark Horse Comics' AND T3.race != 'Human'", "difficulty": "moderate"}
+{"instance_id": "bird_828", "db": "superhero", "question": "Count the fastest superheroes.", "external_knowledge": "superhero.md", "evidence": "fastest refers to attribute_value = 100 WHERE attribute_name = 'Speed';", "SQL": "SELECT COUNT(T3.superhero_name) FROM hero_attribute AS T1 INNER JOIN attribute AS T2 ON T1.attribute_id = T2.id INNER JOIN superhero AS T3 ON T1.hero_id = T3.id WHERE T2.attribute_name = 'Speed' AND T1.attribute_value = 100", "difficulty": "simple"}
+{"instance_id": "bird_829", "db": "superhero", "question": "Which publisher created more superheroes: DC or Marvel Comics? Find the difference in the number of superheroes.", "external_knowledge": "superhero.md", "evidence": "DC refers to publisher_name = 'DC Comics'; Marvel Comics refers to publisher_name = 'Marvel Comics'; difference = SUBTRACT(SUM(publisher_name = 'DC Comics'), SUM(publisher_name = 'Marvel Comics'));", "SQL": "SELECT SUM(CASE WHEN T2.publisher_name = 'DC Comics' THEN 1 ELSE 0 END) - SUM(CASE WHEN T2.publisher_name = 'Marvel Comics' THEN 1 ELSE 0 END) FROM superhero AS T1 INNER JOIN publisher AS T2 ON T1.publisher_id = T2.id", "difficulty": "challenging"}
+{"instance_id": "bird_830", "db": "superhero", "question": "Identify the weakest attribute of the Black Panther.", "external_knowledge": "superhero.md", "evidence": "weakest attribute refers to attribute_name WHERE MIN(attribute_value); Black Panther refers to superhero_name = 'Black Panther';", "SQL": "SELECT T3.attribute_name FROM superhero AS T1 INNER JOIN hero_attribute AS T2 ON T1.id = T2.hero_id INNER JOIN attribute AS T3 ON T2.attribute_id = T3.id WHERE T1.superhero_name = 'Black Panther' ORDER BY T2.attribute_value ASC LIMIT 1", "difficulty": "moderate"}
+{"instance_id": "bird_831", "db": "superhero", "question": "What is Abomination's eye colour?", "external_knowledge": "superhero.md", "evidence": "Abomination refers to superhero_name = 'Abomination'; eye colour refers to colour.colour where eye_colour_id = colour.id;", "SQL": "SELECT T2.colour FROM superhero AS T1 INNER JOIN colour AS T2 ON T1.eye_colour_id = T2.id WHERE T1.superhero_name = 'Abomination'", "difficulty": "simple"}
+{"instance_id": "bird_832", "db": "superhero", "question": "Name the tallest superhero.", "external_knowledge": "superhero.md", "evidence": "tallest superhero refers to MAX(height_cm);", "SQL": "SELECT superhero_name FROM superhero ORDER BY height_cm DESC LIMIT 1", "difficulty": "simple"}
+{"instance_id": "bird_833", "db": "superhero", "question": "Name the superhero, otherwise known as Charles Chandler.", "external_knowledge": "superhero.md", "evidence": "name the superhero refers to superhero_name; Charles Chandler is the full name of superhero;", "SQL": "SELECT superhero_name FROM superhero WHERE full_name = 'Charles Chandler'", "difficulty": "simple"}
+{"instance_id": "bird_834", "db": "superhero", "question": "Among all superheroes created by George Lucas, identify the percentage of female superheroes.", "external_knowledge": "superhero.md", "evidence": "created by George Lucas refers to publisher_name = 'George Lucas'; percentage = MULTIPLY(DIVIDE(SUM(gender = 'Female' WHERE publisher_name = 'George Lucas'), COUNT(publisher_name = 'George Lucas')), 100.0); female refers to gender = 'Female';", "SQL": "SELECT CAST(COUNT(CASE WHEN T3.gender = 'Female' THEN 1 ELSE NULL END) AS REAL) * 100 / COUNT(T1.id) FROM superhero AS T1 INNER JOIN publisher AS T2 ON T1.publisher_id = T2.id INNER JOIN gender AS T3 ON T1.gender_id = T3.id WHERE T2.publisher_name = 'George Lucas'", "difficulty": "challenging"}
+{"instance_id": "bird_835", "db": "superhero", "question": "Among all superheroes in Marvel Comics, identify the percentage of 'good' superheroes.", "external_knowledge": "superhero.md", "evidence": "Marvel Comics refers to publisher_name = 'Marvel Comics'; percentage = MULTIPLY(DIVIDE(SUM(alignment = 'Good' WHERE publisher_name = 'Marvel Comics'), COUNT(publisher_name = 'Marvel Comics')), 100.0); good superheroes refers to alignment = 'Good';", "SQL": "SELECT CAST(COUNT(CASE WHEN T3.alignment = 'Good' THEN T1.id ELSE NULL END) AS REAL) * 100 / COUNT(T1.id) FROM superhero AS T1 INNER JOIN publisher AS T2 ON T1.publisher_id = T2.id INNER JOIN alignment AS T3 ON T1.alignment_id = T3.id WHERE T2.publisher_name = 'Marvel Comics'", "difficulty": "challenging"}
+{"instance_id": "bird_836", "db": "superhero", "question": "What is the total number of superheroes that have John as their first name?", "external_knowledge": "superhero.md", "evidence": "have John as their first name refers to full_name LIKE 'John%';", "SQL": "SELECT COUNT(id) FROM superhero WHERE full_name LIKE 'John%'", "difficulty": "simple"}
+{"instance_id": "bird_837", "db": "superhero", "question": "Give the hero ID of superhero with the lowest attribute value.", "external_knowledge": "superhero.md", "evidence": "lowest attribute value refers to MIN(attribute_value);", "SQL": "SELECT hero_id FROM hero_attribute WHERE attribute_value = ( SELECT MIN(attribute_value) FROM hero_attribute )", "difficulty": "simple"}
+{"instance_id": "bird_838", "db": "superhero", "question": "Provide the full name of the superhero named Alien.", "external_knowledge": "superhero.md", "evidence": "", "SQL": "SELECT full_name FROM superhero WHERE superhero_name = 'Alien'", "difficulty": "simple"}
+{"instance_id": "bird_839", "db": "superhero", "question": "In superheroes with weight less than 100, list the full name of the superheroes with brown eyes.", "external_knowledge": "superhero.md", "evidence": "weight less than 100 refers to weight_kg < 100", "SQL": "SELECT T1.full_name FROM superhero AS T1 INNER JOIN colour AS T2 ON T1.eye_colour_id = T2.id WHERE T1.weight_kg < 100 AND T2.colour = 'Brown'", "difficulty": "simple"}
+{"instance_id": "bird_840", "db": "superhero", "question": "List the attribute value of the superhero named Aquababy.", "external_knowledge": "superhero.md", "evidence": "", "SQL": "SELECT T2.attribute_value FROM superhero AS T1 INNER JOIN hero_attribute AS T2 ON T1.id = T2.hero_id WHERE T1.superhero_name = 'Aquababy'", "difficulty": "simple"}
+{"instance_id": "bird_841", "db": "superhero", "question": "Provide the weight and race of the superhero with superhero ID 40.", "external_knowledge": "superhero.md", "evidence": "weight refers to weight_kg; superhero ID 40 refers to superhero.id = 40;", "SQL": "SELECT T1.weight_kg, T2.race FROM superhero AS T1 INNER JOIN race AS T2 ON T1.race_id = T2.id WHERE T1.id = 40", "difficulty": "simple"}
+{"instance_id": "bird_842", "db": "superhero", "question": "Calculate the average height of all neutral superheroes.", "external_knowledge": "superhero.md", "evidence": "", "SQL": "SELECT AVG(T1.height_cm) FROM superhero AS T1 INNER JOIN alignment AS T2 ON T1.alignment_id = T2.id WHERE T2.alignment = 'Neutral'", "difficulty": "simple"}
+{"instance_id": "bird_843", "db": "superhero", "question": "List the hero ID of superheroes have intellegence as their power.", "external_knowledge": "superhero.md", "evidence": "hero ID refers to superhero.id; have intelligence as their power refers to power_name = 'Intelligence';", "SQL": "SELECT T1.hero_id FROM hero_power AS T1 INNER JOIN superpower AS T2 ON T1.power_id = T2.id WHERE T2.power_name = 'Intelligence'", "difficulty": "simple"}
+{"instance_id": "bird_844", "db": "superhero", "question": "Give the eye colour of Blackwulf.", "external_knowledge": "superhero.md", "evidence": "eye colour refers to colour.colour where eye_colour_id = colour.id; Blackwulf refers to superhero_name = 'Blackwulf';", "SQL": "SELECT T2.colour FROM superhero AS T1 INNER JOIN colour AS T2 ON T1.eye_colour_id = T2.id WHERE T1.superhero_name = 'Blackwulf'", "difficulty": "simple"}
+{"instance_id": "bird_845", "db": "superhero", "question": "List the power of superheroes with height greater than 80% of the average height of all superheroes.", "external_knowledge": "superhero.md", "evidence": "power of superheroes refers to power_name; height greater than 80% of the average height of all superheroes = height_cm > MULTIPLY(AVG(height_cm), 0.8);", "SQL": "SELECT T3.power_name FROM superhero AS T1 INNER JOIN hero_power AS T2 ON T1.id = T2.hero_id INNER JOIN superpower AS T3 ON T2.power_id = T3.id WHERE T1.height_cm * 100 > ( SELECT AVG(height_cm) FROM superhero ) * 80", "difficulty": "moderate"}

data/text2sql/bird/databases/superhero/superhero.sqlite

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:75e94a2c3236ee3bb2c01fb97a1c4b4c1c269bcefd4eab1d04be323d2d0825b1
+size 237568

data/text2sql/spider2-lite/bigquery/austin/austin_bigquery_summary.json

@@ -0,0 +1,39 @@
+{
+  "tables": {
+    "bigquery-public-data.austin_311.311_service_requests": {
+      "name": "bigquery-public-data.austin_311.311_service_requests",
+      "columns": [
+        {"name": "unique_key", "type": "STRING", "is_primary_key": true, "pk_origin": "heuristic", "is_foreign_key": false, "description": "The service request tracking number."},
+        {"name": "complaint_description", "type": "STRING", "is_primary_key": false, "is_foreign_key": false, "description": "Service request type"},
+        {"name": "source", "type": "STRING", "is_primary_key": false, "is_foreign_key": false, "description": "Contact method Service request was received from."},
+        {"name": "status", "type": "STRING", "is_primary_key": false, "is_foreign_key": false, "description": "Service request status."},
+        {"name": "status_change_date", "type": "TIMESTAMP", "is_primary_key": false, "is_foreign_key": false, "description": "Date of last Service request status change."},
+        {"name": "created_date", "type": "TIMESTAMP", "is_primary_key": false, "is_foreign_key": false, "description": "Date Service request was created."},
+        {"name": "last_update_date", "type": "TIMESTAMP", "is_primary_key": false, "is_foreign_key": false, "description": "Date Service request was updated."},
+        {"name": "close_date", "type": "TIMESTAMP", "is_primary_key": false, "is_foreign_key": false, "description": "Date Service request was closed."},
+        {"name": "incident_address", "type": "STRING", "is_primary_key": false, "is_foreign_key": false, "description": "Service location of Service request."},
+        {"name": "street_number", "type": "STRING", "is_primary_key": false, "is_foreign_key": false, "description": "Parsed location information. Street number."},
+        {"name": "street_name", "type": "STRING", "is_primary_key": false, "is_foreign_key": false, "description": "Parsed location information. Street name."},
+        {"name": "city", "type": "STRING", "is_primary_key": false, "is_foreign_key": false, "description": "Parsed location information. City."},
+        {"name": "incident_zip", "type": "INT64", "is_primary_key": false, "is_foreign_key": false, "description": "Parsed location information. Zip code."},
+        {"name": "county", "type": "STRING", "is_primary_key": false, "is_foreign_key": false, "description": "Parsed location information. County."},
+        {"name": "state_plane_x_coordinate", "type": "STRING", "is_primary_key": false, "is_foreign_key": false, "description": "State plane X coordinate."},
+        {"name": "state_plane_y_coordinate", "type": "FLOAT64", "is_primary_key": false, "is_foreign_key": false, "description": "State plane Y coordinate."},
+        {"name": "latitude", "type": "FLOAT64", "is_primary_key": false, "is_foreign_key": false, "description": "Service request location latitude coordinate."},
+        {"name": "longitude", "type": "FLOAT64", "is_primary_key": false, "is_foreign_key": false, "description": "Service request location longitude coordinate."},
+        {"name": "location", "type": "STRING", "is_primary_key": false, "is_foreign_key": false, "description": "Service request location latitude and longitude coordinates."},
+        {"name": "council_district_code", "type": "INT64", "is_primary_key": false, "is_foreign_key": false, "description": "Council district corresponding to Service request location."},
+        {"name": "map_page", "type": "STRING", "is_primary_key": false, "is_foreign_key": false, "description": "Service request location corresponding map page."},
+        {"name": "map_tile", "type": "STRING", "is_primary_key": false, "is_foreign_key": false, "description": "Service request location corresponding map tile."}
+      ],
+      "samples": {
+        "unique_key": ["18-00004837", "18-00004838"],
+        "complaint_description": ["Loose Dog", "Animal Control - Assistance Request"],
+        "status": ["Closed", "Closed"],
+        "city": ["Austin", "Austin"]
+      },
+      "foreign_keys": []
+    }
+  },
+  "relationships": []
+}

data/text2sql/spider2-lite/golden_lite_spider_total.json

@@ -0,0 +1,1443 @@
+{
+  "local003": {
+    "instance_id": "local003",
+    "schema_extraction": {
+      "tables": [
+        "customers",
+        "order_items",
+        "orders"
+      ],
+      "columns": [
+        "customers.customer_unique_id",
+        "customers.customer_id",
+        "order_items.price",
+        "order_items.order_id",
+        "orders.order_purchase_timestamp",
+        "orders.customer_id",
+        "orders.order_status",
+        "orders.order_id"
+      ]
+    }
+  },
+  "local004": {
+    "instance_id": "local004",
+    "schema_extraction": {
+      "tables": [
+        "customers",
+        "order_payments",
+        "orders"
+      ],
+      "columns": [
+        "customers.customer_unique_id",
+        "customers.customer_id",
+        "order_payments.payment_value",
+        "order_payments.order_id",
+        "orders.customer_id",
+        "orders.order_id",
+        "orders.order_purchase_timestamp"
+      ]
+    }
+  },
+  "local008": {
+    "instance_id": "local008",
+    "schema_extraction": {
+      "tables": [
+        "batting",
+        "player"
+      ],
+      "columns": [
+        "batting.player_id",
+        "batting.g",
+        "batting.r",
+        "batting.h",
+        "batting.hr",
+        "player.name_given",
+        "player.player_id"
+      ]
+    }
+  },
+  "local009": {
+    "instance_id": "local009",
+    "schema_extraction": {
+      "tables": [
+        "airports_data",
+        "flights"
+      ],
+      "columns": [
+        "airports_data.city",
+        "airports_data.airport_code",
+        "airports_data.coordinates",
+        "flights.flight_id",
+        "flights.departure_airport",
+        "flights.arrival_airport"
+      ]
+    }
+  },
+  "local010": {
+    "instance_id": "local010",
+    "schema_extraction": {
+      "tables": [
+        "airports_data",
+        "flights"
+      ],
+      "columns": [
+        "airports_data.city",
+        "airports_data.airport_code",
+        "airports_data.coordinates",
+        "flights.flight_id",
+        "flights.departure_airport",
+        "flights.arrival_airport"
+      ]
+    }
+  },
+  "local015": {
+    "instance_id": "local015",
+    "schema_extraction": {
+      "tables": [
+        "collisions",
+        "parties"
+      ],
+      "columns": [
+        "collisions.case_id",
+        "collisions.motorcyclist_killed_count",
+        "collisions.motorcycle_collision",
+        "parties.party_safety_equipment_1",
+        "parties.party_safety_equipment_2",
+        "parties.case_id",
+        "parties.party_age"
+      ]
+    }
+  },
+  "local017": {
+    "instance_id": "local017",
+    "schema_extraction": {
+      "tables": [
+        "collisions"
+      ],
+      "columns": [
+        "collisions.pcf_violation_category",
+        "collisions.case_id",
+        "collisions.collision_date"
+      ]
+    }
+  },
+  "local019": {
+    "instance_id": "local019",
+    "schema_extraction": {
+      "tables": [
+        "Promotions",
+        "Cards",
+        "Locations",
+        "Events",
+        "Matches",
+        "Belts",
+        "Wrestlers"
+      ],
+      "columns": [
+        "Promotions.id",
+        "Promotions.name",
+        "Cards.id",
+        "Cards.location_id",
+        "Cards.event_id",
+        "Cards.promotion_id",
+        "Locations.name",
+        "Locations.id",
+        "Events.name",
+        "Events.id",
+        "Matches.duration",
+        "Matches.win_type",
+        "Matches.title_id",
+        "Matches.winner_id",
+        "Matches.loser_id",
+        "Matches.card_id",
+        "Belts.name",
+        "Belts.id",
+        "Wrestlers.name",
+        "Wrestlers.id"
+      ]
+    }
+  },
+  "local026": {
+    "instance_id": "local026",
+    "schema_extraction": {
+      "tables": [
+        "player",
+        "ball_by_ball",
+        "batsman_scored",
+        "extra_runs"
+      ],
+      "columns": [
+        "player.player_name",
+        "player.player_id",
+        "ball_by_ball.match_id",
+        "ball_by_ball.bowler",
+        "ball_by_ball.over_id",
+        "ball_by_ball.innings_no",
+        "batsman_scored.match_id",
+        "batsman_scored.over_id",
+        "batsman_scored.ball_id",
+        "batsman_scored.innings_no",
+        "batsman_scored.runs_scored",
+        "extra_runs.match_id",
+        "extra_runs.over_id",
+        "extra_runs.ball_id",
+        "extra_runs.innings_no",
+        "extra_runs.extra_runs"
+      ]
+    }
+  },
+  "local021": {
+    "instance_id": "local021",
+    "schema_extraction": {
+      "tables": [
+        "ball_by_ball",
+        "batsman_scored"
+      ],
+      "columns": [
+        "ball_by_ball.match_id",
+        "ball_by_ball.striker",
+        "ball_by_ball.innings_no",
+        "ball_by_ball.ball_id",
+        "ball_by_ball.over_id",
+        "batsman_scored.runs_scored",
+        "batsman_scored.innings_no",
+        "batsman_scored.ball_id",
+        "batsman_scored.match_id",
+        "batsman_scored.over_id"
+      ]
+    }
+  },
+  "local022": {
+    "instance_id": "local022",
+    "schema_extraction": {
+      "tables": [
+        "player",
+        "match",
+        "player_match",
+        "ball_by_ball",
+        "batsman_scored"
+      ],
+      "columns": [
+        "player.player_name",
+        "player.player_id",
+        "match.match_id",
+        "match.team_1",
+        "match.team_2",
+        "match.match_winner",
+        "player_match.team_id",
+        "player_match.player_id",
+        "player_match.match_id",
+        "ball_by_ball.striker",
+        "ball_by_ball.match_id",
+        "ball_by_ball.innings_no",
+        "ball_by_ball.ball_id",
+        "ball_by_ball.over_id",
+        "batsman_scored.runs_scored",
+        "batsman_scored.innings_no",
+        "batsman_scored.ball_id",
+        "batsman_scored.match_id",
+        "batsman_scored.over_id"
+      ]
+    }
+  },
+  "local023": {
+    "instance_id": "local023",
+    "schema_extraction": {
+      "tables": [
+        "player",
+        "match",
+        "ball_by_ball",
+        "batsman_scored"
+      ],
+      "columns": [
+        "player.player_name",
+        "player.player_id",
+        "match.match_id",
+        "match.season_id",
+        "ball_by_ball.striker",
+        "ball_by_ball.match_id",
+        "ball_by_ball.innings_no",
+        "ball_by_ball.ball_id",
+        "ball_by_ball.over_id",
+        "batsman_scored.runs_scored",
+        "batsman_scored.innings_no",
+        "batsman_scored.ball_id",
+        "batsman_scored.match_id",
+        "batsman_scored.over_id"
+      ]
+    }
+  },
+  "local024": {
+    "instance_id": "local024",
+    "schema_extraction": {
+      "tables": [
+        "player",
+        "ball_by_ball",
+        "batsman_scored"
+      ],
+      "columns": [
+        "player.country_name",
+        "player.player_id",
+        "ball_by_ball.striker",
+        "ball_by_ball.match_id",
+        "ball_by_ball.innings_no",
+        "ball_by_ball.ball_id",
+        "ball_by_ball.over_id",
+        "batsman_scored.runs_scored",
+        "batsman_scored.innings_no",
+        "batsman_scored.ball_id",
+        "batsman_scored.match_id",
+        "batsman_scored.over_id"
+      ]
+    }
+  },
+  "local028": {
+    "instance_id": "local028",
+    "schema_extraction": {
+      "tables": [
+        "olist_orders"
+      ],
+      "columns": [
+        "olist_orders.order_delivered_customer_date",
+        "olist_orders.order_status"
+      ]
+    }
+  },
+  "local031": {
+    "instance_id": "local031",
+    "schema_extraction": {
+      "tables": [
+        "olist_orders"
+      ],
+      "columns": [
+        "olist_orders.order_status",
+        "olist_orders.order_delivered_customer_date"
+      ]
+    }
+  },
+  "local029": {
+    "instance_id": "local029",
+    "schema_extraction": {
+      "tables": [
+        "olist_customers",
+        "olist_orders",
+        "olist_order_payments"
+      ],
+      "columns": [
+        "olist_customers.customer_unique_id",
+        "olist_customers.customer_city",
+        "olist_customers.customer_state",
+        "olist_customers.customer_id",
+        "olist_orders.order_id",
+        "olist_orders.customer_id",
+        "olist_orders.order_status",
+        "olist_order_payments.payment_value",
+        "olist_order_payments.order_id"
+      ]
+    }
+  },
+  "local030": {
+    "instance_id": "local030",
+    "schema_extraction": {
+      "tables": [
+        "olist_customers",
+        "olist_orders",
+        "olist_order_payments"
+      ],
+      "columns": [
+        "olist_customers.customer_city",
+        "olist_customers.customer_id",
+        "olist_orders.customer_id",
+        "olist_orders.order_id",
+        "olist_orders.order_status",
+        "olist_order_payments.payment_value",
+        "olist_order_payments.order_id"
+      ]
+    }
+  },
+  "local034": {
+    "instance_id": "local034",
+    "schema_extraction": {
+      "tables": [
+        "olist_order_items",
+        "olist_products",
+        "olist_order_payments"
+      ],
+      "columns": [
+        "olist_order_items.product_id",
+        "olist_order_items.order_id",
+        "olist_products.product_category_name",
+        "olist_products.product_id",
+        "olist_order_payments.payment_type",
+        "olist_order_payments.order_id"
+      ]
+    }
+  },
+  "local038": {
+    "instance_id": "local038",
+    "schema_extraction": {
+      "tables": [
+        "actor",
+        "language",
+        "category",
+        "film",
+        "film_actor",
+        "film_category"
+      ],
+      "columns": [
+        "actor.actor_id",
+        "actor.first_name",
+        "actor.last_name",
+        "language.language_id",
+        "language.name",
+        "category.category_id",
+        "category.name",
+        "film.film_id",
+        "film.language_id",
+        "film.length",
+        "film.rating",
+        "film.release_year",
+        "film_actor.actor_id",
+        "film_actor.film_id",
+        "film_category.film_id",
+        "film_category.category_id"
+      ]
+    }
+  },
+  "local039": {
+    "instance_id": "local039",
+    "schema_extraction": {
+      "tables": [
+        "city",
+        "address",
+        "category",
+        "customer",
+        "film_category",
+        "inventory",
+        "rental"
+      ],
+      "columns": [
+        "city.city_id",
+        "city.city",
+        "address.address_id",
+        "address.city_id",
+        "category.name",
+        "category.category_id",
+        "customer.customer_id",
+        "customer.address_id",
+        "film_category.film_id",
+        "film_category.category_id",
+        "inventory.inventory_id",
+        "inventory.film_id",
+        "rental.customer_id",
+        "rental.inventory_id",
+        "rental.return_date",
+        "rental.rental_date"
+      ]
+    }
+  },
+  "local041": {
+    "instance_id": "local041",
+    "schema_extraction": {
+      "tables": [
+        "trees"
+      ],
+      "columns": [
+        "trees.boroname",
+        "trees.health"
+      ]
+    }
+  },
+  "local049": {
+    "instance_id": "local049",
+    "schema_extraction": {
+      "tables": [
+        "companies_dates",
+        "companies_industries"
+      ],
+      "columns": [
+        "companies_dates.company_id",
+        "companies_dates.date_joined",
+        "companies_industries.industry",
+        "companies_industries.company_id"
+      ]
+    }
+  },
+  "local054": {
+    "instance_id": "local054",
+    "schema_extraction": {
+      "tables": [
+        "albums",
+        "artists",
+        "customers",
+        "invoices",
+        "invoice_items",
+        "tracks"
+      ],
+      "columns": [
+        "albums.ArtistId",
+        "albums.AlbumId",
+        "artists.Name",
+        "artists.ArtistId",
+        "customers.CustomerId",
+        "customers.FirstName",
+        "invoices.InvoiceId",
+        "invoices.CustomerId",
+        "invoice_items.TrackId",
+        "invoice_items.InvoiceId",
+        "invoice_items.UnitPrice",
+        "invoice_items.Quantity",
+        "tracks.AlbumId",
+        "tracks.TrackId"
+      ]
+    }
+  },
+  "local055": {
+    "instance_id": "local055",
+    "schema_extraction": {
+      "tables": [
+        "albums",
+        "artists",
+        "customers",
+        "invoices",
+        "invoice_items",
+        "tracks"
+      ],
+      "columns": [
+        "albums.AlbumId",
+        "albums.ArtistId",
+        "artists.ArtistId",
+        "artists.Name",
+        "customers.CustomerId",
+        "invoices.InvoiceId",
+        "invoices.CustomerId",
+        "invoice_items.TrackId",
+        "invoice_items.InvoiceId",
+        "invoice_items.Quantity",
+        "invoice_items.UnitPrice",
+        "tracks.TrackId",
+        "tracks.AlbumId"
+      ]
+    }
+  },
+  "local198": {
+    "instance_id": "local198",
+    "schema_extraction": {
+      "tables": [
+        "customers",
+        "invoices"
+      ],
+      "columns": [
+        "customers.Country",
+        "invoices.BillingCountry",
+        "invoices.Total"
+      ]
+    }
+  },
+  "local056": {
+    "instance_id": "local056",
+    "schema_extraction": {
+      "tables": [
+        "customer",
+        "payment"
+      ],
+      "columns": [
+        "customer.last_name",
+        "customer.customer_id",
+        "customer.first_name",
+        "payment.amount",
+        "payment.customer_id",
+        "payment.payment_date"
+      ]
+    }
+  },
+  "local058": {
+    "instance_id": "local058",
+    "schema_extraction": {
+      "tables": [
+        "hardware_dim_product",
+        "hardware_fact_sales_monthly"
+      ],
+      "columns": [
+        "hardware_dim_product.segment",
+        "hardware_dim_product.product_code",
+        "hardware_fact_sales_monthly.fiscal_year",
+        "hardware_fact_sales_monthly.product_code"
+      ]
+    }
+  },
+  "local059": {
+    "instance_id": "local059",
+    "schema_extraction": {
+      "tables": [
+        "hardware_dim_product",
+        "hardware_fact_sales_monthly"
+      ],
+      "columns": [
+        "hardware_dim_product.product_code",
+        "hardware_dim_product.division",
+        "hardware_fact_sales_monthly.sold_quantity",
+        "hardware_fact_sales_monthly.product_code",
+        "hardware_fact_sales_monthly.date"
+      ]
+    }
+  },
+  "local071": {
+    "instance_id": "local071",
+    "schema_extraction": {
+      "tables": [
+        "cities"
+      ],
+      "columns": [
+        "cities.country_code_2",
+        "cities.insert_date"
+      ]
+    }
+  },
+  "local066": {
+    "instance_id": "local066",
+    "schema_extraction": {
+      "tables": [
+        "pizza_toppings",
+        "pizza_recipes"
+      ],
+      "columns": [
+        "pizza_toppings.topping_name",
+        "pizza_toppings.topping_id",
+        "pizza_recipes.pizza_id",
+        "pizza_recipes.toppings"
+      ]
+    }
+  },
+  "local065": {
+    "instance_id": "local065",
+    "schema_extraction": {
+      "tables": [
+        "pizza_clean_runner_orders",
+        "pizza_clean_customer_orders"
+      ],
+      "columns": [
+        "pizza_clean_runner_orders.order_id",
+        "pizza_clean_runner_orders.cancellation",
+        "pizza_clean_customer_orders.order_id",
+        "pizza_clean_customer_orders.pizza_id",
+        "pizza_clean_customer_orders.extras"
+      ]
+    }
+  },
+  "local064": {
+    "instance_id": "local064",
+    "schema_extraction": {
+      "tables": [
+        "customer_transactions"
+      ],
+      "columns": [
+        "customer_transactions.customer_id",
+        "customer_transactions.txn_date",
+        "customer_transactions.txn_amount",
+        "customer_transactions.txn_type"
+      ]
+    }
+  },
+  "local299": {
+    "instance_id": "local299",
+    "schema_extraction": {
+      "tables": [
+        "customer_transactions"
+      ],
+      "columns": [
+        "customer_transactions.customer_id",
+        "customer_transactions.txn_date",
+        "customer_transactions.txn_amount",
+        "customer_transactions.txn_type"
+      ]
+    }
+  },
+  "local300": {
+    "instance_id": "local300",
+    "schema_extraction": {
+      "tables": [
+        "customer_transactions"
+      ],
+      "columns": [
+        "customer_transactions.customer_id",
+        "customer_transactions.txn_date",
+        "customer_transactions.txn_amount",
+        "customer_transactions.txn_type"
+      ]
+    }
+  },
+  "local075": {
+    "instance_id": "local075",
+    "schema_extraction": {
+      "tables": [
+        "shopping_cart_events",
+        "shopping_cart_page_hierarchy"
+      ],
+      "columns": [
+        "shopping_cart_events.event_type",
+        "shopping_cart_events.visit_id",
+        "shopping_cart_page_hierarchy.page_id",
+        "shopping_cart_page_hierarchy.page_name",
+        "shopping_cart_page_hierarchy.product_id"
+      ]
+    }
+  },
+  "local078": {
+    "instance_id": "local078",
+    "schema_extraction": {
+      "tables": [
+        "interest_metrics",
+        "interest_map"
+      ],
+      "columns": [
+        "interest_metrics.interest_id",
+        "interest_metrics.month_year",
+        "interest_metrics.composition",
+        "interest_map.interest_name",
+        "interest_map.id"
+      ]
+    }
+  },
+  "local085": {
+    "instance_id": "local085",
+    "schema_extraction": {
+      "tables": [
+        "orders"
+      ],
+      "columns": [
+        "orders.employeeid",
+        "orders.orderid",
+        "orders.shippeddate",
+        "orders.requireddate"
+      ]
+    }
+  },
+  "local099": {
+    "instance_id": "local099",
+    "schema_extraction": {
+      "tables": [
+        "Person",
+        "M_Director",
+        "M_Cast"
+      ],
+      "columns": [
+        "Person.PID",
+        "Person.Name",
+        "M_Director.PID",
+        "M_Director.MID",
+        "M_Cast.PID",
+        "M_Cast.MID"
+      ]
+    }
+  },
+  "local131": {
+    "instance_id": "local131",
+    "schema_extraction": {
+      "tables": [
+        "Musical_Preferences",
+        "Musical_Styles"
+      ],
+      "columns": [
+        "Musical_Preferences.StyleID",
+        "Musical_Preferences.PreferenceSeq",
+        "Musical_Styles.StyleName",
+        "Musical_Styles.StyleID"
+      ]
+    }
+  },
+  "local132": {
+    "instance_id": "local132",
+    "schema_extraction": {
+      "tables": [
+        "Customers",
+        "Entertainer_Styles",
+        "Entertainers",
+        "Musical_Preferences"
+      ],
+      "columns": [
+        "Customers.CustLastName",
+        "Customers.CustomerID",
+        "Entertainer_Styles.EntertainerID",
+        "Entertainer_Styles.StyleID",
+        "Entertainer_Styles.StyleStrength",
+        "Entertainers.EntStageName",
+        "Entertainers.EntertainerID",
+        "Musical_Preferences.CustomerID",
+        "Musical_Preferences.StyleID",
+        "Musical_Preferences.PreferenceSeq"
+      ]
+    }
+  },
+  "local157": {
+    "instance_id": "local157",
+    "schema_extraction": {
+      "tables": [
+        "bitcoin_prices"
+      ],
+      "columns": [
+        "bitcoin_prices.ticker",
+        "bitcoin_prices.market_date",
+        "bitcoin_prices.volume"
+      ]
+    }
+  },
+  "local163": {
+    "instance_id": "local163",
+    "schema_extraction": {
+      "tables": [
+        "university_faculty"
+      ],
+      "columns": [
+        "university_faculty.FacRank",
+        "university_faculty.FacFirstName",
+        "university_faculty.FacLastName",
+        "university_faculty.FacSalary"
+      ]
+    }
+  },
+  "local193": {
+    "instance_id": "local193",
+    "schema_extraction": {
+      "tables": [
+        "payment"
+      ],
+      "columns": [
+        "payment.customer_id",
+        "payment.payment_date",
+        "payment.amount"
+      ]
+    }
+  },
+  "local194": {
+    "instance_id": "local194",
+    "schema_extraction": {
+      "tables": [
+        "film",
+        "film_actor",
+        "inventory",
+        "payment",
+        "rental"
+      ],
+      "columns": [
+        "film.film_id",
+        "film.title",
+        "film_actor.actor_id",
+        "film_actor.film_id",
+        "inventory.film_id",
+        "inventory.inventory_id",
+        "payment.amount",
+        "payment.rental_id",
+        "rental.rental_id",
+        "rental.inventory_id"
+      ]
+    }
+  },
+  "local195": {
+    "instance_id": "local195",
+    "schema_extraction": {
+      "tables": [
+        "actor",
+        "customer",
+        "film",
+        "film_actor",
+        "inventory",
+        "payment",
+        "rental"
+      ],
+      "columns": [
+        "actor.actor_id",
+        "actor.first_name",
+        "actor.last_name",
+        "customer.customer_id",
+        "film.film_id",
+        "film.title",
+        "film_actor.actor_id",
+        "film_actor.film_id",
+        "inventory.film_id",
+        "inventory.inventory_id",
+        "payment.amount",
+        "payment.rental_id",
+        "payment.customer_id",
+        "rental.inventory_id",
+        "rental.rental_id"
+      ]
+    }
+  },
+  "local196": {
+    "instance_id": "local196",
+    "schema_extraction": {
+      "tables": [
+        "inventory",
+        "payment",
+        "rental"
+      ],
+      "columns": [
+        "inventory.inventory_id",
+        "inventory.film_id",
+        "payment.customer_id",
+        "payment.amount",
+        "rental.customer_id",
+        "rental.inventory_id",
+        "rental.rental_date"
+      ]
+    }
+  },
+  "local197": {
+    "instance_id": "local197",
+    "schema_extraction": {
+      "tables": [
+        "payment"
+      ],
+      "columns": [
+        "payment.customer_id",
+        "payment.amount",
+        "payment.payment_date"
+      ]
+    }
+  },
+  "local199": {
+    "instance_id": "local199",
+    "schema_extraction": {
+      "tables": [
+        "staff",
+        "rental"
+      ],
+      "columns": [
+        "staff.store_id",
+        "staff.staff_id",
+        "rental.rental_id",
+        "rental.staff_id",
+        "rental.rental_date"
+      ]
+    }
+  },
+  "local202": {
+    "instance_id": "local202",
+    "schema_extraction": {
+      "tables": [
+        "alien_data"
+      ],
+      "columns": [
+        "alien_data.state",
+        "alien_data.age",
+        "alien_data.aggressive"
+      ]
+    }
+  },
+  "local209": {
+    "instance_id": "local209",
+    "schema_extraction": {
+      "tables": [
+        "deliveries",
+        "stores",
+        "orders"
+      ],
+      "columns": [
+        "deliveries.delivery_order_id",
+        "deliveries.delivery_status",
+        "stores.store_name",
+        "stores.store_id",
+        "orders.order_id",
+        "orders.store_id",
+        "orders.delivery_order_id"
+      ]
+    }
+  },
+  "local210": {
+    "instance_id": "local210",
+    "schema_extraction": {
+      "tables": [
+        "hubs",
+        "stores",
+        "orders"
+      ],
+      "columns": [
+        "hubs.hub_name",
+        "hubs.hub_id",
+        "stores.store_id",
+        "stores.hub_id",
+        "orders.store_id",
+        "orders.order_created_month",
+        "orders.order_status"
+      ]
+    }
+  },
+  "local218": {
+    "instance_id": "local218",
+    "schema_extraction": {
+      "tables": [
+        "Match",
+        "Team"
+      ],
+      "columns": [
+        "Match.season",
+        "Match.away_team_goal",
+        "Match.home_team_api_id",
+        "Match.away_team_api_id",
+        "Match.home_team_goal",
+        "Team.id",
+        "Team.team_long_name",
+        "Team.team_api_id"
+      ]
+    }
+  },
+  "local219": {
+    "instance_id": "local219",
+    "schema_extraction": {
+      "tables": [
+        "Player",
+        "Match",
+        "League",
+        "Team"
+      ],
+      "columns": [
+        "Player.player_name",
+        "Player.player_api_id",
+        "Match.id",
+        "Match.season",
+        "Match.match_api_id",
+        "Match.home_team_goal",
+        "Match.away_team_goal",
+        "Match.goal",
+        "Match.card",
+        "Match.league_id",
+        "Match.home_team_api_id",
+        "Match.away_team_api_id",
+        "Match.home_player_1",
+        "Match.home_player_2",
+        "Match.home_player_3",
+        "Match.home_player_4",
+        "Match.home_player_5",
+        "Match.home_player_6",
+        "Match.home_player_7",
+        "Match.home_player_8",
+        "Match.home_player_9",
+        "Match.home_player_10",
+        "Match.home_player_11",
+        "Match.away_player_1",
+        "Match.away_player_2",
+        "Match.away_player_3",
+        "Match.away_player_4",
+        "Match.away_player_5",
+        "Match.away_player_6",
+        "Match.away_player_7",
+        "Match.away_player_8",
+        "Match.away_player_9",
+        "Match.away_player_10",
+        "Match.away_player_11",
+        "League.name",
+        "League.id",
+        "Team.team_long_name",
+        "Team.team_api_id"
+      ]
+    }
+  },
+  "local229": {
+    "instance_id": "local229",
+    "schema_extraction": {
+      "tables": [
+        "ball_by_ball",
+        "batsman_scored"
+      ],
+      "columns": [
+        "ball_by_ball.match_id",
+        "ball_by_ball.non_striker",
+        "ball_by_ball.striker",
+        "ball_by_ball.innings_no",
+        "ball_by_ball.ball_id",
+        "ball_by_ball.over_id",
+        "batsman_scored.runs_scored",
+        "batsman_scored.innings_no",
+        "batsman_scored.ball_id",
+        "batsman_scored.match_id",
+        "batsman_scored.over_id"
+      ]
+    }
+  },
+  "local244": {
+    "instance_id": "local244",
+    "schema_extraction": {
+      "tables": [
+        "InvoiceLine",
+        "Track"
+      ],
+      "columns": [
+        "InvoiceLine.TrackId",
+        "InvoiceLine.UnitPrice",
+        "InvoiceLine.Quantity",
+        "Track.TrackId",
+        "Track.Milliseconds"
+      ]
+    }
+  },
+  "local263": {
+    "instance_id": "local263",
+    "schema_extraction": {
+      "tables": [
+        "model_score",
+        "model"
+      ],
+      "columns": [
+        "model_score.name",
+        "model_score.version",
+        "model_score.step",
+        "model_score.test_score",
+        "model_score.model",
+        "model.L1_model",
+        "model.name",
+        "model.version"
+      ]
+    }
+  },
+  "local269": {
+    "instance_id": "local269",
+    "schema_extraction": {
+      "tables": [
+        "packaging",
+        "packaging_relations"
+      ],
+      "columns": [
+        "packaging.id",
+        "packaging_relations.contains_id",
+        "packaging_relations.packaging_id",
+        "packaging_relations.qty"
+      ]
+    }
+  },
+  "local274": {
+    "instance_id": "local274",
+    "schema_extraction": {
+      "tables": [
+        "products",
+        "picking_line"
+      ],
+      "columns": [
+        "products.name",
+        "products.id",
+        "picking_line.product_id",
+        "picking_line.location_id",
+        "picking_line.qty",
+        "picking_line.order_id"
+      ]
+    }
+  },
+  "local283": {
+    "instance_id": "local283",
+    "schema_extraction": {
+      "tables": [
+        "Match",
+        "League",
+        "Country",
+        "Team"
+      ],
+      "columns": [
+        "Match.id",
+        "Match.season",
+        "Match.stage",
+        "Match.date",
+        "Match.home_team_goal",
+        "Match.away_team_goal",
+        "Match.country_id",
+        "Match.league_id",
+        "Match.home_team_api_id",
+        "Match.away_team_api_id",
+        "League.name",
+        "League.id",
+        "Country.name",
+        "Country.id",
+        "Team.team_long_name",
+        "Team.team_api_id"
+      ]
+    }
+  },
+  "local284": {
+    "instance_id": "local284",
+    "schema_extraction": {
+      "tables": [
+        "veg_loss_rate_df"
+      ],
+      "columns": [
+        "veg_loss_rate_df.loss_rate_%"
+      ]
+    }
+  },
+  "local285": {
+    "instance_id": "local285",
+    "schema_extraction": {
+      "tables": [
+        "veg_txn_df",
+        "veg_loss_rate_df",
+        "veg_cat",
+        "veg_whsle_df"
+      ],
+      "columns": [
+        "veg_txn_df.item_code",
+        "veg_txn_df.txn_date",
+        "veg_txn_df.qty_sold(kg)",
+        "veg_txn_df.unit_selling_px_rmb/kg",
+        "veg_loss_rate_df.item_code",
+        "veg_loss_rate_df.loss_rate_%",
+        "veg_cat.category_code",
+        "veg_cat.category_name",
+        "veg_cat.item_code",
+        "veg_whsle_df.whsle_px_rmb-kg",
+        "veg_whsle_df.whsle_date",
+        "veg_whsle_df.item_code"
+      ]
+    }
+  },
+  "local301": {
+    "instance_id": "local301",
+    "schema_extraction": {
+      "tables": [
+        "cleaned_weekly_sales"
+      ],
+      "columns": [
+        "cleaned_weekly_sales.week_date",
+        "cleaned_weekly_sales.sales"
+      ]
+    }
+  },
+  "local329": {
+    "instance_id": "local329",
+    "schema_extraction": {
+      "tables": [
+        "form_log"
+      ],
+      "columns": [
+        "form_log.session",
+        "form_log.path",
+        "form_log.stamp"
+      ]
+    }
+  },
+  "local358": {
+    "instance_id": "local358",
+    "schema_extraction": {
+      "tables": [
+        "mst_users"
+      ],
+      "columns": [
+        "mst_users.user_id",
+        "mst_users.birth_date"
+      ]
+    }
+  },
+  "local360": {
+    "instance_id": "local360",
+    "schema_extraction": {
+      "tables": [
+        "activity_log"
+      ],
+      "columns": [
+        "activity_log.session",
+        "activity_log.stamp",
+        "activity_log.path",
+        "activity_log.search_type"
+      ]
+    }
+  },
+  "local336": {
+    "instance_id": "local336",
+    "schema_extraction": {
+      "tables": [
+        "pit_stops",
+        "results",
+        "lap_times_ext",
+        "races_ext",
+        "retirements",
+        "lap_positions"
+      ],
+      "columns": [
+        "pit_stops.driver_id",
+        "pit_stops.lap",
+        "pit_stops.milliseconds",
+        "pit_stops.race_id",
+        "results.race_id",
+        "results.driver_id",
+        "results.grid",
+        "lap_times_ext.lap",
+        "lap_times_ext.race_id",
+        "lap_times_ext.driver_id",
+        "lap_times_ext.running_milliseconds",
+        "races_ext.race_id",
+        "races_ext.is_pit_data_available",
+        "retirements.driver_id",
+        "retirements.race_id",
+        "retirements.lap",
+        "lap_positions.race_id",
+        "lap_positions.driver_id",
+        "lap_positions.lap",
+        "lap_positions.position"
+      ]
+    }
+  },
+  "local309": {
+    "instance_id": "local309",
+    "schema_extraction": {
+      "tables": [
+        "constructors",
+        "drivers",
+        "races",
+        "results"
+      ],
+      "columns": [
+        "constructors.name",
+        "constructors.constructor_id",
+        "drivers.surname",
+        "drivers.driver_id",
+        "drivers.forename",
+        "races.year",
+        "races.race_id",
+        "results.points",
+        "results.race_id",
+        "results.driver_id",
+        "results.constructor_id"
+      ]
+    }
+  },
+  "local311": {
+    "instance_id": "local311",
+    "schema_extraction": {
+      "tables": [
+        "constructors",
+        "drivers",
+        "races",
+        "results"
+      ],
+      "columns": [
+        "constructors.name",
+        "constructors.constructor_id",
+        "drivers.forename",
+        "drivers.surname",
+        "drivers.driver_id",
+        "races.year",
+        "races.race_id",
+        "results.points",
+        "results.race_id",
+        "results.driver_id",
+        "results.constructor_id"
+      ]
+    }
+  },
+  "local354": {
+    "instance_id": "local354",
+    "schema_extraction": {
+      "tables": [
+        "races",
+        "results"
+      ],
+      "columns": [
+        "races.year",
+        "races.round",
+        "races.race_id",
+        "results.driver_id",
+        "results.constructor_id",
+        "results.race_id"
+      ]
+    }
+  },
+  "local355": {
+    "instance_id": "local355",
+    "schema_extraction": {
+      "tables": [
+        "results",
+        "driver_standings_ext",
+        "races_ext"
+      ],
+      "columns": [
+        "results.race_id",
+        "results.driver_id",
+        "results.constructor_id",
+        "driver_standings_ext.race_id",
+        "driver_standings_ext.driver_id",
+        "races_ext.race_id",
+        "races_ext.year",
+        "races_ext.round"
+      ]
+    }
+  },
+  "local074": {
+    "instance_id": "local074",
+    "schema_extraction": {
+      "tables": [
+        "customer_transactions"
+      ],
+      "columns": [
+        "customer_transactions.customer_id",
+        "customer_transactions.txn_date",
+        "customer_transactions.txn_amount"
+      ]
+    }
+  },
+  "local077": {
+    "instance_id": "local077",
+    "schema_extraction": {
+      "tables": [
+        "interest_metrics",
+        "interest_map"
+      ],
+      "columns": [
+        "interest_metrics.month_year",
+        "interest_metrics.interest_id",
+        "interest_metrics.composition",
+        "interest_metrics.index_value",
+        "interest_metrics._year",
+        "interest_metrics._month",
+        "interest_map.id",
+        "interest_map.interest_name"
+      ]
+    }
+  },
+  "local114": {
+    "instance_id": "local114",
+    "schema_extraction": {
+      "tables": [
+        "web_orders",
+        "web_sales_reps",
+        "web_region"
+      ],
+      "columns": [
+        "web_orders.id",
+        "web_orders.account_id",
+        "web_orders.total_amt_usd",
+        "web_sales_reps.id",
+        "web_sales_reps.name",
+        "web_sales_reps.region_id",
+        "web_region.id",
+        "web_region.name"
+      ]
+    }
+  },
+  "local141": {
+    "instance_id": "local141",
+    "schema_extraction": {
+      "tables": [
+        "salesorderheader",
+        "SalesPersonQuotaHistory"
+      ],
+      "columns": [
+        "salesorderheader.salespersonid",
+        "salesorderheader.orderdate",
+        "salesorderheader.totaldue",
+        "SalesPersonQuotaHistory.BusinessEntityID",
+        "SalesPersonQuotaHistory.QuotaDate",
+        "SalesPersonQuotaHistory.SalesQuota"
+      ]
+    }
+  },
+  "local277": {
+    "instance_id": "local277",
+    "schema_extraction": {
+      "tables": [
+        "monthly_sales"
+      ],
+      "columns": [
+        "monthly_sales.product_id",
+        "monthly_sales.mth",
+        "monthly_sales.qty"
+      ]
+    }
+  }
+}

data/text2sql/spider2-lite/snowflake/AUSTIN/AUSTIN_db_summary.json

@@ -0,0 +1,39 @@
+{
+  "tables": {
+    "AUSTIN.AUSTIN_311._311_SERVICE_REQUESTS": {
+      "name": "AUSTIN.AUSTIN_311._311_SERVICE_REQUESTS",
+      "columns": [
+        {"name": "unique_key", "type": "VARCHAR(16777216)", "is_primary_key": true, "pk_origin": "heuristic", "is_foreign_key": false},
+        {"name": "complaint_description", "type": "VARCHAR(16777216)", "is_primary_key": false, "is_foreign_key": false},
+        {"name": "source", "type": "VARCHAR(16777216)", "is_primary_key": false, "is_foreign_key": false},
+        {"name": "status", "type": "VARCHAR(16777216)", "is_primary_key": false, "is_foreign_key": false},
+        {"name": "status_change_date", "type": "NUMBER(38,0)", "is_primary_key": false, "is_foreign_key": false},
+        {"name": "created_date", "type": "NUMBER(38,0)", "is_primary_key": false, "is_foreign_key": false},
+        {"name": "last_update_date", "type": "NUMBER(38,0)", "is_primary_key": false, "is_foreign_key": false},
+        {"name": "close_date", "type": "NUMBER(38,0)", "is_primary_key": false, "is_foreign_key": false},
+        {"name": "incident_address", "type": "VARCHAR(16777216)", "is_primary_key": false, "is_foreign_key": false},
+        {"name": "street_number", "type": "VARCHAR(16777216)", "is_primary_key": false, "is_foreign_key": false},
+        {"name": "street_name", "type": "VARCHAR(16777216)", "is_primary_key": false, "is_foreign_key": false},
+        {"name": "city", "type": "VARCHAR(16777216)", "is_primary_key": false, "is_foreign_key": false},
+        {"name": "incident_zip", "type": "VARCHAR(16777216)", "is_primary_key": false, "is_foreign_key": false},
+        {"name": "county", "type": "VARCHAR(16777216)", "is_primary_key": false, "is_foreign_key": false},
+        {"name": "state_plane_x_coordinate", "type": "VARCHAR(16777216)", "is_primary_key": false, "is_foreign_key": false},
+        {"name": "state_plane_y_coordinate", "type": "FLOAT", "is_primary_key": false, "is_foreign_key": false},
+        {"name": "latitude", "type": "FLOAT", "is_primary_key": false, "is_foreign_key": false},
+        {"name": "longitude", "type": "FLOAT", "is_primary_key": false, "is_foreign_key": false},
+        {"name": "location", "type": "VARCHAR(16777216)", "is_primary_key": false, "is_foreign_key": false},
+        {"name": "council_district_code", "type": "NUMBER(38,0)", "is_primary_key": false, "is_foreign_key": false},
+        {"name": "map_page", "type": "VARCHAR(16777216)", "is_primary_key": false, "is_foreign_key": false},
+        {"name": "map_tile", "type": "VARCHAR(16777216)", "is_primary_key": false, "is_foreign_key": false}
+      ],
+      "samples": {
+        "unique_key": ["18-00004837", "18-00004838"],
+        "complaint_description": ["Loose Dog", "Animal Control - Assistance Request"],
+        "status": ["Closed", "Closed"],
+        "city": ["Austin", "Austin"]
+      },
+      "foreign_keys": []
+    }
+  },
+  "relationships": []
+}

data/text2sql/spider2-lite/sqlite/Pagila/Pagila.sqlite

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dbca3837b9965d75fbc2b4af1b2daec57d841f95942e656cea44db997e05da88
+size 5828608

data/ultradomain/README.md

@@ -0,0 +1,25 @@
+# UltraDomain Data
+
+Datasets: agriculture, biology, cooking, cs, finance, legal, literature, mathematics, mix, music, philosophy, physics
+
+Loaded from [TommyChien/UltraDomain](https://huggingface.co/datasets/TommyChien/UltraDomain) at runtime.
+
+**Graph type:** entity (default) or passage-entity (`--graph_type passage-entity`)
+
+## Pre-built KGs
+
+Pre-built KGs use `openai-small` embedding (requires `OPENAI_API_KEY`):
+
+```bash
+huggingface-cli download qafd/kg --repo-type dataset --include "ultradomain/*" --local-dir ./kg
+```
+
+## Usage
+
+```bash
+# Uses downloaded KG, or builds from scratch if not found
+python benchmarks/run.py --task ultradomain --dataset mix --questions 10
+
+# Use passage-entity graph instead
+python benchmarks/run.py --task ultradomain --dataset mix --graph_type passage-entity --questions 10
+```

docs/index.html

@@ -0,0 +1,452 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+  <meta charset="UTF-8" />
+  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+  <title>QAFD-RAG: Query-Aware Flow Diffusion for Graph-Based RAG</title>
+  <link rel="preconnect" href="https://fonts.googleapis.com" />
+  <link href="https://fonts.googleapis.com/css2?family=DM+Sans:ital,opsz,wght@0,9..40,400;0,9..40,500;0,9..40,600;0,9..40,700&family=JetBrains+Mono:wght@400;500&display=swap" rel="stylesheet" />
+  <style>
+    :root {
+      --navy-900: #0a1628; --navy-800: #0f2140; --navy-700: #1e3a5f;
+      --blue-600: #2563eb; --blue-500: #3b82f6; --blue-400: #60a5fa;
+      --blue-300: #93c5fd; --blue-200: #bfdbfe; --blue-100: #dbeafe; --blue-50: #eff6ff;
+      --slate-900: #0f172a; --slate-800: #1e293b; --slate-700: #334155;
+      --slate-500: #64748b; --slate-400: #94a3b8; --slate-300: #cbd5e1;
+      --slate-200: #e2e8f0; --slate-100: #f1f5f9; --slate-50: #f8fafc;
+      --white: #ffffff; --radius: 10px; --radius-lg: 16px;
+    }
+    * { margin: 0; padding: 0; box-sizing: border-box; }
+    body { font-family: 'DM Sans', -apple-system, sans-serif; color: var(--slate-800); line-height: 1.6; background: var(--white); -webkit-font-smoothing: antialiased; }
+
+    .hero {
+      background: linear-gradient(155deg, #d6e6f7 0%, #dbeafe 40%, #e8f1fc 70%, #eff6ff 100%);
+      color: var(--slate-800); padding: 36px 20px 40px; text-align: center;
+      position: relative; overflow: hidden;
+    }
+    .hero::before {
+      content: ''; position: absolute; top: -50%; left: -50%; width: 200%; height: 200%;
+      background: radial-gradient(ellipse at 30% 50%, rgba(37,99,235,0.04) 0%, transparent 60%),
+                  radial-gradient(ellipse at 70% 30%, rgba(59,130,246,0.03) 0%, transparent 50%);
+      pointer-events: none;
+    }
+    .hero-content { position: relative; z-index: 1; }
+
+    .graph-wrap {
+      margin: 0 auto 16px; width: 480px; height: 280px;
+      animation: logoFadeIn 1s ease-out;
+    }
+    @keyframes logoFadeIn { from { opacity:0; transform: scale(0.9) translateY(10px); } to { opacity:1; transform: scale(1) translateY(0); } }
+    @keyframes waterFlow { 0% { stroke-dashoffset: var(--len); opacity:0.85; } 100% { stroke-dashoffset:0; opacity:0.85; } }
+    @keyframes pipeFadeIn { 0% { opacity:0; } 100% { opacity: var(--pipe-op, 0.7); } }
+    @keyframes sheenFadeIn { 0% { opacity:0; } 100% { opacity: var(--sheen-op, 0.4); } }
+    @keyframes nodeAppear { 0% { transform:scale(0); opacity:0; } 60% { transform:scale(1.2); opacity:1; } 100% { transform:scale(1); opacity:1; } }
+    @keyframes fadeUp { from { opacity:0; transform:translateY(16px); } to { opacity:1; transform:translateY(0); } }
+    @keyframes seedPulse { 0%,100% { r: 14; } 50% { r: 17; } }
+    
+    
+    
+    
+
+    .btn-group { display:flex; gap:10px; justify-content:center; flex-wrap:wrap; animation:fadeUp 0.6s ease-out 0.6s both; }
+    .btn { display:inline-flex; align-items:center; gap:7px; padding:10px 22px; border-radius:50px; font-size:0.9rem; font-weight:600; text-decoration:none; transition:all 0.2s; font-family:inherit; }
+    .btn:hover { transform:translateY(-2px); box-shadow:0 6px 20px rgba(0,0,0,0.2); }
+    .btn-primary { background:var(--blue-600); color:#fff; }
+    .btn-outline { background:rgba(30,58,138,0.06); color:var(--slate-800); border:1.5px solid var(--slate-300); }
+    .btn-outline:hover { border-color:var(--blue-500); background:rgba(37,99,235,0.08); }
+    .btn svg { width:17px; height:17px; flex-shrink:0; }
+
+    .container { max-width:940px; margin:0 auto; padding:0 24px; }
+    section { padding:64px 0; }
+    section.alt { background:var(--slate-50); }
+    section h2 { font-size:1.7rem; font-weight:700; margin-bottom:10px; text-align:center; letter-spacing:-0.3px; }
+    section .section-sub { text-align:center; color:var(--slate-500); font-size:1rem; margin-bottom:32px; max-width:600px; margin-left:auto; margin-right:auto; }
+    .about-text { font-size:1.02rem; line-height:1.85; color:var(--slate-500); text-align:center; max-width:680px; margin:0 auto 36px; }
+    .duo-grid { display:grid; grid-template-columns:1fr 1fr; gap:20px; }
+    .card { background:var(--white); border:1px solid var(--slate-200); border-radius:var(--radius-lg); padding:28px; transition:box-shadow 0.2s; }
+    .card:hover { box-shadow:0 4px 20px rgba(0,0,0,0.06); }
+    .card h3 { color:var(--blue-600); font-size:1.05rem; font-weight:700; margin-bottom:10px; }
+    .card p, .card li { font-size:0.93rem; color:var(--slate-500); line-height:1.7; }
+    .card ol { padding-left:18px; } .card li { margin-bottom:6px; } .card li strong { color:var(--slate-800); }
+    .figure-row { display:flex; gap:18px; justify-content:center; flex-wrap:wrap; margin-top:8px; }
+    .figure-card { text-align:center; background:var(--white); border:1px solid var(--slate-200); border-radius:var(--radius-lg); padding:16px; flex:1; min-width:220px; max-width:280px; transition:box-shadow 0.2s, transform 0.2s; }
+    .figure-card:hover { box-shadow:0 6px 24px rgba(0,0,0,0.07); transform:translateY(-2px); }
+    .figure-card img { width:100%; border-radius:6px; }
+    .figure-card .caption { margin-top:10px; font-size:0.88rem; font-weight:700; color:var(--slate-700); }
+    .figure-card .caption small { font-weight:500; color:var(--slate-400); }
+    .bench-grid { display:grid; grid-template-columns:repeat(auto-fit,minmax(190px,1fr)); gap:14px; }
+    .bench-item { background:var(--white); border:1px solid var(--slate-200); border-radius:var(--radius-lg); padding:22px 18px; text-align:center; transition:box-shadow 0.2s; }
+    .bench-item:hover { box-shadow:0 4px 16px rgba(0,0,0,0.06); }
+    .bench-item .task { font-weight:700; font-size:1rem; color:var(--blue-600); margin-bottom:4px; }
+    .bench-item .dataset { font-size:0.85rem; color:var(--slate-500); }
+    .bench-item .metrics { font-size:0.78rem; color:var(--slate-400); margin-top:6px; }
+    .models-text { text-align:center; color:var(--slate-500); font-size:1rem; line-height:2; }
+    .models-text strong { color:var(--slate-700); }
+    pre { background:var(--slate-900); color:var(--slate-300); border-radius:var(--radius-lg); padding:24px; overflow-x:auto; font-size:0.85rem; line-height:1.65; margin-top:16px; }
+    code { font-family:'JetBrains Mono','SF Mono',monospace; }
+    pre .comment { color:var(--slate-500); } pre .string { color:#7dd3fc; } pre .keyword { color:var(--blue-400); }
+    section h3 { font-size:1.1rem; font-weight:700; margin:28px 0 8px; color:var(--slate-700); }
+    footer { background:var(--slate-900); color:var(--slate-400); padding:36px 20px; text-align:center; font-size:0.85rem; }
+    footer a { color:var(--blue-300); text-decoration:none; } footer a:hover { text-decoration:underline; }
+    footer .sep { margin:0 8px; opacity:0.4; }
+    @media (max-width:768px) {
+      .hero h1 { font-size:2.2rem; }
+      .graph-wrap { width:100%; max-width:480px; height:auto; aspect-ratio:420/280; }
+      .duo-grid { grid-template-columns:1fr; }
+      .figure-card { max-width:100%; }
+    }
+  </style>
+</head>
+<body>
+<header class="hero">
+  <div class="hero-content">
+    <div class="graph-wrap">
+      <svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 420 280" width="100%" height="100%">
+        <defs>
+          <radialGradient id="src" cx="40%" cy="32%"><stop offset="0%" stop-color="#1E40AF"/><stop offset="45%" stop-color="#1E3A8A"/><stop offset="100%" stop-color="#172554"/></radialGradient>
+          <radialGradient id="srcSp" cx="34%" cy="26%"><stop offset="0%" stop-color="#FFF" stop-opacity="0.9"/><stop offset="18%" stop-color="#93C5FD" stop-opacity="0.3"/><stop offset="100%" stop-color="#FFF" stop-opacity="0"/></radialGradient>
+          <radialGradient id="srcGl" cx="50%" cy="50%" r="50%"><stop offset="0%" stop-color="#3B82F6" stop-opacity="0.5"/><stop offset="50%" stop-color="#1E3A8A" stop-opacity="0.15"/><stop offset="100%" stop-color="#172554" stop-opacity="0"/></radialGradient>
+          <radialGradient id="s1" cx="38%" cy="28%"><stop offset="0%" stop-color="#60A5FA"/><stop offset="45%" stop-color="#2563EB"/><stop offset="100%" stop-color="#1E3A8A"/></radialGradient>
+          <radialGradient id="s1sp" cx="32%" cy="24%"><stop offset="0%" stop-color="#FFF" stop-opacity="0.85"/><stop offset="22%" stop-color="#BFDBFE" stop-opacity="0.2"/><stop offset="100%" stop-color="#FFF" stop-opacity="0"/></radialGradient>
+          <radialGradient id="m1" cx="38%" cy="28%"><stop offset="0%" stop-color="#93C5FD"/><stop offset="50%" stop-color="#3B82F6"/><stop offset="100%" stop-color="#2563EB"/></radialGradient>
+          <radialGradient id="m1sp" cx="32%" cy="24%"><stop offset="0%" stop-color="#FFF" stop-opacity="0.7"/><stop offset="20%" stop-color="#FFF" stop-opacity="0.12"/><stop offset="100%" stop-color="#FFF" stop-opacity="0"/></radialGradient>
+          <radialGradient id="w1" cx="38%" cy="28%"><stop offset="0%" stop-color="#BFDBFE"/><stop offset="50%" stop-color="#93C5FD"/><stop offset="100%" stop-color="#60A5FA"/></radialGradient>
+          <radialGradient id="w1sp" cx="32%" cy="24%"><stop offset="0%" stop-color="#FFF" stop-opacity="0.75"/><stop offset="18%" stop-color="#FFF" stop-opacity="0.12"/><stop offset="100%" stop-color="#FFF" stop-opacity="0"/></radialGradient>
+          <radialGradient id="st1" cx="38%" cy="28%"><stop offset="0%" stop-color="#DBEAFE"/><stop offset="50%" stop-color="#BFDBFE"/><stop offset="100%" stop-color="#93C5FD"/></radialGradient>
+          <radialGradient id="lf" cx="38%" cy="28%"><stop offset="0%" stop-color="#EFF6FF"/><stop offset="50%" stop-color="#DBEAFE"/><stop offset="100%" stop-color="#BFDBFE"/></radialGradient>
+          <linearGradient id="pSh" x1="0%" y1="0%" x2="0%" y2="100%"><stop offset="0%" stop-color="#FFF" stop-opacity="0.25"/><stop offset="40%" stop-color="#93C5FD" stop-opacity="0.08"/><stop offset="100%" stop-color="#FFF" stop-opacity="0"/></linearGradient>
+          <filter id="sB" x="-80%" y="-80%" width="260%" height="260%"><feGaussianBlur in="SourceGraphic" stdDeviation="7" result="b1"/><feGaussianBlur in="SourceGraphic" stdDeviation="3" result="b2"/><feMerge><feMergeNode in="b1"/><feMergeNode in="b2"/></feMerge></filter>
+          <filter id="nG" x="-40%" y="-40%" width="180%" height="180%"><feGaussianBlur stdDeviation="2.5" result="b"/><feMerge><feMergeNode in="b"/><feMergeNode in="SourceGraphic"/></feMerge></filter>
+          <filter id="sG" x="-40%" y="-40%" width="180%" height="180%"><feGaussianBlur stdDeviation="4" result="b"/><feMerge><feMergeNode in="b"/><feMergeNode in="SourceGraphic"/></feMerge></filter>
+        </defs>
+
+        <!-- ========= BACKGROUND KNOWLEDGE GRAPH (always visible, gray) ========= -->
+        <g opacity="0.15" stroke="#94A3B8" stroke-width="0.7" fill="none">
+          <line x1="15" y1="60" x2="40" y2="30"/><line x1="40" y1="30" x2="75" y2="22"/>
+          <line x1="15" y1="60" x2="10" y2="110"/><line x1="10" y1="110" x2="25" y2="160"/>
+          <line x1="25" y1="160" x2="12" y2="210"/><line x1="12" y1="210" x2="40" y2="250"/>
+          <line x1="40" y1="30" x2="50" y2="75"/><line x1="50" y1="75" x2="30" y2="120"/>
+          <line x1="30" y1="120" x2="55" y2="155"/><line x1="55" y1="155" x2="25" y2="160"/>
+          <line x1="75" y1="22" x2="95" y2="55"/><line x1="95" y1="55" x2="75" y2="95"/>
+          <line x1="75" y1="95" x2="50" y2="75"/><line x1="95" y1="55" x2="130" y2="40"/>
+          <line x1="75" y1="95" x2="110" y2="120"/><line x1="55" y1="155" x2="85" y2="180"/>
+          <line x1="85" y1="180" x2="65" y2="220"/><line x1="65" y1="220" x2="40" y2="250"/>
+          <line x1="85" y1="180" x2="120" y2="200"/><line x1="120" y1="200" x2="100" y2="240"/>
+          <line x1="100" y1="240" x2="130" y2="265"/><line x1="350" y1="22" x2="385" y2="15"/>
+          <line x1="385" y1="15" x2="410" y2="40"/><line x1="410" y1="40" x2="405" y2="85"/>
+          <line x1="405" y1="85" x2="410" y2="135"/><line x1="410" y1="135" x2="400" y2="180"/>
+          <line x1="400" y1="180" x2="408" y2="230"/><line x1="408" y1="230" x2="390" y2="260"/>
+          <line x1="350" y1="22" x2="370" y2="60"/><line x1="370" y1="60" x2="405" y2="85"/>
+          <line x1="370" y1="60" x2="355" y2="100"/><line x1="355" y1="100" x2="375" y2="140"/>
+          <line x1="375" y1="140" x2="400" y2="180"/><line x1="375" y1="140" x2="350" y2="175"/>
+          <line x1="350" y1="175" x2="370" y2="215"/><line x1="370" y1="215" x2="390" y2="260"/>
+          <line x1="350" y1="175" x2="320" y2="200"/><line x1="320" y1="200" x2="340" y2="240"/>
+          <line x1="340" y1="240" x2="308" y2="260"/><line x1="130" y1="40" x2="170" y2="18"/>
+          <line x1="170" y1="18" x2="220" y2="12"/><line x1="220" y1="12" x2="275" y2="18"/>
+          <line x1="275" y1="18" x2="320" y2="12"/><line x1="320" y1="12" x2="350" y2="22"/>
+          <line x1="130" y1="265" x2="175" y2="270"/><line x1="175" y1="270" x2="220" y2="265"/>
+          <line x1="220" y1="265" x2="260" y2="255"/><line x1="260" y1="255" x2="308" y2="260"/>
+          
+          
+          
+          <line x1="320" y1="200" x2="290" y2="220"/>
+          <line x1="290" y1="220" x2="260" y2="255"/>
+          <line x1="130" y1="40" x2="158" y2="92"/><line x1="110" y1="120" x2="158" y2="148"/>
+          <line x1="120" y1="200" x2="158" y2="175"/><line x1="355" y1="100" x2="325" y2="118"/>
+          <line x1="375" y1="140" x2="335" y2="165"/><line x1="275" y1="18" x2="270" y2="45"/>
+          <line x1="170" y1="18" x2="210" y2="60"/>
+          <line x1="262" y1="92" x2="262" y2="148"/><line x1="210" y1="60" x2="262" y2="92"/>
+          <line x1="158" y1="92" x2="210" y2="60"/><line x1="262" y1="148" x2="210" y2="175"/>
+          <line x1="210" y1="175" x2="158" y2="148"/><line x1="158" y1="92" x2="158" y2="148"/>
+          <line x1="262" y1="92" x2="305" y2="68"/><line x1="305" y1="68" x2="325" y2="118"/>
+          <line x1="325" y1="118" x2="262" y2="148"/><line x1="158" y1="92" x2="112" y2="62"/>
+          <line x1="112" y1="62" x2="92" y2="35"/><line x1="210" y1="60" x2="210" y2="25"/>
+          <line x1="210" y1="175" x2="210" y2="218"/><line x1="210" y1="218" x2="190" y2="250"/>
+          <line x1="325" y1="118" x2="335" y2="165"/><line x1="305" y1="68" x2="322" y2="35"/>
+          <line x1="158" y1="148" x2="118" y2="178"/>
+        </g>
+        <g fill="#B0BEC5" opacity="0.22">
+          <circle cx="15" cy="60" r="2.5"/><circle cx="40" cy="30" r="2.2"/><circle cx="75" cy="22" r="2.2"/>
+          <circle cx="10" cy="110" r="2"/><circle cx="25" cy="160" r="2.2"/><circle cx="12" cy="210" r="2"/>
+          <circle cx="40" cy="250" r="2"/><circle cx="50" cy="75" r="2.2"/><circle cx="30" cy="120" r="2"/>
+          <circle cx="55" cy="155" r="2.2"/><circle cx="95" cy="55" r="2.5"/><circle cx="75" cy="95" r="2.2"/>
+          <circle cx="110" cy="120" r="2"/><circle cx="130" cy="40" r="2.2"/><circle cx="85" cy="180" r="2"/>
+          <circle cx="65" cy="220" r="2"/><circle cx="120" cy="200" r="2.2"/><circle cx="100" cy="240" r="2"/>
+          <circle cx="130" cy="265" r="2"/><circle cx="170" cy="18" r="2.2"/><circle cx="220" cy="12" r="2.2"/>
+          <circle cx="275" cy="18" r="2"/><circle cx="320" cy="12" r="2"/><circle cx="350" cy="22" r="2.2"/>
+          <circle cx="385" cy="15" r="2"/><circle cx="410" cy="40" r="2"/><circle cx="405" cy="85" r="2.2"/>
+          <circle cx="410" cy="135" r="2"/><circle cx="400" cy="180" r="2.2"/><circle cx="408" cy="230" r="2"/>
+          <circle cx="390" cy="260" r="2"/><circle cx="370" cy="60" r="2.5"/><circle cx="355" cy="100" r="2.5"/>
+          <circle cx="375" cy="140" r="2.2"/><circle cx="350" cy="175" r="2.2"/><circle cx="370" cy="215" r="2"/>
+          <circle cx="320" cy="200" r="2.2"/><circle cx="340" cy="240" r="2"/><circle cx="308" cy="260" r="2"/>
+          <circle cx="290" cy="220" r="2"/><circle cx="260" cy="255" r="2"/><circle cx="175" cy="270" r="2"/>
+          <circle cx="220" cy="265" r="2"/><circle cx="270" cy="45" r="2"/><circle cx="210" cy="60" r="2"/>
+          <circle cx="325" cy="118" r="3"/><circle cx="305" cy="68" r="2.8"/>
+          <circle cx="262" cy="148" r="2.8"/><circle cx="262" cy="92" r="2.8"/>
+          <circle cx="210" cy="175" r="2.5"/><circle cx="158" cy="92" r="2.5"/>
+          <circle cx="158" cy="148" r="2.2"/><circle cx="112" cy="62" r="2.2"/>
+          <circle cx="92" cy="35" r="2"/><circle cx="210" cy="25" r="2"/>
+          <circle cx="210" cy="218" r="2.2"/><circle cx="190" cy="250" r="2"/>
+          <circle cx="335" cy="165" r="2"/><circle cx="118" cy="178" r="2"/>
+          <circle cx="322" cy="35" r="2"/>
+        </g>
+
+        <!--
+          PHASE 1: BUILD — water shoots from source, pipe+node appear where it lands
+          PHASE 2: LOOP — after build, continuous flow from source outward
+          
+          Each element uses <animate> with begin= for precise timing.
+          Pipe: starts invisible, fades in when water arrives
+          Node: starts invisible, pops when pipe reaches it
+          Water: draws from source outward using stroke-dashoffset
+        -->
+
+        <!-- ======= SOURCE (0s) ======= -->
+        <circle cx="325" cy="118" r="20" fill="url(#srcGl)" filter="url(#sB)" opacity="0"><animate attributeName="opacity" to="1" dur="0.4s" begin="0.2s" fill="freeze"/></circle>
+        <g filter="url(#sG)" opacity="0"><animate attributeName="opacity" to="1" dur="0.4s" begin="0.2s" fill="freeze"/>
+          <circle cx="325" cy="118" r="14" fill="url(#src)"/><circle cx="325" cy="118" r="14" fill="url(#srcSp)"/>
+          <ellipse cx="320" cy="111" rx="4.5" ry="3" fill="#FFF" opacity="0.5"/>
+        </g>
+
+        <!-- ======= HOP 1: Source→G(305,68) and Source→D(262,148) ======= -->
+        <!-- Water Source→G -->
+        <path d="M325,118 C318,100 312,85 305,68" fill="none" stroke="#7DD3FC" stroke-width="5.5" stroke-linecap="round" stroke-dasharray="55" stroke-dashoffset="55" opacity="0"><animate attributeName="stroke-dashoffset" to="0" dur="0.6s" begin="0.5s" fill="freeze"/><animate attributeName="opacity" to="0.85" dur="0.1s" begin="0.5s" fill="freeze"/></path>
+        <!-- Pipe Source→G fades in -->
+        <path d="M305,68 C312,85 318,100 325,118" fill="none" stroke="#172554" stroke-width="10" stroke-linecap="round" opacity="0"><animate attributeName="opacity" to="0.9" dur="0.3s" begin="0.7s" fill="freeze"/></path>
+        <path d="M305,68 C312,85 318,100 325,118" fill="none" stroke="url(#pSh)" stroke-width="4.5" stroke-linecap="round" opacity="0"><animate attributeName="opacity" to="0.55" dur="0.3s" begin="0.75s" fill="freeze"/></path>
+        <!-- G node pops -->
+        <g filter="url(#nG)" opacity="0"><animate attributeName="opacity" to="1" dur="0.3s" begin="0.85s" fill="freeze"/>
+          <circle cx="305" cy="68" r="9" fill="url(#s1)"/><circle cx="305" cy="68" r="9" fill="url(#s1sp)"/>
+        </g>
+
+        <!-- Water Source→D -->
+        <path d="M325,118 C305,128 282,138 262,148" fill="none" stroke="#7DD3FC" stroke-width="5" stroke-linecap="round" stroke-dasharray="75" stroke-dashoffset="75" opacity="0"><animate attributeName="stroke-dashoffset" to="0" dur="0.7s" begin="0.55s" fill="freeze"/><animate attributeName="opacity" to="0.85" dur="0.1s" begin="0.55s" fill="freeze"/></path>
+        <path d="M262,148 C282,138 305,128 325,118" fill="none" stroke="#172554" stroke-width="9.5" stroke-linecap="round" opacity="0"><animate attributeName="opacity" to="0.88" dur="0.3s" begin="0.8s" fill="freeze"/></path>
+        <path d="M262,148 C282,138 305,128 325,118" fill="none" stroke="url(#pSh)" stroke-width="4.2" stroke-linecap="round" opacity="0"><animate attributeName="opacity" to="0.52" dur="0.3s" begin="0.85s" fill="freeze"/></path>
+        <g filter="url(#nG)" opacity="0"><animate attributeName="opacity" to="1" dur="0.3s" begin="0.95s" fill="freeze"/>
+          <circle cx="262" cy="148" r="10" fill="url(#s1)"/><circle cx="262" cy="148" r="10" fill="url(#s1sp)"/>
+        </g>
+
+        <!-- ======= HOP 2: G→C and D→C (converging) ======= -->
+        <path d="M305,68 C290,72 275,82 262,92" fill="none" stroke="#7DD3FC" stroke-width="5" stroke-linecap="round" stroke-dasharray="55" stroke-dashoffset="55" opacity="0"><animate attributeName="stroke-dashoffset" to="0" dur="0.6s" begin="1.0s" fill="freeze"/><animate attributeName="opacity" to="0.85" dur="0.1s" begin="1.0s" fill="freeze"/></path>
+        <path d="M262,92 C275,82 290,72 305,68" fill="none" stroke="#172554" stroke-width="9.5" stroke-linecap="round" opacity="0"><animate attributeName="opacity" to="0.86" dur="0.3s" begin="1.2s" fill="freeze"/></path>
+        <path d="M262,92 C275,82 290,72 305,68" fill="none" stroke="url(#pSh)" stroke-width="4.2" stroke-linecap="round" opacity="0"><animate attributeName="opacity" to="0.5" dur="0.3s" begin="1.25s" fill="freeze"/></path>
+
+        <path d="M262,148 C262,130 262,110 262,92" fill="none" stroke="#7DD3FC" stroke-width="5" stroke-linecap="round" stroke-dasharray="56" stroke-dashoffset="56" opacity="0"><animate attributeName="stroke-dashoffset" to="0" dur="0.55s" begin="1.05s" fill="freeze"/><animate attributeName="opacity" to="0.85" dur="0.1s" begin="1.05s" fill="freeze"/></path>
+        <path d="M262,92 C262,110 262,130 262,148" fill="none" stroke="#172554" stroke-width="9.5" stroke-linecap="round" opacity="0"><animate attributeName="opacity" to="0.84" dur="0.3s" begin="1.25s" fill="freeze"/></path>
+        <path d="M262,92 C262,110 262,130 262,148" fill="none" stroke="url(#pSh)" stroke-width="4.2" stroke-linecap="round" opacity="0"><animate attributeName="opacity" to="0.5" dur="0.3s" begin="1.3s" fill="freeze"/></path>
+
+        <g filter="url(#nG)" opacity="0"><animate attributeName="opacity" to="1" dur="0.3s" begin="1.35s" fill="freeze"/>
+          <circle cx="262" cy="92" r="10" fill="url(#s1)"/><circle cx="262" cy="92" r="10" fill="url(#s1sp)"/>
+        </g>
+
+        <!-- ======= HOP 3: C→B→A (6-ring top) and D→E→F (6-ring bottom) ======= -->
+        <!-- C→B -->
+        <path d="M262,92 C245,78 228,68 210,60" fill="none" stroke="#93C5FD" stroke-width="4.2" stroke-linecap="round" stroke-dasharray="62" stroke-dashoffset="62" opacity="0"><animate attributeName="stroke-dashoffset" to="0" dur="0.6s" begin="1.5s" fill="freeze"/><animate attributeName="opacity" to="0.85" dur="0.1s" begin="1.5s" fill="freeze"/></path>
+        <path d="M210,60 C228,68 245,78 262,92" fill="none" stroke="#1E3A8A" stroke-width="8.5" stroke-linecap="round" opacity="0"><animate attributeName="opacity" to="0.72" dur="0.3s" begin="1.7s" fill="freeze"/></path>
+        <path d="M210,60 C228,68 245,78 262,92" fill="none" stroke="url(#pSh)" stroke-width="3.8" stroke-linecap="round" opacity="0"><animate attributeName="opacity" to="0.45" dur="0.3s" begin="1.75s" fill="freeze"/></path>
+        <g filter="url(#nG)" opacity="0"><animate attributeName="opacity" to="1" dur="0.3s" begin="1.85s" fill="freeze"/>
+          <circle cx="210" cy="60" r="9" fill="url(#s1)"/><circle cx="210" cy="60" r="9" fill="url(#s1sp)"/>
+        </g>
+
+        <!-- B→A -->
+        <path d="M210,60 C192,68 175,80 158,92" fill="none" stroke="#93C5FD" stroke-width="3.8" stroke-linecap="round" stroke-dasharray="65" stroke-dashoffset="65" opacity="0"><animate attributeName="stroke-dashoffset" to="0" dur="0.6s" begin="1.9s" fill="freeze"/><animate attributeName="opacity" to="0.85" dur="0.1s" begin="1.9s" fill="freeze"/></path>
+        <path d="M158,92 C175,80 192,68 210,60" fill="none" stroke="#1E3A8A" stroke-width="8" stroke-linecap="round" opacity="0"><animate attributeName="opacity" to="0.7" dur="0.3s" begin="2.1s" fill="freeze"/></path>
+        <path d="M158,92 C175,80 192,68 210,60" fill="none" stroke="url(#pSh)" stroke-width="3.5" stroke-linecap="round" opacity="0"><animate attributeName="opacity" to="0.45" dur="0.3s" begin="2.15s" fill="freeze"/></path>
+        <g filter="url(#nG)" opacity="0"><animate attributeName="opacity" to="1" dur="0.3s" begin="2.25s" fill="freeze"/>
+          <circle cx="158" cy="92" r="9" fill="url(#s1)"/><circle cx="158" cy="92" r="9" fill="url(#s1sp)"/>
+        </g>
+
+        <!-- D→E -->
+        <path d="M262,148 C245,158 228,168 210,175" fill="none" stroke="#93C5FD" stroke-width="3.8" stroke-linecap="round" stroke-dasharray="62" stroke-dashoffset="62" opacity="0"><animate attributeName="stroke-dashoffset" to="0" dur="0.6s" begin="1.5s" fill="freeze"/><animate attributeName="opacity" to="0.85" dur="0.1s" begin="1.5s" fill="freeze"/></path>
+        <path d="M210,175 C228,168 245,158 262,148" fill="none" stroke="#1E3A8A" stroke-width="7.5" stroke-linecap="round" opacity="0"><animate attributeName="opacity" to="0.68" dur="0.3s" begin="1.7s" fill="freeze"/></path>
+        <path d="M210,175 C228,168 245,158 262,148" fill="none" stroke="url(#pSh)" stroke-width="3.2" stroke-linecap="round" opacity="0"><animate attributeName="opacity" to="0.42" dur="0.3s" begin="1.75s" fill="freeze"/></path>
+        <g filter="url(#nG)" opacity="0"><animate attributeName="opacity" to="1" dur="0.3s" begin="1.85s" fill="freeze"/>
+          <circle cx="210" cy="175" r="8" fill="url(#m1)"/><circle cx="210" cy="175" r="8" fill="url(#m1sp)"/>
+        </g>
+
+        <!-- E→F -->
+        <path d="M210,175 C192,168 175,158 158,148" fill="none" stroke="#BFDBFE" stroke-width="2.8" stroke-linecap="round" stroke-dasharray="65" stroke-dashoffset="65" opacity="0"><animate attributeName="stroke-dashoffset" to="0" dur="0.6s" begin="1.9s" fill="freeze"/><animate attributeName="opacity" to="0.85" dur="0.1s" begin="1.9s" fill="freeze"/></path>
+        <path d="M158,148 C175,158 192,168 210,175" fill="none" stroke="#1D4ED8" stroke-width="5.5" stroke-linecap="round" opacity="0"><animate attributeName="opacity" to="0.5" dur="0.3s" begin="2.1s" fill="freeze"/></path>
+        <g filter="url(#nG)" opacity="0"><animate attributeName="opacity" to="1" dur="0.3s" begin="2.25s" fill="freeze"/>
+          <circle cx="158" cy="148" r="7" fill="url(#m1)"/><circle cx="158" cy="148" r="7" fill="url(#m1sp)"/>
+        </g>
+
+        <!-- F→A ring close -->
+        <path d="M158,148 C158,130 158,112 158,92" fill="none" stroke="#BFDBFE" stroke-width="2.2" stroke-linecap="round" stroke-dasharray="56" stroke-dashoffset="56" opacity="0"><animate attributeName="stroke-dashoffset" to="0" dur="0.5s" begin="2.3s" fill="freeze"/><animate attributeName="opacity" to="0.85" dur="0.1s" begin="2.3s" fill="freeze"/></path>
+        <path d="M158,92 C158,112 158,130 158,148" fill="none" stroke="#1D4ED8" stroke-width="5" stroke-linecap="round" opacity="0"><animate attributeName="opacity" to="0.48" dur="0.3s" begin="2.5s" fill="freeze"/></path>
+
+        <!-- ======= HOP 4: BRANCHES ======= -->
+        <!-- A→M1 -->
+        <path d="M158,92 C142,80 128,70 112,62" fill="none" stroke="#93C5FD" stroke-width="3.2" stroke-linecap="round" stroke-dasharray="52" stroke-dashoffset="52" opacity="0"><animate attributeName="stroke-dashoffset" to="0" dur="0.5s" begin="2.4s" fill="freeze"/><animate attributeName="opacity" to="0.85" dur="0.1s" begin="2.4s" fill="freeze"/></path>
+        <path d="M112,62 C128,70 142,80 158,92" fill="none" stroke="#1E3A8A" stroke-width="6.5" stroke-linecap="round" opacity="0"><animate attributeName="opacity" to="0.58" dur="0.3s" begin="2.55s" fill="freeze"/></path>
+        <path d="M112,62 C128,70 142,80 158,92" fill="none" stroke="url(#pSh)" stroke-width="2.8" stroke-linecap="round" opacity="0"><animate attributeName="opacity" to="0.38" dur="0.3s" begin="2.6s" fill="freeze"/></path>
+        <g filter="url(#nG)" opacity="0"><animate attributeName="opacity" to="1" dur="0.3s" begin="2.7s" fill="freeze"/>
+          <circle cx="112" cy="62" r="7.5" fill="url(#m1)"/><circle cx="112" cy="62" r="7.5" fill="url(#m1sp)"/>
+        </g>
+        <!-- M1→leaf -->
+        <path d="M112,62 C104,52 98,42 92,35" fill="none" stroke="#DBEAFE" stroke-width="1.5" stroke-linecap="round" stroke-dasharray="35" stroke-dashoffset="35" opacity="0"><animate attributeName="stroke-dashoffset" to="0" dur="0.4s" begin="2.8s" fill="freeze"/><animate attributeName="opacity" to="0.85" dur="0.1s" begin="2.8s" fill="freeze"/></path>
+        <path d="M92,35 C98,42 104,52 112,62" fill="none" stroke="#3B82F6" stroke-width="2.8" stroke-linecap="round" opacity="0"><animate attributeName="opacity" to="0.25" dur="0.3s" begin="2.95s" fill="freeze"/></path>
+        <circle cx="92" cy="35" r="3" fill="url(#lf)" opacity="0"><animate attributeName="opacity" to="1" dur="0.3s" begin="3.05s" fill="freeze"/></circle>
+
+        <!-- B→M2 up -->
+        <path d="M210,60 C210,48 210,35 210,25" fill="none" stroke="#BFDBFE" stroke-width="2" stroke-linecap="round" stroke-dasharray="35" stroke-dashoffset="35" opacity="0"><animate attributeName="stroke-dashoffset" to="0" dur="0.4s" begin="2.1s" fill="freeze"/><animate attributeName="opacity" to="0.85" dur="0.1s" begin="2.1s" fill="freeze"/></path>
+        <path d="M210,25 C210,35 210,48 210,60" fill="none" stroke="#2563EB" stroke-width="4" stroke-linecap="round" opacity="0"><animate attributeName="opacity" to="0.35" dur="0.3s" begin="2.25s" fill="freeze"/></path>
+        <circle cx="210" cy="25" r="5" fill="url(#w1)" opacity="0"><animate attributeName="opacity" to="1" dur="0.3s" begin="2.35s" fill="freeze"/></circle>
+
+        <!-- E→M3 down -->
+        <path d="M210,175 C210,190 210,205 210,218" fill="none" stroke="#BFDBFE" stroke-width="2" stroke-linecap="round" stroke-dasharray="43" stroke-dashoffset="43" opacity="0"><animate attributeName="stroke-dashoffset" to="0" dur="0.4s" begin="2.1s" fill="freeze"/><animate attributeName="opacity" to="0.85" dur="0.1s" begin="2.1s" fill="freeze"/></path>
+        <path d="M210,218 C210,205 210,190 210,175" fill="none" stroke="#2563EB" stroke-width="4" stroke-linecap="round" opacity="0"><animate attributeName="opacity" to="0.38" dur="0.3s" begin="2.25s" fill="freeze"/></path>
+        <circle cx="210" cy="218" r="5.5" fill="url(#w1)" opacity="0"><animate attributeName="opacity" to="1" dur="0.3s" begin="2.35s" fill="freeze"/></circle>
+        <!-- M3→leaf -->
+        <path d="M210,218 C202,232 196,242 190,250" fill="none" stroke="#DBEAFE" stroke-width="1.3" stroke-linecap="round" stroke-dasharray="38" stroke-dashoffset="38" opacity="0"><animate attributeName="stroke-dashoffset" to="0" dur="0.4s" begin="2.5s" fill="freeze"/><animate attributeName="opacity" to="0.85" dur="0.1s" begin="2.5s" fill="freeze"/></path>
+        <path d="M190,250 C196,242 202,232 210,218" fill="none" stroke="#93C5FD" stroke-width="1.8" stroke-linecap="round" opacity="0"><animate attributeName="opacity" to="0.16" dur="0.3s" begin="2.65s" fill="freeze"/></path>
+        <circle cx="190" cy="250" r="2.5" fill="url(#lf)" opacity="0"><animate attributeName="opacity" to="1" dur="0.3s" begin="2.75s" fill="freeze"/></circle>
+
+        <!-- Source→ext -->
+        <path d="M325,118 C328,132 332,150 335,165" fill="none" stroke="#BFDBFE" stroke-width="1.8" stroke-linecap="round" stroke-dasharray="50" stroke-dashoffset="50" opacity="0"><animate attributeName="stroke-dashoffset" to="0" dur="0.5s" begin="2.0s" fill="freeze"/><animate attributeName="opacity" to="0.85" dur="0.1s" begin="2.0s" fill="freeze"/></path>
+        <path d="M335,165 C332,150 328,132 325,118" fill="none" stroke="#2563EB" stroke-width="3.2" stroke-linecap="round" opacity="0"><animate attributeName="opacity" to="0.28" dur="0.3s" begin="2.15s" fill="freeze"/></path>
+        <circle cx="335" cy="165" r="4" fill="url(#w1)" opacity="0"><animate attributeName="opacity" to="1" dur="0.3s" begin="2.3s" fill="freeze"/></circle>
+
+        <!-- STARVED -->
+        <path d="M158,148 C142,158 130,168 118,178" fill="none" stroke="#DBEAFE" stroke-width="1" stroke-linecap="round" stroke-dasharray="45" stroke-dashoffset="45" opacity="0"><animate attributeName="stroke-dashoffset" to="0" dur="0.7s" begin="3.0s" fill="freeze"/><animate attributeName="opacity" to="0.85" dur="0.1s" begin="3.0s" fill="freeze"/></path>
+        <path d="M118,178 C130,168 142,158 158,148" fill="none" stroke="#60A5FA" stroke-width="2" stroke-linecap="round" opacity="0"><animate attributeName="opacity" to="0.13" dur="0.3s" begin="3.2s" fill="freeze"/></path>
+        <circle cx="118" cy="178" r="3.5" fill="url(#st1)" opacity="0"><animate attributeName="opacity" to="1" dur="0.3s" begin="3.3s" fill="freeze"/></circle>
+
+        <path d="M305,68 C310,55 316,42 322,35" fill="none" stroke="#DBEAFE" stroke-width="1" stroke-linecap="round" stroke-dasharray="40" stroke-dashoffset="40" opacity="0"><animate attributeName="stroke-dashoffset" to="0" dur="0.7s" begin="3.1s" fill="freeze"/><animate attributeName="opacity" to="0.85" dur="0.1s" begin="3.1s" fill="freeze"/></path>
+        <path d="M322,35 C316,42 310,55 305,68" fill="none" stroke="#60A5FA" stroke-width="2.5" stroke-linecap="round" opacity="0"><animate attributeName="opacity" to="0.18" dur="0.3s" begin="3.3s" fill="freeze"/></path>
+        <circle cx="322" cy="35" r="3.5" fill="url(#st1)" opacity="0"><animate attributeName="opacity" to="1" dur="0.3s" begin="3.4s" fill="freeze"/></circle>
+
+        <!-- ======= PHASE 2: CONTINUOUS WATER LOOP from source outward (4s+) ======= -->
+        <g fill="none" stroke-linecap="round" opacity="0">
+          <animate attributeName="opacity" to="1" dur="0.5s" begin="4s" fill="freeze"/>
+          <!-- 5-ring -->
+          <path d="M325,118 C318,100 312,85 305,68" stroke="#7DD3FC" stroke-width="5.5" stroke-dasharray="12 8"><animate attributeName="stroke-dashoffset" values="0;-40" dur="1s" begin="4s" repeatCount="indefinite"/></path>
+          <path d="M325,118 C305,128 282,138 262,148" stroke="#7DD3FC" stroke-width="5" stroke-dasharray="12 8"><animate attributeName="stroke-dashoffset" values="0;-40" dur="1.2s" begin="4s" repeatCount="indefinite"/></path>
+          <path d="M305,68 C290,72 275,82 262,92" stroke="#7DD3FC" stroke-width="5" stroke-dasharray="12 8"><animate attributeName="stroke-dashoffset" values="0;-40" dur="1.1s" begin="4s" repeatCount="indefinite"/></path>
+          <path d="M262,148 C262,130 262,110 262,92" stroke="#7DD3FC" stroke-width="5" stroke-dasharray="12 8"><animate attributeName="stroke-dashoffset" values="0;-40" dur="1s" begin="4s" repeatCount="indefinite"/></path>
+          <!-- 6-ring top (outward from C) -->
+          <path d="M262,92 C245,78 228,68 210,60" stroke="#93C5FD" stroke-width="4" stroke-dasharray="10 8"><animate attributeName="stroke-dashoffset" values="0;-36" dur="1.4s" begin="4s" repeatCount="indefinite"/></path>
+          <path d="M210,60 C192,68 175,80 158,92" stroke="#93C5FD" stroke-width="3.8" stroke-dasharray="10 8"><animate attributeName="stroke-dashoffset" values="0;-36" dur="1.5s" begin="4s" repeatCount="indefinite"/></path>
+          <!-- 6-ring bottom (outward from D) -->
+          <path d="M262,148 C245,158 228,168 210,175" stroke="#93C5FD" stroke-width="3.5" stroke-dasharray="10 8"><animate attributeName="stroke-dashoffset" values="0;-36" dur="1.3s" begin="4s" repeatCount="indefinite"/></path>
+          <path d="M210,175 C192,168 175,158 158,148" stroke="#BFDBFE" stroke-width="2.5" stroke-dasharray="8 8"><animate attributeName="stroke-dashoffset" values="0;-32" dur="1.6s" begin="4s" repeatCount="indefinite"/></path>
+          <path d="M158,148 C158,130 158,112 158,92" stroke="#BFDBFE" stroke-width="2" stroke-dasharray="8 8"><animate attributeName="stroke-dashoffset" values="0;-32" dur="1.8s" begin="4s" repeatCount="indefinite"/></path>
+          <!-- Branches (outward) -->
+          <path d="M158,92 C142,80 128,70 112,62" stroke="#93C5FD" stroke-width="3" stroke-dasharray="8 7"><animate attributeName="stroke-dashoffset" values="0;-30" dur="1.4s" begin="4s" repeatCount="indefinite"/></path>
+          <path d="M210,60 C210,48 210,35 210,25" stroke="#BFDBFE" stroke-width="2" stroke-dasharray="6 7"><animate attributeName="stroke-dashoffset" values="0;-26" dur="1.6s" begin="4s" repeatCount="indefinite"/></path>
+          <path d="M210,175 C210,190 210,205 210,218" stroke="#BFDBFE" stroke-width="2" stroke-dasharray="6 7"><animate attributeName="stroke-dashoffset" values="0;-26" dur="1.5s" begin="4s" repeatCount="indefinite"/></path>
+          <path d="M325,118 C328,132 332,150 335,165" stroke="#BFDBFE" stroke-width="1.5" stroke-dasharray="5 6"><animate attributeName="stroke-dashoffset" values="0;-22" dur="1.7s" begin="4s" repeatCount="indefinite"/></path>
+        </g>
+      </svg>
+    </div>
+    <h1 style="font-size:2.8rem;font-weight:700;letter-spacing:-1px;margin-bottom:24px;color:var(--navy-900);animation:fadeUp 0.6s ease-out 0.3s both;">QAFD-RAG</h1>
+    <div class="btn-group">
+      <a class="btn btn-primary" href="https://openreview.net/pdf?id=n28wnc2QTc"><svg viewBox="0 0 24 24" fill="currentColor"><path d="M14 2H6a2 2 0 0 0-2 2v16a2 2 0 0 0 2 2h12a2 2 0 0 0 2-2V8l-6-6zm-1 2l5 5h-5V4zM6 20V4h5v7h7v9H6z"/></svg> Paper</a>
+      <span class="btn btn-outline" style="cursor:default;opacity:0.7;"><svg viewBox="0 0 24 24" fill="currentColor"><path d="M12 0C5.37 0 0 5.37 0 12c0 5.3 3.438 9.8 8.205 11.385.6.113.82-.258.82-.577 0-.285-.01-1.04-.015-2.04-3.338.724-4.042-1.61-4.042-1.61-.546-1.385-1.335-1.755-1.335-1.755-1.087-.744.084-.729.084-.729 1.205.084 1.838 1.236 1.838 1.236 1.07 1.835 2.809 1.305 3.495.998.108-.776.417-1.305.76-1.605-2.665-.3-5.466-1.332-5.466-5.93 0-1.31.465-2.38 1.235-3.22-.135-.303-.54-1.523.105-3.176 0 0 1.005-.322 3.3 1.23.96-.267 1.98-.399 3-.405 1.02.006 2.04.138 3 .405 2.28-1.552 3.285-1.23 3.285-1.23.645 1.653.24 2.873.12 3.176.765.84 1.23 1.91 1.23 3.22 0 4.61-2.805 5.625-5.475 5.92.42.36.81 1.096.81 2.22 0 1.605-.015 2.896-.015 3.286 0 .315.21.69.825.57C20.565 21.795 24 17.295 24 12 24 5.37 18.63 0 12 0z"/></svg> Code (Coming Soon)</span>
+      <a class="btn btn-outline" href="https://huggingface.co/datasets/qafd/kg"><svg xmlns="http://www.w3.org/2000/svg" width="17" height="17" viewBox="0 0 95 88" fill="none"><path fill="#FFD21E" d="M47.21 76.5a34.75 34.75 0 1 0 0-69.5 34.75 34.75 0 0 0 0 69.5Z"/><path fill="#FF9D0B" d="M81.96 41.75a34.75 34.75 0 1 0-69.5 0 34.75 34.75 0 0 0 69.5 0Zm-73.5 0a38.75 38.75 0 1 1 77.5 0 38.75 38.75 0 0 1-77.5 0Z"/><path fill="#3A3B45" d="M58.5 32.3c1.28.44 1.78 3.06 3.07 2.38a5 5 0 1 0-6.76-2.07c.61 1.15 2.55-.72 3.7-.32ZM34.95 32.3c-1.28.44-1.79 3.06-3.07 2.38a5 5 0 1 1 6.76-2.07c-.61 1.15-2.56-.72-3.7-.32Z"/><path fill="#FF323D" d="M46.96 56.29c9.83 0 13-8.76 13-13.26 0-2.34-1.57-1.6-4.09-.36-2.33 1.15-5.46 2.74-8.9 2.74-7.19 0-13-6.88-13-2.38s3.16 13.26 13 13.26Z"/><path fill="#3A3B45" fill-rule="evenodd" d="M39.43 54a8.7 8.7 0 0 1 5.3-4.49c.4-.12.81.57 1.24 1.28.4.68.82 1.37 1.24 1.37.45 0 .9-.68 1.33-1.35.45-.7.89-1.38 1.32-1.25a8.61 8.61 0 0 1 5 4.17c3.73-2.94 5.1-7.74 5.1-10.7 0-2.34-1.57-1.6-4.09-.36l-.14.07c-2.31 1.15-5.39 2.67-8.77 2.67s-6.45-1.52-8.77-2.67c-2.6-1.29-4.23-2.1-4.23.29 0 3.05 1.46 8.06 5.47 10.97Z" clip-rule="evenodd"/><path fill="#FF9D0B" d="M70.71 37a3.25 3.25 0 1 0 0-6.5 3.25 3.25 0 0 0 0 6.5ZM24.21 37a3.25 3.25 0 1 0 0-6.5 3.25 3.25 0 0 0 0 6.5ZM17.52 48c-1.62 0-3.06.66-4.07 1.87a5.97 5.97 0 0 0-1.33 3.76 7.1 7.1 0 0 0-1.94-.3c-1.55 0-2.95.59-3.94 1.66a5.8 5.8 0 0 0-.8 7 5.3 5.3 0 0 0-1.79 2.82c-.24.9-.48 2.8.8 4.74a5.22 5.22 0 0 0-.37 5.02c1.02 2.32 3.57 4.14 8.52 6.1 3.07 1.22 5.89 2 5.91 2.01a44.33 44.33 0 0 0 10.93 1.6c5.86 0 10.05-1.8 12.46-5.34 3.88-5.69 3.33-10.9-1.7-15.92-2.77-2.78-4.62-6.87-5-7.77-.78-2.66-2.84-5.62-6.25-5.62a5.7 5.7 0 0 0-4.6 2.46c-1-1.26-1.98-2.25-2.86-2.82A7.4 7.4 0 0 0 17.52 48Zm0 4c.51 0 1.14.22 1.82.65 2.14 1.36 6.25 8.43 7.76 11.18.5.92 1.37 1.31 2.14 1.31 1.55 0 2.75-1.53.15-3.48-3.92-2.93-2.55-7.72-.68-8.01.08-.02.17-.02.24-.02 1.7 0 2.45 2.93 2.45 2.93s2.2 5.52 5.98 9.3c3.77 3.77 3.97 6.8 1.22 10.83-1.88 2.75-5.47 3.58-9.16 3.58-3.81 0-7.73-.9-9.92-1.46-.11-.03-13.45-3.8-11.76-7 .28-.54.75-.76 1.34-.76 2.38 0 6.7 3.54 8.57 3.54.41 0 .7-.17.83-.6.79-2.85-12.06-4.05-10.98-8.17.2-.73.71-1.02 1.44-1.02 3.14 0 10.2 5.53 11.68 5.53.11 0 .2-.03.24-.1.74-1.2.33-2.04-4.9-5.2-5.21-3.16-8.88-5.06-6.8-7.33.24-.26.58-.38 1-.38 3.17 0 10.66 6.82 10.66 6.82s2.02 2.1 3.25 2.1c.28 0 .52-.1.68-.38.86-1.46-8.06-8.22-8.56-11.01-.34-1.9.24-2.85 1.31-2.85Z"/><path fill="#FFD21E" d="M38.6 76.69c2.75-4.04 2.55-7.07-1.22-10.84-3.78-3.77-5.98-9.3-5.98-9.3s-.82-3.2-2.69-2.9c-1.87.3-3.24 5.08.68 8.01 3.91 2.93-.78 4.92-2.29 2.17-1.5-2.75-5.62-9.82-7.76-11.18-2.13-1.35-3.63-.6-3.13 2.2.5 2.79 9.43 9.55 8.56 11-.87 1.47-3.93-1.71-3.93-1.71s-9.57-8.71-11.66-6.44c-2.08 2.27 1.59 4.17 6.8 7.33 5.23 3.16 5.64 4 4.9 5.2-.75 1.2-12.28-8.53-13.36-4.4-1.08 4.11 11.77 5.3 10.98 8.15-.8 2.85-9.06-5.38-10.74-2.18-1.7 3.21 11.65 6.98 11.76 7.01 4.3 1.12 15.25 3.49 19.08-2.12Z"/><path fill="#FF9D0B" d="M77.4 48c1.62 0 3.07.66 4.07 1.87a5.97 5.97 0 0 1 1.33 3.76 7.1 7.1 0 0 1 1.95-.3c1.55 0 2.95.59 3.94 1.66a5.8 5.8 0 0 1 .8 7 5.3 5.3 0 0 1 1.78 2.82c.24.9.48 2.8-.8 4.74a5.22 5.22 0 0 1 .37 5.02c-1.02 2.32-3.57 4.14-8.51 6.1-3.08 1.22-5.9 2-5.92 2.01a44.33 44.33 0 0 1-10.93 1.6c-5.86 0-10.05-1.8-12.46-5.34-3.88-5.69-3.33-10.9 1.7-15.92 2.78-2.78 4.63-6.87 5.01-7.77.78-2.66 2.83-5.62 6.24-5.62a5.7 5.7 0 0 1 4.6 2.46c1-1.26 1.98-2.25 2.87-2.82A7.4 7.4 0 0 1 77.4 48Zm0 4c-.51 0-1.13.22-1.82.65-2.13 1.36-6.25 8.43-7.76 11.18a2.43 2.43 0 0 1-2.14 1.31c-1.54 0-2.75-1.53-.14-3.48 3.91-2.93 2.54-7.72.67-8.01a1.54 1.54 0 0 0-.24-.02c-1.7 0-2.45 2.93-2.45 2.93s-2.2 5.52-5.97 9.3c-3.78 3.77-3.98 6.8-1.22 10.83 1.87 2.75 5.47 3.58 9.15 3.58 3.82 0 7.73-.9 9.93-1.46.1-.03 13.45-3.8 11.76-7-.29-.54-.75-.76-1.34-.76-2.38 0-6.71 3.54-8.57 3.54-.42 0-.71-.17-.83-.6-.8-2.85 12.05-4.05 10.97-8.17-.19-.73-.7-1.02-1.44-1.02-3.14 0-10.2 5.53-11.68 5.53-.1 0-.19-.03-.23-.1-.74-1.2-.34-2.04 4.88-5.2 5.23-3.16 8.9-5.06 6.8-7.33-.23-.26-.57-.38-.98-.38-3.18 0-10.67 6.82-10.67 6.82s-2.02 2.1-3.24 2.1a.74.74 0 0 1-.68-.38c-.87-1.46 8.05-8.22 8.55-11.01.34-1.9-.24-2.85-1.31-2.85Z"/><path fill="#FFD21E" d="M56.33 76.69c-2.75-4.04-2.56-7.07 1.22-10.84 3.77-3.77 5.97-9.3 5.97-9.3s.82-3.2 2.7-2.9c1.86.3 3.23 5.08-.68 8.01-3.92 2.93.78 4.92 2.28 2.17 1.51-2.75 5.63-9.82 7.76-11.18 2.13-1.35 3.64-.6 3.13 2.2-.5 2.79-9.42 9.55-8.55 11 .86 1.47 3.92-1.71 3.92-1.71s9.58-8.71 11.66-6.44c2.08 2.27-1.58 4.17-6.8 7.33-5.23 3.16-5.63 4-4.9 5.2.75 1.2 12.28-8.53 13.36-4.4 1.08 4.11-11.76 5.3-10.97 8.15.8 2.85 9.05-5.38 10.74-2.18 1.69 3.21-11.65 6.98-11.76 7.01-4.31 1.12-15.26 3.49-19.08-2.12Z"/></svg> Data</a>
+    </div>
+  </div>
+</header>
+
+<section>
+  <div class="container">
+    <h2 style="font-size:1.55rem;font-weight:700;color:var(--slate-800);margin-bottom:16px;">Query-Aware Flow Diffusion for Graph-Based<br/>Retrieval-Augmented Generation</h2>
+    <p class="about-text"><strong style="color:var(--blue-600);">QAFD-RAG</strong> is a graph-based retrieval-augmented generation framework that uses <strong>query-aware flow diffusion</strong> to retrieve contextually relevant subgraphs from a knowledge graph with <strong>statistical retrieval guarantees</strong>.</p>
+    <div class="duo-grid">
+      <div class="card"><h3>The Problem</h3><p>Existing graph-based RAG methods rely on <strong>static retrieval</strong>&mdash;community detection (GraphRAG) or one-hop entity lookup (LightRAG)&mdash;that ignore query context and miss relevant multi-hop connections.</p></div>
+      <div class="card"><h3>Our Solution</h3><p>QAFD-RAG <strong>dynamically re-weights</strong> graph edges based on query relevance and <strong>propagates flow</strong> through the knowledge graph to discover multi-hop context, assembling a query-specific subgraph for the LLM.</p></div>
+    </div>
+  </div>
+</section>
+
+<section class="alt">
+  <div class="container">
+    <h2>Approach Comparison</h2>
+    <p class="section-sub">How different retrieval strategies explore a knowledge graph</p>
+    <div class="figure-row">
+      <div class="figure-card"><img src="figs/GraphRAG.png" alt="GraphRAG"/><div class="caption">GraphRAG<br/><small>Community Detection</small></div></div>
+      <div class="figure-card"><img src="figs/LightRAG.png" alt="LightRAG"/><div class="caption">LightRAG<br/><small>One-Hop Entity-Centric</small></div></div>
+      <div class="figure-card"><img src="figs/QAFD-RAG.png" alt="QAFD-RAG"/><div class="caption">QAFD-RAG (Ours)<br/><small>Query-Aware Flow Diffusion</small></div></div>
+    </div>
+    <p style="margin-top:24px;font-size:0.92rem;color:var(--slate-500);line-height:1.75;max-width:860px;margin-left:auto;margin-right:auto;">Comparison on Wikipedia pages (Apple fruit, Apple Inc., Amazon River, Amazon.com). Query: <em>"Introduce Steve Jobs's products in Apple."</em> <strong style="color:var(--slate-700);">GraphRAG</strong> retrieves entire communities, mixing relevant nodes with irrelevant ones. <strong style="color:var(--slate-700);">LightRAG</strong> focuses on 1-hop neighborhoods. <strong style="color:var(--slate-700);">QAFD-RAG</strong> reweights edges by query meaning, suppressing irrelevant neighborhoods.</p>
+  </div>
+</section>
+
+<section>
+  <div class="container">
+    <h2>How It Works</h2>
+    <p class="section-sub">A two-stage pipeline from raw documents to grounded LLM answers</p>
+    <div class="duo-grid">
+      <div class="card"><h3>Stage 1 &mdash; Knowledge Graph Construction</h3><ol><li><strong>Chunking</strong> &mdash; Documents split into overlapping token-based chunks.</li><li><strong>Entity Extraction</strong> &mdash; LLM identifies entities and relationships.</li><li><strong>Graph Assembly</strong> &mdash; Entities become nodes, relationships become weighted edges.</li></ol></div>
+      <div class="card"><h3>Stage 2 &mdash; Query-Aware Retrieval</h3><ol><li><strong>Entity Matching</strong> &mdash; Query keywords matched to KG entities via vector similarity.</li><li><strong>Flow Diffusion</strong> &mdash; Mass propagates with query-aware edge re-weighting.</li><li><strong>Context Assembly</strong> &mdash; Top-ranked nodes form context for the LLM.</li></ol></div>
+    </div>
+  </div>
+</section>
+
+<section class="alt">
+  <div class="container">
+    <h2>Benchmarks</h2>
+    <p class="section-sub">Evaluated across diverse retrieval and generation tasks</p>
+    <div class="bench-grid">
+      <div class="bench-item"><div class="task">UltraDomain</div><div class="dataset">Diverse Domain QA</div><div class="metrics">Comprehensiveness · Diversity · Relevance · Logicality · Coherence</div></div>
+      <div class="bench-item"><div class="task">Multi-hop QA</div><div class="dataset">MuSiQue · HotpotQA · 2WikiMultiHopQA</div><div class="metrics">F1 · Exact Match</div></div>
+      <div class="bench-item"><div class="task">Text-to-SQL</div><div class="dataset">Spider2-lite (Pagila, etc.)</div><div class="metrics">Schema Retrieval Accuracy</div></div>
+      <div class="bench-item"><div class="task">Summarization</div><div class="dataset">SQuALITY</div><div class="metrics">BLEU · ROUGE · METEOR</div></div>
+    </div>
+  </div>
+</section>
+
+<section>
+  <div class="container">
+    <h2>Supported Models</h2>
+    <p class="models-text"><strong>Embeddings:</strong> openai-small · openai-large · jina-v3 · gritlm · nvidia-nv-embed-v2<br/><strong>LLMs:</strong> gpt-4o-mini · gpt-4o · gpt-5-nano · gpt-5-mini · gpt-5 · gpt-oss-120b</p>
+  </div>
+</section>
+
+<section class="alt">
+  <div class="container">
+    <h2>Quick Start</h2>
+<pre><code><span class="comment"># 1. Install dependencies</span>
+pip install -r requirements.txt
+<span class="comment"># 2. Set your OpenAI API key</span>
+export OPENAI_API_KEY=<span class="string">"sk-..."</span>
+<span class="comment"># 3. Build a knowledge graph</span>
+./run.sh ultradomain --build --max-documents 100
+<span class="comment"># 4. Run a benchmark</span>
+./run.sh ultradomain --questions 10</code></pre>
+    <h3>Python API</h3>
+<pre><code><span class="keyword">from</span> src <span class="keyword">import</span> QAFD_RAG, QueryParam
+rag = QAFD_RAG(
+    working_dir=<span class="string">"./my_kg"</span>,
+    llm_model_name=<span class="string">"gpt-4o-mini"</span>,
+    embedding_model_key=<span class="string">"jina-v3"</span>,
+)
+<span class="comment"># Index documents</span>
+rag.insert([<span class="string">"Document text 1..."</span>, <span class="string">"Document text 2..."</span>])
+<span class="comment"># Query</span>
+answer = rag.query(<span class="string">"What is X?"</span>, param=QueryParam(mode=<span class="string">"hybrid"</span>))
+print(answer)</code></pre>
+  </div>
+</section>
+
+<section>
+  <div class="container">
+    <h2>Citation</h2>
+<pre><code>@inproceedings{zhou2026qafd,
+  title={Query-Aware Flow Diffusion for Graph-Based RAG with Retrieval Guarantees},
+  author={Zhuoping Zhou and Davoud Ataee Tarzanagh and Sima Didari and Wenjun Hu
+          and Baruch Gutow and Oxana Verkholyak and Masoud Faraki and Heng Hao
+          and Hankyu Moon and Seungjai Min},
+  booktitle={International Conference on Learning Representations (ICLR)},
+  year={2026}
+}</code></pre>
+  </div>
+</section>
+
+<footer>
+  <p>QAFD-RAG <span class="sep">·</span><a href="https://openreview.net/pdf?id=n28wnc2QTc">Paper</a> <span class="sep">·</span><a href="https://github.com/Tarzanagh/QAFD-RAG">GitHub</a> <span class="sep">·</span><a href="https://huggingface.co/datasets/qafd/kg">HuggingFace</a></p>
+  <p style="margin-top:8px;opacity:0.6;">Contact: tarzanagh [at] gmail [dot] com</p>
+</footer>
+</body>
+</html>

experiments/edge_weight_ablation.py

@@ -0,0 +1,401 @@
+"""
+Edge Weight Ablation Study for QAFD-RAG
+========================================
+
+Runs all experiments from a single script to avoid reloading the KG each time.
+
+Experiments:
+1. Query-aware vs query-agnostic
+2. Weight scheme comparison (original/multiply/add)
+3. (a, b) sensitivity sweep for Hybrid
+4. Qualitative per-query diagnostics
+"""
+
+import json
+import logging
+import os
+import sys
+import time
+from typing import Dict, List, Tuple
+
+import numpy as np
+
+# ---------------------------------------------------------------------------
+# Setup paths
+# ---------------------------------------------------------------------------
+_project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
+sys.path.insert(0, _project_root)
+
+import types as _types
+for _pkg_path in ["src", "src.retrievers", "src.hipporag_pipeline"]:
+    if _pkg_path not in sys.modules:
+        _m = _types.ModuleType(_pkg_path)
+        _m.__path__ = [os.path.join(_project_root, *_pkg_path.split("."))]
+        _m.__package__ = _pkg_path
+        sys.modules[_pkg_path] = _m
+
+import importlib.util as _ilu
+def _load_mod(fqn, filepath):
+    spec = _ilu.spec_from_file_location(fqn, filepath)
+    mod = _ilu.module_from_spec(spec)
+    sys.modules[fqn] = mod
+    spec.loader.exec_module(mod)
+    return mod
+
+_src = os.path.join(_project_root, "src")
+_load_mod("src.retrievers.base", os.path.join(_src, "retrievers", "base.py"))
+_load_mod("src.retrievers.flow_diffusion", os.path.join(_src, "retrievers", "flow_diffusion.py"))
+
+from src.hipporag_pipeline.config import HippoRAGConfig
+from src.hipporag_pipeline.embedding_store import EmbeddingModelWrapper
+from src.hipporag_pipeline.kg_builder import KGBuilder
+from src.hipporag_pipeline.openie import OpenIE
+from src.hipporag_pipeline.reranker import FactReranker
+from src.hipporag_pipeline.retriever import HippoRAGRetriever
+from src.hipporag_pipeline.graph_adapter import run_igraph_qafd
+from src.hipporag_pipeline.benchmark_runner import (
+    get_gold_docs, get_gold_answers, recall_at_k,
+    exact_match, f1_score, run_qa, _openai_embed, _openai_complete,
+)
+
+logging.basicConfig(level=logging.WARNING)
+logger = logging.getLogger(__name__)
+
+# ---------------------------------------------------------------------------
+# Load KG once
+# ---------------------------------------------------------------------------
+
+def load_everything(dataset="musique", num_queries=10):
+    """Load KG, embeddings, data — once for all experiments."""
+    api_key = os.environ.get("OPENAI_API_KEY", "")
+
+    config = HippoRAGConfig(
+        llm_model="gpt-4o-mini",
+        embedding_model_key="openai-small",
+        dataset=dataset,
+        save_dir="outputs",
+    )
+
+    import asyncio
+
+    async def llm_func(prompt, system_prompt=None, history_messages=[], **kwargs):
+        return await _openai_complete(
+            model="gpt-4o-mini", prompt=prompt,
+            system_prompt=system_prompt, history_messages=history_messages,
+            api_key=api_key, **kwargs
+        )
+
+    async def embed_func(texts):
+        return await _openai_embed(texts, model="text-embedding-3-small", api_key=api_key)
+
+    embedding_model = EmbeddingModelWrapper(embed_func, batch_size=16)
+    openie = OpenIE(llm_func)
+    builder = KGBuilder(config, embedding_model, openie)
+
+    # Load data
+    data_dir = os.path.join(_project_root, "data", "multihop")
+    with open(os.path.join(data_dir, f"{dataset}_corpus.json")) as f:
+        corpus = json.load(f)
+    docs = [f"{d['title']}\n{d['text']}" for d in corpus]
+
+    with open(os.path.join(data_dir, f"{dataset}.json")) as f:
+        samples = json.load(f)[:num_queries]
+
+    queries = [s["question"] for s in samples]
+    gold_answers = get_gold_answers(samples)
+    gold_docs = get_gold_docs(samples, dataset)
+
+    # Build / load KG
+    builder.index(docs)
+
+    # Build retriever
+    reranker = FactReranker(llm_func)
+    retriever = HippoRAGRetriever(
+        config=config,
+        embedding_model=embedding_model,
+        reranker=reranker,
+        graph=builder.graph,
+        chunk_embedding_store=builder.chunk_embedding_store,
+        entity_embedding_store=builder.entity_embedding_store,
+        fact_embedding_store=builder.fact_embedding_store,
+        openie_results_path=builder.openie_results_path,
+    )
+    retriever.prepare()
+
+    return config, retriever, queries, gold_answers, gold_docs, llm_func
+
+
+def run_retrieval_with_params(
+    retriever: HippoRAGRetriever,
+    queries: List[str],
+    gold_docs: List[List[str]],
+    gold_answers,
+    weight_scheme: str = "original",
+    hybrid_a: float = 1.0,
+    hybrid_b: float = 0.5,
+    query_aware: bool = True,
+) -> Dict:
+    """Run retrieval with specific edge weight params. Returns metrics dict."""
+    from src.hipporag_pipeline.utils import compute_mdhash_id, min_max_normalize
+
+    retriever._encode_queries(queries)
+    k_list = [1, 2, 5, 10, 20, 50, 100, 200]
+    all_retrieved = []
+    qafd_times = []
+    convergence_iters = []
+
+    for q in queries:
+        fact_scores = retriever._get_fact_scores(q)
+        top_indices, top_facts, _ = retriever._rerank_facts(q, fact_scores)
+
+        if len(top_facts) == 0:
+            sorted_ids, sorted_scores = retriever._dense_passage_retrieval(q)
+            all_retrieved.append([
+                retriever.chunk_store.get_row(retriever.passage_node_keys[idx])["content"]
+                for idx in sorted_ids[:200]
+            ])
+            qafd_times.append(0.0)
+            convergence_iters.append(-1)  # DPR fallback
+            continue
+
+        # Compute seed weights (same as retriever._graph_search)
+        n_nodes = retriever.graph.vcount()
+        phrase_weights = np.zeros(n_nodes)
+        passage_weights = np.zeros(n_nodes)
+        number_of_occurs = np.zeros(n_nodes)
+
+        for rank, f in enumerate(top_facts):
+            subj, obj = f[0].lower(), f[2].lower()
+            fs = fact_scores[top_indices[rank]] if fact_scores.ndim > 0 else float(fact_scores)
+            for phrase in [subj, obj]:
+                pk = compute_mdhash_id(phrase, prefix="entity-")
+                pid = retriever.node_name_to_vertex_idx.get(pk)
+                if pid is not None:
+                    wfs = fs
+                    num_chunks = len(retriever.ent_node_to_chunk_ids.get(pk, set()))
+                    if num_chunks > 0:
+                        wfs /= num_chunks
+                    phrase_weights[pid] += wfs
+                    number_of_occurs[pid] += 1
+
+        nonzero = number_of_occurs > 0
+        phrase_weights[nonzero] /= number_of_occurs[nonzero]
+
+        dpr_ids, dpr_scores = retriever._dense_passage_retrieval(q)
+        norm_dpr = min_max_normalize(dpr_scores)
+        pw = retriever.config.passage_node_weight
+        for i, did in enumerate(dpr_ids.tolist()):
+            pk = retriever.passage_node_keys[did]
+            pid = retriever.node_name_to_vertex_idx.get(pk)
+            if pid is not None:
+                passage_weights[pid] = norm_dpr[i] * pw
+
+        node_weights = phrase_weights + passage_weights
+        if np.sum(node_weights) == 0:
+            sorted_ids, sorted_scores = dpr_ids, dpr_scores
+            all_retrieved.append([
+                retriever.chunk_store.get_row(retriever.passage_node_keys[idx])["content"]
+                for idx in sorted_ids[:200]
+            ])
+            qafd_times.append(0.0)
+            convergence_iters.append(-1)
+            continue
+
+        # Build node embeddings
+        if not hasattr(retriever, '_node_emb_dict') or retriever._node_emb_dict is None:
+            retriever._node_emb_dict = {}
+            for i, nk in enumerate(retriever.entity_node_keys):
+                if i < len(retriever.entity_embeddings):
+                    retriever._node_emb_dict[nk] = retriever.entity_embeddings[i]
+            for i, nk in enumerate(retriever.passage_node_keys):
+                if i < len(retriever.passage_embeddings):
+                    retriever._node_emb_dict[nk] = retriever.passage_embeddings[i]
+
+        query_emb = retriever._query_emb_fact.get(q) if query_aware else None
+
+        t0 = time.time()
+        sorted_ids, sorted_scores = run_igraph_qafd(
+            graph=retriever.graph,
+            node_name_to_idx=retriever.node_name_to_vertex_idx,
+            passage_node_idxs=retriever.passage_node_idxs,
+            source_weights=node_weights,
+            node_embeddings=retriever._node_emb_dict if query_aware else {},
+            query_embedding=query_emb,
+            alpha=retriever.config.qafd_alpha,
+            epsilon=retriever.config.qafd_epsilon,
+            max_iterations=retriever.config.qafd_max_iterations,
+            step_size=retriever.config.qafd_step_size,
+            weight_scheme=weight_scheme,
+            hybrid_a=hybrid_a,
+            hybrid_b=hybrid_b,
+            use_node_degree=retriever.config.qafd_use_node_degree,
+            random_seed=retriever.config.qafd_random_seed,
+        )
+        elapsed = time.time() - t0
+        qafd_times.append(elapsed)
+
+        top_docs = [
+            retriever.chunk_store.get_row(retriever.passage_node_keys[idx])["content"]
+            for idx in sorted_ids[:200]
+        ]
+        all_retrieved.append(top_docs)
+
+    # Compute metrics
+    recall_metrics = recall_at_k(gold_docs, all_retrieved, k_list) if gold_docs else {}
+
+    return {
+        "recall": recall_metrics,
+        "avg_qafd_time": np.mean([t for t in qafd_times if t > 0]) if any(t > 0 for t in qafd_times) else 0,
+        "dpr_fallback_count": sum(1 for t in convergence_iters if t == -1),
+        "qafd_queries": sum(1 for t in qafd_times if t > 0),
+    }
+
+
+# ===========================================================================
+# Main experiments
+# ===========================================================================
+
+def main():
+    import argparse
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--dataset", default="musique")
+    parser.add_argument("--num_queries", type=int, default=10)
+    args = parser.parse_args()
+
+    print("=" * 70)
+    print("  QAFD-RAG Edge Weight Ablation Study")
+    print(f"  Dataset: {args.dataset}, Queries: {args.num_queries}")
+    print("=" * 70)
+
+    config, retriever, queries, gold_answers, gold_docs, llm_func = load_everything(
+        args.dataset, args.num_queries
+    )
+
+    results = {}
+
+    # ── Experiment 1: Query-aware vs Query-agnostic ──────────────────
+    print("\n[1/4] Query-aware vs Query-agnostic ablation")
+
+    print("  Running: Hybrid (a=1, b=0.5) — query-aware ...")
+    r = run_retrieval_with_params(
+        retriever, queries, gold_docs, gold_answers,
+        weight_scheme="original", hybrid_a=1.0, hybrid_b=0.5, query_aware=True,
+    )
+    results["hybrid_query_aware"] = r
+    print(f"    R@10={r['recall'].get('Recall@10', 0):.4f}  R@100={r['recall'].get('Recall@100', 0):.4f}  QAFD={r['avg_qafd_time']:.3f}s  DPR_fallback={r['dpr_fallback_count']}")
+
+    print("  Running: Query-agnostic (b=0) ...")
+    r = run_retrieval_with_params(
+        retriever, queries, gold_docs, gold_answers,
+        weight_scheme="original", hybrid_a=1.0, hybrid_b=0.0, query_aware=False,
+    )
+    results["query_agnostic"] = r
+    print(f"    R@10={r['recall'].get('Recall@10', 0):.4f}  R@100={r['recall'].get('Recall@100', 0):.4f}  QAFD={r['avg_qafd_time']:.3f}s  DPR_fallback={r['dpr_fallback_count']}")
+
+    # ── Experiment 2: Weight scheme comparison ───────────────────────
+    print("\n[2/4] Weight scheme comparison")
+
+    for scheme in ["original", "multiply", "add"]:
+        print(f"  Running: {scheme} ...")
+        r = run_retrieval_with_params(
+            retriever, queries, gold_docs, gold_answers,
+            weight_scheme=scheme, query_aware=True,
+        )
+        results[f"scheme_{scheme}"] = r
+        print(f"    R@10={r['recall'].get('Recall@10', 0):.4f}  R@100={r['recall'].get('Recall@100', 0):.4f}  QAFD={r['avg_qafd_time']:.3f}s")
+
+    # ── Experiment 3: (a, b) sensitivity sweep ───────────────────────
+    print("\n[3/4] (a, b) sensitivity sweep for Hybrid")
+
+    a_values = [0.5, 1.0, 2.0]
+    b_values = [0.0, 0.1, 0.25, 0.5, 1.0]
+
+    sweep_results = {}
+    for a in a_values:
+        for b in b_values:
+            label = f"a={a},b={b}"
+            r = run_retrieval_with_params(
+                retriever, queries, gold_docs, gold_answers,
+                weight_scheme="original", hybrid_a=a, hybrid_b=b, query_aware=(b > 0),
+            )
+            sweep_results[label] = r
+            r10 = r['recall'].get('Recall@10', 0)
+            r100 = r['recall'].get('Recall@100', 0)
+            print(f"  {label:>15}  R@10={r10:.4f}  R@100={r100:.4f}  QAFD={r['avg_qafd_time']:.3f}s")
+
+    results["sweep"] = sweep_results
+
+    # ── Experiment 4: Per-query diagnostics ──────────────────────────
+    print("\n[4/4] Per-query diagnostics (first 5 queries)")
+
+    # Compare query-aware vs agnostic per query
+    diag_queries = queries[:5]
+    diag_results = []
+
+    for qi, q in enumerate(diag_queries):
+        # Query-aware
+        r_aware = run_retrieval_with_params(
+            retriever, [q], [gold_docs[qi]], [gold_answers[qi]],
+            weight_scheme="original", hybrid_a=1.0, hybrid_b=0.5, query_aware=True,
+        )
+        # Query-agnostic
+        r_agnostic = run_retrieval_with_params(
+            retriever, [q], [gold_docs[qi]], [gold_answers[qi]],
+            weight_scheme="original", hybrid_a=1.0, hybrid_b=0.0, query_aware=False,
+        )
+
+        r10_aware = r_aware['recall'].get('Recall@10', 0)
+        r10_agnostic = r_agnostic['recall'].get('Recall@10', 0)
+        delta = r10_aware - r10_agnostic
+
+        status = "HELPS" if delta > 0 else ("HURTS" if delta < 0 else "SAME")
+        print(f"  Q{qi}: R@10 aware={r10_aware:.3f} agnostic={r10_agnostic:.3f}  delta={delta:+.3f} [{status}]")
+        print(f"       Q: {q[:80]}...")
+
+        diag_results.append({
+            "query": q,
+            "r10_aware": r10_aware,
+            "r10_agnostic": r10_agnostic,
+            "delta": delta,
+            "status": status,
+        })
+
+    results["diagnostics"] = diag_results
+
+    # ── Summary ──────────────────────────────────────────────────────
+    print("\n" + "=" * 70)
+    print("  SUMMARY")
+    print("=" * 70)
+
+    print("\n  Ablation: Query-aware vs Query-agnostic")
+    print(f"    Query-aware (Hybrid a=1,b=0.5):  R@10={results['hybrid_query_aware']['recall'].get('Recall@10',0):.4f}  R@100={results['hybrid_query_aware']['recall'].get('Recall@100',0):.4f}")
+    print(f"    Query-agnostic (b=0):             R@10={results['query_agnostic']['recall'].get('Recall@10',0):.4f}  R@100={results['query_agnostic']['recall'].get('Recall@100',0):.4f}")
+
+    print("\n  Weight scheme comparison")
+    for scheme in ["original", "multiply", "add"]:
+        r = results[f"scheme_{scheme}"]
+        print(f"    {scheme:>10}:  R@10={r['recall'].get('Recall@10',0):.4f}  R@100={r['recall'].get('Recall@100',0):.4f}")
+
+    print("\n  Best (a,b) from sweep:")
+    best_label = max(sweep_results, key=lambda k: sweep_results[k]['recall'].get('Recall@10', 0))
+    best = sweep_results[best_label]
+    print(f"    {best_label}:  R@10={best['recall'].get('Recall@10',0):.4f}  R@100={best['recall'].get('Recall@100',0):.4f}")
+
+    print("\n  Per-query diagnostics:")
+    helps = sum(1 for d in diag_results if d["status"] == "HELPS")
+    hurts = sum(1 for d in diag_results if d["status"] == "HURTS")
+    same = sum(1 for d in diag_results if d["status"] == "SAME")
+    print(f"    Query-awareness HELPS: {helps}/{len(diag_results)}, HURTS: {hurts}/{len(diag_results)}, SAME: {same}/{len(diag_results)}")
+
+    # Save
+    out_dir = os.path.join(_project_root, "experiments", "results")
+    os.makedirs(out_dir, exist_ok=True)
+    out_path = os.path.join(out_dir, f"edge_weight_ablation_{args.dataset}.json")
+    with open(out_path, "w") as f:
+        json.dump(results, f, indent=2, default=str)
+    print(f"\n  Results saved to {out_path}")
+    print("=" * 70)
+
+
+if __name__ == "__main__":
+    main()

experiments/query_aware_ablation.py

@@ -0,0 +1,478 @@
+"""
+Ablation: Query-Aware vs Query-Agnostic Flow Diffusion
+=======================================================
+
+Condition A (Query-Agnostic): b=0 → w̄ = H_sim(h(u), h(v)) · a
+Condition B (Query-Aware):    b=0.25 → w̄ = H_sim(h(u), h(v)) · (a + b·(sim(u,q)+sim(v,q)))
+
+Measures:
+  - Downstream task quality (Recall@K, F1, EM)
+  - Subgraph size (nodes with nonzero flow)
+  - Leakage ratio (mass at irrelevant vs relevant nodes)
+  - Convergence iterations
+  - Per-query qualitative diagnostics
+"""
+
+import json
+import logging
+import os
+import sys
+import time
+from collections import defaultdict
+from typing import Dict, List, Optional, Tuple
+
+import numpy as np
+
+# ---------------------------------------------------------------------------
+_project_root = os.path.abspath(os.path.join(os.path.dirname(__file__), ".."))
+sys.path.insert(0, _project_root)
+
+import types as _types
+for _pkg_path in ["src", "src.retrievers", "src.hipporag_pipeline"]:
+    if _pkg_path not in sys.modules:
+        _m = _types.ModuleType(_pkg_path)
+        _m.__path__ = [os.path.join(_project_root, *_pkg_path.split("."))]
+        _m.__package__ = _pkg_path
+        sys.modules[_pkg_path] = _m
+
+import importlib.util as _ilu
+def _load_mod(fqn, filepath):
+    spec = _ilu.spec_from_file_location(fqn, filepath)
+    mod = _ilu.module_from_spec(spec)
+    sys.modules[fqn] = mod
+    spec.loader.exec_module(mod)
+    return mod
+
+_src = os.path.join(_project_root, "src")
+_load_mod("src.retrievers.base", os.path.join(_src, "retrievers", "base.py"))
+_load_mod("src.retrievers.flow_diffusion", os.path.join(_src, "retrievers", "flow_diffusion.py"))
+
+from src.hipporag_pipeline.config import HippoRAGConfig
+from src.hipporag_pipeline.embedding_store import EmbeddingModelWrapper
+from src.hipporag_pipeline.kg_builder import KGBuilder
+from src.hipporag_pipeline.openie import OpenIE
+from src.hipporag_pipeline.reranker import FactReranker
+from src.hipporag_pipeline.retriever import HippoRAGRetriever
+from src.hipporag_pipeline.graph_adapter import IGraphQAFD
+from src.hipporag_pipeline.benchmark_runner import (
+    get_gold_docs, get_gold_answers, recall_at_k,
+    exact_match, f1_score, run_qa, _openai_embed, _openai_complete,
+)
+from src.hipporag_pipeline.utils import compute_mdhash_id, min_max_normalize
+
+logging.basicConfig(level=logging.WARNING)
+
+
+# ===========================================================================
+# Data + KG loading (once per dataset)
+# ===========================================================================
+
+def load_dataset_and_kg(dataset="musique", num_queries=10):
+    api_key = os.environ.get("OPENAI_API_KEY", "")
+    config = HippoRAGConfig(
+        llm_model="gpt-4o-mini",
+        embedding_model_key="openai-small",
+        dataset=dataset,
+        save_dir="outputs",
+    )
+    import asyncio
+
+    async def llm_func(prompt, system_prompt=None, history_messages=[], **kwargs):
+        return await _openai_complete(
+            model="gpt-4o-mini", prompt=prompt,
+            system_prompt=system_prompt, history_messages=history_messages,
+            api_key=api_key, **kwargs
+        )
+
+    async def embed_func(texts):
+        return await _openai_embed(texts, model="text-embedding-3-small", api_key=api_key)
+
+    embedding_model = EmbeddingModelWrapper(embed_func, batch_size=16)
+    openie = OpenIE(llm_func)
+    builder = KGBuilder(config, embedding_model, openie)
+
+    data_dir = os.path.join(_project_root, "data", "multihop")
+    with open(os.path.join(data_dir, f"{dataset}_corpus.json")) as f:
+        corpus = json.load(f)
+    docs = [f"{d['title']}\n{d['text']}" for d in corpus]
+
+    with open(os.path.join(data_dir, f"{dataset}.json")) as f:
+        samples = json.load(f)[:num_queries]
+
+    queries = [s["question"] for s in samples]
+    gold_answers = get_gold_answers(samples)
+    gold_docs = get_gold_docs(samples, dataset)
+
+    builder.index(docs)
+
+    reranker = FactReranker(llm_func)
+    retriever = HippoRAGRetriever(
+        config=config, embedding_model=embedding_model, reranker=reranker,
+        graph=builder.graph,
+        chunk_embedding_store=builder.chunk_embedding_store,
+        entity_embedding_store=builder.entity_embedding_store,
+        fact_embedding_store=builder.fact_embedding_store,
+        openie_results_path=builder.openie_results_path,
+    )
+    retriever.prepare()
+
+    # Cache node embeddings
+    retriever._node_emb_dict = {}
+    for i, nk in enumerate(retriever.entity_node_keys):
+        if i < len(retriever.entity_embeddings):
+            retriever._node_emb_dict[nk] = retriever.entity_embeddings[i]
+    for i, nk in enumerate(retriever.passage_node_keys):
+        if i < len(retriever.passage_embeddings):
+            retriever._node_emb_dict[nk] = retriever.passage_embeddings[i]
+
+    return config, retriever, queries, gold_answers, gold_docs, llm_func
+
+
+# ===========================================================================
+# Core: run QAFD with detailed diagnostics
+# ===========================================================================
+
+def run_qafd_detailed(
+    retriever: HippoRAGRetriever,
+    query: str,
+    gold_doc_set: set,
+    alpha: float,
+    weight_scheme: str,
+    hybrid_a: float,
+    hybrid_b: float,
+    query_aware: bool,
+) -> Dict:
+    """Run QAFD on a single query, return detailed metrics."""
+
+    retriever._encode_queries([query])
+    fact_scores = retriever._get_fact_scores(query)
+    top_indices, top_facts, _ = retriever._rerank_facts(query, fact_scores)
+
+    # DPR fallback
+    if len(top_facts) == 0:
+        sorted_ids, sorted_scores = retriever._dense_passage_retrieval(query)
+        top_docs = [
+            retriever.chunk_store.get_row(retriever.passage_node_keys[idx])["content"]
+            for idx in sorted_ids[:200]
+        ]
+        return {
+            "method": "DPR_fallback",
+            "top_docs": top_docs,
+            "subgraph_size": 0,
+            "leakage_ratio": 1.0,
+            "convergence_iters": -1,
+            "qafd_time": 0.0,
+            "flow_at_relevant": 0.0,
+            "flow_at_irrelevant": 0.0,
+        }
+
+    # Compute seed weights
+    n_nodes = retriever.graph.vcount()
+    phrase_weights = np.zeros(n_nodes)
+    passage_weights = np.zeros(n_nodes)
+    number_of_occurs = np.zeros(n_nodes)
+
+    for rank, f in enumerate(top_facts):
+        subj, obj = f[0].lower(), f[2].lower()
+        fs = fact_scores[top_indices[rank]] if fact_scores.ndim > 0 else float(fact_scores)
+        for phrase in [subj, obj]:
+            pk = compute_mdhash_id(phrase, prefix="entity-")
+            pid = retriever.node_name_to_vertex_idx.get(pk)
+            if pid is not None:
+                wfs = fs
+                nc = len(retriever.ent_node_to_chunk_ids.get(pk, set()))
+                if nc > 0:
+                    wfs /= nc
+                phrase_weights[pid] += wfs
+                number_of_occurs[pid] += 1
+
+    nonzero = number_of_occurs > 0
+    phrase_weights[nonzero] /= number_of_occurs[nonzero]
+
+    dpr_ids, dpr_scores = retriever._dense_passage_retrieval(query)
+    norm_dpr = min_max_normalize(dpr_scores)
+    pw = retriever.config.passage_node_weight
+    for i, did in enumerate(dpr_ids.tolist()):
+        pk = retriever.passage_node_keys[did]
+        pid = retriever.node_name_to_vertex_idx.get(pk)
+        if pid is not None:
+            passage_weights[pid] = norm_dpr[i] * pw
+
+    node_weights = phrase_weights + passage_weights
+    if np.sum(node_weights) == 0:
+        sorted_ids, sorted_scores = dpr_ids, dpr_scores
+        top_docs = [
+            retriever.chunk_store.get_row(retriever.passage_node_keys[idx])["content"]
+            for idx in sorted_ids[:200]
+        ]
+        return {
+            "method": "DPR_fallback_zero_seeds",
+            "top_docs": top_docs,
+            "subgraph_size": 0,
+            "leakage_ratio": 1.0,
+            "convergence_iters": -1,
+            "qafd_time": 0.0,
+            "flow_at_relevant": 0.0,
+            "flow_at_irrelevant": 0.0,
+        }
+
+    query_emb = retriever._query_emb_fact.get(query) if query_aware else None
+    node_embs = retriever._node_emb_dict if query_aware else {}
+
+    # Run QAFD directly to get raw node scores
+    t0 = time.time()
+    qafd = IGraphQAFD(
+        graph=retriever.graph,
+        node_name_to_idx=retriever.node_name_to_vertex_idx,
+        source_weights=node_weights,
+        node_embeddings=node_embs,
+        query_embedding=query_emb,
+        alpha=alpha,
+        epsilon=retriever.config.qafd_epsilon,
+        max_iterations=retriever.config.qafd_max_iterations,
+        step_size=retriever.config.qafd_step_size,
+        weight_scheme=weight_scheme,
+        hybrid_a=hybrid_a,
+        hybrid_b=hybrid_b,
+        use_node_degree=retriever.config.qafd_use_node_degree,
+        random_seed=retriever.config.qafd_random_seed,
+    )
+    raw_scores = qafd.run()
+    elapsed = time.time() - t0
+
+    # Extract passage scores
+    doc_scores = np.array([raw_scores[idx] for idx in retriever.passage_node_idxs])
+    total = np.sum(doc_scores)
+    if total > 0:
+        doc_scores_norm = doc_scores / total
+    else:
+        doc_scores_norm = np.ones(len(doc_scores)) / max(len(doc_scores), 1)
+
+    sorted_ids = np.argsort(doc_scores_norm)[::-1]
+    sorted_scores = doc_scores_norm[sorted_ids]
+
+    top_docs = [
+        retriever.chunk_store.get_row(retriever.passage_node_keys[idx])["content"]
+        for idx in sorted_ids[:200]
+    ]
+
+    # --- Compute detailed metrics ---
+
+    # Subgraph size: nodes with nonzero flow
+    subgraph_size = int(np.sum(raw_scores > 1e-10))
+
+    # Leakage ratio: flow at irrelevant vs relevant passage nodes
+    flow_relevant = 0.0
+    flow_irrelevant = 0.0
+    relevant_count = 0
+    irrelevant_count = 0
+
+    for i, pk in enumerate(retriever.passage_node_keys):
+        content = retriever.chunk_store.get_row(pk)["content"]
+        score = doc_scores[i]
+        is_gold = any(g in content or content in g for g in gold_doc_set)
+        if is_gold:
+            flow_relevant += score
+            relevant_count += 1
+        else:
+            flow_irrelevant += score
+            irrelevant_count += 1
+
+    total_flow = flow_relevant + flow_irrelevant
+    leakage = flow_irrelevant / total_flow if total_flow > 0 else 1.0
+
+    return {
+        "method": "QAFD",
+        "top_docs": top_docs,
+        "subgraph_size": subgraph_size,
+        "leakage_ratio": round(leakage, 4),
+        "flow_at_relevant": round(flow_relevant, 6),
+        "flow_at_irrelevant": round(flow_irrelevant, 6),
+        "relevant_passages": relevant_count,
+        "qafd_time": round(elapsed, 4),
+    }
+
+
+# ===========================================================================
+# Run full experiment
+# ===========================================================================
+
+def run_experiment(
+    retriever, queries, gold_docs, gold_answers,
+    alpha: float, hybrid_a: float, hybrid_b: float,
+    query_aware: bool, label: str,
+) -> Dict:
+    """Run all queries with given params, return aggregate metrics."""
+    all_docs = []
+    subgraph_sizes = []
+    leakage_ratios = []
+    flow_relevants = []
+    flow_irrelevants = []
+    qafd_times = []
+    per_query = []
+
+    for qi, q in enumerate(queries):
+        gold_set = set(gold_docs[qi]) if gold_docs else set()
+        r = run_qafd_detailed(
+            retriever, q, gold_set,
+            alpha=alpha,
+            weight_scheme="original",
+            hybrid_a=hybrid_a,
+            hybrid_b=hybrid_b,
+            query_aware=query_aware,
+        )
+        all_docs.append(r["top_docs"])
+        subgraph_sizes.append(r["subgraph_size"])
+        leakage_ratios.append(r["leakage_ratio"])
+        flow_relevants.append(r["flow_at_relevant"])
+        flow_irrelevants.append(r["flow_at_irrelevant"])
+        qafd_times.append(r["qafd_time"])
+        per_query.append({
+            "query": q[:100],
+            "method": r["method"],
+            "subgraph_size": r["subgraph_size"],
+            "leakage_ratio": r["leakage_ratio"],
+            "flow_relevant": r["flow_at_relevant"],
+            "flow_irrelevant": r["flow_at_irrelevant"],
+        })
+
+    # Recall
+    k_list = [1, 2, 5, 10, 20, 50, 100, 200]
+    recall_metrics = recall_at_k(gold_docs, all_docs, k_list) if gold_docs else {}
+
+    return {
+        "label": label,
+        "alpha": alpha,
+        "hybrid_a": hybrid_a,
+        "hybrid_b": hybrid_b,
+        "query_aware": query_aware,
+        "recall": recall_metrics,
+        "avg_subgraph_size": round(np.mean(subgraph_sizes), 1),
+        "avg_leakage_ratio": round(np.mean(leakage_ratios), 4),
+        "avg_flow_relevant": round(np.mean(flow_relevants), 6),
+        "avg_flow_irrelevant": round(np.mean(flow_irrelevants), 6),
+        "avg_qafd_time": round(np.mean(qafd_times), 4),
+        "per_query": per_query,
+    }
+
+
+def main():
+    import argparse
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--dataset", default="musique")
+    parser.add_argument("--num_queries", type=int, default=10)
+    args = parser.parse_args()
+
+    print("=" * 70)
+    print("  Query-Aware vs Query-Agnostic Ablation Study")
+    print(f"  Dataset: {args.dataset}, Queries: {args.num_queries}")
+    print("=" * 70)
+
+    config, retriever, queries, gold_answers, gold_docs, llm_func = \
+        load_dataset_and_kg(args.dataset, args.num_queries)
+
+    print(f"  Graph: {retriever.graph.vcount()} nodes, {retriever.graph.ecount()} edges")
+    print(f"  Entities: {len(retriever.entity_node_keys)}, Passages: {len(retriever.passage_node_keys)}")
+
+    all_results = {}
+
+    # ── Run across multiple alpha values ─────────────────────────────
+    alpha_values = [2.0, 10.0, 50.0]
+
+    for alpha in alpha_values:
+        print(f"\n{'─' * 70}")
+        print(f"  Alpha = {alpha}")
+        print(f"{'─' * 70}")
+
+        # Condition A: Query-Agnostic (b=0)
+        print(f"  Running Query-Agnostic (b=0) ...")
+        r_agnostic = run_experiment(
+            retriever, queries, gold_docs, gold_answers,
+            alpha=alpha, hybrid_a=1.0, hybrid_b=0.0,
+            query_aware=False, label=f"agnostic_a{alpha}",
+        )
+        all_results[f"agnostic_a{alpha}"] = r_agnostic
+
+        # Condition B: Query-Aware (a=1, b=0.25)
+        print(f"  Running Query-Aware (a=1, b=0.25) ...")
+        r_aware_025 = run_experiment(
+            retriever, queries, gold_docs, gold_answers,
+            alpha=alpha, hybrid_a=1.0, hybrid_b=0.25,
+            query_aware=True, label=f"aware025_a{alpha}",
+        )
+        all_results[f"aware025_a{alpha}"] = r_aware_025
+
+        # Condition C: Query-Aware (a=1, b=0.5) — current default
+        print(f"  Running Query-Aware (a=1, b=0.5) ...")
+        r_aware_050 = run_experiment(
+            retriever, queries, gold_docs, gold_answers,
+            alpha=alpha, hybrid_a=1.0, hybrid_b=0.5,
+            query_aware=True, label=f"aware050_a{alpha}",
+        )
+        all_results[f"aware050_a{alpha}"] = r_aware_050
+
+        # Print comparison table
+        print(f"\n  {'Metric':<25} {'Agnostic(b=0)':>15} {'Aware(b=0.25)':>15} {'Aware(b=0.5)':>15} {'Δ(0.25 vs 0)':>15}")
+        print(f"  {'─' * 85}")
+
+        for k in [10, 50, 100]:
+            key = f"Recall@{k}"
+            va = r_agnostic['recall'].get(key, 0)
+            vb = r_aware_025['recall'].get(key, 0)
+            vc = r_aware_050['recall'].get(key, 0)
+            d = vb - va
+            marker = " ↑" if d > 0 else (" ↓" if d < 0 else "")
+            print(f"  {key:<25} {va:>15.4f} {vb:>15.4f} {vc:>15.4f} {d:>+14.4f}{marker}")
+
+        print(f"  {'Subgraph size':<25} {r_agnostic['avg_subgraph_size']:>15.1f} {r_aware_025['avg_subgraph_size']:>15.1f} {r_aware_050['avg_subgraph_size']:>15.1f} {r_aware_025['avg_subgraph_size'] - r_agnostic['avg_subgraph_size']:>+14.1f}")
+        print(f"  {'Leakage ratio':<25} {r_agnostic['avg_leakage_ratio']:>15.4f} {r_aware_025['avg_leakage_ratio']:>15.4f} {r_aware_050['avg_leakage_ratio']:>15.4f} {r_aware_025['avg_leakage_ratio'] - r_agnostic['avg_leakage_ratio']:>+14.4f}")
+        print(f"  {'Flow@relevant':<25} {r_agnostic['avg_flow_relevant']:>15.6f} {r_aware_025['avg_flow_relevant']:>15.6f} {r_aware_050['avg_flow_relevant']:>15.6f} {r_aware_025['avg_flow_relevant'] - r_agnostic['avg_flow_relevant']:>+14.6f}")
+        print(f"  {'Flow@irrelevant':<25} {r_agnostic['avg_flow_irrelevant']:>15.6f} {r_aware_025['avg_flow_irrelevant']:>15.6f} {r_aware_050['avg_flow_irrelevant']:>15.6f} {r_aware_025['avg_flow_irrelevant'] - r_agnostic['avg_flow_irrelevant']:>+14.6f}")
+        print(f"  {'QAFD time (s)':<25} {r_agnostic['avg_qafd_time']:>15.4f} {r_aware_025['avg_qafd_time']:>15.4f} {r_aware_050['avg_qafd_time']:>15.4f}")
+
+    # ── Per-query diagnostics (alpha=10, first 5 queries) ────────────
+    print(f"\n{'=' * 70}")
+    print("  Per-Query Diagnostics (alpha=10.0)")
+    print(f"{'=' * 70}")
+
+    r_ag = all_results.get("agnostic_a10.0", {})
+    r_aw = all_results.get("aware025_a10.0", {})
+
+    if r_ag and r_aw:
+        ag_pq = r_ag.get("per_query", [])
+        aw_pq = r_aw.get("per_query", [])
+        for qi in range(min(5, len(ag_pq))):
+            ag = ag_pq[qi]
+            aw = aw_pq[qi]
+            print(f"\n  Q{qi}: {ag['query']}")
+            print(f"    {'':>20} {'Agnostic':>12} {'Aware':>12} {'Delta':>12}")
+            print(f"    {'Subgraph size':>20} {ag['subgraph_size']:>12} {aw['subgraph_size']:>12} {aw['subgraph_size']-ag['subgraph_size']:>+12}")
+            print(f"    {'Leakage ratio':>20} {ag['leakage_ratio']:>12.4f} {aw['leakage_ratio']:>12.4f} {aw['leakage_ratio']-ag['leakage_ratio']:>+12.4f}")
+            print(f"    {'Flow@relevant':>20} {ag['flow_relevant']:>12.6f} {aw['flow_relevant']:>12.6f} {aw['flow_relevant']-ag['flow_relevant']:>+12.6f}")
+            print(f"    {'Flow@irrelevant':>20} {ag['flow_irrelevant']:>12.6f} {aw['flow_irrelevant']:>12.6f} {aw['flow_irrelevant']-ag['flow_irrelevant']:>+12.6f}")
+
+            status = "HELPS" if aw['leakage_ratio'] < ag['leakage_ratio'] else (
+                "HURTS" if aw['leakage_ratio'] > ag['leakage_ratio'] else "SAME"
+            )
+            print(f"    → Query awareness {status} (leakage {'decreased' if status == 'HELPS' else 'increased' if status == 'HURTS' else 'unchanged'})")
+
+    # ── Save ─────────────────────────────────────────────────────────
+    out_dir = os.path.join(_project_root, "experiments", "results")
+    os.makedirs(out_dir, exist_ok=True)
+    out_path = os.path.join(out_dir, f"query_aware_ablation_{args.dataset}.json")
+
+    # Remove top_docs from saved output (too large)
+    save_results = {}
+    for k, v in all_results.items():
+        sv = dict(v)
+        sv.pop("per_query", None)
+        save_results[k] = sv
+
+    with open(out_path, "w") as f:
+        json.dump(save_results, f, indent=2, default=str)
+    print(f"\n  Results saved to {out_path}")
+    print("=" * 70)
+
+
+if __name__ == "__main__":
+    main()

experiments/run_all_ablations.sh

@@ -0,0 +1,49 @@
+#!/bin/bash
+# Run all query-awareness ablation experiments on musique (100 queries)
+# Each run reuses the pre-built KG — only retrieval is re-run.
+
+DATASET="musique"
+N=100
+BASE="python src/hipporag_pipeline/benchmark_runner.py --task multihop --dataset $DATASET --num_queries $N --embedding_model openai-small --skip_qa"
+
+echo "======================================================================"
+echo "  QAFD-RAG Query-Awareness Ablation (${DATASET}, ${N} queries)"
+echo "======================================================================"
+
+# Baseline (all flags = 0)
+echo ""
+echo "[1/7] Baseline (original, all QA flags = 0)"
+$BASE --save_dir outputs_ablation/baseline 2>&1 | grep -E "Recall@|QAFD completed|Retrieval done"
+
+# Phase 1: Query-aware sink capacity
+echo ""
+echo "[2/7] Phase 1: qa_sink_gamma=0.5"
+$BASE --qa_sink_gamma 0.5 --save_dir outputs_ablation/sink_05 2>&1 | grep -E "Recall@|QAFD completed|Retrieval done"
+
+echo ""
+echo "[3/7] Phase 1: qa_sink_gamma=1.0"
+$BASE --qa_sink_gamma 1.0 --save_dir outputs_ablation/sink_10 2>&1 | grep -E "Recall@|QAFD completed|Retrieval done"
+
+# Phase 2: Query-aware warm start
+echo ""
+echo "[4/7] Phase 2: qa_warm_delta=0.5"
+$BASE --qa_warm_delta 0.5 --save_dir outputs_ablation/warm_05 2>&1 | grep -E "Recall@|QAFD completed|Retrieval done"
+
+# Phase 3: Post-diffusion reranking
+echo ""
+echo "[5/7] Phase 3: qa_post_lambda=0.5"
+$BASE --qa_post_lambda 0.5 --save_dir outputs_ablation/post_05 2>&1 | grep -E "Recall@|QAFD completed|Retrieval done"
+
+echo ""
+echo "[6/7] Phase 3: qa_post_lambda=1.0"
+$BASE --qa_post_lambda 1.0 --save_dir outputs_ablation/post_10 2>&1 | grep -E "Recall@|QAFD completed|Retrieval done"
+
+# Combined best
+echo ""
+echo "[7/7] Combined: sink=0.5 + warm=0.5 + post=0.5"
+$BASE --qa_sink_gamma 0.5 --qa_warm_delta 0.5 --qa_post_lambda 0.5 --save_dir outputs_ablation/combined 2>&1 | grep -E "Recall@|QAFD completed|Retrieval done"
+
+echo ""
+echo "======================================================================"
+echo "  ALL EXPERIMENTS COMPLETE"
+echo "======================================================================"

experiments/run_alpha_qa_sweep.sh

@@ -0,0 +1,29 @@
+#!/bin/bash
+# Alpha × QA sweep: find the sweet spot where decoupled QA edge weights help
+cd /home/davoud/QAFD-RAG
+N=100
+BASE="python src/hipporag_pipeline/benchmark_runner.py --task multihop --num_queries $N --embedding_model openai-small --skip_qa"
+
+echo "======================================================================"
+echo "  Alpha x Query-Awareness Sweep (decoupled push, $N queries)"
+echo "======================================================================"
+
+for DATASET in musique hotpotqa 2wikimultihopqa; do
+    echo ""
+    echo "====== Dataset: $DATASET ======"
+
+    for ALPHA in 2 3 5 8; do
+        echo "  --- alpha=$ALPHA ---"
+
+        echo "    QA-aware (original, a=1, b=0.5)"
+        $BASE --dataset $DATASET --qafd_alpha $ALPHA 2>&1 | grep "Recall@10:\|Recall@100:"
+
+        echo "    QA-agnostic (b=0)"
+        $BASE --dataset $DATASET --qafd_alpha $ALPHA --qafd_weight_scheme none 2>&1 | grep "Recall@10:\|Recall@100:"
+    done
+done
+
+echo ""
+echo "======================================================================"
+echo "  ALL EXPERIMENTS COMPLETE"
+echo "======================================================================"

experiments/run_batch_push_ablation.sh

@@ -0,0 +1,29 @@
+#!/bin/bash
+# Batch push ablation: does batch push make edge weights matter?
+# Compare: single vs batch, agnostic vs aware
+cd /home/davoud/QAFD-RAG
+N=100
+BASE="python src/hipporag_pipeline/benchmark_runner.py --task multihop --num_queries $N --embedding_model openai-small --skip_qa"
+
+echo "======================================================================"
+echo "  Batch Push Ablation ($N queries per dataset)"
+echo "======================================================================"
+
+for DATASET in musique hotpotqa 2wikimultihopqa; do
+    echo ""
+    echo "====== Dataset: $DATASET ======"
+
+    echo "  [1] Single push, query-aware (current default)"
+    $BASE --dataset $DATASET 2>&1 | grep -E "Recall@|QAFD:"
+
+    echo "  [2] Batch push, query-aware"
+    $BASE --dataset $DATASET --batch_push 2>&1 | grep -E "Recall@|QAFD:"
+
+    echo "  [3] Batch push, query-agnostic (b=0)"
+    $BASE --dataset $DATASET --batch_push --qafd_weight_scheme none 2>&1 | grep -E "Recall@|QAFD:"
+done
+
+echo ""
+echo "======================================================================"
+echo "  ALL EXPERIMENTS COMPLETE"
+echo "======================================================================"

experiments/run_batch_push_nvidia.sh

@@ -0,0 +1,55 @@
+#!/bin/bash
+# Batch push sweep on nvidia KG (dense, 1.6M edges)
+# Goal: find config where QA-aware beats agnostic AND recall is high
+cd /home/davoud/QAFD-RAG
+N=100
+BASE="python src/hipporag_pipeline/benchmark_runner.py --task multihop --num_queries $N --embedding_model nvidia-nv-embed-v2 --skip_qa"
+LOG=experiments/results/batch_push_nvidia.log
+mkdir -p experiments/results
+exec > >(tee -a $LOG) 2>&1
+
+echo "======================================================================"
+echo "  Batch Push Sweep on nvidia KG ($N queries)"
+echo "  Started: $(date)"
+echo "======================================================================"
+
+for DATASET in musique hotpotqa; do
+    echo ""
+    echo "====== $DATASET ======"
+
+    # Baseline: single push, alpha=2, eps=0.01
+    echo "  [1] single, a=2, eps=0.01, QA-aware"
+    $BASE --dataset $DATASET --qafd_alpha 2 --qafd_epsilon 0.01 2>&1 | grep "Recall@"
+    echo "  [2] single, a=2, eps=0.01, agnostic"
+    $BASE --dataset $DATASET --qafd_alpha 2 --qafd_epsilon 0.01 --qafd_weight_scheme none 2>&1 | grep "Recall@"
+
+    # Batch push, alpha=2, eps=0.01
+    echo "  [3] batch, a=2, eps=0.01, QA-aware"
+    $BASE --dataset $DATASET --qafd_alpha 2 --qafd_epsilon 0.01 --batch_push 2>&1 | grep "Recall@"
+    echo "  [4] batch, a=2, eps=0.01, agnostic"
+    $BASE --dataset $DATASET --qafd_alpha 2 --qafd_epsilon 0.01 --batch_push --qafd_weight_scheme none 2>&1 | grep "Recall@"
+
+    # Batch push, alpha=2, eps=0.005
+    echo "  [5] batch, a=2, eps=0.005, QA-aware"
+    $BASE --dataset $DATASET --qafd_alpha 2 --qafd_epsilon 0.005 --batch_push 2>&1 | grep "Recall@"
+    echo "  [6] batch, a=2, eps=0.005, agnostic"
+    $BASE --dataset $DATASET --qafd_alpha 2 --qafd_epsilon 0.005 --batch_push --qafd_weight_scheme none 2>&1 | grep "Recall@"
+
+    # Batch push, alpha=2, eps=0.003
+    echo "  [7] batch, a=2, eps=0.003, QA-aware"
+    $BASE --dataset $DATASET --qafd_alpha 2 --qafd_epsilon 0.003 --batch_push 2>&1 | grep "Recall@"
+    echo "  [8] batch, a=2, eps=0.003, agnostic"
+    $BASE --dataset $DATASET --qafd_alpha 2 --qafd_epsilon 0.003 --batch_push --qafd_weight_scheme none 2>&1 | grep "Recall@"
+
+    # Batch push, alpha=3, eps=0.005
+    echo "  [9] batch, a=3, eps=0.005, QA-aware"
+    $BASE --dataset $DATASET --qafd_alpha 3 --qafd_epsilon 0.005 --batch_push 2>&1 | grep "Recall@"
+    echo "  [10] batch, a=3, eps=0.005, agnostic"
+    $BASE --dataset $DATASET --qafd_alpha 3 --qafd_epsilon 0.005 --batch_push --qafd_weight_scheme none 2>&1 | grep "Recall@"
+done
+
+echo ""
+echo "======================================================================"
+echo "  ALL EXPERIMENTS COMPLETE"
+echo "  Finished: $(date)"
+echo "======================================================================"

experiments/run_decoupled_ablation.sh

@@ -0,0 +1,32 @@
+#!/bin/bash
+# Decoupled push ablation: does separating accumulation from routing
+# make query-aware edge weights effective?
+cd /home/davoud/QAFD-RAG
+N=100
+BASE="python src/hipporag_pipeline/benchmark_runner.py --task multihop --num_queries $N --embedding_model openai-small --skip_qa"
+
+echo "======================================================================"
+echo "  Decoupled Push Ablation ($N queries per dataset)"
+echo "======================================================================"
+
+for DATASET in musique hotpotqa 2wikimultihopqa; do
+    echo ""
+    echo "====== Dataset: $DATASET ======"
+
+    echo "  [1] Single push, query-aware (decoupled)"
+    $BASE --dataset $DATASET 2>&1 | grep -E "Recall@|QAFD:"
+
+    echo "  [2] Single push, query-agnostic (decoupled, b=0 baseline)"
+    $BASE --dataset $DATASET --qafd_weight_scheme none 2>&1 | grep -E "Recall@|QAFD:"
+
+    echo "  [3] Batch push, query-aware (decoupled)"
+    $BASE --dataset $DATASET --batch_push 2>&1 | grep -E "Recall@|QAFD:"
+
+    echo "  [4] Batch push, query-agnostic (decoupled, b=0 baseline)"
+    $BASE --dataset $DATASET --batch_push --qafd_weight_scheme none 2>&1 | grep -E "Recall@|QAFD:"
+done
+
+echo ""
+echo "======================================================================"
+echo "  ALL EXPERIMENTS COMPLETE"
+echo "======================================================================"

experiments/run_nvidia_qa_ablation.sh

@@ -0,0 +1,43 @@
+#!/bin/bash
+# Query-awareness ablation on nvidia-nv-embed-v2 KGs (dense, 1.6M edges)
+# Tests: QA-aware vs agnostic, different alpha/epsilon settings
+cd /home/davoud/QAFD-RAG
+N=100
+BASE="python src/hipporag_pipeline/benchmark_runner.py --task multihop --num_queries $N --embedding_model nvidia-nv-embed-v2 --skip_qa"
+
+echo "======================================================================"
+echo "  Query-Awareness on nvidia KG (dense graph, $N queries)"
+echo "  Started: $(date)"
+echo "======================================================================"
+
+for DATASET in musique hotpotqa 2wikimultihopqa; do
+    echo ""
+    echo "====== $DATASET ======"
+
+    # Alpha=2, epsilon=0.01 (original settings)
+    echo "  --- alpha=2, eps=0.01 ---"
+    echo "  [QA-aware]"
+    $BASE --dataset $DATASET --qafd_alpha 2.0 --qafd_epsilon 0.01 2>&1 | grep "Recall@10:\|Recall@100:\|QAFD:"
+    echo "  [QA-agnostic]"
+    $BASE --dataset $DATASET --qafd_alpha 2.0 --qafd_epsilon 0.01 --qafd_weight_scheme none 2>&1 | grep "Recall@10:\|Recall@100:\|QAFD:"
+
+    # Alpha=3, epsilon=0.01
+    echo "  --- alpha=3, eps=0.01 ---"
+    echo "  [QA-aware]"
+    $BASE --dataset $DATASET --qafd_alpha 3.0 --qafd_epsilon 0.01 2>&1 | grep "Recall@10:\|Recall@100:\|QAFD:"
+    echo "  [QA-agnostic]"
+    $BASE --dataset $DATASET --qafd_alpha 3.0 --qafd_epsilon 0.01 --qafd_weight_scheme none 2>&1 | grep "Recall@10:\|Recall@100:\|QAFD:"
+
+    # Alpha=3, epsilon=0.001 (tighter convergence)
+    echo "  --- alpha=3, eps=0.001 ---"
+    echo "  [QA-aware]"
+    $BASE --dataset $DATASET --qafd_alpha 3.0 --qafd_epsilon 0.001 2>&1 | grep "Recall@10:\|Recall@100:\|QAFD:"
+    echo "  [QA-agnostic]"
+    $BASE --dataset $DATASET --qafd_alpha 3.0 --qafd_epsilon 0.001 --qafd_weight_scheme none 2>&1 | grep "Recall@10:\|Recall@100:\|QAFD:"
+done
+
+echo ""
+echo "======================================================================"
+echo "  ALL EXPERIMENTS COMPLETE"
+echo "  Finished: $(date)"
+echo "======================================================================"

experiments/run_overnight_qa_awareness.sh

@@ -0,0 +1,136 @@
+#!/bin/bash
+# Overnight query-awareness experiments
+# Tests decoupled push with alpha=3: QA-aware vs agnostic
+# Across tasks, graph types, and embedding models
+cd /home/davoud/QAFD-RAG
+LOG=/home/davoud/QAFD-RAG/experiments/results/overnight_qa_awareness.log
+mkdir -p experiments/results
+exec > >(tee -a $LOG) 2>&1
+echo "======================================================================"
+echo "  Overnight Query-Awareness Experiments"
+echo "  Started: $(date)"
+echo "======================================================================"
+
+BASE_PE="python src/hipporag_pipeline/benchmark_runner.py"
+
+# =====================================================================
+# 1. PASSAGE-ENTITY: Multihop with OpenAI embedding (100q, alpha=3)
+# =====================================================================
+echo ""
+echo "############ PASSAGE-ENTITY: MULTIHOP (openai-small) ############"
+
+for DATASET in musique hotpotqa 2wikimultihopqa; do
+    echo ""
+    echo "====== PE / $DATASET / openai-small ======"
+
+    echo "  [QA-aware] alpha=3"
+    $BASE_PE --task multihop --dataset $DATASET --num_queries 100 --skip_qa \
+        --embedding_model openai-small --qafd_alpha 3.0 2>&1 | grep "Recall@"
+
+    echo "  [QA-agnostic] alpha=3"
+    $BASE_PE --task multihop --dataset $DATASET --num_queries 100 --skip_qa \
+        --embedding_model openai-small --qafd_alpha 3.0 --qafd_weight_scheme none 2>&1 | grep "Recall@"
+done
+
+# =====================================================================
+# 2. PASSAGE-ENTITY: Multihop with LOCAL embeddings (50q, alpha=3)
+# =====================================================================
+echo ""
+echo "############ PASSAGE-ENTITY: MULTIHOP (local embeddings) ############"
+
+echo ""
+echo "====== PE / musique / jina-v3 ======"
+echo "  [QA-aware]"
+$BASE_PE --task multihop --dataset musique --num_queries 50 --skip_qa \
+    --embedding_model jina-v3 --qafd_alpha 3.0 2>&1 | grep "Recall@"
+echo "  [QA-agnostic]"
+$BASE_PE --task multihop --dataset musique --num_queries 50 --skip_qa \
+    --embedding_model jina-v3 --qafd_alpha 3.0 --qafd_weight_scheme none 2>&1 | grep "Recall@"
+
+echo ""
+echo "====== PE / musique / nvidia-nv-embed-v2 ======"
+echo "  [QA-aware]"
+$BASE_PE --task multihop --dataset musique --num_queries 50 --skip_qa \
+    --embedding_model nvidia-nv-embed-v2 --qafd_alpha 3.0 2>&1 | grep "Recall@"
+echo "  [QA-agnostic]"
+$BASE_PE --task multihop --dataset musique --num_queries 50 --skip_qa \
+    --embedding_model nvidia-nv-embed-v2 --qafd_alpha 3.0 --qafd_weight_scheme none 2>&1 | grep "Recall@"
+
+echo ""
+echo "====== PE / hotpotqa / gritlm ======"
+echo "  [QA-aware]"
+$BASE_PE --task multihop --dataset hotpotqa --num_queries 50 --skip_qa \
+    --embedding_model gritlm --qafd_alpha 3.0 2>&1 | grep "Recall@"
+echo "  [QA-agnostic]"
+$BASE_PE --task multihop --dataset hotpotqa --num_queries 50 --skip_qa \
+    --embedding_model gritlm --qafd_alpha 3.0 --qafd_weight_scheme none 2>&1 | grep "Recall@"
+
+# =====================================================================
+# 3. PASSAGE-ENTITY: UltraDomain (10q, quality scores)
+# =====================================================================
+echo ""
+echo "############ PASSAGE-ENTITY: ULTRADOMAIN ############"
+
+for DATASET in mix agriculture cs; do
+    echo ""
+    echo "====== PE / ultradomain / $DATASET / openai-small ======"
+
+    echo "  [QA-aware]"
+    $BASE_PE --task ultradomain --dataset $DATASET --num_queries 10 \
+        --embedding_model openai-small --qafd_alpha 3.0 2>&1 | grep -E "comprehensiveness|diversity|logicality|relevance|coherence|Overall"
+
+    echo "  [QA-agnostic]"
+    $BASE_PE --task ultradomain --dataset $DATASET --num_queries 10 \
+        --embedding_model openai-small --qafd_alpha 3.0 --qafd_weight_scheme none 2>&1 | grep -E "comprehensiveness|diversity|logicality|relevance|coherence|Overall"
+done
+
+echo ""
+echo "====== PE / ultradomain / mix / jina-v3 ======"
+echo "  [QA-aware]"
+$BASE_PE --task ultradomain --dataset mix --num_queries 10 \
+    --embedding_model jina-v3 --qafd_alpha 3.0 2>&1 | grep -E "comprehensiveness|diversity|logicality|relevance|coherence|Overall"
+echo "  [QA-agnostic]"
+$BASE_PE --task ultradomain --dataset mix --num_queries 10 \
+    --embedding_model jina-v3 --qafd_alpha 3.0 --qafd_weight_scheme none 2>&1 | grep -E "comprehensiveness|diversity|logicality|relevance|coherence|Overall"
+
+# =====================================================================
+# 4. ENTITY GRAPH: Multihop (100q, alpha=3 vs alpha=2)
+# =====================================================================
+echo ""
+echo "############ ENTITY GRAPH: MULTIHOP ############"
+
+for DATASET in musique hotpotqa; do
+    echo ""
+    echo "====== Entity / $DATASET ======"
+
+    echo "  [alpha=3]"
+    QAFD_CONDA_ENV=qafd-rag ./run.sh multihop --dataset $DATASET --questions 100 --alpha 3.0 2>&1 | grep -E "F1 Score|Exact Match"
+
+    echo "  [alpha=2]"
+    QAFD_CONDA_ENV=qafd-rag ./run.sh multihop --dataset $DATASET --questions 100 --alpha 2.0 2>&1 | grep -E "F1 Score|Exact Match"
+done
+
+# =====================================================================
+# 5. BATCH PUSH + ALPHA=3 (passage-entity, 100q)
+# =====================================================================
+echo ""
+echo "############ BATCH PUSH + ALPHA=3 ############"
+
+for DATASET in musique hotpotqa; do
+    echo ""
+    echo "====== Batch / $DATASET / openai-small ======"
+
+    echo "  [QA-aware, batch]"
+    $BASE_PE --task multihop --dataset $DATASET --num_queries 100 --skip_qa \
+        --embedding_model openai-small --qafd_alpha 3.0 --batch_push 2>&1 | grep "Recall@"
+
+    echo "  [QA-agnostic, batch]"
+    $BASE_PE --task multihop --dataset $DATASET --num_queries 100 --skip_qa \
+        --embedding_model openai-small --qafd_alpha 3.0 --batch_push --qafd_weight_scheme none 2>&1 | grep "Recall@"
+done
+
+echo ""
+echo "======================================================================"
+echo "  ALL EXPERIMENTS COMPLETE"
+echo "  Finished: $(date)"
+echo "======================================================================"

experiments/run_qa_awareness_v2.sh

@@ -0,0 +1,34 @@
+#!/bin/bash
+# Query-Awareness Ablation V2: warm walk, accumulation, more steps
+cd /home/davoud/QAFD-RAG
+N=100
+BASE="python src/hipporag_pipeline/benchmark_runner.py --task multihop --num_queries $N --embedding_model openai-small --skip_qa"
+
+echo "======================================================================"
+echo "  Query-Awareness V2 Ablation ($N queries per dataset)"
+echo "======================================================================"
+
+for DATASET in musique hotpotqa 2wikimultihopqa; do
+    echo ""
+    echo "====== Dataset: $DATASET ======"
+
+    echo "  [1] Baseline (original)"
+    $BASE --dataset $DATASET 2>&1 | grep -E "Recall@|Retrieval done"
+
+    echo "  [2] QA Warm Walk (edge weights in warm-start)"
+    $BASE --dataset $DATASET --qa_warm_walk 2>&1 | grep -E "Recall@|Retrieval done"
+
+    echo "  [3] QA Warm Walk + 5 steps"
+    $BASE --dataset $DATASET --qa_warm_walk --qa_warm_steps 5 2>&1 | grep -E "Recall@|Retrieval done"
+
+    echo "  [4] QA Accumulation gamma=0.5"
+    $BASE --dataset $DATASET --qa_accum_gamma 0.5 2>&1 | grep -E "Recall@|Retrieval done"
+
+    echo "  [5] QA Warm Walk + Accumulation gamma=0.5"
+    $BASE --dataset $DATASET --qa_warm_walk --qa_accum_gamma 0.5 2>&1 | grep -E "Recall@|Retrieval done"
+done
+
+echo ""
+echo "======================================================================"
+echo "  ALL EXPERIMENTS COMPLETE"
+echo "======================================================================"

experiments/run_sim_mode_ablation.sh

@@ -0,0 +1,29 @@
+#!/bin/bash
+# Edge weight ablation: similarity contrast functions across 3 datasets
+# Tests whether sharper similarity contrast makes query-aware edge weights effective
+
+cd /home/davoud/QAFD-RAG
+N=100
+BASE="python src/hipporag_pipeline/benchmark_runner.py --task multihop --num_queries $N --embedding_model openai-small --skip_qa"
+
+echo "======================================================================"
+echo "  Similarity Mode Ablation ($N queries per dataset)"
+echo "======================================================================"
+
+for DATASET in musique hotpotqa 2wikimultihopqa; do
+    echo ""
+    echo "======================================================================"
+    echo "  Dataset: $DATASET"
+    echo "======================================================================"
+
+    for SIM in normalized relu relu_sq; do
+        echo ""
+        echo "  [$DATASET] sim_mode=$SIM"
+        $BASE --dataset $DATASET --sim_mode $SIM 2>&1 | grep -E "Recall@|Retrieval done"
+    done
+done
+
+echo ""
+echo "======================================================================"
+echo "  ALL EXPERIMENTS COMPLETE"
+echo "======================================================================"

requirements.txt

@@ -0,0 +1,46 @@
+# QAFD-RAG Dependencies
+# Install:
+#   conda create -n qafd-rag python=3.10 -y
+#   conda activate qafd-rag
+#   pip install -r requirements.txt
+
+# --- Core ---
+openai>=1.30.0,<2.0
+tenacity>=8.0.0,<9.0
+aiohttp>=3.9.0,<4.0
+pydantic>=2.0.0,<3.0
+numpy>=1.26.0,<2.0
+tiktoken>=0.7.0,<1.0
+networkx>=3.0,<4.0
+graspologic>=3.4.0,<4.0
+nano-vectordb>=0.0.4
+tqdm>=4.60.0
+nest-asyncio>=1.5.0
+python-dotenv>=1.0.0
+httpx>=0.27.0,<1.0
+huggingface_hub>=0.25.0
+
+# --- Graph (passage-entity pipeline) ---
+python-igraph>=0.11.0,<1.0
+pyarrow>=15.0.0
+datasets>=3.0.0
+
+# --- Embedding Models ---
+torch>=2.0.0
+transformers==4.45.2
+sentence-transformers>=3.0.0,<4.0
+einops>=0.7.0
+gritlm>=1.0.0
+regex>=2024.0.0
+
+# --- Data Processing ---
+pandas>=2.0.0
+pyvis>=0.3.0
+
+# --- LLM Backends ---
+ollama>=0.5.0
+aioboto3>=11.0.0
+urllib3>=1.26.0,<2.0
+
+# --- Benchmark Evaluation ---
+rouge-score>=0.1.2

run.sh

@@ -0,0 +1,113 @@
+#!/bin/bash
+# QAFD-RAG Benchmark Runner
+# Unified interface for all benchmarks
+
+set -euo pipefail
+
+cd "$(dirname "${BASH_SOURCE[0]}")" || { echo "Error: Cannot cd to script directory"; exit 1; }
+
+# Conda environment (has torch, transformers, sentence-transformers, etc.)
+CONDA_ENV="${QAFD_CONDA_ENV:-qafd-rag}"
+PYTHON="conda run --live-stream -n ${CONDA_ENV} python"
+
+# Set NVIDIA library paths (needed for torch with pip-installed CUDA libs)
+NVIDIA_LIB="$(conda info --base)/envs/${CONDA_ENV}/lib/python3.10/site-packages/nvidia"
+if [ -d "$NVIDIA_LIB" ]; then
+    export LD_LIBRARY_PATH="${NVIDIA_LIB}/cudnn/lib:${NVIDIA_LIB}/cublas/lib:${NVIDIA_LIB}/cuda_runtime/lib:${NVIDIA_LIB}/cuda_cupti/lib:${NVIDIA_LIB}/cuda_nvrtc/lib:${NVIDIA_LIB}/cufft/lib:${NVIDIA_LIB}/curand/lib:${NVIDIA_LIB}/cusolver/lib:${NVIDIA_LIB}/cusparse/lib:${NVIDIA_LIB}/nccl/lib:${NVIDIA_LIB}/nvjitlink/lib:${LD_LIBRARY_PATH:-}"
+fi
+
+# Load API key from .env if not already set
+if [ -z "${OPENAI_API_KEY:-}" ]; then
+    if [ -f .env ]; then
+        eval "$(grep -E '^OPENAI_API_KEY=' .env | head -1)"
+        export OPENAI_API_KEY
+    fi
+    if [ -z "${OPENAI_API_KEY:-}" ]; then
+        echo "Error: OPENAI_API_KEY not set. Export it or add it to a .env file."
+        exit 1
+    fi
+fi
+
+# Show help
+show_help() {
+    echo "Usage: ./run.sh <task> [options]"
+    echo ""
+    echo "Tasks (graph type):"
+    echo "  ultradomain      QA benchmark on UltraDomain dataset       [entity graph]"
+    echo "  text2sql         Text-to-SQL benchmark                     [entity graph]"
+    echo "  summarization    Document summarization QA benchmark        [entity graph]"
+    echo "  multihop         Multi-hop QA benchmark (MuSiQue, etc.)    [entity graph]"
+    echo ""
+    echo "Note: run.sh uses the entity graph pipeline. For the passage-entity graph"
+    echo "  (recommended for multihop), use the unified runner instead:"
+    echo "  python benchmarks/run.py --task multihop --dataset musique --questions 100"
+    echo ""
+    echo "Common Options (all tasks):"
+    echo "  --questions N        Number of questions to benchmark"
+    echo "  --build              Build KG only, don't run benchmark"
+    echo "  --force-build        Force rebuild KG even if exists"
+    echo "  --max-documents N    Max documents for KG building"
+    echo "  --embedding MODEL    Embedding model (openai-small, openai-large, jina-v3)"
+    echo "  --llm MODEL          LLM model (gpt-4o-mini, gpt-4o, gpt-5-nano, gpt-5-mini, gpt-5, gpt-oss-120b)"
+    echo ""
+    echo "Summarization-specific Options:"
+    echo "  --dataset NAME       Dataset: squality (default: squality)"
+    echo ""
+    echo "Multihop-specific Options:"
+    echo "  --dataset NAME       Dataset: musique, hotpotqa, 2wikimultihopqa"
+    echo ""
+    echo "Text2SQL-specific Options:"
+    echo "  --benchmark NAME     Benchmark: spider2-lite (default) or bird"
+    echo "  --db NAME            Database name (e.g., Pagila, superhero)"
+    echo ""
+    echo "Text2SQL End-to-End Pipeline:"
+    echo "  Place your .sqlite file in data/text2sql/<benchmark>/databases/<DB>/<DB>.sqlite"
+    echo "  The DB summary (JSON) is auto-generated on first run."
+    echo ""
+    echo "Examples:"
+    echo "  # Build KG"
+    echo "  ./run.sh ultradomain --build --max-documents 100"
+    echo "  ./run.sh summarization --build --max-documents 10"
+    echo "  ./run.sh multihop --dataset musique --build --max-documents 500"
+    echo "  ./run.sh multihop --dataset hotpotqa --build --max-documents 500"
+    echo ""
+    echo "  # Run benchmark"
+    echo "  ./run.sh ultradomain --questions 10"
+    echo "  ./run.sh summarization --questions 50"
+    echo "  ./run.sh multihop --dataset musique --questions 100"
+    echo "  ./run.sh multihop --dataset hotpotqa --questions 100"
+    echo "  ./run.sh multihop --dataset 2wikimultihopqa --questions 100"
+    echo "  ./run.sh text2sql --questions 5 --db Pagila"
+    echo "  ./run.sh text2sql --benchmark bird --questions 5 --db superhero"
+    echo ""
+    echo "  # Use a local OpenAI-compatible model"
+    echo "  export LOCAL_LLM_BASE_URL='http://localhost:8000/v1'"
+    echo "  ./run.sh ultradomain --questions 10 --llm gpt-oss-120b"
+    echo "  ./run.sh multihop --dataset musique --questions 10 --llm gpt-oss-120b"
+}
+
+case "${1:-help}" in
+    ultradomain)
+        shift
+        $PYTHON benchmarks/ultradomain/benchmark_ultradomain.py "$@"
+        ;;
+    text2sql)
+        shift
+        $PYTHON benchmarks/text2sql/benchmark_text2sql.py "$@"
+        ;;
+    summarization)
+        shift
+        $PYTHON benchmarks/summarization/benchmark_summarization.py "$@"
+        ;;
+    multihop)
+        shift
+        $PYTHON benchmarks/multihop/benchmark_multihop.py "$@"
+        ;;
+    help|--help|-h)
+        show_help
+        ;;
+    *)
+        echo "Error: Unknown task '${1}'. Run './run.sh --help' for usage."
+        exit 1
+        ;;
+esac

src/QAFD_RAG.py

@@ -0,0 +1,747 @@
+import asyncio
+import os
+from tqdm.asyncio import tqdm as tqdm_async
+from dataclasses import asdict, dataclass, field
+from datetime import datetime
+from functools import partial
+from typing import Type, cast, List, Dict, Any, Optional
+
+from .llm import (
+    gpt_4o_mini_complete,
+    gpt_oss_120b_complete,
+    local_sentence_embedding,
+    openai_cloud_embedding,
+    is_local_model,
+    is_cloud_model,
+)
+
+# Try to import the new functions, but don't fail if they don't exist
+try:
+    from .llm import get_embedding_func_for_model, EMBEDDING_CONFIGS
+    HAS_EMBEDDING_CONFIGS = True
+except ImportError:
+    HAS_EMBEDDING_CONFIGS = False
+    print("Warning: get_embedding_func_for_model not found in llm.py")
+    print("Please update your llm.py file with the new version")
+
+from .operate import (
+    chunking_by_token_size,
+    extract_entities,
+    kg_query,
+)
+from .indexing import DatabaseSchemaBuilder
+
+from .utils import (
+    EmbeddingFunc,
+    compute_mdhash_id,
+    limit_async_func_call,
+    convert_response_to_json,
+    logger,
+    set_logger,
+)
+from .base import (
+    BaseGraphStorage,
+    BaseKVStorage,
+    BaseVectorStorage,
+    StorageNameSpace,
+    QueryParam,
+)
+
+from .storage import (
+    JsonKVStorage,
+    NanoVectorDBStorage,
+    NetworkXStorage,
+)
+
+
+async def abuild_from_excel_files(self, excel_paths: List[str]) -> Dict[str, Any]:
+    """Build KG from Excel files"""
+    from .indexing import ExcelSchemaBuilder
+    
+    builder = ExcelSchemaBuilder(
+        graph_storage=self.chunk_entity_relation_graph,
+        entities_vdb=self.entities_vdb,
+        relationships_vdb=self.relationships_vdb
+    )
+    
+    result = await builder.build_from_excel_files(excel_paths)
+    await self._insert_done()
+    
+    logger.info(f"Excel KG build completed: {result}")
+    return result
+
+
+def build_from_excel_files(self, excel_paths: List[str]):
+    """Sync wrapper"""
+    loop = always_get_an_event_loop()
+    return loop.run_until_complete(self.abuild_from_excel_files(excel_paths))
+
+
+def lazy_external_import(module_name: str, class_name: str):
+    """Lazily import a class from an external module based on the package of the caller."""
+    import inspect
+
+    caller_frame = inspect.currentframe().f_back
+    module = inspect.getmodule(caller_frame)
+    package = module.__package__ if module else None
+
+    def import_class(*args, **kwargs):
+        import importlib
+        module = importlib.import_module(module_name, package=package)
+        cls = getattr(module, class_name)
+        return cls(*args, **kwargs)
+
+    return import_class
+
+
+Neo4JStorage = lazy_external_import(".kg.neo4j_impl", "Neo4JStorage")
+OracleKVStorage = lazy_external_import(".kg.oracle_impl", "OracleKVStorage")
+OracleGraphStorage = lazy_external_import(".kg.oracle_impl", "OracleGraphStorage")
+OracleVectorDBStorage = lazy_external_import(".kg.oracle_impl", "OracleVectorDBStorage")
+MilvusVectorDBStorge = lazy_external_import(".kg.milvus_impl", "MilvusVectorDBStorge")
+MongoKVStorage = lazy_external_import(".kg.mongo_impl", "MongoKVStorage")
+ChromaVectorDBStorage = lazy_external_import(".kg.chroma_impl", "ChromaVectorDBStorage")
+TiDBKVStorage = lazy_external_import(".kg.tidb_impl", "TiDBKVStorage")
+TiDBVectorDBStorage = lazy_external_import(".kg.tidb_impl", "TiDBVectorDBStorage")
+AGEStorage = lazy_external_import(".kg.age_impl", "AGEStorage")
+
+
+def always_get_an_event_loop() -> asyncio.AbstractEventLoop:
+    """
+    Ensure that there is always an event loop available.
+
+    This function tries to get the current event loop. If the current event loop is closed or does not exist,
+    it creates a new event loop and sets it as the current event loop.
+
+    Returns:
+        asyncio.AbstractEventLoop: The current or newly created event loop.
+    """
+    try:
+        current_loop = asyncio.get_event_loop()
+        if current_loop.is_closed():
+            raise RuntimeError("Event loop is closed.")
+        return current_loop
+    except RuntimeError:
+        logger.info("Creating a new event loop in main thread.")
+        new_loop = asyncio.new_event_loop()
+        asyncio.set_event_loop(new_loop)
+        return new_loop
+
+
+@dataclass
+class QAFD_RAG:
+    working_dir: str = field(
+        default_factory=lambda: f"./QAFD_RAG_cache_{datetime.now().strftime('%Y-%m-%d-%H:%M:%S')}"
+    )
+
+    embedding_cache_config: dict = field(
+        default_factory=lambda: {
+            "enabled": False,
+            "similarity_threshold": 0.95,
+            "use_llm_check": False,
+        }
+    )
+    kv_storage: str = field(default="JsonKVStorage")
+    vector_storage: str = field(default="NanoVectorDBStorage")
+    graph_storage: str = field(default="NetworkXStorage")
+
+    current_log_level = logger.level
+    log_level: str = field(default=current_log_level)
+
+    # Chunking parameters
+    chunk_token_size: int = 1200
+    chunk_overlap_token_size: int = 100
+    tiktoken_model_name: str = "gpt-4o-mini"
+
+    # Entity extraction parameters
+    entity_extract_max_gleaning: int = 1
+    entity_summary_to_max_tokens: int = 5000
+
+    # Node embedding algorithm
+    node_embedding_algorithm: str = "node2vec"
+    node2vec_params: dict = field(
+        default_factory=lambda: {
+            "dimensions": 1536,
+            "num_walks": 10,
+            "walk_length": 40,
+            "window_size": 2,
+            "iterations": 3,
+            "random_seed": 3,
+        }
+    )
+
+    # ============================================================================
+    # EMBEDDING CONFIGURATION (NEW: Configurable embedding models)
+    # ============================================================================
+    
+    # Embedding model key (from EMBEDDING_CONFIGS in llm.py)
+    embedding_model_key: Optional[str] = None
+    
+    # Embedding function and dimensions (auto-configured from embedding_model_key)
+    embedding_func: Optional[EmbeddingFunc] = None
+    embedding_dim: Optional[int] = None
+    
+    # Embedding parameters
+    embedding_batch_num: int = 32
+    embedding_func_max_async: int = 16
+    max_embed_tokens: int = 8192  # Maximum tokens per embedding request
+
+    # ============================================================================
+    # LLM CONFIGURATION
+    # ============================================================================
+    
+    llm_model_func: callable = gpt_4o_mini_complete
+    llm_model_name: str = "gpt-4o-mini"
+    llm_model_max_token_size: int = 32768
+    llm_model_max_async: int = 16
+    llm_model_kwargs: dict = field(default_factory=dict)
+
+    # Vector DB storage parameters
+    vector_db_storage_cls_kwargs: dict = field(default_factory=dict)
+
+    enable_llm_cache: bool = True
+
+    # Additional parameters
+    addon_params: dict = field(default_factory=dict)
+    convert_response_to_json_func: callable = convert_response_to_json
+
+    def __post_init__(self):
+        log_file = os.path.join("QAFD_RAG.log")
+        set_logger(log_file)
+        logger.setLevel(self.log_level)
+
+        logger.info(f"Logger initialized for working directory: {self.working_dir}")
+
+        # ============================================================================
+        # EMBEDDING MODEL CONFIGURATION LOGIC (NEW)
+        # ============================================================================
+        
+        if HAS_EMBEDDING_CONFIGS:
+            # Priority 1: Use explicitly passed embedding_model_key
+            if self.embedding_model_key:
+                logger.info(f"[Embedding Config] Using explicit embedding_model_key: {self.embedding_model_key}")
+                embedding_func, embedding_dim, emb_config = get_embedding_func_for_model(self.embedding_model_key)
+                self.embedding_func = embedding_func
+                self.embedding_dim = embedding_dim
+                logger.info(f"[Embedding Config] {emb_config['description']}")
+                logger.info(f"[Embedding Config] Dimensions: {embedding_dim}, Max tokens: {emb_config['max_tokens']}")
+            
+            # Priority 2: Check environment variable EMBEDDING_MODEL_KEY
+            elif os.environ.get("EMBEDDING_MODEL_KEY"):
+                embedding_key = os.environ.get("EMBEDDING_MODEL_KEY")
+                logger.info(f"[Embedding Config] Using env EMBEDDING_MODEL_KEY: {embedding_key}")
+                embedding_func, embedding_dim, emb_config = get_embedding_func_for_model(embedding_key)
+                self.embedding_func = embedding_func
+                self.embedding_dim = embedding_dim
+                self.embedding_model_key = embedding_key
+                logger.info(f"[Embedding Config] {emb_config['description']}")
+            
+            # Priority 3: Check environment variable USE_OPENAI_EMBEDDINGS (legacy)
+            elif os.environ.get("USE_OPENAI_EMBEDDINGS") == "1":
+                logger.info(f"[Embedding Config] Using legacy USE_OPENAI_EMBEDDINGS=1")
+                self.embedding_func = openai_cloud_embedding
+                self.embedding_dim = 1024
+                self.embedding_model_key = "openai-large"
+                logger.info(f"[Embedding Config] OpenAI cloud embeddings (1024-dim)")
+            
+            elif os.environ.get("USE_OPENAI_EMBEDDINGS") == "0":
+                logger.info(f"[Embedding Config] Using legacy USE_OPENAI_EMBEDDINGS=0")
+                self.embedding_func = local_sentence_embedding
+                self.embedding_dim = 1024
+                self.embedding_model_key = "jina-v3"
+                logger.info(f"[Embedding Config] Local Jina v3 embeddings (1024-dim)")
+            
+            # Priority 4: Auto-detect based on LLM model name
+            elif self.llm_model_name:
+                model_name = self.llm_model_name.lower()
+                if is_local_model(model_name):
+                    logger.info(f"[Embedding Config] Local LLM detected ({model_name}) → using local embeddings")
+                    self.embedding_func = local_sentence_embedding
+                    self.embedding_dim = 1024
+                    self.embedding_model_key = "jina-v3"
+                else:
+                    logger.info(f"[Embedding Config] Cloud LLM detected ({model_name}) → using OpenAI embeddings")
+                    self.embedding_func = openai_cloud_embedding
+                    self.embedding_dim = 1024
+                    self.embedding_model_key = "openai-large"
+            
+            # Priority 5: Default to local Jina v3
+            else:
+                logger.info(f"[Embedding Config] No configuration found → defaulting to Jina v3 (local)")
+                self.embedding_func = local_sentence_embedding
+                self.embedding_dim = 1024
+                self.embedding_model_key = "jina-v3"
+        else:
+            # Fallback to old behavior if new functions not available
+            logger.warning("[Embedding Config] Using legacy embedding configuration")
+            env_embedding_setting = os.environ.get("USE_OPENAI_EMBEDDINGS")
+            
+            if env_embedding_setting == "0":
+                self.embedding_func = local_sentence_embedding
+                self.embedding_dim = 1024
+                logger.info(f"[Embedding Override] Using local embeddings (1024-dim) - forced by USE_OPENAI_EMBEDDINGS=0")
+            elif env_embedding_setting == "1":
+                self.embedding_func = openai_cloud_embedding
+                self.embedding_dim = 1024
+                logger.info(f"[Embedding Override] Using OpenAI embeddings (1024-dim) - forced by USE_OPENAI_EMBEDDINGS=1")
+            elif is_local_model(self.llm_model_name.lower() if self.llm_model_name else ""):
+                self.embedding_func = local_sentence_embedding
+                self.embedding_dim = 1024
+                logger.info(f"[Embedding] Local model detected → using local embeddings (1024-dim)")
+            else:
+                self.embedding_func = openai_cloud_embedding
+                self.embedding_dim = 1024
+                logger.info(f"[Embedding] Cloud model detected → using OpenAI embeddings (1024-dim)")
+        
+        # Validate embedding function was set
+        if self.embedding_func is None:
+            logger.error("[Embedding Config] Failed to configure embedding function!")
+            raise ValueError("Embedding function not configured")
+        
+        if self.embedding_dim is None:
+            self.embedding_dim = 1024  # Default
+            logger.warning(f"[Embedding Config] embedding_dim not set, defaulting to 1024")
+        
+        logger.info(f"[Embedding Config] ✅ Final: {self.embedding_model_key if self.embedding_model_key else 'auto'} ({self.embedding_dim}-dim)")
+
+        # ============================================================================
+        # STORAGE INITIALIZATION
+        # ============================================================================
+        
+        self.key_string_value_json_storage_cls: Type[BaseKVStorage] = (
+            self._get_storage_class()[self.kv_storage]
+        )
+        self.vector_db_storage_cls: Type[BaseVectorStorage] = self._get_storage_class()[
+            self.vector_storage
+        ]
+        self.graph_storage_cls: Type[BaseGraphStorage] = self._get_storage_class()[
+            self.graph_storage
+        ]
+
+        if not os.path.exists(self.working_dir):
+            logger.info(f"Creating working directory {self.working_dir}")
+            os.makedirs(self.working_dir)
+
+        self.llm_response_cache = (
+            self.key_string_value_json_storage_cls(
+                namespace="llm_response_cache",
+                global_config=asdict(self),
+                embedding_func=None,
+            )
+            if self.enable_llm_cache
+            else None
+        )
+        
+        # Limit async calls for embedding function
+        self.embedding_func = limit_async_func_call(self.embedding_func_max_async)(
+            self.embedding_func
+        )
+
+        # Initialize storage components with embedding function
+        self.full_docs = self.key_string_value_json_storage_cls(
+            namespace="full_docs",
+            global_config=asdict(self),
+            embedding_func=self.embedding_func,
+        )
+        self.text_chunks = self.key_string_value_json_storage_cls(
+            namespace="text_chunks",
+            global_config=asdict(self),
+            embedding_func=self.embedding_func,
+        )
+        self.chunk_entity_relation_graph = self.graph_storage_cls(
+            namespace="chunk_entity_relation",
+            global_config=asdict(self),
+            embedding_func=self.embedding_func,
+        )
+
+        # Vector databases for entities, relationships, and chunks
+        self.entities_vdb = self.vector_db_storage_cls(
+            namespace="entities",
+            global_config=asdict(self),
+            embedding_func=self.embedding_func,
+            meta_fields={"entity_name"},
+        )
+        self.relationships_vdb = self.vector_db_storage_cls(
+            namespace="relationships",
+            global_config=asdict(self),
+            embedding_func=self.embedding_func,
+            meta_fields={"src_id", "tgt_id"},
+        )
+        self.chunks_vdb = self.vector_db_storage_cls(
+            namespace="chunks",
+            global_config=asdict(self),
+            embedding_func=self.embedding_func,
+        )
+
+        # Configure LLM function
+        self.llm_model_func = limit_async_func_call(self.llm_model_max_async)(
+            partial(
+                self.llm_model_func,
+                hashing_kv=self.llm_response_cache
+                if self.llm_response_cache
+                and hasattr(self.llm_response_cache, "global_config")
+                else self.key_string_value_json_storage_cls(
+                    global_config=asdict(self),
+                ),
+                **self.llm_model_kwargs,
+            )
+        )
+        
+        # Initialize database schema builder
+        self.schema_builder = DatabaseSchemaBuilder(
+            graph_storage=self.chunk_entity_relation_graph,
+            entities_vdb=self.entities_vdb,
+            relationships_vdb=self.relationships_vdb,
+            llm_model_func=self.llm_model_func
+        )
+
+    def _get_storage_class(self) -> dict[str, Type]:
+        return {
+            # Key-Value Storage
+            "JsonKVStorage": JsonKVStorage,
+            "OracleKVStorage": OracleKVStorage,
+            "MongoKVStorage": MongoKVStorage,
+            "TiDBKVStorage": TiDBKVStorage,
+            # Vector Storage
+            "NanoVectorDBStorage": NanoVectorDBStorage,
+            "OracleVectorDBStorage": OracleVectorDBStorage,
+            "MilvusVectorDBStorge": MilvusVectorDBStorge,
+            "ChromaVectorDBStorage": ChromaVectorDBStorage,
+            "TiDBVectorDBStorage": TiDBVectorDBStorage,
+            # Graph Storage
+            "NetworkXStorage": NetworkXStorage,
+            "Neo4JStorage": Neo4JStorage,
+            "OracleGraphStorage": OracleGraphStorage,
+            "AGEStorage": AGEStorage,
+        }
+
+    def insert(self, string_or_strings, addon_params=None):
+        loop = always_get_an_event_loop()
+        return loop.run_until_complete(self.ainsert(string_or_strings, addon_params))
+
+    async def ainsert(self, string_or_strings, addon_params=None):
+        update_storage = False
+        try:
+            if isinstance(string_or_strings, str):
+                string_or_strings = [string_or_strings]
+
+            new_docs = {
+                compute_mdhash_id(c.strip(), prefix="doc-"): {"content": c.strip()}
+                for c in string_or_strings
+            }
+            _add_doc_keys = await self.full_docs.filter_keys(list(new_docs.keys()))
+            new_docs = {k: v for k, v in new_docs.items() if k in _add_doc_keys}
+            if not len(new_docs):
+                logger.warning("All docs are already in the storage")
+                return
+            update_storage = True
+            logger.info(f"[New Docs] inserting {len(new_docs)} docs")
+
+            inserting_chunks = {}
+            for doc_key, doc in tqdm_async(
+                new_docs.items(), desc="Chunking documents", unit="doc"
+            ):
+                chunks = {
+                    compute_mdhash_id(dp["content"], prefix="chunk-"): {
+                        **dp,
+                        "full_doc_id": doc_key,
+                    }
+                    for dp in chunking_by_token_size(
+                        doc["content"],
+                        overlap_token_size=self.chunk_overlap_token_size,
+                        max_token_size=self.chunk_token_size,
+                        tiktoken_model=self.tiktoken_model_name,
+                    )
+                }
+                inserting_chunks.update(chunks)
+            _add_chunk_keys = await self.text_chunks.filter_keys(
+                list(inserting_chunks.keys())
+            )
+            inserting_chunks = {
+                k: v for k, v in inserting_chunks.items() if k in _add_chunk_keys
+            }
+            if not len(inserting_chunks):
+                logger.warning("All chunks are already in the storage")
+                return
+            logger.info(f"[New Chunks] inserting {len(inserting_chunks)} chunks")
+
+            await self.chunks_vdb.upsert(inserting_chunks)
+
+            logger.info("[Entity Extraction]...")
+            
+            # Create a temporary config with custom addon_params if provided
+            temp_config = asdict(self)
+            if addon_params is not None:
+                temp_config["addon_params"] = addon_params
+            
+            maybe_new_kg = await extract_entities(
+                inserting_chunks,
+                knowledge_graph_inst=self.chunk_entity_relation_graph,
+                entity_vdb=self.entities_vdb,
+                relationships_vdb=self.relationships_vdb,
+                global_config=temp_config,
+            )
+            if maybe_new_kg is None:
+                logger.warning("No new entities and relationships found")
+                return
+            self.chunk_entity_relation_graph = maybe_new_kg
+
+            await self.full_docs.upsert(new_docs)
+            await self.text_chunks.upsert(inserting_chunks)
+        finally:
+            if update_storage:
+                await self._insert_done()
+
+    async def _insert_done(self):
+        tasks = []
+        for storage_inst in [
+            self.full_docs,
+            self.text_chunks,
+            self.llm_response_cache,
+            self.entities_vdb,
+            self.relationships_vdb,
+            self.chunks_vdb,
+            self.chunk_entity_relation_graph,
+        ]:
+            if storage_inst is None:
+                continue
+            tasks.append(cast(StorageNameSpace, storage_inst).index_done_callback())
+        await asyncio.gather(*tasks)
+
+    def insert_custom_kg(self, custom_kg: dict):
+        loop = always_get_an_event_loop()
+        return loop.run_until_complete(self.ainsert_custom_kg(custom_kg))
+
+    async def ainsert_custom_kg(self, custom_kg: dict):
+        update_storage = False
+        try:
+            all_chunks_data = {}
+            chunk_to_source_map = {}
+            for chunk_data in custom_kg.get("chunks", []):
+                chunk_content = chunk_data["content"]
+                source_id = chunk_data["source_id"]
+                chunk_id = compute_mdhash_id(chunk_content.strip(), prefix="chunk-")
+
+                chunk_entry = {"content": chunk_content.strip(), "source_id": source_id}
+                all_chunks_data[chunk_id] = chunk_entry
+                chunk_to_source_map[source_id] = chunk_id
+                update_storage = True
+
+            if self.chunks_vdb is not None and all_chunks_data:
+                await self.chunks_vdb.upsert(all_chunks_data)
+            if self.text_chunks is not None and all_chunks_data:
+                await self.text_chunks.upsert(all_chunks_data)
+
+            all_entities_data = []
+            for entity_data in custom_kg.get("entities", []):
+                entity_name = f'"{entity_data["entity_name"].lower()}"'
+                entity_type = entity_data.get("entity_type", "UNKNOWN")
+                description = entity_data.get("description", "No description provided")
+
+                source_chunk_id = entity_data.get("source_id", "UNKNOWN")
+                source_id = chunk_to_source_map.get(source_chunk_id, "UNKNOWN")
+
+                if source_id == "UNKNOWN":
+                    logger.warning(
+                        f"Entity '{entity_name}' has an UNKNOWN source_id. Please check the source mapping."
+                    )
+
+                node_data = {
+                    "entity_type": entity_type,
+                    "description": description,
+                    "source_id": source_id,
+                }
+
+                await self.chunk_entity_relation_graph.upsert_node(
+                    entity_name, node_data=node_data
+                )
+                node_data["entity_name"] = entity_name
+                all_entities_data.append(node_data)
+                update_storage = True
+
+            all_relationships_data = []
+            for relationship_data in custom_kg.get("relationships", []):
+                src_id = f'"{relationship_data["src_id"].lower()}"'
+                tgt_id = f'"{relationship_data["tgt_id"].lower()}"'
+                description = relationship_data["description"]
+                keywords = relationship_data["keywords"]
+                weight = relationship_data.get("weight", 1.0)
+
+                source_chunk_id = relationship_data.get("source_id", "UNKNOWN")
+                source_id = chunk_to_source_map.get(source_chunk_id, "UNKNOWN")
+
+                if source_id == "UNKNOWN":
+                    logger.warning(
+                        f"Relationship from '{src_id}' to '{tgt_id}' has an UNKNOWN source_id. Please check the source mapping."
+                    )
+
+                for need_insert_id in [src_id, tgt_id]:
+                    if not (
+                        await self.chunk_entity_relation_graph.has_node(need_insert_id)
+                    ):
+                        await self.chunk_entity_relation_graph.upsert_node(
+                            need_insert_id,
+                            node_data={
+                                "source_id": source_id,
+                                "description": "UNKNOWN",
+                                "entity_type": "UNKNOWN",
+                            },
+                        )
+
+                await self.chunk_entity_relation_graph.upsert_edge(
+                    src_id,
+                    tgt_id,
+                    edge_data={
+                        "weight": weight,
+                        "description": description,
+                        "keywords": keywords,
+                        "source_id": source_id,
+                    },
+                )
+                edge_data = {
+                    "src_id": src_id,
+                    "tgt_id": tgt_id,
+                    "description": description,
+                    "keywords": keywords,
+                }
+                all_relationships_data.append(edge_data)
+                update_storage = True
+
+            if self.entities_vdb is not None:
+                data_for_vdb = {
+                    compute_mdhash_id(dp["entity_name"], prefix="ent-"): {
+                        "content": dp["entity_name"] + dp["description"],
+                        "entity_name": dp["entity_name"],
+                    }
+                    for dp in all_entities_data
+                }
+                await self.entities_vdb.upsert(data_for_vdb)
+
+            if self.relationships_vdb is not None:
+                data_for_vdb = {
+                    compute_mdhash_id(dp["src_id"] + dp["tgt_id"], prefix="rel-"): {
+                        "src_id": dp["src_id"],
+                        "tgt_id": dp["tgt_id"],
+                        "content": dp["keywords"]
+                        + dp["src_id"]
+                        + dp["tgt_id"]
+                        + dp["description"],
+                    }
+                    for dp in all_relationships_data
+                }
+                await self.relationships_vdb.upsert(data_for_vdb)
+        finally:
+            if update_storage:
+                await self._insert_done()
+    
+    def query(self, query: str, param: QueryParam = QueryParam()):
+        loop = always_get_an_event_loop()
+        return loop.run_until_complete(self.aquery(query, param))
+    
+    async def aquery(self, query: str, param: QueryParam = QueryParam()):
+        if param.mode in ["local", "global", "hybrid"]:
+            response = await kg_query(
+                query,
+                self.chunk_entity_relation_graph,
+                self.entities_vdb,
+                self.relationships_vdb,
+                self.text_chunks,
+                param,
+                asdict(self),
+                hashing_kv=self.llm_response_cache
+                if self.llm_response_cache
+                and hasattr(self.llm_response_cache, "global_config")
+                else self.key_string_value_json_storage_cls(
+                    global_config=asdict(self),
+                ),
+            )
+        else:
+            raise ValueError(f"Unknown mode {param.mode}")
+        await self._query_done()
+        return response
+
+    async def _query_done(self):
+        tasks = []
+        for storage_inst in [self.llm_response_cache]:
+            if storage_inst is None:
+                continue
+            tasks.append(cast(StorageNameSpace, storage_inst).index_done_callback())
+        await asyncio.gather(*tasks)
+
+    def delete_by_entity(self, entity_name: str):
+        loop = always_get_an_event_loop()
+        return loop.run_until_complete(self.adelete_by_entity(entity_name))
+
+    async def adelete_by_entity(self, entity_name: str):
+        entity_name = f'"{entity_name.lower()}"'
+
+        try:
+            await self.entities_vdb.delete_entity(entity_name)
+            await self.relationships_vdb.delete_relation(entity_name)
+            await self.chunk_entity_relation_graph.delete_node(entity_name)
+
+            logger.info(
+                f"Entity '{entity_name}' and its relationships have been deleted."
+            )
+            await self._delete_by_entity_done()
+        except Exception as e:
+            logger.error(f"Error while deleting entity '{entity_name}': {e}")
+
+    async def _delete_by_entity_done(self):
+        tasks = []
+        for storage_inst in [
+            self.entities_vdb,
+            self.relationships_vdb,
+            self.chunk_entity_relation_graph,
+        ]:
+            if storage_inst is None:
+                continue
+            tasks.append(cast(StorageNameSpace, storage_inst).index_done_callback())
+        await asyncio.gather(*tasks)
+
+    def build_from_database_schema(self, 
+                                  schema_file_path: str, 
+                                  metadata_file_path: str = None,
+                                  language: str = "English"):
+        """
+        Build knowledge graph from database schema JSON file
+        
+        This method manually constructs the knowledge graph from a JSON schema file,
+        avoiding the chunking issues that can cause LLM errors. It follows the approach
+        used in CoFD for database schema processing.
+        
+        Args:
+            schema_file_path: Path to the JSON schema file
+            metadata_file_path: Optional path to metadata file
+            language: Output language for descriptions
+            
+        Returns:
+            Dictionary containing build statistics
+        """
+        loop = always_get_an_event_loop()
+        return loop.run_until_complete(self.abuild_from_database_schema(
+            schema_file_path, metadata_file_path, language
+        ))
+
+    async def abuild_from_database_schema(self, 
+                                        schema_file_path: str, 
+                                        metadata_file_path: str = None,
+                                        language: str = "English"):
+        """
+        Async version of build_from_database_schema
+        """
+        try:
+            # Use the schema builder to construct the knowledge graph
+            result = await self.schema_builder.build_from_json_schema(
+                schema_file_path, metadata_file_path, language
+            )
+            
+            # Update storage after building
+            await self._insert_done()
+            
+            logger.info(f"Database schema build completed: {result}")
+            return result
+            
+        except Exception as e:
+            logger.error(f"Error building from database schema: {e}")
+            raise

src/__init__.py

@@ -0,0 +1,4 @@
+from .QAFD_RAG import QAFD_RAG as QAFD_RAG, QueryParam as QueryParam
+
+
+from .QAFD_RAG import QAFD_RAG, QueryParam

src/answering/__init__.py

@@ -0,0 +1,42 @@
+"""
+Answering module for QAFD-RAG.
+
+This module handles the query-time operations: from user question to final answer.
+It coordinates keyword extraction, entity retrieval, graph traversal, context assembly,
+and response generation.
+
+Submodules:
+-----------
+- handler: Main query entry point (kg_query)
+- context: Context building based on query modes
+- clusters: Flow diffusion cluster operations
+- text_units: Text chunk retrieval
+
+Note: Evaluation functions have been moved to src/evaluation.py
+"""
+
+from .handler import kg_query
+from .context import build_query_context
+from .clusters import (
+    get_embeddings_for_flow_diffusion,
+    convert_subgraph_to_json,
+    find_flow_diffusion_clusters_and_summarize,
+    summarize_clusters_batch_with_llm,
+    summarize_cluster_with_llm,
+)
+from .text_units import find_most_related_text_unit_from_entities
+
+__all__ = [
+    # Handler
+    "kg_query",
+    # Context
+    "build_query_context",
+    # Clusters
+    "get_embeddings_for_flow_diffusion",
+    "convert_subgraph_to_json",
+    "find_flow_diffusion_clusters_and_summarize",
+    "summarize_clusters_batch_with_llm",
+    "summarize_cluster_with_llm",
+    # Text units
+    "find_most_related_text_unit_from_entities",
+]

src/answering/clusters.py

@@ -0,0 +1,739 @@
+"""
+Flow diffusion cluster operations for knowledge graph traversal.
+
+This module provides functions to find clusters using flow diffusion,
+convert subgraphs to JSON format and summarize clusters using LLM.
+"""
+
+import asyncio
+import tiktoken
+import networkx as nx
+from ..base import (
+    BaseGraphStorage,
+    QueryParam,
+)
+from ..retrievers import QueryAwareWeightedFlowDiffusion
+from ..utils import (
+    logger,
+    truncate_list_by_token_size,
+)
+
+
+async def get_embeddings_for_flow_diffusion(
+    graph: nx.Graph,
+    query: str,
+    knowledge_graph_inst: BaseGraphStorage,
+    global_config: dict,
+    query_param: QueryParam = None,
+) -> tuple[dict, list]:
+    """
+    Get embeddings for nodes and query for query-aware flow diffusion.
+
+    Returns:
+    --------
+    tuple
+        (node_embeddings, subquery_embedding)
+    """
+    node_embeddings = {}
+    subquery_embedding = None
+
+    # Check if query-aware flow diffusion is enabled
+    if query_param and not query_param.enable_query_aware_flow_diffusion:
+        logger.info("Query-aware flow diffusion is disabled, skipping embedding calculation")
+        return {}, None
+
+    # Try to get embeddings from global config if available
+    if "embedding_func" in global_config and global_config["embedding_func"]:
+        try:
+            # Prioritize using cached node embeddings
+            if hasattr(knowledge_graph_inst, 'get_cached_node_embeddings'):
+                logger.info("Attempting to use cached node embeddings...")
+                node_embeddings = await knowledge_graph_inst.get_cached_node_embeddings(global_config)
+                if node_embeddings:
+                    logger.info(f"Successfully using cached embeddings for {len(node_embeddings)} nodes")
+                else:
+                    logger.info("No cached embeddings found, will compute new ones")
+            else:
+                logger.info("Storage class does not support cached embeddings, computing new ones...")
+                # Get embeddings for all nodes in the graph with batching
+                all_nodes = list(graph.nodes())
+                if all_nodes:
+                    # Get batch size and token limit from config
+                    batch_size = global_config.get("embedding_batch_num", 32)
+                    max_tokens_per_request = global_config.get("max_embed_tokens", 8192)
+
+                    # Prepare node texts
+                    node_texts = []
+                    for node in all_nodes:
+                        node_info = await knowledge_graph_inst.get_node(node)
+                        if node_info and "description" in node_info:
+                            node_texts.append(f"{node} {node_info['description']}")
+                        else:
+                            node_texts.append(node)
+
+                    # Process in batches to avoid token limit
+                    logger.info(f"Computing embeddings for {len(node_texts)} node texts in batches of {batch_size}...")
+
+                    # Import tiktoken for token counting
+                    tiktoken_model = global_config.get("tiktoken_model_name", "gpt-4o-mini")
+                    try:
+                        encoding = tiktoken.encoding_for_model(tiktoken_model)
+                    except:
+                        encoding = tiktoken.get_encoding("cl100k_base")  # fallback
+
+                    # Process batches
+                    all_embeddings = []
+                    for i in range(0, len(node_texts), batch_size):
+                        batch_texts = node_texts[i:i + batch_size]
+                        batch_nodes = all_nodes[i:i + batch_size]
+
+                        # Check token count for this batch
+                        total_tokens = sum(len(encoding.encode(text)) for text in batch_texts)
+
+                        if total_tokens > max_tokens_per_request:
+                            logger.warning(f"Batch {i//batch_size + 1} exceeds token limit ({total_tokens} > {max_tokens_per_request}), reducing batch size...")
+                            # Process this batch with smaller chunks
+                            sub_batch_size = max(1, batch_size // 2)
+                            for j in range(0, len(batch_texts), sub_batch_size):
+                                sub_batch_texts = batch_texts[j:j + sub_batch_size]
+                                sub_batch_nodes = batch_nodes[j:j + sub_batch_size]
+
+                                # Check sub-batch token count
+                                sub_total_tokens = sum(len(encoding.encode(text)) for text in sub_batch_texts)
+                                if sub_total_tokens > max_tokens_per_request:
+                                    logger.warning(f"Sub-batch still exceeds token limit ({sub_total_tokens} > {max_tokens_per_request}), processing one by one...")
+                                    # Process one by one
+                                    for k, (text, node) in enumerate(zip(sub_batch_texts, sub_batch_nodes)):
+                                        try:
+                                            embedding_array = await global_config["embedding_func"]([text])
+                                            all_embeddings.append(embedding_array[0])
+                                            logger.debug(f"Processed node {k+1}/{len(sub_batch_texts)} in sub-batch")
+                                        except Exception as e:
+                                            logger.error(f"Failed to process node {node}: {e}")
+                                            # Add zero embedding as fallback
+                                            embedding_dim = 1536  # default dimension
+                                            all_embeddings.append([0.0] * embedding_dim)
+                                else:
+                                    try:
+                                        embedding_array = await global_config["embedding_func"](sub_batch_texts)
+                                        all_embeddings.extend(embedding_array)
+                                        logger.debug(f"Processed sub-batch {j//sub_batch_size + 1} with {len(sub_batch_texts)} nodes")
+                                    except Exception as e:
+                                        logger.error(f"Failed to process sub-batch: {e}")
+                                        # Add zero embeddings as fallback
+                                        embedding_dim = 1536  # default dimension
+                                        for _ in sub_batch_texts:
+                                            all_embeddings.append([0.0] * embedding_dim)
+                        else:
+                            try:
+                                embedding_array = await global_config["embedding_func"](batch_texts)
+                                all_embeddings.extend(embedding_array)
+                                logger.debug(f"Processed batch {i//batch_size + 1} with {len(batch_texts)} nodes")
+                            except Exception as e:
+                                logger.error(f"Failed to process batch: {e}")
+                                # Add zero embeddings as fallback
+                                embedding_dim = 1536  # default dimension
+                                for _ in batch_texts:
+                                    all_embeddings.append([0.0] * embedding_dim)
+
+                    # Store embeddings
+                    for i, node in enumerate(all_nodes):
+                        if i < len(all_embeddings):
+                            node_embeddings[node] = all_embeddings[i].tolist()
+                        else:
+                            logger.warning(f"Missing embedding for node {node}")
+
+                    logger.info(f"Computed embeddings for {len(node_embeddings)} nodes")
+
+            # Get query embedding
+            if query:
+                # Prioritize using cached query embedding
+                if hasattr(knowledge_graph_inst, 'get_cached_query_embedding'):
+                    logger.info("Attempting to use cached query embedding...")
+                    subquery_embedding = await knowledge_graph_inst.get_cached_query_embedding(query, global_config)
+                    if subquery_embedding is None:
+                        logger.warning("Failed to get cached query embedding, computing new one...")
+                        query_embedding_array = await global_config["embedding_func"]([query])
+                        subquery_embedding = query_embedding_array[0].tolist()
+                        logger.info("Query embedding computed successfully")
+                else:
+                    logger.info("Storage class does not support cached query embeddings, computing new one...")
+                    query_embedding_array = await global_config["embedding_func"]([query])
+                    subquery_embedding = query_embedding_array[0].tolist()
+                    logger.info("Query embedding computed successfully")
+
+        except Exception as e:
+            logger.warning(f"Failed to get embeddings for query-aware flow diffusion: {e}")
+            # Fallback to non-query-aware mode
+            node_embeddings = {}
+            subquery_embedding = None
+    else:
+        logger.warning("No embedding function available in global config")
+
+    return node_embeddings, subquery_embedding
+
+
+def convert_subgraph_to_json(G: nx.Graph, cluster_nodes: list, diffused_nodes: dict, source_node: str) -> dict:
+    """
+    Convert a subgraph to JSON format with nodes and edges information.
+
+    Parameters:
+    -----------
+    G : nx.Graph
+        The original graph
+    cluster_nodes : list
+        List of nodes in the cluster
+    diffused_nodes : dict
+        Dictionary mapping nodes to their flow values
+    source_node : str
+        The source node for this cluster
+
+    Returns:
+    --------
+    dict
+        JSON representation of the subgraph
+    """
+    # Create subgraph from cluster nodes
+    support_nodes = set(cluster_nodes)
+    subgraph = G.subgraph(support_nodes)
+
+    # Convert nodes to JSON format
+    nodes_json = []
+    # Build a local index map so node "id" matches the CSV-style index
+    node_index_map = {node: idx for idx, node in enumerate(cluster_nodes)}
+    for node in cluster_nodes:
+        node_attrs = G.nodes[node] if node in G.nodes else {}
+        node_degree = subgraph.degree(node) if node in subgraph else 0
+        nodes_json.append({
+            "id": node_index_map[node],
+            "entity": node,
+            "entity_type": node_attrs.get("entity_type", "UNKNOWN"),
+            "description": node_attrs.get("description", "UNKNOWN"),
+            "rank": node_degree,
+        })
+
+    # Convert edges to JSON format
+    edges_json = []
+    for u, v, data in subgraph.edges(data=True):
+        edges_json.append({
+            "source": u,
+            "target": v,
+            "source_id": node_index_map.get(u),
+            "target_id": node_index_map.get(v),
+            "weight": data.get('weight', 1.0)
+        })
+
+    return {
+        "source_node": source_node,
+        "cluster_size": len(cluster_nodes),
+        "max_flow_value": max(diffused_nodes.values()) if diffused_nodes else 0.0,
+        "nodes": nodes_json,
+        "edges": edges_json,
+        "total_edges": len(edges_json)
+    }
+
+
+async def find_flow_diffusion_clusters_and_summarize(
+    node_datas: list[dict],
+    query: str,
+    query_param: QueryParam,
+    knowledge_graph_inst: BaseGraphStorage,
+    global_config: dict,
+):
+    """
+    Apply flow diffusion to find clusters and summarize them using LLM.
+
+    Parameters:
+    -----------
+    node_datas : list[dict]
+        List of node data dictionaries
+    query : str
+        The original query that provides context for the relationship analysis
+    query_param : QueryParam
+        Query parameters (includes flow diffusion configuration)
+    knowledge_graph_inst : BaseGraphStorage
+        Knowledge graph storage instance
+    global_config : dict
+        Global configuration
+
+    Returns:
+    --------
+    list
+        If return_raw_clusters=True: List of cluster JSON objects with subgraph data
+        If return_raw_clusters=False: List of summarized cluster relationships
+    """
+    # Use cached NetworkX graph to avoid repeated construction
+    if hasattr(knowledge_graph_inst, 'get_cached_nx_graph'):
+        logger.info("Attempting to use cached NetworkX graph...")
+        G = await knowledge_graph_inst.get_cached_nx_graph()
+        logger.info(f"Successfully obtained NetworkX graph with {G.number_of_nodes()} nodes and {G.number_of_edges()} edges")
+    else:
+        logger.info("Storage class does not support cached graphs, building new one...")
+        # Compatibility handling: if no cache method, use original approach
+        G = nx.Graph()
+        edges = await knowledge_graph_inst.edges()
+        nodes = await knowledge_graph_inst.nodes()
+
+        # Add edges with weights
+        for u, v in edges:
+            edge_data = await knowledge_graph_inst.get_edge(u, v)
+            if edge_data and 'weight' in edge_data:
+                G.add_edge(u, v, weight=edge_data['weight'])
+            else:
+                G.add_edge(u, v, weight=1.0)  # Default weight: 1.0 if not specified
+
+        G.add_nodes_from(nodes)
+        logger.info(f"Built new NetworkX graph with {G.number_of_nodes()} nodes and {G.number_of_edges()} edges")
+
+    # Get source nodes from node_datas (limit to configured maximum)
+    source_nodes = [dp["entity_name"] for dp in node_datas[:query_param.max_source_nodes]]
+
+    # Apply flow diffusion from each source node independently
+    all_clusters = []
+    use_llm_func = global_config["llm_model_func"]
+
+    logger.info(f"Starting flow diffusion from {len(source_nodes)} source nodes independently")
+
+    # Pre-compute node embeddings and query embeddings to avoid repeated computation in loops
+    logger.info("Pre-computing embeddings for flow diffusion...")
+    node_embeddings, subquery_embedding = await get_embeddings_for_flow_diffusion(
+        G, query, knowledge_graph_inst, global_config, query_param
+    )
+    logger.info(f"Pre-computed embeddings: {len(node_embeddings)} nodes, query embedding: {'Yes' if subquery_embedding else 'No'}")
+
+    # Initialize cluster processing based on configuration
+    all_clusters = []
+
+    if query_param.use_batch_cluster_summarization:
+        logger.info("Using batch cluster summarization mode (more efficient)")
+    else:
+        logger.info("Using individual cluster summarization mode (original approach)")
+
+    if query_param.use_batch_cluster_summarization:
+        # Collect all clusters first for batch processing
+        clusters_to_summarize = []
+        raw_clusters = []
+
+        # Run flow diffusion from each source node
+        for source_node in source_nodes:
+
+            if not G.has_node(source_node):
+                continue
+
+            # Get source node information
+            source_node_info = await knowledge_graph_inst.get_node(source_node)
+            if not source_node_info:
+                source_node_info = {"entity_type": "UNKNOWN", "description": "No description available"}
+
+            # Calculate confidence based on source node's relevance to the query
+            confidence = 0.7  # Default confidence
+
+            # Apply flow diffusion from this source node
+            wfd = QueryAwareWeightedFlowDiffusion(
+                G, source_node, source_node, confidence,
+                node_embeddings=node_embeddings,
+                subquery_embedding=subquery_embedding,
+                weight_func=query_param.weight_func
+            )
+            wfd.initialize(alpha=query_param.alpha)
+            diffused_nodes = wfd.flow_diffusion()
+
+            if len(diffused_nodes) > 1:  # Multi-node cluster
+                # Get cluster nodes and their flow values
+                cluster_nodes = list(diffused_nodes.keys())
+                cluster_flow_values = list(diffused_nodes.values())
+
+                # Only process if cluster has significant flow (using configured threshold)
+                if cluster_flow_values:
+                    max_flow = max(cluster_flow_values)
+                    if max_flow < query_param.min_flow_threshold:
+                        continue
+                else:
+                    continue  # Skip if no flow values
+
+                # Get node information for the cluster
+                cluster_node_data = []
+                for node in cluster_nodes:
+                    node_info = await knowledge_graph_inst.get_node(node)
+                    if node_info:
+                        cluster_node_data.append({
+                            'name': node,
+                            'entity_type': node_info.get('entity_type', 'UNKNOWN'),
+                            'description': node_info.get('description', 'UNKNOWN'),
+                            'flow_value': diffused_nodes.get(node, 0.0)
+                        })
+
+                # Sort by flow value
+                cluster_node_data.sort(key=lambda x: x['flow_value'], reverse=True)
+
+                if query_param.return_raw_clusters:
+                    # Convert subgraph to JSON format
+                    cluster_json = convert_subgraph_to_json(G, cluster_nodes, diffused_nodes, source_node)
+                    # Add node details to the JSON
+                    cluster_json["node_details"] = cluster_node_data
+                    # Enrich source_node with type + description
+                    for nd in cluster_node_data:
+                        if nd["name"] == source_node:
+                            cluster_json["source_node"] = {
+                                "entity": source_node,
+                                "entity_type": nd.get("entity_type", "UNKNOWN"),
+                                "description": nd.get("description", "UNKNOWN")
+                            }
+                            break
+                    # Remove edges before appending
+                    cluster_json.pop("edges", None)
+                    cluster_json.pop("total_edges", None)
+                    cluster_json.pop("node_details", None)
+                    raw_clusters.append(cluster_json)
+                else:
+                    # Collect cluster data for batch processing
+                    clusters_to_summarize.append((cluster_node_data, source_node))
+
+            else:  # --- NEW FALLBACK: single-node cluster ---
+                node_info = await knowledge_graph_inst.get_node(source_node)
+                cluster_node_data = [{
+                    'name': source_node,
+                    'entity_type': node_info.get('entity_type', 'UNKNOWN') if node_info else 'UNKNOWN',
+                    'description': node_info.get('description', 'UNKNOWN') if node_info else 'UNKNOWN',
+                    'flow_value': diffused_nodes.get(source_node, 0.0) if diffused_nodes else 0.0
+                }]
+
+                if query_param.return_raw_clusters:
+                    cluster_json = {
+                        "source_node": {
+                            "entity": source_node,
+                            "entity_type": cluster_node_data[0]["entity_type"],
+                            "description": cluster_node_data[0]["description"]
+                        },
+                        "cluster_size": 1,
+                        "max_flow_value": 0.0,
+                        "nodes": [{
+                            "id": 0,
+                            "entity": source_node,
+                            "entity_type": cluster_node_data[0]["entity_type"],
+                            "description": cluster_node_data[0]["description"],
+                            "rank": 0
+                        }],
+                        "edges": [],
+                        "total_edges": 0,
+                        "node_details": cluster_node_data
+                    }
+                    # Remove edges before appending
+                    cluster_json.pop("edges", None)
+                    cluster_json.pop("total_edges", None)
+                    cluster_json.pop("node_details", None)
+                    raw_clusters.append(cluster_json)
+                else:
+                    clusters_to_summarize.append((cluster_node_data, source_node))
+
+        # Process clusters based on return type
+        if query_param.return_raw_clusters:
+            all_clusters = raw_clusters
+        else:
+            # Batch process clusters in chunks to avoid token limits
+            if clusters_to_summarize:
+                all_clusters = []
+                batch_size = query_param.batch_cluster_size
+
+                # Process clusters in batches
+                for i in range(0, len(clusters_to_summarize), batch_size):
+                    batch_clusters = clusters_to_summarize[i:i + batch_size]
+                    logger.info(f"Processing batch {i//batch_size + 1}/{(len(clusters_to_summarize) + batch_size - 1)//batch_size} with {len(batch_clusters)} clusters")
+
+                    cluster_summaries = await summarize_clusters_batch_with_llm(
+                        batch_clusters, use_llm_func, global_config
+                    )
+                    all_clusters.extend([summary for summary in cluster_summaries if summary])
+            else:
+                all_clusters = []
+
+    else:
+        # Process clusters individually (original approach)
+        for source_node in source_nodes:
+
+            if not G.has_node(source_node):
+                continue
+
+            if G.nodes[source_node].get("entity_type", "").lower() == "complete_table":
+                continue
+
+            # Get source node information
+            source_node_info = await knowledge_graph_inst.get_node(source_node)
+            if not source_node_info:
+                source_node_info = {"entity_type": "UNKNOWN", "description": "No description available"}
+
+            # Calculate confidence based on source node's relevance to the query
+            confidence = 0.7  # Default confidence
+
+            # Apply flow diffusion from this source node
+            wfd = QueryAwareWeightedFlowDiffusion(
+                G, source_node, source_node, confidence,
+                node_embeddings=node_embeddings,
+                subquery_embedding=subquery_embedding,
+                weight_func=query_param.weight_func
+            )
+            wfd.initialize(alpha=query_param.alpha)
+            diffused_nodes = wfd.flow_diffusion()
+
+            if len(diffused_nodes) > 1:  # Only consider clusters with multiple nodes
+                # Get cluster nodes and their flow values
+                cluster_nodes = list(diffused_nodes.keys())
+                cluster_flow_values = list(diffused_nodes.values())
+
+                # Only process if cluster has significant flow (using configured threshold)
+                if cluster_flow_values:
+                    max_flow = max(cluster_flow_values)
+                    if max_flow < query_param.min_flow_threshold:
+                        continue
+                else:
+                    continue  # Skip if no flow values
+
+                # Get node information for the cluster
+                cluster_node_data = []
+                for node in cluster_nodes:
+                    node_info = await knowledge_graph_inst.get_node(node)
+                    if node_info:
+                        cluster_node_data.append({
+                            'name': node,
+                            'entity_type': node_info.get('entity_type', 'UNKNOWN'),
+                            'description': node_info.get('description', 'UNKNOWN'),
+                            'flow_value': diffused_nodes.get(node, 0.0)
+                        })
+
+                # Sort by flow value
+                cluster_node_data.sort(key=lambda x: x['flow_value'], reverse=True)
+
+                # Check if we should return raw cluster data instead of LLM summaries
+                if query_param.return_raw_clusters:
+                    # Convert subgraph to JSON format
+                    cluster_json = convert_subgraph_to_json(G, cluster_nodes, diffused_nodes, source_node)
+                    # Add node details to the JSON
+                    cluster_json["node_details"] = cluster_node_data
+                    all_clusters.append(cluster_json)
+                else:
+                    # Create cluster summary using LLM (individual processing)
+                    cluster_summary = await summarize_cluster_with_llm(
+                        cluster_node_data, source_node, use_llm_func, global_config
+                    )
+
+                    if cluster_summary:
+                        all_clusters.append(cluster_summary)
+
+    logger.info(f"Flow diffusion completed, {len(all_clusters)} clusters found")
+
+    # Only apply token constraints if we're returning LLM summaries
+    if not query_param.return_raw_clusters:
+        # Limit the number of clusters based on token constraints
+        original_cluster_count = len(all_clusters)
+        all_clusters = truncate_list_by_token_size(
+            all_clusters,
+            key=lambda x: x,
+            max_token_size=query_param.max_token_for_local_context,
+        )
+
+        if original_cluster_count != len(all_clusters):
+            logger.info(f"Clusters truncated from {original_cluster_count} to {len(all_clusters)} due to token limit")
+
+    return all_clusters
+
+
+async def summarize_clusters_batch_with_llm(
+    clusters_data: list[tuple[list[dict], str]],
+    use_llm_func: callable,
+    global_config: dict = None,
+) -> list[str]:
+    """
+    Summarize multiple clusters using a single LLM call.
+
+    Parameters:
+    -----------
+    clusters_data : list[tuple[list[dict], str]]
+        List of tuples containing (cluster_node_data, source_node) for each cluster
+    use_llm_func : callable
+        LLM function to use for summarization
+    global_config : dict
+        Global configuration (optional)
+
+    Returns:
+    --------
+    list[str]
+        List of summarized cluster relationships
+    """
+    if not clusters_data:
+        return []
+
+    # Filter out clusters with less than 2 nodes
+    valid_clusters = []
+    for cluster_node_data, source_node in clusters_data:
+        if len(cluster_node_data) >= 2:
+            valid_clusters.append((cluster_node_data, source_node))
+
+    if not valid_clusters:
+        return []
+
+    # Create batch prompt for all clusters
+    clusters_info = []
+    for i, (cluster_node_data, source_node) in enumerate(valid_clusters, 1):
+        nodes_info = []
+        for node_data in cluster_node_data:
+            nodes_info.append(
+                f"- {node_data['name']} ({node_data['entity_type']}): {node_data['description']} "
+                f"(flow strength: {node_data['flow_value']:.3f})"
+            )
+
+        nodes_text = "\n".join(nodes_info)
+        clusters_info.append(f"""Cluster {i}:
+Source Node: {source_node}
+Cluster Nodes:
+{nodes_text}""")
+
+    all_clusters_text = "\n\n".join(clusters_info)
+
+    prompt = f"""Please analyze the following clusters of entities and their relationships, then provide a concise summary for each cluster.
+
+Clusters Information:
+{all_clusters_text}
+
+For each cluster, please provide a summary that explains:
+1. How these entities are related to each other
+2. What concept or theme this cluster represents
+3. The significance of the connections between these entities
+4. How the source node connects to the other entities in this cluster
+
+Please format your response as follows:
+Cluster 1: [summary for cluster 1]
+Cluster 2: [summary for cluster 2]
+...
+Cluster N: [summary for cluster N]
+
+Keep each summary concise but informative, focusing on the relationships and thematic connections."""
+
+    # Use configuration-based max_tokens if available, otherwise use a reasonable default
+    max_tokens = 2000  # Increased for batch processing
+    if global_config and "entity_summary_to_max_tokens" in global_config:
+        max_tokens = min(global_config["entity_summary_to_max_tokens"] * len(valid_clusters), 16384)
+
+    logger.info(f"Processing batch of {len(valid_clusters)} clusters with max_tokens={max_tokens}")
+
+    try:
+        batch_summary = await use_llm_func(prompt, max_tokens=max_tokens)
+        batch_summary = batch_summary.strip()
+
+        # Parse the response to extract individual cluster summaries
+        summaries = []
+        lines = batch_summary.split('\n')
+        current_summary = ""
+
+        for line in lines:
+            line = line.strip()
+            if line.startswith('Cluster ') and ':' in line:
+                # Save previous summary if exists
+                if current_summary:
+                    summaries.append(current_summary.strip())
+                # Start new summary
+                current_summary = line.split(':', 1)[1].strip()
+            elif line and current_summary:
+                current_summary += " " + line
+
+        # Add the last summary
+        if current_summary:
+            summaries.append(current_summary.strip())
+
+        # Ensure we have the right number of summaries
+        while len(summaries) < len(valid_clusters):
+            summaries.append("Cluster summary not available")
+
+        return summaries[:len(valid_clusters)]
+
+    except Exception as e:
+        logger.error(f"Error summarizing clusters batch: {e}")
+        # Fallback: create individual summaries for each cluster
+        fallback_summaries = []
+        for cluster_node_data, source_node in valid_clusters:
+            if len(cluster_node_data) > 0:
+                top_nodes = cluster_node_data[:3]  # Get top 3 nodes by flow value
+                node_names = [node['name'] for node in top_nodes]
+                if len(cluster_node_data) > 1:
+                    fallback_summary = f"Cluster centered around {source_node} connecting to {', '.join(node_names)} and {len(cluster_node_data)-1} other entities with flow values ranging from {cluster_node_data[0]['flow_value']:.3f} to {cluster_node_data[-1]['flow_value']:.3f}"
+                else:
+                    fallback_summary = f"Cluster centered around {source_node} connecting to {', '.join(node_names)}"
+                fallback_summaries.append(fallback_summary)
+            else:
+                fallback_summaries.append(f"Cluster connecting {source_node} with related entities")
+
+        return fallback_summaries
+
+
+async def summarize_cluster_with_llm(
+    cluster_node_data: list[dict],
+    source_node: str,
+    use_llm_func: callable,
+    global_config: dict = None,
+) -> str:
+    """
+    Summarize a cluster of nodes using LLM.
+
+    Parameters:
+    -----------
+    cluster_node_data : list[dict]
+        List of node data in the cluster
+    source_node : str
+        Source node name
+    use_llm_func : callable
+        LLM function to use for summarization
+    global_config : dict
+        Global configuration (optional)
+
+    Returns:
+    --------
+    str
+        Summarized cluster relationship
+    """
+    if len(cluster_node_data) < 2:
+        return None
+
+    # Create prompt for cluster summarization
+    nodes_info = []
+    for node_data in cluster_node_data:
+        nodes_info.append(
+            f"- {node_data['name']} ({node_data['entity_type']}): {node_data['description']} "
+            f"(flow strength: {node_data['flow_value']:.3f})"
+        )
+
+    nodes_text = "\n".join(nodes_info)
+
+    prompt = f"""Please analyze the following cluster of entities and their relationships, then provide a concise summary of how they are connected and what this cluster represents.
+
+Cluster Information:
+Source Node: {source_node}
+Cluster Nodes:
+{nodes_text}
+
+Please provide a summary that explains:
+1. How these entities are related to each other
+2. What concept or theme this cluster represents
+3. The significance of the connections between these entities
+4. How the source node connects to the other entities in this cluster
+
+Keep the summary concise but informative, focusing on the relationships and thematic connections."""
+
+    # Use configuration-based max_tokens if available, otherwise use a reasonable default
+    max_tokens = 1000  # Default value
+    if global_config and "entity_summary_to_max_tokens" in global_config:
+        max_tokens = global_config["entity_summary_to_max_tokens"]
+
+    try:
+        summary = await use_llm_func(prompt, max_tokens=max_tokens)
+        return summary.strip()
+    except Exception as e:
+        logger.error(f"Error summarizing cluster: {e}")
+        # Fallback to more informative description
+        if len(cluster_node_data) > 0:
+            top_nodes = cluster_node_data[:3]  # Get top 3 nodes by flow value
+            node_names = [node['name'] for node in top_nodes]
+            if len(cluster_node_data) > 1:
+                return f"Cluster centered around {source_node} connecting to {', '.join(node_names)} and {len(cluster_node_data)-1} other entities with flow values ranging from {cluster_node_data[0]['flow_value']:.3f} to {cluster_node_data[-1]['flow_value']:.3f}"
+            else:
+                return f"Cluster centered around {source_node} connecting to {', '.join(node_names)}"
+        else:
+            return f"Cluster connecting {source_node} with related entities"

src/answering/context.py

@@ -0,0 +1,322 @@
+"""
+Context building for RAG query processing.
+
+This module provides functions to build query context from the knowledge graph
+based on different query modes (local, global, hybrid).
+"""
+
+import asyncio
+from ..base import (
+    BaseGraphStorage,
+    BaseKVStorage,
+    BaseVectorStorage,
+    TextChunkSchema,
+    QueryParam,
+)
+from ..utils import (
+    logger,
+    list_of_list_to_csv,
+    csv_string_to_list,
+)
+from .clusters import find_flow_diffusion_clusters_and_summarize
+from .text_units import find_most_related_text_unit_from_entities
+
+
+async def build_query_context(
+    query: list,
+    knowledge_graph_inst: BaseGraphStorage,
+    entities_vdb: BaseVectorStorage,
+    relationships_vdb: BaseVectorStorage,
+    text_chunks_db: BaseKVStorage[TextChunkSchema],
+    query_param: QueryParam,
+    global_config: dict,
+):
+    """
+    Build query context based on extracted keywords and query mode.
+
+    Parameters:
+    -----------
+    query : list
+        List containing [ll_keywords, hl_keywords]
+    knowledge_graph_inst : BaseGraphStorage
+        Knowledge graph storage instance
+    entities_vdb : BaseVectorStorage
+        Entity vector database
+    relationships_vdb : BaseVectorStorage
+        Relationships vector database
+    text_chunks_db : BaseKVStorage[TextChunkSchema]
+        Text chunks database
+    query_param : QueryParam
+        Query parameters including mode (local/global/hybrid)
+    global_config : dict
+        Global configuration
+
+    Returns:
+    --------
+    str
+        Formatted context string for LLM response generation
+    """
+    ll_keywords, hl_keywords = query[0], query[1]
+
+    # Initialize context variables
+    entities_context, relations_context, text_units_context = "", "", ""
+
+    if query_param.mode == "local":
+        # Local mode: use ll_keywords only
+        if ll_keywords == "":
+            logger.warning("Low level keywords is empty for local mode")
+            return "", "", ""
+
+        (
+            entities_context,
+            relations_context,
+            text_units_context,
+        ) = await _get_node_data_with_flow_diffusion(
+            ll_keywords,
+            knowledge_graph_inst,
+            entities_vdb,
+            text_chunks_db,
+            query_param,
+            global_config,
+        )
+
+    elif query_param.mode == "global":
+        # Global mode: use hl_keywords only
+        if hl_keywords == "":
+            logger.warning("High level keywords is empty for global mode")
+            return "", "", ""
+
+        (
+            entities_context,
+            relations_context,
+            text_units_context,
+        ) = await _get_node_data_with_flow_diffusion(
+            hl_keywords,
+            knowledge_graph_inst,
+            entities_vdb,
+            text_chunks_db,
+            query_param,
+            global_config,
+        )
+
+    elif query_param.mode == "hybrid":
+        # Hybrid mode: use combined keywords (both ll_keywords and hl_keywords)
+        if ll_keywords == "" and hl_keywords == "":
+            logger.warning("Both Low Level and High Level keywords are empty for hybrid mode")
+            return "", "", ""
+
+        # Get local information using ll_keywords
+        local_entities_context, local_relations_context, local_text_units_context = "", "", ""
+        if ll_keywords:
+            (
+                local_entities_context,
+                local_relations_context,
+                local_text_units_context,
+            ) = await _get_node_data_with_flow_diffusion(
+                ll_keywords,
+                knowledge_graph_inst,
+                entities_vdb,
+                text_chunks_db,
+                query_param,
+                global_config,
+            )
+
+        # Get global information using hl_keywords
+        global_entities_context, global_relations_context, global_text_units_context = "", "", ""
+        if hl_keywords:
+            (
+                global_entities_context,
+                global_relations_context,
+                global_text_units_context,
+            ) = await _get_node_data_with_flow_diffusion(
+                hl_keywords,
+                knowledge_graph_inst,
+                entities_vdb,
+                text_chunks_db,
+                query_param,
+                global_config,
+            )
+
+    # Return context based on mode
+    if query_param.mode == "local":
+        if query_param.return_raw_entities:
+            return entities_context
+        elif query_param.return_raw_clusters:
+            return relations_context
+
+        return f"""
+-----local-information-----
+-----low-level entity information-----
+```csv
+{entities_context}
+```
+-----low-level relationship information-----
+```csv
+{relations_context}
+```
+-----Sources-----
+```csv
+{text_units_context}
+```
+"""
+    elif query_param.mode == "global":
+        if query_param.return_raw_entities:
+            return entities_context
+        elif query_param.return_raw_clusters:
+            return relations_context
+
+        return f"""
+-----global-information-----
+-----high-level entity information-----
+```csv
+{entities_context}
+```
+-----high-level relationship information-----
+```csv
+{relations_context}
+```
+-----Sources-----
+```csv
+{text_units_context}
+```
+"""
+    elif query_param.mode == "hybrid":
+        if query_param.return_raw_entities:
+            # Merge local + global entities CSV into one CSV and reindex id
+            merged_rows = []
+            if local_entities_context:
+                merged_rows += csv_string_to_list(local_entities_context)[1:]
+            if global_entities_context:
+                merged_rows += csv_string_to_list(global_entities_context)[1:]
+            for idx, row in enumerate(merged_rows):
+                if row:
+                    row[0] = str(idx)
+            merged_entities_csv = list_of_list_to_csv(
+                [["id", "entity", "entity_type", "description", "rank"]] + merged_rows
+            )
+            return merged_entities_csv
+        elif query_param.return_raw_clusters:
+            return local_relations_context + global_relations_context
+
+        return f"""
+-----hybrid-information-----
+-----local information (from low-level keywords)-----
+-----local entity information-----
+```csv
+{local_entities_context}
+```
+-----local relationship information-----
+```csv
+{local_relations_context}
+```
+-----local sources-----
+```csv
+{local_text_units_context}
+```
+-----global information (from high-level keywords)-----
+-----global entity information-----
+```csv
+{global_entities_context}
+```
+-----global relationship information-----
+```csv
+{global_relations_context}
+```
+-----global sources-----
+```csv
+{global_text_units_context}
+```
+"""
+    else:
+        return ""
+
+
+async def _get_node_data_with_flow_diffusion(
+    query,
+    knowledge_graph_inst: BaseGraphStorage,
+    entities_vdb: BaseVectorStorage,
+    text_chunks_db: BaseKVStorage[TextChunkSchema],
+    query_param: QueryParam,
+    global_config: dict,
+):
+    """
+    Get node data using flow diffusion for finding relationships.
+
+    Parameters:
+    -----------
+    query : str
+        Query string (can be either ll_keywords or hl_keywords)
+    knowledge_graph_inst : BaseGraphStorage
+        Knowledge graph storage instance
+    entities_vdb : BaseVectorStorage
+        Entity vector database
+    text_chunks_db : BaseKVStorage[TextChunkSchema]
+        Text chunks database
+    query_param : QueryParam
+        Query parameters
+    global_config : dict
+        Global configuration
+
+    Returns:
+    --------
+    tuple
+        (entities_context, relations_context, text_units_context)
+    """
+    results = await entities_vdb.query(query, top_k=query_param.max_source_nodes)
+    if not len(results):
+        return "", "", ""
+
+    node_datas = await asyncio.gather(
+        *[knowledge_graph_inst.get_node(r["entity_name"]) for r in results]
+    )
+    if not all([n is not None for n in node_datas]):
+        logger.warning("Some nodes are missing, maybe the storage is damaged")
+
+    node_degrees = await asyncio.gather(
+        *[knowledge_graph_inst.node_degree(r["entity_name"]) for r in results]
+    )
+    node_datas = [
+        {**n, "entity_name": k["entity_name"], "rank": d}
+        for k, n, d in zip(results, node_datas, node_degrees)
+        if n is not None
+    ]
+
+    use_text_units = await find_most_related_text_unit_from_entities(
+        node_datas, query_param, text_chunks_db, knowledge_graph_inst
+    )
+
+    # Use flow diffusion instead of the original relationship finding method
+    use_relations = await find_flow_diffusion_clusters_and_summarize(
+        node_datas, query, query_param, knowledge_graph_inst, global_config
+    )
+
+    logger.info(
+        f"Flow diffusion query uses {len(node_datas)} entities, {len(use_relations)} cluster summaries, {len(use_text_units)} text units"
+    )
+
+    entites_section_list = [["id", "entity", "entity_type", "description", "rank"]]
+    for i, n in enumerate(node_datas):
+        entites_section_list.append([
+            i,
+            n["entity_name"],
+            n.get("entity_type", "UNKNOWN"),
+            n.get("description", "UNKNOWN"),
+            n["rank"],
+        ])
+    entities_context = list_of_list_to_csv(entites_section_list)
+
+    # Relations context: return JSON clusters when requested; otherwise CSV
+    if query_param.return_raw_clusters:
+        relations_context = use_relations
+    else:
+        relations_section_list = [["id", "cluster_summary"]]
+        for i, summary in enumerate(use_relations):
+            relations_section_list.append([i, summary])
+        relations_context = list_of_list_to_csv(relations_section_list)
+
+    text_units_section_list = [["id", "content"]]
+    for i, t in enumerate(use_text_units):
+        text_units_section_list.append([i, t["content"]])
+    text_units_context = list_of_list_to_csv(text_units_section_list)
+
+    return entities_context, relations_context, text_units_context