FOTO paper — reproduction scripts
Reproduces the benchmark tables in the FOTO paper from scratch. Run order:
| step | script | produces | needs |
|---|---|---|---|
| 1 | census.py |
astro_captions.parquet (ArxivCap astro-ph text slice) |
HF_TOKEN (recommended) |
| 2 | build_index.py --strategy title_caption |
OpenAI index (indexes/title_caption) |
OPENAI_API_KEY |
| 2 | build_index.py --strategy title_caption --backend local |
bge index (indexes/title_caption_bge-base-en-v1.5) |
— |
| 3 | benchmark_grid.py |
results_grid.jsonl (Table 4 data) |
ANTHROPIC_API_KEY, SEMANTIC_SCHOLAR_API_KEY, OPENAI_API_KEY |
| 4 | summarize_grid.py results_grid.jsonl |
Table 4 R@k tables + heatmap/curves | — |
| 5 | embedder_comparison.py |
Table 3 (bge vs openai, query prefix applied) | OPENAI_API_KEY |
| 6 | verify_bench.py |
verification table (vision/text/deepseek) | ANTHROPIC_API_KEY, DEEPSEEK_API_KEY, HF_TOKEN |
To run:
python census.py
python build_index.py --strategy title_caption # openai, 512d
python build_index.py --strategy title_caption --backend local # bge, 768d
python benchmark_grid.py \
--index-dir indexes/title_caption_bge-base-en-v1.5 \
--pathfinder-emb ./pathfinder/embeddings.npy \
--pathfinder-meta ./pathfinder/meta.parquet \
--n-figures 500 --out results_grid
python summarize_grid.py results_grid.jsonl # task level (paper Table 4)
python summarize_grid.py results_grid.jsonl --level fig
python summarize_grid.py results_grid.jsonl --level paper
python embedder_comparison.py --results results_grid.jsonl \
--index-bge indexes/title_caption_bge-base-en-v1.5 \
--index-openai indexes/title_caption
python verify_bench.py --results results_grid.jsonl \
--index-bge indexes/title_caption_bge-base-en-v1.5 --out verify2