| # FOTO paper — reproduction scripts |
|
|
| Reproduces the benchmark tables in the FOTO paper from scratch. Run order: |
|
|
| | step | script | produces | needs | |
| |---|---|---|---| |
| | 1 | `census.py` | `astro_captions.parquet` (ArxivCap astro-ph text slice) | HF_TOKEN (recommended) | |
| | 2 | `build_index.py --strategy title_caption` | OpenAI index (`indexes/title_caption`) | OPENAI_API_KEY | |
| | 2 | `build_index.py --strategy title_caption --backend local` | bge index (`indexes/title_caption_bge-base-en-v1.5`) | — | |
| | 3 | `benchmark_grid.py` | `results_grid.jsonl` (Table 4 data) | ANTHROPIC_API_KEY, SEMANTIC_SCHOLAR_API_KEY, OPENAI_API_KEY | |
| | 4 | `summarize_grid.py results_grid.jsonl` | Table 4 R@k tables + heatmap/curves | — | |
| | 5 | `embedder_comparison.py` | Table 3 (bge vs openai, query prefix applied) | OPENAI_API_KEY | |
| | 6 | `verify_bench.py` | verification table (vision/text/deepseek) | ANTHROPIC_API_KEY, DEEPSEEK_API_KEY, HF_TOKEN | |
| |
| To run: |
| |
| ```bash |
| python census.py |
| python build_index.py --strategy title_caption # openai, 512d |
| python build_index.py --strategy title_caption --backend local # bge, 768d |
| |
| python benchmark_grid.py \ |
| --index-dir indexes/title_caption_bge-base-en-v1.5 \ |
| --pathfinder-emb ./pathfinder/embeddings.npy \ |
| --pathfinder-meta ./pathfinder/meta.parquet \ |
| --n-figures 500 --out results_grid |
| |
| python summarize_grid.py results_grid.jsonl # task level (paper Table 4) |
| python summarize_grid.py results_grid.jsonl --level fig |
| python summarize_grid.py results_grid.jsonl --level paper |
|
|
| python embedder_comparison.py --results results_grid.jsonl \ |
| --index-bge indexes/title_caption_bge-base-en-v1.5 \ |
| --index-openai indexes/title_caption |
| |
| python verify_bench.py --results results_grid.jsonl \ |
| --index-bge indexes/title_caption_bge-base-en-v1.5 --out verify2 |
| ``` |
| |
