| --- |
| license: apache-2.0 |
| base_model: |
| - Qwen/Qwen3.5-4B |
| tags: |
| - lora |
| - knowledge-graph |
| - tool-use |
| - graph-traversal |
| - question-answering |
| - scientific-reasoning |
| --- |
| |
| # Prospector |
|
|
| Prospector is a compact LLM policy post-trained to answer questions over a |
| scientific knowledge graph. The policy calls a small set of tools to traverse |
| the graph turn by turn, building its context from what each call returns, |
| and answers with a chain of edges that is checked against what its tools |
| retrieved. Prospector is Qwen3.5 at 0.8B, 2B and 4B parameters with LoRA |
| adapters, trained on one graph built from the semiconductor-manufacturing |
| literature. With frozen weights, the same adapters transfer to graphs from |
| three other fields. |
|
|
| This repository holds the weights, the training rows and the stored |
| evaluation runs of the paper, laid out as the code repository expects them. |
| The code that serves, evaluates and trains them is at |
| [github.com/lamm-mit/Prospector](https://github.com/lamm-mit/Prospector). |
| Its README covers the evaluation harness, serving a policy, training one on |
| your own graph, and the command behind every number in the paper. |
|
|
| ## Get started |
|
|
| The code repository's quick start, on a Mac with Apple silicon |
| or a Linux machine with an NVIDIA GPU: |
|
|
| ```bash |
| # install uv, unless you have it |
| curl -LsSf https://astral.sh/uv/install.sh | sh |
| export PATH="$HOME/.local/bin:$PATH" |
| git clone https://github.com/lamm-mit/Prospector && cd Prospector |
| # download this repository's adapters and data into the checkout, about 9 GB |
| uvx --from huggingface_hub hf download lamm-mit/Prospector --local-dir . \ |
| --include "models/*" --include "data/*" |
| # build the environment and evaluate Prospector-4B on six questions |
| ./quickstart.sh |
| ``` |
|
|
| The first run also downloads the base model from the Hub and merges the |
| adapters into it. The semiconductor graph is then under `data/raw/`, and the |
| code repository's `data/raw/README.md` downloads the three other graphs. The |
| code repository's README, under "Reproducing the paper", gives for each |
| number in the paper the command that recomputes it and the stored run the |
| number is computed from. |
|
|
| ## Results |
|
|
| The gate accepts a cited chain when every cited edge was returned by a tool |
| and the answer names a node of the chain. Gated accuracy is the fraction of |
| questions whose chain passes the gate and whose answer names a gold label. |
| The table gives means over the seeds of the runs stored here, ± one standard |
| deviation. |
|
|
| | policy, graph | gated accuracy | |
| |---|---| |
| | Prospector-4B, semiconductor, eight seeds (`flash20cap1024-s0` to `-s7`) | 0.839 ± 0.015 | |
| | Prospector-2B, semiconductor, eight seeds (`q2bd2eval-ck120-s0` to `-s7`) | 0.755 ± 0.016 | |
| | Prospector-0.8B, semiconductor, eight seeds (`q0.8bd2eval-ck120-s0` to `-s7`) | 0.694 ± 0.021 | |
| | Prospector-4B, bio-inspired materials, three seeds (`flash20stackbio-s0` to `-s2`) | 0.803 ± 0.017 | |
| | Prospector-4B, PFAS, three seeds (`stack1kpfas-ctx32k-s0` to `-s2`) | 0.766 ± 0.022 | |
| | Prospector-4B, materials properties, three seeds (`stack1kmatprop-s0` to `-s2`) | 0.809 ± 0.011 | |
|
|
| ## The served policies |
|
|
| The paper's recipe trains each size in four stages: a supervised stage on a |
| frontier teacher's demonstrations, a GRPO round over whole episodes, a |
| second supervised stage on the demonstrations blended with the policy's own |
| kept episodes, and two decision rounds of GRPO that resume stored |
| mid-episode states. A verifier is then trained on the served policy's own |
| samples. Each size serves the second supervised stage's adapter with the |
| checkpoint that each decision round's evaluation grid picked applied on top, |
| in this load order, over `Qwen/Qwen3.5-<size>`. The policies run with the |
| `plain_chatml` chat template at a budget of 20 tool calls and 1,024 |
| generated tokens a turn. |
|
|
| | size | policy (`--adapter`, in load order) | verifier | |
| |---|---|---| |
| | 4B | `models/traj_sft_4b_flash20g2/final`, `models/traj_grpo_4b_flash20d1/checkpoint-120`, `models/traj_grpo_4b_flash20d2/checkpoint-120` | `models/verifier_sft_4b_flash20v/final` | |
| | 2B | `models/traj_sft_2b_flash20g2/final`, `models/traj_grpo_2b_flash20d1/checkpoint-100`, `models/traj_grpo_2b_flash20d2/checkpoint-120` | `models/verifier_sft_2b_flash20v/final` | |
| | 0.8B | `models/traj_sft_0.8b_flash20g2/final`, `models/traj_grpo_0.8b_flash20d1/checkpoint-100`, `models/traj_grpo_0.8b_flash20d2/checkpoint-120` | `models/verifier_sft_0.8b_flash20v/final` | |
|
|
| The adapters are PEFT LoRA adapters in transformers' fused naming for Qwen3.5 |
| (`in_proj_qkv`), which the code repository's pinned `transformers` and `peft` |
| load. Each `run.json` records the chat template its adapter was trained |
| under, and each GRPO round's `peak.txt` records the checkpoint its grid |
| picked. |
|
|
| ## What is here |
|
|
| Downloaded into a checkout's root, every file lands where the code expects |
| it. `MANIFEST` lists every file by pattern with its reason. |
|
|
| ``` |
| data/raw/ the semiconductor graph, the paper's own; the code repository's |
| data/raw/README.md downloads the three other graphs from their sources |
| models/ the recipe's six training runs at each size, each adapter with the |
| run.json that records what it was trained from, and |
| recipe_backfill_v1.json, which records how those run.json files were |
| written and each later change made by hand to a stored file here |
| data/traj/ the training rows of each stage, each with a .provenance.json sidecar |
| naming what it was built from, where one exists |
| data/runs/ the stored evaluation runs: every run the paper reads or the code |
| repository's README names, the timing runs behind the cost figure, |
| and the latent-state probe's derived files under _probe/ |
| data/analysis/ inference_speed.json, which prices the timing runs at the posted |
| rates it dates |
| MANIFEST the list of what is here and why, pattern by pattern |
| ``` |
|
|
| The set is the smallest that serves the three policies, lets `prospector |
| run` re-evaluate a size over the released adapters without training |
| anything, and recomputes every figure in the paper from stored runs, with |
| the probe's hidden states (about 1.7 GB a run) regenerated by its `states` |
| stage from the stored transcripts. |
|
|
| **The training rows.** `data/traj/` holds the rows each stage trained from, |
| named as the code derives them. Stage 1's demonstrations |
| (`tsmc_sft_mix_flash20.jsonl`) are shared by the three sizes. The second |
| supervised stage's mix, the decision rounds' stored states and the |
| verifier's rows exist per size. The teacher runs the demonstrations came |
| from (`flash20-teach-4x-g0` to `-g7` and `-p2`) are under `data/runs/`. The |
| 0.8B's and 2B's mixes and stored states were rebuilt by the code's builders |
| from the stored runs they were sampled from, and the same builders reproduce |
| the 4B's files byte for byte. With these adapters in place, `prospector run` |
| trains nothing. At 0.8B and 2B the command reuses the stored evaluations |
| here, which carry the names it derives, and evaluates what it does not find. |
| At 4B, whose stored runs carry older names, the command evaluates afresh |
| under its own names. |
|
|
| **The stored runs.** A run is a directory `data/runs/<name>--<benchmark>/` |
| in the shape the code repository's README describes. A closed-book or |
| retrieved-subgraph run, which has no episodes, holds one reply per question |
| and, where it was scored, the scorer's output under the same gate. |
|
|
| ## License and citation |
|
|
| The adapters, the rows, the runs and the semiconductor graph are released |
| under the Apache License 2.0. |
|
|
| ```bibtex |
| @article{yang2026prospector, |
| title = {Prospector: learning a compact graph traversal policy for scientific reasoning}, |
| author = {Yang, Haiqian and Buehler, Markus J.}, |
| journal = {arXiv preprint}, |
| year = {2026} |
| } |
| ``` |
|
|
