Spaces:
Running
Retrieval fixes: robust nprobe tuning + goals-only query
Browse files1. nprobe: set via faiss.extract_index_ivf() instead of attribute
assignment on the top-level index. FAISS silently ignores nprobe set
on wrapper indexes (IndexIDMap etc.), so reaching the IVF layer
directly guards against a future wrap making the tuning a no-op.
Verified the persisted index defaults to nprobe=16 and load bumps it
to 32. Non-IVF indexes and test doubles are tolerated (RuntimeError /
TypeError respectively) since search-breadth tuning must never break
index loading.
2. Retrieval query: goals only, newline-joined. Previously the retrieve
node space-joined goals + Lean error messages into one string. The
LeanDojo ByT5 encoder was trained on canonical proof states
("h : T\n⊢ goal"); error text is off-distribution noise that skews
the embedding. Errors still reach the LLM through the generation
prompt — they just no longer pollute retrieval. With no open goals
(pure syntax error) the query is empty and the retriever returns [],
which generation already handles.
Sanity check ("n : ℕ\n⊢ n + 0 = n"): top-5 = Nat.add_comm,
Nat.add_right_cancel, Nat.succ_add, Num.add_succ, Nat.add_assoc.
116/116 tests pass.
- src/langgraph_agent.py +7 -1
- src/retriever.py +9 -3
|
@@ -121,7 +121,13 @@ def make_verify_node(lean_env: LeanEnvironment):
|
|
| 121 |
|
| 122 |
def make_retrieve_node(retriever: MathLibRetriever):
|
| 123 |
def retrieve_node(state: ProofState) -> ProofState:
|
| 124 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 125 |
print("Retrieving relevant Mathlib lemmas…")
|
| 126 |
lemmas = retriever.retrieve(query)
|
| 127 |
print(f" Retrieved {len(lemmas)} lemma(s).")
|
|
|
|
| 121 |
|
| 122 |
def make_retrieve_node(retriever: MathLibRetriever):
|
| 123 |
def retrieve_node(state: ProofState) -> ProofState:
|
| 124 |
+
# Query with goals only, newline-joined: the LeanDojo encoder was
|
| 125 |
+
# trained on canonical proof states ("h1 : T1\nh2 : T2\n⊢ goal"), so
|
| 126 |
+
# Lean error text is off-distribution noise in the embedding. Errors
|
| 127 |
+
# still reach the LLM via the generation prompt — just not retrieval.
|
| 128 |
+
# No open goals (e.g. pure syntax error) → empty query → retriever
|
| 129 |
+
# returns [] and generation proceeds without premises.
|
| 130 |
+
query = "\n\n".join(state["goals"])
|
| 131 |
print("Retrieving relevant Mathlib lemmas…")
|
| 132 |
lemmas = retriever.retrieve(query)
|
| 133 |
print(f" Retrieved {len(lemmas)} lemma(s).")
|
|
@@ -143,7 +143,13 @@ class MathLibRetriever:
|
|
| 143 |
)
|
| 144 |
# Tune IVFPQ search breadth. nprobe=32 / nlist=512 = 6% of clusters
|
| 145 |
# searched — good recall/speed tradeoff for this index size.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 146 |
try:
|
| 147 |
-
self._faiss_store.index.nprobe = self.nprobe
|
| 148 |
-
except
|
| 149 |
-
|
|
|
|
|
|
|
|
|
| 143 |
)
|
| 144 |
# Tune IVFPQ search breadth. nprobe=32 / nlist=512 = 6% of clusters
|
| 145 |
# searched — good recall/speed tradeoff for this index size.
|
| 146 |
+
# extract_index_ivf reaches the IVF layer even if the index is later
|
| 147 |
+
# wrapped (e.g. IndexIDMap); setting nprobe on a wrapper is silently
|
| 148 |
+
# ignored by FAISS, which this guards against.
|
| 149 |
+
import faiss
|
| 150 |
try:
|
| 151 |
+
faiss.extract_index_ivf(self._faiss_store.index).nprobe = self.nprobe
|
| 152 |
+
except (RuntimeError, TypeError):
|
| 153 |
+
# RuntimeError: no IVF layer (e.g. flat index) — nothing to tune.
|
| 154 |
+
# TypeError: not a real faiss index (e.g. a test double).
|
| 155 |
+
pass
|