p4r5kpftnp-cmd commited on
Commit
1dc8f6a
·
1 Parent(s): 7181ddb

Retrieval fixes: robust nprobe tuning + goals-only query

Browse files

1. nprobe: set via faiss.extract_index_ivf() instead of attribute
assignment on the top-level index. FAISS silently ignores nprobe set
on wrapper indexes (IndexIDMap etc.), so reaching the IVF layer
directly guards against a future wrap making the tuning a no-op.
Verified the persisted index defaults to nprobe=16 and load bumps it
to 32. Non-IVF indexes and test doubles are tolerated (RuntimeError /
TypeError respectively) since search-breadth tuning must never break
index loading.

2. Retrieval query: goals only, newline-joined. Previously the retrieve
node space-joined goals + Lean error messages into one string. The
LeanDojo ByT5 encoder was trained on canonical proof states
("h : T\n⊢ goal"); error text is off-distribution noise that skews
the embedding. Errors still reach the LLM through the generation
prompt — they just no longer pollute retrieval. With no open goals
(pure syntax error) the query is empty and the retriever returns [],
which generation already handles.

Sanity check ("n : ℕ\n⊢ n + 0 = n"): top-5 = Nat.add_comm,
Nat.add_right_cancel, Nat.succ_add, Num.add_succ, Nat.add_assoc.
116/116 tests pass.

Files changed (2) hide show
  1. src/langgraph_agent.py +7 -1
  2. src/retriever.py +9 -3
src/langgraph_agent.py CHANGED
@@ -121,7 +121,13 @@ def make_verify_node(lean_env: LeanEnvironment):
121
 
122
  def make_retrieve_node(retriever: MathLibRetriever):
123
  def retrieve_node(state: ProofState) -> ProofState:
124
- query = " ".join(state["goals"] + state["errors"])
 
 
 
 
 
 
125
  print("Retrieving relevant Mathlib lemmas…")
126
  lemmas = retriever.retrieve(query)
127
  print(f" Retrieved {len(lemmas)} lemma(s).")
 
121
 
122
  def make_retrieve_node(retriever: MathLibRetriever):
123
  def retrieve_node(state: ProofState) -> ProofState:
124
+ # Query with goals only, newline-joined: the LeanDojo encoder was
125
+ # trained on canonical proof states ("h1 : T1\nh2 : T2\n⊢ goal"), so
126
+ # Lean error text is off-distribution noise in the embedding. Errors
127
+ # still reach the LLM via the generation prompt — just not retrieval.
128
+ # No open goals (e.g. pure syntax error) → empty query → retriever
129
+ # returns [] and generation proceeds without premises.
130
+ query = "\n\n".join(state["goals"])
131
  print("Retrieving relevant Mathlib lemmas…")
132
  lemmas = retriever.retrieve(query)
133
  print(f" Retrieved {len(lemmas)} lemma(s).")
src/retriever.py CHANGED
@@ -143,7 +143,13 @@ class MathLibRetriever:
143
  )
144
  # Tune IVFPQ search breadth. nprobe=32 / nlist=512 = 6% of clusters
145
  # searched — good recall/speed tradeoff for this index size.
 
 
 
 
146
  try:
147
- self._faiss_store.index.nprobe = self.nprobe
148
- except AttributeError:
149
- pass # not an IVF index — nothing to tune
 
 
 
143
  )
144
  # Tune IVFPQ search breadth. nprobe=32 / nlist=512 = 6% of clusters
145
  # searched — good recall/speed tradeoff for this index size.
146
+ # extract_index_ivf reaches the IVF layer even if the index is later
147
+ # wrapped (e.g. IndexIDMap); setting nprobe on a wrapper is silently
148
+ # ignored by FAISS, which this guards against.
149
+ import faiss
150
  try:
151
+ faiss.extract_index_ivf(self._faiss_store.index).nprobe = self.nprobe
152
+ except (RuntimeError, TypeError):
153
+ # RuntimeError: no IVF layer (e.g. flat index) — nothing to tune.
154
+ # TypeError: not a real faiss index (e.g. a test double).
155
+ pass