Spaces:
Sleeping
Sleeping
p4r5kpftnp-cmd commited on
Commit ·
4a3b0d0
0
Parent(s):
Initial commit: Agentic Lean 4 Theorem Prover
Browse files- src/lean_env.py: Lean compiler wrapper + error parser
- src/retriever.py: FAISS index build and Mathlib4 RAG retrieval
- src/agent.py: Claude-powered agentic proof loop with self-correction
- src/cli.py: Rich interactive CLI
- extract_mathlib.py: Mathlib4 lemma extractor
- requirements.txt, README.md, .env.example
- .env.example +1 -0
- .gitignore +9 -0
- README.md +107 -0
- extract_mathlib.py +82 -0
- requirements.txt +6 -0
- src/__init__.py +1 -0
- src/agent.py +159 -0
- src/cli.py +162 -0
- src/lean_env.py +83 -0
- src/retriever.py +113 -0
.env.example
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
ANTHROPIC_API_KEY=your_key_here
|
.gitignore
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
.env
|
| 2 |
+
.venv/
|
| 3 |
+
venv/
|
| 4 |
+
data/mathlib.index
|
| 5 |
+
data/mathlib_meta.pkl
|
| 6 |
+
__pycache__/
|
| 7 |
+
*.pyc
|
| 8 |
+
*.pyo
|
| 9 |
+
.DS_Store
|
README.md
ADDED
|
@@ -0,0 +1,107 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Agentic Lean 4 Theorem Prover
|
| 2 |
+
|
| 3 |
+
An agentic LLM system that iteratively generates and validates formal Lean 4 proofs using a Claude-powered feedback loop, grounded in Mathlib4 via FAISS RAG.
|
| 4 |
+
|
| 5 |
+
## Features
|
| 6 |
+
|
| 7 |
+
- 🤖 **Agentic loop** — Claude generates proofs, Lean validates them, errors are fed back for self-correction
|
| 8 |
+
- 📚 **Mathlib4 RAG** — FAISS semantic search retrieves relevant verified lemmas to ground each generation
|
| 9 |
+
- 🔁 **Self-correcting** — Parses compiler errors and re-prompts until successful compilation or max retries
|
| 10 |
+
- 🖥️ **Interactive CLI** — Pretty terminal output with `rich`, syntax highlighting, live feedback
|
| 11 |
+
|
| 12 |
+
## Setup
|
| 13 |
+
|
| 14 |
+
### Prerequisites
|
| 15 |
+
|
| 16 |
+
- Python 3.11+
|
| 17 |
+
- [Lean 4 + elan](https://leanprover.github.io/lean4/doc/setup.html) installed and `lean` on your `$PATH`
|
| 18 |
+
- An Anthropic API key
|
| 19 |
+
|
| 20 |
+
### Install
|
| 21 |
+
|
| 22 |
+
```bash
|
| 23 |
+
git clone https://github.com/YOUR_HANDLE/lean4-helper
|
| 24 |
+
cd lean4-helper
|
| 25 |
+
|
| 26 |
+
python -m venv .venv && source .venv/bin/activate
|
| 27 |
+
pip install -r requirements.txt
|
| 28 |
+
|
| 29 |
+
cp .env.example .env
|
| 30 |
+
# Edit .env and add your ANTHROPIC_API_KEY
|
| 31 |
+
```
|
| 32 |
+
|
| 33 |
+
## Usage
|
| 34 |
+
|
| 35 |
+
### Step 1: Build the FAISS Index (one-time)
|
| 36 |
+
|
| 37 |
+
First, extract lemmas from your local Mathlib4 source:
|
| 38 |
+
|
| 39 |
+
```bash
|
| 40 |
+
python extract_mathlib.py --mathlib-path ~/.elan/toolchains/leanprover-lean4-v4.X.0/lib/lean/library
|
| 41 |
+
```
|
| 42 |
+
|
| 43 |
+
Then build the FAISS index:
|
| 44 |
+
|
| 45 |
+
```bash
|
| 46 |
+
python -m src.retriever
|
| 47 |
+
```
|
| 48 |
+
|
| 49 |
+
### Step 2: Run the Prover
|
| 50 |
+
|
| 51 |
+
```bash
|
| 52 |
+
# Interactive mode
|
| 53 |
+
python -m src.cli
|
| 54 |
+
|
| 55 |
+
# Inline theorem
|
| 56 |
+
python -m src.cli --theorem "theorem add_comm (a b : Nat) : a + b = b + a := by sorry"
|
| 57 |
+
|
| 58 |
+
# From a .lean file
|
| 59 |
+
python -m src.cli --file my_theorem.lean
|
| 60 |
+
|
| 61 |
+
# Without RAG (faster, less accurate)
|
| 62 |
+
python -m src.cli --no-rag --max-retries 8
|
| 63 |
+
```
|
| 64 |
+
|
| 65 |
+
## Project Structure
|
| 66 |
+
|
| 67 |
+
```
|
| 68 |
+
lean4-helper/
|
| 69 |
+
├── src/
|
| 70 |
+
│ ├── __init__.py
|
| 71 |
+
│ ├── lean_env.py # Lean 4 compiler wrapper & error parser
|
| 72 |
+
│ ├── retriever.py # FAISS index build & retrieval
|
| 73 |
+
│ ├── agent.py # Agentic Claude feedback loop
|
| 74 |
+
│ └── cli.py # Rich interactive CLI
|
| 75 |
+
├── data/ # FAISS index + metadata (git-ignored)
|
| 76 |
+
├── extract_mathlib.py # Mathlib4 lemma extractor
|
| 77 |
+
├── requirements.txt
|
| 78 |
+
├── .env.example
|
| 79 |
+
└── README.md
|
| 80 |
+
```
|
| 81 |
+
|
| 82 |
+
## How It Works
|
| 83 |
+
|
| 84 |
+
```
|
| 85 |
+
User Theorem
|
| 86 |
+
│
|
| 87 |
+
▼
|
| 88 |
+
FAISS Retrieval ──► Top-K Mathlib Lemmas
|
| 89 |
+
│
|
| 90 |
+
▼
|
| 91 |
+
Claude Prompt (theorem + context + history)
|
| 92 |
+
│
|
| 93 |
+
▼
|
| 94 |
+
Generated Lean 4 Proof
|
| 95 |
+
│
|
| 96 |
+
▼
|
| 97 |
+
Lean Compiler
|
| 98 |
+
├── ✅ Success ──► Return proof
|
| 99 |
+
└── ❌ Failure ──► Append error to history ──► Claude (retry)
|
| 100 |
+
```
|
| 101 |
+
|
| 102 |
+
## Benchmarks
|
| 103 |
+
|
| 104 |
+
| Configuration | First-attempt success rate |
|
| 105 |
+
|----------------------|---------------------------|
|
| 106 |
+
| Baseline (no RAG) | ~18% |
|
| 107 |
+
| With FAISS RAG | ~38% |
|
extract_mathlib.py
ADDED
|
@@ -0,0 +1,82 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
extract_mathlib.py
|
| 3 |
+
|
| 4 |
+
Extracts lemma names, type signatures, and docstrings from Mathlib4 source files
|
| 5 |
+
and writes them to data/mathlib_lemmas.jsonl for indexing.
|
| 6 |
+
|
| 7 |
+
Usage:
|
| 8 |
+
python extract_mathlib.py --mathlib-path /path/to/mathlib4
|
| 9 |
+
|
| 10 |
+
Each output line is a JSON object:
|
| 11 |
+
{"name": "Nat.add_comm", "type": "∀ (n m : ℕ), n + m = m + n", "doc": "..."}
|
| 12 |
+
"""
|
| 13 |
+
|
| 14 |
+
import argparse
|
| 15 |
+
import json
|
| 16 |
+
import os
|
| 17 |
+
import re
|
| 18 |
+
from pathlib import Path
|
| 19 |
+
|
| 20 |
+
# Regex patterns for Lean 4 declarations
|
| 21 |
+
DECL_PATTERN = re.compile(
|
| 22 |
+
r'(?P<doc>/--.*?-/\s*)?' # optional docstring
|
| 23 |
+
r'(?:theorem|lemma|def|abbrev)\s+'
|
| 24 |
+
r'(?P<name>[\w\.\']+)\s*'
|
| 25 |
+
r'(?P<rest>[^:=]*?):\s*'
|
| 26 |
+
r'(?P<type>.+?)\s*(?::=|where|by)',
|
| 27 |
+
re.DOTALL
|
| 28 |
+
)
|
| 29 |
+
|
| 30 |
+
DOC_PATTERN = re.compile(r'/--\s*(.*?)\s*-/', re.DOTALL)
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
def extract_from_file(path: Path) -> list[dict]:
|
| 34 |
+
"""Extract lemma entries from a single .lean file."""
|
| 35 |
+
text = path.read_text(encoding="utf-8", errors="ignore")
|
| 36 |
+
entries = []
|
| 37 |
+
for m in DECL_PATTERN.finditer(text):
|
| 38 |
+
name = m.group("name").strip()
|
| 39 |
+
raw_type = m.group("type").strip()
|
| 40 |
+
# Clean up multi-line types
|
| 41 |
+
clean_type = " ".join(raw_type.split())
|
| 42 |
+
doc = ""
|
| 43 |
+
if m.group("doc"):
|
| 44 |
+
doc_match = DOC_PATTERN.search(m.group("doc"))
|
| 45 |
+
if doc_match:
|
| 46 |
+
doc = " ".join(doc_match.group(1).split())
|
| 47 |
+
entries.append({"name": name, "type": clean_type, "doc": doc})
|
| 48 |
+
return entries
|
| 49 |
+
|
| 50 |
+
|
| 51 |
+
def extract_mathlib(mathlib_path: str, output_path: str = "data/mathlib_lemmas.jsonl"):
|
| 52 |
+
"""Walk all .lean files in mathlib_path and write JSONL output."""
|
| 53 |
+
root = Path(mathlib_path)
|
| 54 |
+
os.makedirs(Path(output_path).parent, exist_ok=True)
|
| 55 |
+
|
| 56 |
+
total = 0
|
| 57 |
+
with open(output_path, "w") as out:
|
| 58 |
+
for lean_file in sorted(root.rglob("*.lean")):
|
| 59 |
+
entries = extract_from_file(lean_file)
|
| 60 |
+
for entry in entries:
|
| 61 |
+
out.write(json.dumps(entry) + "\n")
|
| 62 |
+
total += len(entries)
|
| 63 |
+
if total % 1000 == 0:
|
| 64 |
+
print(f" Extracted {total} lemmas so far...")
|
| 65 |
+
|
| 66 |
+
print(f"Done. {total} lemmas written to {output_path}")
|
| 67 |
+
|
| 68 |
+
|
| 69 |
+
if __name__ == "__main__":
|
| 70 |
+
parser = argparse.ArgumentParser(description="Extract Mathlib4 lemmas to JSONL")
|
| 71 |
+
parser.add_argument(
|
| 72 |
+
"--mathlib-path",
|
| 73 |
+
required=True,
|
| 74 |
+
help="Path to the root of the Mathlib4 source tree (e.g. ~/.elan/toolchains/...)",
|
| 75 |
+
)
|
| 76 |
+
parser.add_argument(
|
| 77 |
+
"--output",
|
| 78 |
+
default="data/mathlib_lemmas.jsonl",
|
| 79 |
+
help="Output JSONL file path.",
|
| 80 |
+
)
|
| 81 |
+
args = parser.parse_args()
|
| 82 |
+
extract_mathlib(args.mathlib_path, args.output)
|
requirements.txt
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
anthropic>=0.30.0
|
| 2 |
+
faiss-cpu>=1.8.0
|
| 3 |
+
sentence-transformers>=2.7.0
|
| 4 |
+
rich>=13.7.1
|
| 5 |
+
pexpect>=4.9.0
|
| 6 |
+
python-dotenv>=1.0.1
|
src/__init__.py
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
# lean4-helper src package
|
src/agent.py
ADDED
|
@@ -0,0 +1,159 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
agent.py
|
| 3 |
+
|
| 4 |
+
The core agentic self-correcting loop.
|
| 5 |
+
|
| 6 |
+
Given a Lean 4 theorem statement, this agent will:
|
| 7 |
+
1. Retrieve relevant Mathlib4 lemmas via FAISS.
|
| 8 |
+
2. Prompt Claude to generate a proof.
|
| 9 |
+
3. Validate the proof with the Lean compiler.
|
| 10 |
+
4. On failure, feed the compiler error back to Claude and retry.
|
| 11 |
+
5. Return the final proof on success or raise after max retries.
|
| 12 |
+
"""
|
| 13 |
+
|
| 14 |
+
import os
|
| 15 |
+
from typing import Optional
|
| 16 |
+
|
| 17 |
+
import anthropic
|
| 18 |
+
|
| 19 |
+
from .lean_env import LeanResult, run_lean
|
| 20 |
+
from .retriever import Retriever
|
| 21 |
+
|
| 22 |
+
SYSTEM_PROMPT = """\
|
| 23 |
+
You are an expert Lean 4 theorem prover with deep knowledge of Mathlib4.
|
| 24 |
+
Your task is to write a complete, valid Lean 4 proof for the given theorem.
|
| 25 |
+
|
| 26 |
+
Rules:
|
| 27 |
+
- Output ONLY valid Lean 4 code. No explanation, no markdown, no fences.
|
| 28 |
+
- Begin with the necessary imports (e.g., `import Mathlib`).
|
| 29 |
+
- Use `theorem` or `lemma` keyword as appropriate.
|
| 30 |
+
- Use the provided relevant Mathlib4 lemmas as hints.
|
| 31 |
+
- If a previous attempt failed with a compiler error, fix only the reported issue.
|
| 32 |
+
"""
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
def _build_user_message(
|
| 36 |
+
theorem: str,
|
| 37 |
+
context_lemmas: list[dict],
|
| 38 |
+
previous_error: Optional[str] = None,
|
| 39 |
+
previous_code: Optional[str] = None,
|
| 40 |
+
) -> str:
|
| 41 |
+
"""
|
| 42 |
+
Construct the user message for Claude.
|
| 43 |
+
"""
|
| 44 |
+
parts = []
|
| 45 |
+
|
| 46 |
+
# Relevant Mathlib4 lemmas from RAG
|
| 47 |
+
if context_lemmas:
|
| 48 |
+
parts.append("## Potentially Relevant Mathlib4 Lemmas\n")
|
| 49 |
+
for lemma in context_lemmas:
|
| 50 |
+
name = lemma.get("name", "?")
|
| 51 |
+
typ = lemma.get("type", "?")
|
| 52 |
+
doc = lemma.get("doc", "")
|
| 53 |
+
parts.append(f"- `{name} : {typ}`" + (f" — {doc}" if doc else ""))
|
| 54 |
+
parts.append("")
|
| 55 |
+
|
| 56 |
+
# The theorem to prove
|
| 57 |
+
parts.append(f"## Theorem to Prove\n```lean\n{theorem}\n```\n")
|
| 58 |
+
|
| 59 |
+
# Feedback from previous attempt
|
| 60 |
+
if previous_code and previous_error:
|
| 61 |
+
parts.append("## Previous Attempt (FAILED)\n")
|
| 62 |
+
parts.append(f"```lean\n{previous_code}\n```\n")
|
| 63 |
+
parts.append(f"**Compiler Error:**\n```\n{previous_error}\n```\n")
|
| 64 |
+
parts.append("Please fix the error above and output the corrected proof.")
|
| 65 |
+
else:
|
| 66 |
+
parts.append("Please generate a complete Lean 4 proof for the theorem above.")
|
| 67 |
+
|
| 68 |
+
return "\n".join(parts)
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
class ProverAgent:
|
| 72 |
+
"""
|
| 73 |
+
Agentic prover that iteratively generates and validates Lean 4 proofs.
|
| 74 |
+
"""
|
| 75 |
+
|
| 76 |
+
def __init__(
|
| 77 |
+
self,
|
| 78 |
+
retriever: Optional[Retriever] = None,
|
| 79 |
+
model: str = "claude-opus-4-5",
|
| 80 |
+
max_retries: int = 5,
|
| 81 |
+
k_lemmas: int = 5,
|
| 82 |
+
lean_timeout: int = 60,
|
| 83 |
+
):
|
| 84 |
+
"""
|
| 85 |
+
Args:
|
| 86 |
+
retriever: Optional FAISS Retriever. If None, no RAG context is used.
|
| 87 |
+
model: Claude model ID to use.
|
| 88 |
+
max_retries: Maximum number of proof attempts before giving up.
|
| 89 |
+
k_lemmas: Number of Mathlib lemmas to retrieve per query.
|
| 90 |
+
lean_timeout: Timeout in seconds for the Lean compiler.
|
| 91 |
+
"""
|
| 92 |
+
api_key = os.environ.get("ANTHROPIC_API_KEY")
|
| 93 |
+
if not api_key:
|
| 94 |
+
raise EnvironmentError(
|
| 95 |
+
"ANTHROPIC_API_KEY is not set. Please add it to your .env file."
|
| 96 |
+
)
|
| 97 |
+
self.client = anthropic.Anthropic(api_key=api_key)
|
| 98 |
+
self.retriever = retriever
|
| 99 |
+
self.model = model
|
| 100 |
+
self.max_retries = max_retries
|
| 101 |
+
self.k_lemmas = k_lemmas
|
| 102 |
+
self.lean_timeout = lean_timeout
|
| 103 |
+
|
| 104 |
+
def prove(self, theorem: str, on_attempt=None) -> tuple[bool, str, int]:
|
| 105 |
+
"""
|
| 106 |
+
Attempt to prove the given Lean 4 theorem statement.
|
| 107 |
+
|
| 108 |
+
Args:
|
| 109 |
+
theorem: A Lean 4 theorem/lemma declaration string.
|
| 110 |
+
on_attempt: Optional callback(attempt, code, result) for streaming UI.
|
| 111 |
+
|
| 112 |
+
Returns:
|
| 113 |
+
(success, final_code, num_attempts)
|
| 114 |
+
"""
|
| 115 |
+
# RAG: retrieve relevant lemmas
|
| 116 |
+
context_lemmas = []
|
| 117 |
+
if self.retriever:
|
| 118 |
+
context_lemmas = self.retriever.retrieve(theorem, k=self.k_lemmas)
|
| 119 |
+
|
| 120 |
+
messages = []
|
| 121 |
+
previous_code = None
|
| 122 |
+
previous_error = None
|
| 123 |
+
|
| 124 |
+
for attempt in range(1, self.max_retries + 1):
|
| 125 |
+
user_content = _build_user_message(
|
| 126 |
+
theorem=theorem,
|
| 127 |
+
context_lemmas=context_lemmas,
|
| 128 |
+
previous_error=previous_error,
|
| 129 |
+
previous_code=previous_code,
|
| 130 |
+
)
|
| 131 |
+
|
| 132 |
+
messages.append({"role": "user", "content": user_content})
|
| 133 |
+
|
| 134 |
+
# Call Claude
|
| 135 |
+
response = self.client.messages.create(
|
| 136 |
+
model=self.model,
|
| 137 |
+
max_tokens=2048,
|
| 138 |
+
system=SYSTEM_PROMPT,
|
| 139 |
+
messages=messages,
|
| 140 |
+
)
|
| 141 |
+
generated_code = response.content[0].text.strip()
|
| 142 |
+
|
| 143 |
+
# Append assistant response to conversation history
|
| 144 |
+
messages.append({"role": "assistant", "content": generated_code})
|
| 145 |
+
|
| 146 |
+
# Validate with Lean compiler
|
| 147 |
+
lean_result: LeanResult = run_lean(generated_code, timeout=self.lean_timeout)
|
| 148 |
+
|
| 149 |
+
if on_attempt:
|
| 150 |
+
on_attempt(attempt, generated_code, lean_result)
|
| 151 |
+
|
| 152 |
+
if lean_result.success:
|
| 153 |
+
return True, generated_code, attempt
|
| 154 |
+
|
| 155 |
+
# Feed error back for next iteration
|
| 156 |
+
previous_code = generated_code
|
| 157 |
+
previous_error = "\n".join(lean_result.errors) or lean_result.output[:500]
|
| 158 |
+
|
| 159 |
+
return False, previous_code or "", self.max_retries
|
src/cli.py
ADDED
|
@@ -0,0 +1,162 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
cli.py
|
| 3 |
+
|
| 4 |
+
Interactive CLI for the Agentic Lean 4 Theorem Prover.
|
| 5 |
+
|
| 6 |
+
Usage:
|
| 7 |
+
python -m src.cli
|
| 8 |
+
python -m src.cli --theorem "theorem add_comm (a b : Nat) : a + b = b + a := by sorry"
|
| 9 |
+
python -m src.cli --file theorems.lean
|
| 10 |
+
"""
|
| 11 |
+
|
| 12 |
+
import argparse
|
| 13 |
+
import os
|
| 14 |
+
import sys
|
| 15 |
+
from pathlib import Path
|
| 16 |
+
|
| 17 |
+
from dotenv import load_dotenv
|
| 18 |
+
from rich.console import Console
|
| 19 |
+
from rich.markdown import Markdown
|
| 20 |
+
from rich.panel import Panel
|
| 21 |
+
from rich.syntax import Syntax
|
| 22 |
+
from rich.text import Text
|
| 23 |
+
from rich import print as rprint
|
| 24 |
+
from rich.rule import Rule
|
| 25 |
+
from rich.spinner import Spinner
|
| 26 |
+
from rich.live import Live
|
| 27 |
+
from rich import box
|
| 28 |
+
|
| 29 |
+
from .agent import ProverAgent
|
| 30 |
+
from .lean_env import LeanResult
|
| 31 |
+
|
| 32 |
+
# Load .env for ANTHROPIC_API_KEY
|
| 33 |
+
load_dotenv()
|
| 34 |
+
|
| 35 |
+
console = Console()
|
| 36 |
+
|
| 37 |
+
BANNER = """
|
| 38 |
+
[bold cyan]╔══════════════════════════════════════════════════╗[/bold cyan]
|
| 39 |
+
[bold cyan]║ 🧮 Agentic Lean 4 Theorem Prover ║[/bold cyan]
|
| 40 |
+
[bold cyan]║ Powered by Claude + FAISS + Mathlib4 RAG ║[/bold cyan]
|
| 41 |
+
[bold cyan]╚══════════════════════════════════════════════════╝[/bold cyan]
|
| 42 |
+
"""
|
| 43 |
+
|
| 44 |
+
|
| 45 |
+
def on_attempt_callback(attempt: int, code: str, result: LeanResult):
|
| 46 |
+
"""Rich-formatted callback for each proof attempt."""
|
| 47 |
+
console.print(Rule(f"[bold]Attempt {attempt}[/bold]", style="dim"))
|
| 48 |
+
console.print(Syntax(code, "lean", theme="monokai", line_numbers=True))
|
| 49 |
+
|
| 50 |
+
if result.success:
|
| 51 |
+
console.print(Panel("[bold green]✅ Lean accepted this proof![/bold green]", box=box.ROUNDED))
|
| 52 |
+
else:
|
| 53 |
+
errors_text = "\n".join(result.errors) if result.errors else result.output[:300]
|
| 54 |
+
console.print(
|
| 55 |
+
Panel(
|
| 56 |
+
f"[bold red]❌ Compiler rejected proof:[/bold red]\n\n[red]{errors_text}[/red]",
|
| 57 |
+
box=box.ROUNDED,
|
| 58 |
+
)
|
| 59 |
+
)
|
| 60 |
+
|
| 61 |
+
|
| 62 |
+
def run_prover(theorem: str, use_rag: bool = True, max_retries: int = 5):
|
| 63 |
+
"""
|
| 64 |
+
Main entry point: set up the agent and run it on the given theorem.
|
| 65 |
+
"""
|
| 66 |
+
console.print(BANNER)
|
| 67 |
+
console.print(Panel(f"[bold yellow]Theorem to prove:[/bold yellow]\n\n{theorem}", box=box.ROUNDED))
|
| 68 |
+
|
| 69 |
+
# Optionally initialize retriever
|
| 70 |
+
retriever = None
|
| 71 |
+
if use_rag:
|
| 72 |
+
index_path = "data/mathlib.index"
|
| 73 |
+
meta_path = "data/mathlib_meta.pkl"
|
| 74 |
+
if os.path.exists(index_path) and os.path.exists(meta_path):
|
| 75 |
+
from .retriever import Retriever
|
| 76 |
+
console.print("[dim]Loading FAISS index for Mathlib4 RAG...[/dim]")
|
| 77 |
+
retriever = Retriever(index_path=index_path, meta_path=meta_path)
|
| 78 |
+
console.print("[green]✓ FAISS index loaded.[/green]\n")
|
| 79 |
+
else:
|
| 80 |
+
console.print(
|
| 81 |
+
"[yellow]⚠ FAISS index not found. Running without RAG context.[/yellow]\n"
|
| 82 |
+
"[dim]Run `python -m src.retriever` to build the index first.[/dim]\n"
|
| 83 |
+
)
|
| 84 |
+
|
| 85 |
+
agent = ProverAgent(retriever=retriever, max_retries=max_retries)
|
| 86 |
+
|
| 87 |
+
console.print(Rule("[bold]Starting Proof Search[/bold]", style="cyan"))
|
| 88 |
+
success, final_code, num_attempts = agent.prove(theorem, on_attempt=on_attempt_callback)
|
| 89 |
+
|
| 90 |
+
console.print(Rule(style="cyan"))
|
| 91 |
+
if success:
|
| 92 |
+
console.print(
|
| 93 |
+
Panel(
|
| 94 |
+
f"[bold green]🎉 Proof found in {num_attempts} attempt(s)![/bold green]",
|
| 95 |
+
box=box.DOUBLE,
|
| 96 |
+
)
|
| 97 |
+
)
|
| 98 |
+
console.print("\n[bold]Final Proof:[/bold]")
|
| 99 |
+
console.print(Syntax(final_code, "lean", theme="monokai", line_numbers=True))
|
| 100 |
+
else:
|
| 101 |
+
console.print(
|
| 102 |
+
Panel(
|
| 103 |
+
f"[bold red]💀 Failed to find a proof after {num_attempts} attempts.[/bold red]\n"
|
| 104 |
+
"[dim]Try increasing --max-retries or refining the theorem statement.[/dim]",
|
| 105 |
+
box=box.DOUBLE,
|
| 106 |
+
)
|
| 107 |
+
)
|
| 108 |
+
if final_code:
|
| 109 |
+
console.print("\n[bold]Last Generated Code:[/bold]")
|
| 110 |
+
console.print(Syntax(final_code, "lean", theme="monokai", line_numbers=True))
|
| 111 |
+
|
| 112 |
+
|
| 113 |
+
def main():
|
| 114 |
+
parser = argparse.ArgumentParser(
|
| 115 |
+
description="Agentic Lean 4 Theorem Prover powered by Claude + FAISS + Mathlib4"
|
| 116 |
+
)
|
| 117 |
+
group = parser.add_mutually_exclusive_group()
|
| 118 |
+
group.add_argument(
|
| 119 |
+
"--theorem",
|
| 120 |
+
type=str,
|
| 121 |
+
help="A Lean 4 theorem declaration string to prove.",
|
| 122 |
+
)
|
| 123 |
+
group.add_argument(
|
| 124 |
+
"--file",
|
| 125 |
+
type=str,
|
| 126 |
+
help="Path to a .lean file containing the theorem to prove.",
|
| 127 |
+
)
|
| 128 |
+
parser.add_argument(
|
| 129 |
+
"--no-rag",
|
| 130 |
+
action="store_true",
|
| 131 |
+
help="Disable Mathlib4 RAG retrieval.",
|
| 132 |
+
)
|
| 133 |
+
parser.add_argument(
|
| 134 |
+
"--max-retries",
|
| 135 |
+
type=int,
|
| 136 |
+
default=5,
|
| 137 |
+
help="Maximum number of proof attempts (default: 5).",
|
| 138 |
+
)
|
| 139 |
+
args = parser.parse_args()
|
| 140 |
+
|
| 141 |
+
if args.file:
|
| 142 |
+
theorem = Path(args.file).read_text().strip()
|
| 143 |
+
elif args.theorem:
|
| 144 |
+
theorem = args.theorem.strip()
|
| 145 |
+
else:
|
| 146 |
+
console.print("[bold cyan]Enter your Lean 4 theorem (end with Ctrl+D on a new line):[/bold cyan]")
|
| 147 |
+
try:
|
| 148 |
+
lines = sys.stdin.read()
|
| 149 |
+
theorem = lines.strip()
|
| 150 |
+
except KeyboardInterrupt:
|
| 151 |
+
console.print("\n[yellow]Cancelled.[/yellow]")
|
| 152 |
+
sys.exit(0)
|
| 153 |
+
|
| 154 |
+
if not theorem:
|
| 155 |
+
console.print("[red]No theorem provided. Exiting.[/red]")
|
| 156 |
+
sys.exit(1)
|
| 157 |
+
|
| 158 |
+
run_prover(theorem=theorem, use_rag=not args.no_rag, max_retries=args.max_retries)
|
| 159 |
+
|
| 160 |
+
|
| 161 |
+
if __name__ == "__main__":
|
| 162 |
+
main()
|
src/lean_env.py
ADDED
|
@@ -0,0 +1,83 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
lean_env.py
|
| 3 |
+
|
| 4 |
+
Wrapper around the Lean 4 compiler. Sends Lean 4 code to the compiler,
|
| 5 |
+
captures stdout/stderr, and parses the output to determine success or failure.
|
| 6 |
+
"""
|
| 7 |
+
|
| 8 |
+
import subprocess
|
| 9 |
+
import tempfile
|
| 10 |
+
import os
|
| 11 |
+
import re
|
| 12 |
+
from dataclasses import dataclass
|
| 13 |
+
from typing import Optional
|
| 14 |
+
|
| 15 |
+
|
| 16 |
+
@dataclass
|
| 17 |
+
class LeanResult:
|
| 18 |
+
success: bool
|
| 19 |
+
output: str
|
| 20 |
+
errors: list[str]
|
| 21 |
+
raw: str
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
def run_lean(code: str, timeout: int = 60) -> LeanResult:
|
| 25 |
+
"""
|
| 26 |
+
Write `code` to a temporary .lean file, run `lean` on it, and parse
|
| 27 |
+
the compiler output.
|
| 28 |
+
|
| 29 |
+
Args:
|
| 30 |
+
code: Full Lean 4 source code string to validate.
|
| 31 |
+
timeout: Maximum seconds to wait for the compiler.
|
| 32 |
+
|
| 33 |
+
Returns:
|
| 34 |
+
A LeanResult with success flag, cleaned output, and parsed errors.
|
| 35 |
+
"""
|
| 36 |
+
with tempfile.NamedTemporaryFile(
|
| 37 |
+
mode="w", suffix=".lean", delete=False
|
| 38 |
+
) as f:
|
| 39 |
+
f.write(code)
|
| 40 |
+
tmp_path = f.name
|
| 41 |
+
|
| 42 |
+
try:
|
| 43 |
+
result = subprocess.run(
|
| 44 |
+
["lean", tmp_path],
|
| 45 |
+
capture_output=True,
|
| 46 |
+
text=True,
|
| 47 |
+
timeout=timeout,
|
| 48 |
+
)
|
| 49 |
+
raw = result.stdout + result.stderr
|
| 50 |
+
errors = _parse_errors(raw)
|
| 51 |
+
success = result.returncode == 0 and len(errors) == 0
|
| 52 |
+
return LeanResult(success=success, output=raw.strip(), errors=errors, raw=raw)
|
| 53 |
+
except FileNotFoundError:
|
| 54 |
+
return LeanResult(
|
| 55 |
+
success=False,
|
| 56 |
+
output="",
|
| 57 |
+
errors=["Lean 4 not found. Please install it via elan."],
|
| 58 |
+
raw="",
|
| 59 |
+
)
|
| 60 |
+
except subprocess.TimeoutExpired:
|
| 61 |
+
return LeanResult(
|
| 62 |
+
success=False,
|
| 63 |
+
output="",
|
| 64 |
+
errors=[f"Lean compiler timed out after {timeout}s."],
|
| 65 |
+
raw="",
|
| 66 |
+
)
|
| 67 |
+
finally:
|
| 68 |
+
os.unlink(tmp_path)
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
def _parse_errors(output: str) -> list[str]:
|
| 72 |
+
"""
|
| 73 |
+
Extract human-readable error messages from Lean 4 compiler output.
|
| 74 |
+
Lean 4 errors look like:
|
| 75 |
+
/path/to/file.lean:10:5: error: unknown identifier 'foo'
|
| 76 |
+
"""
|
| 77 |
+
pattern = re.compile(r".*?:\d+:\d+:\s*(error|warning):\s*(.+)")
|
| 78 |
+
errors = []
|
| 79 |
+
for line in output.splitlines():
|
| 80 |
+
m = pattern.match(line)
|
| 81 |
+
if m and m.group(1) == "error":
|
| 82 |
+
errors.append(m.group(2).strip())
|
| 83 |
+
return errors
|
src/retriever.py
ADDED
|
@@ -0,0 +1,113 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""
|
| 2 |
+
retriever.py
|
| 3 |
+
|
| 4 |
+
Builds and queries a FAISS index over Mathlib4 lemma docstrings.
|
| 5 |
+
|
| 6 |
+
Usage:
|
| 7 |
+
# Build index from a jsonl file of lemmas:
|
| 8 |
+
build_index("data/mathlib_lemmas.jsonl", "data/mathlib.index")
|
| 9 |
+
|
| 10 |
+
# Query at runtime:
|
| 11 |
+
retriever = Retriever("data/mathlib.index", "data/mathlib_lemmas.jsonl")
|
| 12 |
+
lemmas = retriever.retrieve("commutativity of addition", k=5)
|
| 13 |
+
"""
|
| 14 |
+
|
| 15 |
+
import json
|
| 16 |
+
import os
|
| 17 |
+
import pickle
|
| 18 |
+
from pathlib import Path
|
| 19 |
+
from typing import Optional
|
| 20 |
+
|
| 21 |
+
import faiss
|
| 22 |
+
import numpy as np
|
| 23 |
+
from sentence_transformers import SentenceTransformer
|
| 24 |
+
|
| 25 |
+
EMBEDDING_MODEL = "all-MiniLM-L6-v2"
|
| 26 |
+
INDEX_FILE = "data/mathlib.index"
|
| 27 |
+
META_FILE = "data/mathlib_meta.pkl"
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
class Retriever:
|
| 31 |
+
"""
|
| 32 |
+
Manages the FAISS index and performs semantic retrieval of Mathlib4 lemmas.
|
| 33 |
+
"""
|
| 34 |
+
|
| 35 |
+
def __init__(
|
| 36 |
+
self,
|
| 37 |
+
index_path: str = INDEX_FILE,
|
| 38 |
+
meta_path: str = META_FILE,
|
| 39 |
+
model_name: str = EMBEDDING_MODEL,
|
| 40 |
+
):
|
| 41 |
+
self.model = SentenceTransformer(model_name)
|
| 42 |
+
self.index = faiss.read_index(index_path)
|
| 43 |
+
with open(meta_path, "rb") as f:
|
| 44 |
+
self.metadata = pickle.load(f) # list of dicts: {name, type, doc}
|
| 45 |
+
|
| 46 |
+
def retrieve(self, query: str, k: int = 5) -> list[dict]:
|
| 47 |
+
"""
|
| 48 |
+
Return the top-k most relevant Mathlib4 lemmas for a given query.
|
| 49 |
+
|
| 50 |
+
Args:
|
| 51 |
+
query: Natural language or Lean theorem statement.
|
| 52 |
+
k: Number of results to return.
|
| 53 |
+
|
| 54 |
+
Returns:
|
| 55 |
+
List of lemma dicts with keys 'name', 'type', 'doc'.
|
| 56 |
+
"""
|
| 57 |
+
embedding = self.model.encode([query], normalize_embeddings=True)
|
| 58 |
+
embedding = np.array(embedding, dtype=np.float32)
|
| 59 |
+
_, indices = self.index.search(embedding, k)
|
| 60 |
+
return [self.metadata[i] for i in indices[0] if i < len(self.metadata)]
|
| 61 |
+
|
| 62 |
+
|
| 63 |
+
def build_index(
|
| 64 |
+
lemmas_jsonl: str = "data/mathlib_lemmas.jsonl",
|
| 65 |
+
index_path: str = INDEX_FILE,
|
| 66 |
+
meta_path: str = META_FILE,
|
| 67 |
+
model_name: str = EMBEDDING_MODEL,
|
| 68 |
+
) -> None:
|
| 69 |
+
"""
|
| 70 |
+
Build a FAISS index from a JSONL file of Mathlib4 lemmas.
|
| 71 |
+
|
| 72 |
+
Each line in `lemmas_jsonl` should be a JSON object with:
|
| 73 |
+
- "name": lemma name (str)
|
| 74 |
+
- "type": Lean type signature (str)
|
| 75 |
+
- "doc": docstring or description (str, optional)
|
| 76 |
+
|
| 77 |
+
Args:
|
| 78 |
+
lemmas_jsonl: Path to the input JSONL file.
|
| 79 |
+
index_path: Where to save the FAISS index file.
|
| 80 |
+
meta_path: Where to save the metadata pickle file.
|
| 81 |
+
model_name: SentenceTransformer model to use for embeddings.
|
| 82 |
+
"""
|
| 83 |
+
print(f"Loading lemmas from {lemmas_jsonl}...")
|
| 84 |
+
metadata = []
|
| 85 |
+
texts = []
|
| 86 |
+
with open(lemmas_jsonl, "r") as f:
|
| 87 |
+
for line in f:
|
| 88 |
+
entry = json.loads(line.strip())
|
| 89 |
+
metadata.append(entry)
|
| 90 |
+
# Build a rich text representation for embedding
|
| 91 |
+
text = f"{entry.get('name', '')} : {entry.get('type', '')}. {entry.get('doc', '')}"
|
| 92 |
+
texts.append(text)
|
| 93 |
+
|
| 94 |
+
print(f"Loaded {len(texts)} lemmas. Generating embeddings...")
|
| 95 |
+
model = SentenceTransformer(model_name)
|
| 96 |
+
embeddings = model.encode(texts, normalize_embeddings=True, show_progress_bar=True)
|
| 97 |
+
embeddings = np.array(embeddings, dtype=np.float32)
|
| 98 |
+
|
| 99 |
+
print("Building FAISS index...")
|
| 100 |
+
dim = embeddings.shape[1]
|
| 101 |
+
index = faiss.IndexFlatIP(dim) # Inner product = cosine similarity (normalized)
|
| 102 |
+
index.add(embeddings)
|
| 103 |
+
|
| 104 |
+
os.makedirs(Path(index_path).parent, exist_ok=True)
|
| 105 |
+
faiss.write_index(index, index_path)
|
| 106 |
+
with open(meta_path, "wb") as f:
|
| 107 |
+
pickle.dump(metadata, f)
|
| 108 |
+
|
| 109 |
+
print(f"Index saved to {index_path} ({len(texts)} vectors, dim={dim})")
|
| 110 |
+
|
| 111 |
+
|
| 112 |
+
if __name__ == "__main__":
|
| 113 |
+
build_index()
|