README.md

MaterialMind

Decisions, not summaries. MaterialMind is an LLM-powered materials-selection assistant that turns engineering requirements into an evidence-backed, ranked shortlist with page-level citations. It runs locally (Flask + Ollama) and searches your own PDF corpus via a lightweight RAG index.

What MaterialMind does

• Retrieves → Ranks → Explains: finds the most relevant pages, scores candidate materials against your constraints, and explains trade-offs with citations.
• Local and private: your PDFs never leave your machine.
• Robust ingestion: handles real-world PDFs (papers, standards, reports, textbooks); incremental updates.
• Friendly UI: clean dark theme, background tiles, live weight controls (accepts 40/30/20/10 or 0.4/0.3/0.2/0.1).
• Exact math: backend normalizes weights to sum to exactly 1.0000.

Repository layout

materialmind/ (your corpus lives here)
├─ sources/ (drop your PDFs here)
└─ index/ (vector DB; auto-created)

rag_mini.py (RAG: index/search/ask/answer/export)
app_user.py (Flask app for end users)
templates/ (base.html, index.html, results.html)
static/ (styles.css and images: 11.png, 22.jpg)

Note: We work with PDFs you drop into materialmind/sources (papers, standards, reports). You don’t need vendor datasheets; if you have them as PDFs, drop them in like any other PDF.

Prerequisites

• Python 3.10+
• pip and a virtual environment
• Ollama (for model-generated answers). Example model: qwen2.5:7b-instruct
• Optional: ocrmypdf + tesseract (only for scanned PDFs with no selectable text)

Installation

Create and activate a virtual environment.

macOS / Linux:
python3 -m venv .venv
source .venv/bin/activate

Windows (PowerShell):
python -m venv .venv
.venv\Scripts\Activate.ps1

Install dependencies.

Apple Silicon (macOS):
pip install -U fastembed onnxruntime-silicon chromadb pypdf pymupdf markdown filelock flask flask-cors

Linux/Windows:
pip install -U fastembed onnxruntime chromadb pypdf pymupdf markdown filelock flask flask-cors

Install and start Ollama; pull a local model.

macOS (example):
brew install ollama
ollama serve &
ollama pull qwen2.5:7b-instruct

Confirm paths.

rag_mini.py uses a folder named “materialmind” next to the script:
BASE_DIR = Path(file).resolve().parent / "materialmind"

Ensure the folder exists:
materialmind/sources (create this and drop PDFs here)

Add your PDFs

Drop research papers, standards, reports, and textbooks into:
materialmind/sources

Tips:
• Prefer publisher PDFs (selectable text).
• If a PDF is scanned (no text), OCR it:
ocrmypdf input.pdf output_ocr.pdf
then place output_ocr.pdf in materialmind/sources

Build or update the index

First time (full index):
python rag_mini.py --rebuild

Later (only changed/new PDFs):
python rag_mini.py --update

Optional backup:
python rag_mini.py --backup

Optional export to JSONL (for inspection):
python rag_mini.py --export-json ./materialmind_dump.jsonl

Quick retrieval check (no model required)

Example:
python rag_mini.py --ask "Which alloys resist pitting in seawater better than 316L?"

You will see top-k snippets with file:page citations from your PDFs.

Run the UI

Start the Flask app:
python app_user.py

Open:
http://127.0.0.1:5000/

Usage:
• Fill environment/temperature/constraints.
• Set weights as percentages (e.g., 40/30/20/10) or fractions (0.4/0.3/0.2/0.1). The form enables the button only when the sum equals 100% (or 1.0).
• Click “Get ranked shortlist” to see the ranked table, material cards, and page-level citations.

How it works

• Retrieval: FastEmbed (ONNX) creates embeddings; Chroma stores and retrieves chunks with file + page metadata.
• Decision layer: your constraints and weights guide the model to produce a structured shortlist JSON (name, score, reasons, trade-offs, citations).
• Local LLM: by default qwen2.5:7b-instruct via Ollama; you can swap models freely.

CLI reference (rag_mini.py)

--rebuild rebuild the entire index from PDFs in materialmind/sources
--update incremental index of only changed/new PDFs
--backup copy the current index to a timestamped folder
--export-json PATH dump records to JSONL (optional)
--ask "question" retrieval only with citations
--answer "question" --model NAME --k N --show
retrieval plus local LLM answer (Ollama required)

Customization

Change model (Ollama):
ollama pull mistral:7b-instruct
then set “Model” in the UI to mistral:7b-instruct

Tune retrieval (rag_mini.py):
CHUNK_CHARS, CHUNK_OVERLAP, DEFAULT_TOPK

Swap embedding model (rag_mini.py):
EMB_MODEL = "BAAI/bge-small-en-v1.5" (good default). Other FastEmbed models also work.

Theme colors:
Edit CSS variables in static/styles.css (:root { … }).

Troubleshooting

“Get ranked shortlist” button disabled:
Enter weights as 40/30/20/10 or 0.4/0.3/0.2/0.1 until the sum reads 100%.

Ollama not found / model errors:
Install/start/pull as above. Retrieval still works without Ollama; only model-generated answers require it.

Chroma lock / mutex errors:
Close other processes using the index. If stuck:
python rag_mini.py --backup
rm -rf materialmind/index/chroma_v3
python rag_mini.py --rebuild

No text in a PDF:
OCR it with ocrmypdf, then re-index.

Port already in use:
Change the host/port in app_user.py.

Privacy

Everything runs locally by default. Your PDFs are indexed on disk and never uploaded. Remove PDFs from materialmind/sources and run --update to de-index.

Roadmap

Team/on-prem sharing, rule/property packs (e.g., PREN/oxidation windows/standards checks), uploads & tagging UI, LoRA adapter for stricter JSON, evaluation harness for grounding and JSON validity.

License

MIT (or your chosen permissive license). Add a LICENSE file to the repository.

One-minute demo

Drop a few corrosion or seawater PDFs into materialmind/sources.

Build: python rag_mini.py --rebuild

Run UI: python app_user.py

Query seawater at 20–25 °C, UTS ≥ 600 MPa, weights 50/30/10/10.

Show ranked shortlist, open a citation (file:page).

Add another PDF, run --update, rerun the query, and show the new citation.