Space README + Gradio entrypoint

.claude/skills/verify/SKILL.md

@@ -0,0 +1,44 @@
+---
+name: verify
+description: How to build and run lumen-rag to verify a change
+---
+
+# Verifying lumen-rag changes
+
+No Chrome/CDP available in this WSL box (`which google-chrome chromium` → nothing) —
+browser-harness will fail with `DevToolsActivePort not found`. Don't waste time on it.
+
+## CLI / core engine
+```
+uv venv /tmp/.../scratchpad/.venv
+uv pip install --python /tmp/.../scratchpad/.venv/bin/python -e .
+source /tmp/.../scratchpad/.venv/bin/activate
+lumen ingest data/docs && lumen ask "..." && lumen eval data/eval.jsonl --k 3
+```
+
+## Gradio app (deploy/hf-space/gradio_app.py)
+No browser to drive it visually — instead exercise the exact same Blocks event
+endpoints the UI buttons call, via `gradio_client` (same code path as a click,
+not a unit-test import):
+
+```
+uv pip install --python .../venv/bin/python -e . gradio gradio_client
+GRADIO_SERVER_PORT=7871 .../venv/bin/python deploy/hf-space/gradio_app.py &
+python - <<'PY'
+from gradio_client import Client, handle_file
+c = Client("http://127.0.0.1:7871/")
+c.predict(api_name="/load_sample")
+c.predict("question", 5, "hybrid", api_name="/ask")
+c.predict(5, api_name="/run_eval")
+c.predict(api_name="/reset_index")
+c.predict([handle_file("path")], api_name="/upload_files")
+PY
+```
+Endpoint names are the Python function names (`/load_sample`, `/ask`, `/run_eval`,
+`/reset_index`, `/upload_files`) as declared by the `.click()`/`.upload()` wiring
+in `gradio_app.py`. Kill the server after with `pkill -f gradio_app.py`.
+
+## Deploy script (deploy/hf-space/push.sh)
+Force-pushes to a real HF Space — destructive/irreversible, don't run it live.
+Review by diff only; check README.md `app_file:` matches whatever push.sh copies
+to `app.py` in the worktree.

.env.example

@@ -0,0 +1,10 @@
+# LLM + embedding provider (optional).
+# Leave OPENAI_API_KEY unset to run fully offline with the built-in
+# deterministic hashing embedder — great for tests, demos, and CI.
+OPENAI_API_KEY=""
+OPENAI_BASE_URL=""
+EMBED_MODEL="text-embedding-3-small"
+CHAT_MODEL="gpt-4o-mini"
+
+# Where the vector index is persisted.
+LUMEN_INDEX_DIR=".lumen_index"

.github/workflows/ci.yml

@@ -0,0 +1,28 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.10", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: pip
+      - run: pip install -e ".[dev]"
+      - name: Lint
+        run: ruff check lumen_rag tests
+      - name: Test (offline mode — no API key)
+        run: pytest -q
+      - name: Eval regression guard
+        run: |
+          lumen ingest data/docs
+          python scripts/check_eval.py data/eval.jsonl --k 3 --mode hybrid

.gitignore

@@ -0,0 +1,15 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+.eggs/
+build/
+dist/
+.venv/
+venv/
+.env
+.pytest_cache/
+.ruff_cache/
+*.npz
+.lumen_index/
+.DS_Store
+.claude-flow/

Dockerfile

@@ -0,0 +1,11 @@
+FROM python:3.12-slim
+
+WORKDIR /app
+ENV PYTHONUNBUFFERED=1 PIP_NO_CACHE_DIR=1
+
+COPY pyproject.toml README.md ./
+COPY lumen_rag ./lumen_rag
+RUN pip install --no-cache-dir ".[openai]"
+
+EXPOSE 8000
+CMD ["lumen", "serve", "--host", "0.0.0.0", "--port", "8000"]

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Rishabh Verma
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

Makefile

@@ -0,0 +1,18 @@
+.PHONY: install test lint demo serve
+
+install:
+	pip install -e ".[dev,openai]"
+
+test:
+	pytest -q
+
+lint:
+	ruff check lumen_rag tests
+
+demo:
+	lumen ingest data/docs
+	lumen ask "How many approvals does a billing change need?"
+	lumen eval data/eval.jsonl --k 3
+
+serve:
+	lumen serve

README.md

@@ -1,13 +1,20 @@
 ---
-title: Lumen Rag
-emoji: 🌖
-colorFrom: red
-colorTo: yellow
+title: Lumen RAG
+emoji: 🔦
+colorFrom: blue
+colorTo: indigo
 sdk: gradio
-sdk_version: 6.20.0
-python_version: '3.12'
+sdk_version: "6.20.0"
 app_file: app.py
+short_description: Transparent, evaluated hybrid RAG demo
 pinned: false
+license: mit
 ---
 
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# Lumen RAG — live demo
+
+Upload documents, ask questions, and watch retrieval quality (recall@k, MRR,
+nDCG@k) render live from the eval harness. Runs 100% offline on a
+deterministic hashing embedder — no API key required.
+
+Source: https://github.com/WickTech/lumen-rag

app.py

@@ -0,0 +1,113 @@
+"""Gradio front-end for the Lumen RAG Space (free-tier CPU, no Docker needed).
+
+Wraps `lumen_rag.RagEngine` directly in-process — no HTTP layer.
+"""
+from __future__ import annotations
+
+from pathlib import Path
+
+import gradio as gr
+
+from lumen_rag.engine import RagEngine
+from lumen_rag.eval import evaluate
+from lumen_rag.eval.harness import load_cases
+from lumen_rag.ingestion.loaders import _LOADERS, load_file
+from lumen_rag.retrieval import Retriever
+
+_SAMPLE_DIR = Path(__file__).resolve().parent.parent.parent / "data" / "docs"
+_EVAL_PATH = Path(__file__).resolve().parent.parent.parent / "data" / "eval.jsonl"
+
+engine = RagEngine()
+
+
+def load_sample() -> str:
+    files = sorted(p for p in _SAMPLE_DIR.iterdir() if p.suffix.lower() in _LOADERS)
+    docs = [load_file(p) for p in files]
+    total = engine.add_documents(docs)
+    return f"Indexed {len(docs)} sample files, {total} chunks."
+
+
+def upload_files(files: list[str]) -> str:
+    docs = []
+    for f in files:
+        suffix = Path(f).suffix.lower()
+        if suffix not in _LOADERS:
+            continue
+        doc = load_file(f)
+        doc["id"] = Path(f).stem
+        docs.append(doc)
+    if not docs:
+        return f"No supported files. Supported: {sorted(_LOADERS)}"
+    total = engine.add_documents(docs)
+    return f"Indexed {len(docs)} files, {total} chunks."
+
+
+def reset_index() -> str:
+    global engine
+    engine = RagEngine()
+    return "Index reset."
+
+
+def ask(question: str, k: int, mode: str):
+    if len(engine.store) == 0:
+        return "Index is empty — load the sample corpus or upload files first.", ""
+    result = engine.query(question, k=int(k), mode=mode)
+    citations = "\n".join(
+        f"[{c['n']}] {c['doc_id']} — {c['source']} (score={c['score']:.3f})"
+        for c in result.citations
+    )
+    return result.text, citations
+
+
+def run_eval(k: int) -> str:
+    if not _EVAL_PATH.exists():
+        return "Bundled eval set not found."
+    if len(engine.store) == 0:
+        return "Index is empty — load the sample corpus first."
+    cases = load_cases(_EVAL_PATH)
+    report = evaluate(Retriever(engine.store, engine.embedder), cases, k=int(k))
+    d = report.as_dict()
+    return "\n".join(f"{key}: {value}" for key, value in d.items())
+
+
+with gr.Blocks(title="Lumen RAG") as demo:
+    gr.Markdown(
+        "# Lumen RAG\n"
+        "Transparent, evaluated RAG: ingest documents, retrieve with hybrid "
+        "vector+BM25 search, answer with citations. Runs 100% offline on a "
+        "deterministic hashing embedder — no API key required."
+    )
+
+    with gr.Row():
+        sample_btn = gr.Button("Load sample corpus")
+        reset_btn = gr.Button("Reset index")
+    upload = gr.File(label="Or upload documents", file_count="multiple")
+    status = gr.Textbox(label="Index status", interactive=False)
+
+    sample_btn.click(load_sample, outputs=status)
+    reset_btn.click(reset_index, outputs=status)
+    upload.upload(upload_files, inputs=upload, outputs=status)
+
+    gr.Markdown("---")
+
+    question = gr.Textbox(label="Question")
+    with gr.Row():
+        k = gr.Slider(1, 20, value=5, step=1, label="k")
+        mode = gr.Dropdown(["hybrid", "vector", "bm25"], value="hybrid", label="Retrieval mode")
+    ask_btn = gr.Button("Ask", variant="primary")
+    answer_box = gr.Textbox(label="Answer", lines=4)
+    citations_box = gr.Textbox(label="Citations", lines=4)
+
+    ask_btn.click(ask, inputs=[question, k, mode], outputs=[answer_box, citations_box])
+
+    gr.Markdown("---")
+
+    with gr.Row():
+        eval_k = gr.Slider(1, 20, value=5, step=1, label="eval k")
+        eval_btn = gr.Button("Run retrieval eval (recall@k, MRR, nDCG@k)")
+    eval_box = gr.Textbox(label="Eval report", lines=6)
+    eval_btn.click(run_eval, inputs=eval_k, outputs=eval_box)
+
+
+if __name__ == "__main__":
+    demo.launch()

benchmark_results.json

@@ -0,0 +1,36 @@
+{
+  "k": 5,
+  "n_cases": 19,
+  "results": [
+    {
+      "config": "naive (1 chunk/doc, vector-only)",
+      "k": 5,
+      "n_cases": 19,
+      "recall@k": 0.9737,
+      "precision@k": 0.2,
+      "mrr": 0.9342,
+      "ndcg@k": 0.9418,
+      "hit_rate": 1.0
+    },
+    {
+      "config": "+ sentence chunking (vector-only)",
+      "k": 5,
+      "n_cases": 19,
+      "recall@k": 0.9737,
+      "precision@k": 0.2,
+      "mrr": 0.9737,
+      "ndcg@k": 0.9677,
+      "hit_rate": 1.0
+    },
+    {
+      "config": "+ hybrid (BM25 + RRF)",
+      "k": 5,
+      "n_cases": 19,
+      "recall@k": 0.9737,
+      "precision@k": 0.2,
+      "mrr": 0.9737,
+      "ndcg@k": 0.9677,
+      "hit_rate": 1.0
+    }
+  ]
+}

data/docs/code-security.md

@@ -0,0 +1,56 @@
+# Engineering handbook — code & security
+
+## Code review
+Every pull request requires at least one approval. Changes to billing or
+auth code require two approvals, including one from a senior engineer. CI
+must be green before merge.
+
+## Billing system changes
+Any change touching the billing pipeline, invoicing, or payment provider
+integration requires sign-off from the billing tech lead in addition to the
+standard two code-review approvals. Billing migrations must be dry-run
+against a staging replica before running in production. Refund logic
+changes require a finance stakeholder review.
+
+## Security incident response
+Suspected security incidents (credential leaks, unauthorized access, data
+exposure) must be reported to the security channel within 15 minutes of
+discovery, faster than the standard on-call acknowledgment window. The
+security lead triages severity and decides whether to invoke the incident
+commander process. All security incidents get a postmortem regardless of
+severity, unlike ordinary incidents which only require one at severity-1.
+
+## Secrets management
+Secrets (API keys, database credentials, signing keys) are stored in the
+central secrets manager, never in source control or environment files
+committed to git. Secrets are rotated automatically every 90 days;
+production database credentials rotate every 30 days. Access to production
+secrets requires a just-in-time approval, logged and reviewed weekly.
+
+## Linter and formatting rules
+All Python code is formatted with ruff and must pass linting in CI before
+merge; JavaScript code uses prettier with the shared team config. Style
+disagreements that aren't caught by the linter are left to reviewer
+discretion rather than escalated.
+
+## Dependency updates
+Automated dependency update bots open pull requests weekly for minor and
+patch version bumps; major version bumps require a manual review from the
+package owner and a changelog read-through before merge.
+
+## Team communication norms
+Teams default to public channels over DMs for anything work-related, so
+context stays searchable. Cross-team requests go through a dedicated
+request channel rather than pinging individuals directly, and response-time
+expectations there are best-effort, not an SLA.
+
+## Tooling procurement
+New SaaS tool requests over $500/year go through a lightweight procurement
+review covering security and data-handling questions before purchase.
+Renewals under the same terms skip the review and are approved by finance
+automatically.
+
+## Internal wiki hygiene
+Wiki pages without an update in 12 months are flagged stale and surfaced in
+a quarterly cleanup pass; owners either refresh or archive them. Search
+ranking on the wiki favors recently-edited pages over older ones.

data/docs/data-eng.md

@@ -0,0 +1,57 @@
+# Engineering handbook — data & platform
+
+## Data retention policy
+Customer data is retained for the duration of the account plus 90 days
+after deletion, to allow for accidental-deletion recovery. Application logs
+are retained for 30 days; security audit logs are retained for 1 year to
+satisfy compliance requirements. Backups are encrypted at rest and rotated
+every 24 hours with a 14-day retention window.
+
+## Database migrations
+Schema migrations must be backward-compatible with the previous application
+version to support zero-downtime deploys: add columns as nullable first,
+backfill, then enforce constraints in a follow-up migration. Migrations
+affecting tables over 10 million rows require a review from the database
+team and must run online without locking writes.
+
+## API versioning
+Public APIs are versioned in the URL path (/v1/, /v2/) and each version is
+supported for at least 12 months after the next version ships. Breaking
+changes require a new major version; additive changes (new optional fields)
+can ship within the current version. Deprecation notices go out at least 90
+days before a version is sunset.
+
+## Feature flags
+New user-facing features must ship behind a feature flag unless the change
+is a pure bugfix. Flags default to off in production and are rolled out
+gradually: 1% -> 10% -> 50% -> 100%, with at least a 24-hour soak at each
+stage for risky changes. Stale flags older than 90 days are flagged for
+cleanup in the quarterly flag audit.
+
+## Internal dashboards
+Engineering metrics dashboards refresh every 15 minutes and pull from the
+same warehouse tables used for the quarterly business review. Dashboard
+access is open to all engineers; editing dashboard definitions requires the
+data-platform team's review.
+
+## Data warehouse costs
+Ad-hoc warehouse queries over 1TB scanned trigger an automatic Slack alert
+to the requester and the data-platform on-call, since large ad-hoc queries
+are the leading cause of unexpected warehouse cost spikes.
+
+## Team communication norms
+Teams default to public channels over DMs for anything work-related, so
+context stays searchable. Cross-team requests go through a dedicated
+request channel rather than pinging individuals directly, and response-time
+expectations there are best-effort, not an SLA.
+
+## Tooling procurement
+New SaaS tool requests over $500/year go through a lightweight procurement
+review covering security and data-handling questions before purchase.
+Renewals under the same terms skip the review and are approved by finance
+automatically.
+
+## Internal wiki hygiene
+Wiki pages without an update in 12 months are flagged stale and surfaced in
+a quarterly cleanup pass; owners either refresh or archive them. Search
+ranking on the wiki favors recently-edited pages over older ones.

data/docs/deploys-ops.md

@@ -0,0 +1,51 @@
+# Engineering handbook — deploys & operations
+
+## Deployments
+We deploy to production every weekday at 4pm using a blue-green strategy.
+Rollbacks are automatic if error rates exceed 2% within the first five
+minutes. Hotfixes may be deployed outside the window with approval from an
+on-call lead.
+
+## On-call
+On-call rotations last one week and run Monday to Monday. The primary
+on-call engineer must acknowledge pages within 15 minutes. Secondary on-call
+is the fallback after 30 minutes of no response.
+
+## Incident response
+Severity-1 incidents require a written postmortem within 48 hours.
+Postmortems are blameless and focus on systemic fixes, not individual fault.
+
+## Disaster recovery
+The disaster recovery plan targets a recovery time objective (RTO) of 4
+hours and a recovery point objective (RPO) of 15 minutes for the primary
+database. Full DR drills, including a simulated region failover, run twice a
+year. Runbooks are stored outside the primary cloud region so they remain
+accessible during a regional outage.
+
+## Internal tooling access
+Engineers request access to internal admin dashboards through the access
+portal; approval routes to the resource owner and typically completes
+within one business day. Read-only access to production dashboards is
+granted by default to all engineers on day one.
+
+## Office network maintenance
+Scheduled network maintenance windows run the first Sunday of each month
+from 2am to 4am local time. Engineers relying on VPN for weekend work
+should check the maintenance calendar before starting a task.
+
+## Team communication norms
+Teams default to public channels over DMs for anything work-related, so
+context stays searchable. Cross-team requests go through a dedicated
+request channel rather than pinging individuals directly, and response-time
+expectations there are best-effort, not an SLA.
+
+## Tooling procurement
+New SaaS tool requests over $500/year go through a lightweight procurement
+review covering security and data-handling questions before purchase.
+Renewals under the same terms skip the review and are approved by finance
+automatically.
+
+## Internal wiki hygiene
+Wiki pages without an update in 12 months are flagged stale and surfaced in
+a quarterly cleanup pass; owners either refresh or archive them. Search
+ranking on the wiki favors recently-edited pages over older ones.

data/docs/finance-misc.md

@@ -0,0 +1,22 @@
+# Engineering handbook — finance & referrals
+
+## Referral bonus
+Employees who refer a candidate who is hired and passes the 90-day mark
+receive a $3000 referral bonus, paid in the following payroll cycle.
+Referrals for open-source contributors converting to full-time hires get an
+additional $500.
+
+## Conference budget
+Each engineer has a $2000/year conference and training budget, separate
+from the vacation and expense-policy stipends. Unused conference budget
+does not carry over into the next year.
+
+## Timesheets
+Hourly contractors submit timesheets weekly by Friday 5pm; salaried
+engineers do not track hours. Timesheet approval is required from a
+manager before payroll processes on the following Monday.
+
+## Brand guidelines
+All external-facing decks and blog posts must use the approved logo and
+color palette from the brand kit. Marketing reviews any content mentioning
+specific customer names before publication.

data/docs/office-facilities.md

@@ -0,0 +1,23 @@
+# Engineering handbook — office & facilities
+
+## Parking and badges
+Office badges are issued on day one and deactivated automatically 24 hours
+after an employee's last day. Parking passes are first-come, first-served
+and must be renewed every 6 months at the front desk.
+
+## Swag and equipment
+New hires receive a laptop, monitor, and welcome swag box in their first
+week. Laptops are refreshed every 3 years, or sooner if hardware fails
+under warranty. Monitor and peripheral requests go through the IT ticket
+queue with a 5-day turnaround.
+
+## Holidays calendar
+The company observes 10 fixed holidays per year plus 2 floating holidays
+employees can schedule with manager approval. The office is closed between
+Christmas and New Year's Day company-wide.
+
+## VPN and network access
+VPN access is required for any internal tool not exposed publicly.
+Credentials for VPN are separate from SSO and expire after 180 days of
+inactivity. Guest wifi is isolated from the internal network and has no
+access to internal services.

data/docs/people-ops.md

@@ -0,0 +1,58 @@
+# Engineering handbook — people ops
+
+## Vacation policy
+Engineers accrue 20 days of paid vacation per year. Unused days carry over
+up to a maximum of 10 days into the next calendar year.
+
+## Expense policy
+Employees can expense up to $100/month for home office supplies and up to
+$50 for a monthly phone/internet stipend, separate from the vacation and
+time-off benefits. Travel expenses require manager pre-approval for trips
+over $500. Receipts must be submitted within 30 days of the expense.
+
+## Remote work policy
+Engineers may work remotely full-time; there is no mandated office
+attendance. Remote employees must overlap at least 4 hours with their
+team's core hours (10am-2pm in the team's primary timezone). Equipment
+stipends of $1000 are available every 3 years for home office setup, on top
+of the monthly $100 supply allowance.
+
+## Hiring process
+Engineering candidates go through a recruiter screen, a technical phone
+screen, and a four-part onsite loop covering coding, system design,
+debugging, and values fit. Hiring decisions require consensus from at least
+3 of 4 interviewers; a single strong no blocks an offer regardless of other
+scores. Offers require approval from the hiring manager's director.
+
+## Performance reviews
+Performance reviews run twice a year, in April and October. Each review
+includes self-assessment, peer feedback from 3-5 nominated colleagues, and a
+manager writeup. Ratings range from "not meeting" to "significantly
+exceeds" and directly inform the twice-yearly compensation adjustment cycle.
+
+## Company swag
+New team members receive a welcome kit with a branded hoodie, water bottle,
+and stickers, shipped to their home address within the first two weeks.
+Additional swag requests for conferences go through the marketing team.
+
+## Meeting norms
+Recurring meetings without a clear agenda are cancelled automatically by
+the calendar bot after two no-shows. Engineers are encouraged to default to
+async written updates over synchronous meetings when possible.
+
+## Team communication norms
+Teams default to public channels over DMs for anything work-related, so
+context stays searchable. Cross-team requests go through a dedicated
+request channel rather than pinging individuals directly, and response-time
+expectations there are best-effort, not an SLA.
+
+## Tooling procurement
+New SaaS tool requests over $500/year go through a lightweight procurement
+review covering security and data-handling questions before purchase.
+Renewals under the same terms skip the review and are approved by finance
+automatically.
+
+## Internal wiki hygiene
+Wiki pages without an update in 12 months are flagged stale and surfaced in
+a quarterly cleanup pass; owners either refresh or archive them. Search
+ranking on the wiki favors recently-edited pages over older ones.

data/docs/support-postmortem.md

@@ -0,0 +1,49 @@
+# Engineering handbook — support & postmortems
+
+## Authentication
+All internal services authenticate via short-lived OAuth2 tokens issued by
+the identity provider; tokens expire after 60 minutes and must be refreshed
+by the client. Service-to-service calls use mutual TLS in addition to
+tokens. Personal access tokens for CLI tools expire after 90 days and are
+scoped to a single project.
+
+## Customer support escalation
+Support tickets tagged "urgent" must be triaged within 1 hour during
+business hours. Enterprise customers on the premium support tier get a
+15-minute first-response SLA, the same acknowledgment window as internal
+on-call pages. Escalations to engineering go through the on-call engineer,
+not directly to individual contributors.
+
+## Postmortem template
+Every postmortem must include a timeline, root cause, blast radius, and at
+least three concrete follow-up action items with owners and due dates.
+Postmortems for severity-1 incidents are reviewed in the weekly engineering
+sync; severity-2 postmortems are reviewed asynchronously. Templates live in
+the incident-response wiki space.
+
+## Documentation standards
+Public-facing API docs are generated from OpenAPI specs and rebuilt on
+every merge to main; internal runbooks are written in markdown and stored
+alongside the service they document, not in a separate wiki.
+
+## Support tooling
+Support agents use a shared ticketing queue with automatic tagging based on
+keyword rules; tickets that go untagged for more than 10 minutes are
+flagged for manual triage by a team lead.
+
+## Team communication norms
+Teams default to public channels over DMs for anything work-related, so
+context stays searchable. Cross-team requests go through a dedicated
+request channel rather than pinging individuals directly, and response-time
+expectations there are best-effort, not an SLA.
+
+## Tooling procurement
+New SaaS tool requests over $500/year go through a lightweight procurement
+review covering security and data-handling questions before purchase.
+Renewals under the same terms skip the review and are approved by finance
+automatically.
+
+## Internal wiki hygiene
+Wiki pages without an update in 12 months are flagged stale and surfaced in
+a quarterly cleanup pass; owners either refresh or archive them. Search
+ranking on the wiki favors recently-edited pages over older ones.

data/eval.jsonl

@@ -0,0 +1,19 @@
+{"question": "When do we deploy to production?", "relevant_doc_ids": ["deploys-ops"]}
+{"question": "How many approvals does a billing change need?", "relevant_doc_ids": ["code-security"]}
+{"question": "How long is an on-call rotation?", "relevant_doc_ids": ["deploys-ops"]}
+{"question": "How many vacation days carry over?", "relevant_doc_ids": ["people-ops"]}
+{"question": "When is a postmortem required?", "relevant_doc_ids": ["deploys-ops", "code-security"]}
+{"question": "How long do OAuth tokens last before they need to be refreshed?", "relevant_doc_ids": ["support-postmortem"]}
+{"question": "How fast do I need to report a suspected credential leak?", "relevant_doc_ids": ["code-security"]}
+{"question": "What's the monthly stipend for home office supplies?", "relevant_doc_ids": ["people-ops"]}
+{"question": "How many core hours must remote engineers overlap with their team?", "relevant_doc_ids": ["people-ops"]}
+{"question": "How many interviewers need to agree for a hiring offer?", "relevant_doc_ids": ["people-ops"]}
+{"question": "How often do performance reviews happen?", "relevant_doc_ids": ["people-ops"]}
+{"question": "How long are security audit logs kept?", "relevant_doc_ids": ["data-eng"]}
+{"question": "What does a postmortem document need to include?", "relevant_doc_ids": ["support-postmortem"]}
+{"question": "What's the rollout sequence for a new feature flag?", "relevant_doc_ids": ["data-eng"]}
+{"question": "How do you migrate a database schema without downtime?", "relevant_doc_ids": ["data-eng"]}
+{"question": "How long is an API version supported after the next version ships?", "relevant_doc_ids": ["data-eng"]}
+{"question": "What's the first-response SLA for premium support customers?", "relevant_doc_ids": ["support-postmortem"]}
+{"question": "What's the recovery time objective for the primary database?", "relevant_doc_ids": ["deploys-ops"]}
+{"question": "How often do production database credentials rotate?", "relevant_doc_ids": ["code-security"]}

docs/case-study.md

@@ -0,0 +1,72 @@
+# Case study: what chunking and hybrid retrieval actually buy you
+
+Lumen ships an eval harness (`lumen_rag/eval/`) specifically so claims like
+"chunking helps" or "hybrid retrieval helps" don't have to stay vibes. This
+is the measurement behind the numbers in the README.
+
+## Setup
+
+- **Corpus**: `data/docs/` — 7 markdown documents (~2,250 words total), each
+  covering multiple unrelated handbook topics per file (e.g. `deploys-ops.md`
+  bundles deployments, on-call, incident response, disaster recovery, plus
+  three unrelated filler sections). This mirrors a real internal wiki page:
+  long, multi-topic, with the answer to any given question living in one
+  paragraph out of several.
+- **Eval set**: `data/eval.jsonl` — 19 labelled questions, each with the
+  correct source document id(s).
+- **Embedder**: the offline deterministic `HashingEmbedder` (bag-of-words
+  feature hashing, L2-normalised) — no API key, fully reproducible, what CI
+  and the hosted demo run by default.
+- **Metric**: `lumen_rag.eval.harness.evaluate`, doc-level (chunk hits are
+  collapsed to unique parent documents before scoring), k=5.
+- **Reproduce**: `python scripts/benchmark.py`
+
+## Results
+
+| Configuration | recall@5 | precision@5 | MRR | nDCG@5 | hit rate |
+|---|---|---|---|---|---|
+| naive — 1 chunk per doc, vector-only | 0.97 | 0.20 | 0.93 | 0.94 | 1.00 |
+| + sentence-aware chunking, vector-only | 0.97 | 0.20 | **0.97** | **0.97** | 1.00 |
+| + hybrid (BM25 + Reciprocal Rank Fusion) | 0.97 | 0.20 | **0.97** | **0.97** | 1.00 |
+
+## Reading the numbers honestly
+
+**Recall and hit-rate are already saturated** at 0.97/1.00 in the naive
+config — with only 7 candidate documents and distinct enough vocabulary
+per topic, the correct document almost always lands somewhere in the top 5
+regardless of technique. Publishing only recall would (falsely) suggest
+chunking doesn't matter here. It does — just not on that metric.
+
+**MRR and nDCG@5 are where the effect shows up.** Both measure *where* the
+correct document ranks, not just whether it's present. Moving from naive to
+chunked lifts MRR from 0.93 to 0.97 — a small absolute jump that maps to a
+concrete failure mode disappearing: when a whole 300–450 word multi-topic
+document is embedded as a single vector, the sections irrelevant to the
+query dilute the average, and the correct document occasionally ranks 2nd
+or 3rd behind a partial-vocabulary-match decoy instead of 1st. Sentence-aware
+chunking (120-word windows, 20-word overlap) embeds each section on its own,
+so the answer-bearing chunk competes on its own signal instead of being
+outvoted by the rest of the document.
+
+**Hybrid ties chunked-vector, not because RRF doesn't work, but because the
+offline `HashingEmbedder` is itself a bag-of-words signal** — term-frequency
+counts, L2-normalised. That's structurally close to what BM25 computes, so
+fusing the two rankers mostly agrees with itself. Hybrid's actual value
+proposition — catching exact rare-term or numeric matches that a *semantic*
+embedding model under-weights in favor of topical similarity — needs a real
+semantic embedder (`OPENAI_API_KEY` set, `text-embedding-3-small`) and a
+larger, noisier corpus to demonstrate honestly. That's flagged as a follow-up
+rather than asserted with numbers we didn't measure.
+
+## Takeaways for anyone building on Lumen
+
+1. **Don't just report recall/hit-rate.** They saturate fast on small
+   corpora and hide ranking-quality regressions. MRR/nDCG catch what recall
+   misses.
+2. **Chunking's benefit scales with document length and topic density**, not
+   corpus size. A corpus of short, single-topic documents won't show this
+   effect — you need documents where the answer is a minority of the content.
+3. **Hybrid retrieval's payoff is embedder-dependent.** With a lexical/hash
+   embedder it's close to redundant with vector search; with a real semantic
+   embedder it complements it. Re-run `scripts/benchmark.py` with
+   `OPENAI_API_KEY` set to see the difference on your own corpus.

lumen_rag/__init__.py

@@ -0,0 +1,3 @@
+"""Lumen RAG — a small, transparent, evaluated retrieval-augmented generation engine."""
+
+__version__ = "0.1.0"

lumen_rag/api/__init__.py

@@ -0,0 +1,3 @@
+from .app import app
+
+__all__ = ["app"]

lumen_rag/api/app.py

@@ -0,0 +1,174 @@
+"""FastAPI surface for the RAG engine: ingest, query, stream, stats, health."""
+from __future__ import annotations
+
+import json
+import tempfile
+from contextlib import asynccontextmanager
+from pathlib import Path
+from typing import AsyncGenerator
+
+from fastapi import FastAPI, HTTPException, UploadFile
+from fastapi.responses import StreamingResponse
+from fastapi.staticfiles import StaticFiles
+from pydantic import BaseModel, Field
+
+from ..config import settings
+from ..engine import RagEngine
+from ..eval import evaluate
+from ..eval.harness import load_cases
+from ..ingestion.loaders import _LOADERS, load_file
+from ..llm import answer_stream
+from ..retrieval import Retriever, RetrievalMode
+
+_STATIC_DIR = Path(__file__).resolve().parent / "static"
+
+# Replaced by lifespan; initialised here so the name always exists (e.g. in tests).
+engine: RagEngine = RagEngine()
+
+
+@asynccontextmanager
+async def lifespan(app: FastAPI):
+    global engine
+    index = Path(settings.index_dir)
+    if (index / "chunks.json").exists():
+        engine = RagEngine.load(index)
+    else:
+        engine = RagEngine()
+    yield
+
+
+app = FastAPI(
+    title="Lumen RAG",
+    version="0.1.0",
+    description="Ingest documents, retrieve with vector search, answer with citations.",
+    lifespan=lifespan,
+)
+
+if _STATIC_DIR.is_dir():
+    app.mount("/demo", StaticFiles(directory=_STATIC_DIR, html=True), name="demo")
+
+
+class Document(BaseModel):
+    id: str | None = None
+    text: str = Field(min_length=1)
+    metadata: dict = Field(default_factory=dict)
+
+
+class IngestRequest(BaseModel):
+    documents: list[Document]
+    chunk_size: int = 120
+    overlap: int = 20
+
+
+class QueryRequest(BaseModel):
+    question: str = Field(min_length=1)
+    k: int = Field(default=5, ge=1, le=20)
+    mode: RetrievalMode = "hybrid"
+
+
+@app.get("/")
+def root():
+    from fastapi.responses import RedirectResponse
+
+    return RedirectResponse(url="/demo/")
+
+
+@app.get("/health")
+def health() -> dict:
+    return {"status": "ok", "offline_mode": settings.offline}
+
+
+@app.get("/stats")
+def stats() -> dict:
+    return {"chunks_indexed": len(engine.store), "embedding_dim": engine.store.dim}
+
+
+@app.post("/ingest")
+def ingest(req: IngestRequest) -> dict:
+    docs = [d.model_dump() for d in req.documents]
+    total = engine.add_documents(docs, chunk_size=req.chunk_size, overlap=req.overlap)
+    engine.save()
+    return {"chunks_indexed": total}
+
+
+@app.post("/query")
+def query(req: QueryRequest) -> dict:
+    if len(engine.store) == 0:
+        raise HTTPException(status_code=409, detail="Index is empty. Ingest documents first.")
+    result = engine.query(req.question, k=req.k, mode=req.mode)
+    return {"answer": result.text, "citations": result.citations}
+
+
+@app.post("/ingest/upload")
+async def ingest_upload(files: list[UploadFile], chunk_size: int = 120, overlap: int = 20) -> dict:
+    docs = []
+    for f in files:
+        suffix = Path(f.filename or "").suffix.lower()
+        if suffix not in _LOADERS:
+            raise HTTPException(status_code=400, detail=f"Unsupported file type: {f.filename}")
+        data = await f.read()
+        with tempfile.NamedTemporaryFile(suffix=suffix, delete=True) as tmp:
+            tmp.write(data)
+            tmp.flush()
+            doc = load_file(tmp.name)
+            doc["id"] = Path(f.filename).stem
+            doc["metadata"]["source"] = f.filename
+            docs.append(doc)
+    total = engine.add_documents(docs, chunk_size=chunk_size, overlap=overlap)
+    engine.save()
+    return {"files_indexed": len(docs), "chunks_indexed": total}
+
+
+@app.post("/ingest/sample")
+def ingest_sample() -> dict:
+    sample_dir = Path(__file__).resolve().parent.parent.parent / "data" / "docs"
+    if not sample_dir.is_dir():
+        raise HTTPException(status_code=404, detail="Bundled sample corpus not found.")
+    files = sorted(p for p in sample_dir.iterdir() if p.suffix.lower() in _LOADERS)
+    docs = [load_file(p) for p in files]
+    total = engine.add_documents(docs)
+    engine.save()
+    return {"files_indexed": len(docs), "chunks_indexed": total}
+
+
+@app.post("/reset")
+def reset() -> dict:
+    global engine
+    engine = RagEngine()
+    engine.save()
+    return {"status": "reset"}
+
+
+@app.get("/eval")
+def run_eval(dataset: str = "data/eval.jsonl", k: int = 5) -> dict:
+    path = Path(dataset)
+    if not path.exists():
+        raise HTTPException(status_code=404, detail=f"Eval set not found: {dataset}")
+    if len(engine.store) == 0:
+        raise HTTPException(status_code=409, detail="Index is empty. Ingest documents first.")
+    cases = load_cases(path)
+    report = evaluate(Retriever(engine.store, engine.embedder), cases, k=k)
+    return report.as_dict() | {"per_case": report.per_case}
+
+
+@app.post("/query/stream")
+async def query_stream(req: QueryRequest) -> StreamingResponse:
+    """Stream the answer as Server-Sent Events.
+
+    Each event is ``data: <json>\\n\\n``. Token events carry ``{"token": "..."}``
+    and the final event carries ``{"done": true, "citations": [...]}``.
+    """
+    if len(engine.store) == 0:
+        raise HTTPException(status_code=409, detail="Index is empty. Ingest documents first.")
+
+    chunks = engine.retriever.retrieve(req.question, k=req.k, mode=req.mode)
+
+    async def _sse() -> AsyncGenerator[str, None]:
+        for token, citations in answer_stream(req.question, chunks):
+            if citations is not None:
+                payload = json.dumps({"done": True, "citations": citations})
+            else:
+                payload = json.dumps({"token": token})
+            yield f"data: {payload}\n\n"
+
+    return StreamingResponse(_sse(), media_type="text/event-stream")

lumen_rag/api/static/index.html

@@ -0,0 +1,168 @@
+<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<title>Lumen RAG — Live Demo</title>
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<style>
+  :root { --bg:#0b0d12; --panel:#12151c; --border:#232733; --text:#e6e8ec; --muted:#8a8f9c; --accent:#5b8cff; --good:#4ade80; }
+  * { box-sizing: border-box; }
+  body { margin:0; background:var(--bg); color:var(--text); font:15px/1.5 -apple-system,Segoe UI,Roboto,sans-serif; }
+  header { padding:24px 32px; border-bottom:1px solid var(--border); }
+  header h1 { margin:0 0 4px; font-size:22px; }
+  header p { margin:0; color:var(--muted); font-size:13px; }
+  main { max-width:980px; margin:0 auto; padding:24px 32px 64px; display:grid; gap:24px; }
+  .card { background:var(--panel); border:1px solid var(--border); border-radius:10px; padding:20px; }
+  .card h2 { margin:0 0 12px; font-size:15px; color:var(--muted); text-transform:uppercase; letter-spacing:.04em; }
+  .row { display:flex; gap:10px; flex-wrap:wrap; align-items:center; }
+  input[type=text] { flex:1; min-width:200px; background:#0e1117; border:1px solid var(--border); color:var(--text); padding:10px 12px; border-radius:6px; font-size:14px; }
+  input[type=file] { color:var(--muted); font-size:13px; }
+  select { background:#0e1117; border:1px solid var(--border); color:var(--text); padding:9px; border-radius:6px; }
+  button { background:var(--accent); color:#fff; border:none; padding:10px 16px; border-radius:6px; cursor:pointer; font-size:14px; font-weight:600; }
+  button:hover { opacity:.9; }
+  button.ghost { background:transparent; border:1px solid var(--border); color:var(--text); }
+  button:disabled { opacity:.5; cursor:default; }
+  .answer { white-space:pre-wrap; margin-top:14px; padding:14px; background:#0e1117; border-radius:6px; border:1px solid var(--border); min-height:24px; }
+  .citation { display:inline-block; margin:2px 6px 2px 0; padding:3px 8px; background:#1a2030; border-radius:4px; font-size:12px; color:var(--muted); }
+  .metrics { display:grid; grid-template-columns:repeat(auto-fit,minmax(110px,1fr)); gap:12px; margin-top:8px; }
+  .metric { background:#0e1117; border:1px solid var(--border); border-radius:8px; padding:12px; text-align:center; }
+  .metric .val { font-size:22px; font-weight:700; color:var(--good); }
+  .metric .lbl { font-size:11px; color:var(--muted); text-transform:uppercase; margin-top:2px; }
+  .status { font-size:13px; color:var(--muted); margin-top:8px; }
+  .status.err { color:#f87171; }
+  footer { text-align:center; color:var(--muted); font-size:12px; padding:24px; }
+  a { color:var(--accent); }
+</style>
+</head>
+<body>
+<header>
+  <h1>🔦 Lumen RAG — Live Demo</h1>
+  <p>Upload docs → hybrid retrieval (vector + BM25) → cited answer. Runs 100% offline.</p>
+</header>
+<main>
+
+  <div class="card">
+    <h2>1 · Load documents</h2>
+    <div class="row">
+      <button id="loadSample">Load sample corpus (5 policy docs)</button>
+      <span style="color:var(--muted)">or</span>
+      <input type="file" id="fileInput" multiple accept=".txt,.md,.html,.htm,.pdf,.docx">
+      <button id="uploadBtn" class="ghost">Upload</button>
+      <button id="resetBtn" class="ghost">Reset index</button>
+    </div>
+    <div class="status" id="ingestStatus"></div>
+  </div>
+
+  <div class="card">
+    <h2>2 · Ask</h2>
+    <div class="row">
+      <input type="text" id="question" placeholder="e.g. How long is an on-call rotation?">
+      <select id="mode">
+        <option value="hybrid" selected>hybrid</option>
+        <option value="vector">vector</option>
+        <option value="bm25">bm25</option>
+      </select>
+      <button id="askBtn">Ask</button>
+    </div>
+    <div class="answer" id="answer">Answer will appear here.</div>
+    <div id="citations"></div>
+  </div>
+
+  <div class="card">
+    <h2>3 · Retrieval quality (recall@k on the labelled sample eval set)</h2>
+    <div class="row">
+      <button id="evalBtn">Run eval</button>
+    </div>
+    <div class="metrics" id="metrics"></div>
+    <div class="status" id="evalStatus"></div>
+  </div>
+
+</main>
+<footer>
+  <a href="https://github.com/WickTech/lumen-rag" target="_blank">Source on GitHub</a> ·
+  API docs at <a href="/docs">/docs</a>
+</footer>
+<script>
+const $ = id => document.getElementById(id);
+
+async function api(path, opts) {
+  const res = await fetch(path, opts);
+  const body = await res.json().catch(() => ({}));
+  if (!res.ok) throw new Error(body.detail || res.statusText);
+  return body;
+}
+
+$('loadSample').onclick = async () => {
+  $('ingestStatus').textContent = 'Loading sample corpus…';
+  $('ingestStatus').className = 'status';
+  try {
+    const r = await api('/ingest/sample', { method: 'POST' });
+    $('ingestStatus').textContent = `Indexed ${r.files_indexed} file(s) → ${r.chunks_indexed} chunks.`;
+  } catch (e) {
+    $('ingestStatus').textContent = 'Error: ' + e.message;
+    $('ingestStatus').className = 'status err';
+  }
+};
+
+$('uploadBtn').onclick = async () => {
+  const files = $('fileInput').files;
+  if (!files.length) { $('ingestStatus').textContent = 'Pick file(s) first.'; return; }
+  const fd = new FormData();
+  for (const f of files) fd.append('files', f);
+  $('ingestStatus').textContent = 'Uploading + indexing…';
+  $('ingestStatus').className = 'status';
+  try {
+    const r = await api('/ingest/upload', { method: 'POST', body: fd });
+    $('ingestStatus').textContent = `Indexed ${r.files_indexed} file(s) → ${r.chunks_indexed} chunks.`;
+  } catch (e) {
+    $('ingestStatus').textContent = 'Error: ' + e.message;
+    $('ingestStatus').className = 'status err';
+  }
+};
+
+$('resetBtn').onclick = async () => {
+  await api('/reset', { method: 'POST' });
+  $('ingestStatus').textContent = 'Index cleared.';
+  $('answer').textContent = 'Answer will appear here.';
+  $('citations').innerHTML = '';
+  $('metrics').innerHTML = '';
+};
+
+$('askBtn').onclick = async () => {
+  const question = $('question').value.trim();
+  if (!question) return;
+  $('answer').textContent = 'Thinking…';
+  $('citations').innerHTML = '';
+  try {
+    const r = await api('/query', {
+      method: 'POST', headers: { 'Content-Type': 'application/json' },
+      body: JSON.stringify({ question, mode: $('mode').value, k: 5 })
+    });
+    $('answer').textContent = r.answer;
+    $('citations').innerHTML = r.citations.map(c =>
+      `<span class="citation">[${c.n}] ${c.doc_id} · score ${Number(c.score).toFixed(3)}</span>`).join('');
+  } catch (e) {
+    $('answer').textContent = 'Error: ' + e.message;
+  }
+};
+
+$('question').addEventListener('keydown', e => { if (e.key === 'Enter') $('askBtn').click(); });
+
+$('evalBtn').onclick = async () => {
+  $('evalStatus').textContent = 'Running eval harness…';
+  $('evalStatus').className = 'status';
+  $('metrics').innerHTML = '';
+  try {
+    const r = await api('/eval');
+    const order = ['recall@k', 'precision@k', 'mrr', 'ndcg@k', 'hit_rate'];
+    $('metrics').innerHTML = order.map(k =>
+      `<div class="metric"><div class="val">${r[k]}</div><div class="lbl">${k}</div></div>`).join('');
+    $('evalStatus').textContent = `${r.n_cases} labelled questions, k=${r.k}.`;
+  } catch (e) {
+    $('evalStatus').textContent = 'Error: ' + e.message;
+    $('evalStatus').className = 'status err';
+  }
+};
+</script>
+</body>
+</html>

lumen_rag/cli.py

@@ -0,0 +1,89 @@
+"""Command-line interface: ingest, ask, eval, serve."""
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+
+import typer
+
+# Ensure Unicode output (arrows, box-drawing) doesn't crash on Windows' cp1252 console.
+for _stream in (sys.stdout, sys.stderr):
+    try:
+        _stream.reconfigure(encoding="utf-8")  # type: ignore[union-attr]
+    except (AttributeError, ValueError):
+        pass
+
+from .config import settings
+from .engine import RagEngine
+from .eval import evaluate
+from .eval.harness import load_cases
+from .ingestion.loaders import _LOADERS
+from .retrieval import Retriever, RetrievalMode
+
+app = typer.Typer(help="Lumen RAG — ingest, ask, and evaluate a RAG pipeline.")
+
+
+@app.command()
+def ingest(
+    paths: list[str] = typer.Argument(..., help="Files or directories (.txt/.md)."),
+    chunk_size: int = 120,
+    overlap: int = 20,
+) -> None:
+    """Index documents into the persistent vector store."""
+    files: list[Path] = []
+    for p in paths:
+        path = Path(p)
+        if path.is_dir():
+            for ext in _LOADERS:
+                files.extend(path.rglob(f"*{ext}"))
+        else:
+            files.append(path)
+
+    engine = (
+        RagEngine.load() if Path(settings.index_dir, "chunks.json").exists() else RagEngine()
+    )
+    docs = [
+        {"id": f.stem, "text": f.read_text(encoding="utf-8"), "metadata": {"source": str(f)}}
+        for f in files
+    ]
+    total = engine.add_documents(docs, chunk_size=chunk_size, overlap=overlap)
+    engine.save()
+    typer.echo(f"Indexed {len(files)} file(s) → {total} chunks in {settings.index_dir}/")
+
+
+@app.command()
+def ask(
+    question: str,
+    k: int = 5,
+    mode: str = typer.Option("hybrid", help="Retrieval mode: vector | bm25 | hybrid"),
+) -> None:
+    """Query the index and print an answer with citations."""
+    engine = RagEngine.load()
+    result = engine.query(question, k=k, mode=mode)  # type: ignore[arg-type]
+    typer.echo("\n" + result.text + "\n")
+    typer.echo("Sources:")
+    for c in result.citations:
+        typer.echo(f"  [{c['n']}] {c['doc_id']} (score={c['score']})")
+
+
+@app.command(name="eval")
+def run_eval(dataset: str, k: int = 5) -> None:
+    """Score the retriever against a JSONL of labelled questions."""
+    engine = RagEngine.load()
+    cases = load_cases(dataset)
+    report = evaluate(Retriever(engine.store, engine.embedder), cases, k=k)
+    typer.echo("\n" + report.pretty() + "\n")
+    typer.echo(json.dumps(report.as_dict()))
+
+
+@app.command()
+def serve(host: str = "0.0.0.0", port: int = 8000) -> None:
+    """Run the FastAPI server."""
+    import uvicorn
+
+    uvicorn.run("lumen_rag.api.app:app", host=host, port=port, reload=False)
+
+
+if __name__ == "__main__":
+    app()

lumen_rag/config.py

@@ -0,0 +1,26 @@
+"""Central configuration, loaded from environment with sensible offline defaults."""
+from __future__ import annotations
+
+import os
+from dataclasses import dataclass
+
+from dotenv import load_dotenv
+
+load_dotenv()
+
+
+@dataclass
+class Settings:
+    openai_api_key: str = os.getenv("OPENAI_API_KEY", "")
+    openai_base_url: str | None = os.getenv("OPENAI_BASE_URL") or None
+    embed_model: str = os.getenv("EMBED_MODEL", "text-embedding-3-small")
+    chat_model: str = os.getenv("CHAT_MODEL", "gpt-4o-mini")
+    index_dir: str = os.getenv("LUMEN_INDEX_DIR", ".lumen_index")
+
+    @property
+    def offline(self) -> bool:
+        """True when no API key is configured; engine falls back to local embeddings."""
+        return not self.openai_api_key
+
+
+settings = Settings()

lumen_rag/embeddings.py

@@ -0,0 +1,79 @@
+"""Embedding providers.
+
+Two implementations behind one interface:
+
+* ``OpenAIEmbedder``    — real embeddings when an API key is present.
+* ``HashingEmbedder``   — a deterministic, dependency-free fallback so the
+                          whole engine (and its test suite) runs offline.
+
+The hashing embedder is a bag-of-words feature hasher with L2 normalisation.
+It is *not* semantically smart, but it is stable and good enough to exercise
+ranking logic and the eval harness without network access.
+"""
+from __future__ import annotations
+
+import hashlib
+import re
+from typing import Protocol
+
+import numpy as np
+
+from .config import settings
+
+_TOKEN_RE = re.compile(r"[a-z0-9]+")
+
+
+class Embedder(Protocol):
+    dim: int
+
+    def embed(self, texts: list[str]) -> np.ndarray:  # (n, dim) float32, L2-normalised
+        ...
+
+
+def _normalize(mat: np.ndarray) -> np.ndarray:
+    norms = np.linalg.norm(mat, axis=1, keepdims=True)
+    norms[norms == 0] = 1.0
+    return (mat / norms).astype(np.float32)
+
+
+class HashingEmbedder:
+    """Deterministic feature-hashing embedder. No network, no model download."""
+
+    def __init__(self, dim: int = 512) -> None:
+        self.dim = dim
+
+    def embed(self, texts: list[str]) -> np.ndarray:
+        out = np.zeros((len(texts), self.dim), dtype=np.float32)
+        for i, text in enumerate(texts):
+            for tok in _TOKEN_RE.findall(text.lower()):
+                h = int(hashlib.md5(tok.encode()).hexdigest(), 16)
+                idx = h % self.dim
+                sign = 1.0 if (h >> 8) & 1 else -1.0
+                out[i, idx] += sign
+        return _normalize(out)
+
+
+class OpenAIEmbedder:
+    """Wraps the OpenAI embeddings endpoint (or any compatible base URL)."""
+
+    def __init__(self) -> None:
+        from openai import OpenAI  # imported lazily; optional dependency
+
+        self._client = OpenAI(
+            api_key=settings.openai_api_key,
+            base_url=settings.openai_base_url,
+        )
+        self._model = settings.embed_model
+        self.dim = 1536  # text-embedding-3-small
+
+    def embed(self, texts: list[str]) -> np.ndarray:
+        resp = self._client.embeddings.create(model=self._model, input=texts)
+        vecs = np.array([d.embedding for d in resp.data], dtype=np.float32)
+        return _normalize(vecs)
+
+
+def get_embedder() -> Embedder:
+    """Pick the real embedder if configured, else the offline fallback."""
+    if settings.offline:
+        return HashingEmbedder()
+    return OpenAIEmbedder()

lumen_rag/engine.py

@@ -0,0 +1,36 @@
+"""High-level facade tying ingestion, retrieval, and answering together."""
+from __future__ import annotations
+
+from pathlib import Path
+
+from .config import settings
+from .embeddings import get_embedder
+from .ingestion import ingest_documents
+from .llm import Answer, answer
+from .retrieval import Retriever, RetrievalMode
+from .store import VectorStore
+
+
+class RagEngine:
+    def __init__(self, store: VectorStore | None = None) -> None:
+        self.embedder = get_embedder()
+        self.store = store or VectorStore(dim=self.embedder.dim)
+        self.retriever = Retriever(self.store, self.embedder)
+
+    def add_documents(self, documents: list[dict], **kwargs) -> int:
+        ingest_documents(documents, store=self.store, embedder=self.embedder, **kwargs)
+        self.retriever._bm25 = None  # invalidate cached BM25 index
+        return len(self.store)
+
+    def query(self, question: str, k: int = 5, mode: RetrievalMode = "hybrid") -> Answer:
+        chunks = self.retriever.retrieve(question, k=k, mode=mode)
+        return answer(question, chunks)
+
+    # --- persistence -------------------------------------------------------
+    def save(self, directory: str | Path | None = None) -> None:
+        self.store.save(directory or settings.index_dir)
+
+    @classmethod
+    def load(cls, directory: str | Path | None = None) -> "RagEngine":
+        store = VectorStore.load(directory or settings.index_dir)
+        return cls(store=store)

lumen_rag/eval/__init__.py

@@ -0,0 +1,13 @@
+from .metrics import hit_rate, mrr, ndcg_at_k, precision_at_k, recall_at_k
+from .harness import EvalCase, EvalReport, evaluate
+
+__all__ = [
+    "hit_rate",
+    "mrr",
+    "ndcg_at_k",
+    "precision_at_k",
+    "recall_at_k",
+    "EvalCase",
+    "EvalReport",
+    "evaluate",
+]

lumen_rag/eval/harness.py

@@ -0,0 +1,107 @@
+"""Run a labelled question set through a retriever and aggregate metrics.
+
+This is what turns "the demo felt good" into "recall@5 is 0.82". Point it at a
+JSONL of questions with known-relevant doc ids and it prints a scorecard you
+can track across changes to chunking, embeddings, or reranking.
+"""
+from __future__ import annotations
+
+import json
+from dataclasses import dataclass, field
+from pathlib import Path
+
+from ..retrieval import Retriever
+from . import metrics
+
+
+@dataclass
+class EvalCase:
+    question: str
+    relevant_doc_ids: list[str]
+
+
+@dataclass
+class EvalReport:
+    k: int
+    n_cases: int
+    recall_at_k: float
+    precision_at_k: float
+    mrr: float
+    ndcg_at_k: float
+    hit_rate: float
+    per_case: list[dict] = field(default_factory=list)
+
+    def as_dict(self) -> dict:
+        return {
+            "k": self.k,
+            "n_cases": self.n_cases,
+            "recall@k": round(self.recall_at_k, 4),
+            "precision@k": round(self.precision_at_k, 4),
+            "mrr": round(self.mrr, 4),
+            "ndcg@k": round(self.ndcg_at_k, 4),
+            "hit_rate": round(self.hit_rate, 4),
+        }
+
+    def pretty(self) -> str:
+        d = self.as_dict()
+        lines = [f"  Retrieval eval — {d['n_cases']} cases @ k={d['k']}", "  " + "-" * 34]
+        for key in ("recall@k", "precision@k", "mrr", "ndcg@k", "hit_rate"):
+            lines.append(f"  {key:<14} {d[key]:.4f}")
+        return "\n".join(lines)
+
+
+def _unique_preserving_order(items) -> list[str]:
+    seen: set[str] = set()
+    out: list[str] = []
+    for item in items:
+        if item not in seen:
+            seen.add(item)
+            out.append(item)
+    return out
+
+
+def load_cases(path: str | Path) -> list[EvalCase]:
+    cases = []
+    for line in Path(path).read_text(encoding="utf-8").splitlines():
+        line = line.strip()
+        if not line:
+            continue
+        obj = json.loads(line)
+        cases.append(EvalCase(obj["question"], list(obj["relevant_doc_ids"])))
+    return cases
+
+
+def evaluate(retriever: Retriever, cases: list[EvalCase], k: int = 5) -> EvalReport:
+    agg = {"recall": 0.0, "precision": 0.0, "mrr": 0.0, "ndcg": 0.0, "hit": 0.0}
+    per_case = []
+
+    for case in cases:
+        results = retriever.retrieve(case.question, k=k)
+        # Retrieval is chunk-level but relevance is doc-level: collapse to the
+        # rank-ordered list of *unique* doc ids so a doc with several retrieved
+        # chunks counts once (otherwise recall/nDCG can exceed 1.0).
+        retrieved_ids = _unique_preserving_order(r.chunk.doc_id for r in results)
+        row = {
+            "question": case.question,
+            "recall": metrics.recall_at_k(retrieved_ids, case.relevant_doc_ids, k),
+            "precision": metrics.precision_at_k(retrieved_ids, case.relevant_doc_ids, k),
+            "mrr": metrics.mrr(retrieved_ids, case.relevant_doc_ids),
+            "ndcg": metrics.ndcg_at_k(retrieved_ids, case.relevant_doc_ids, k),
+            "hit": metrics.hit_rate(retrieved_ids, case.relevant_doc_ids, k),
+            "retrieved": retrieved_ids,
+        }
+        for key in agg:
+            agg[key] += row[key]
+        per_case.append(row)
+
+    n = max(1, len(cases))
+    return EvalReport(
+        k=k,
+        n_cases=len(cases),
+        recall_at_k=agg["recall"] / n,
+        precision_at_k=agg["precision"] / n,
+        mrr=agg["mrr"] / n,
+        ndcg_at_k=agg["ndcg"] / n,
+        hit_rate=agg["hit"] / n,
+        per_case=per_case,
+    )

lumen_rag/eval/metrics.py

@@ -0,0 +1,58 @@
+"""Information-retrieval metrics for evaluating a retriever.
+
+All functions take ``retrieved`` (an ordered list of doc ids, best first) and
+``relevant`` (the set of ids that *should* be retrieved). This is the standard
+way to score a RAG retriever against a labelled question set.
+"""
+from __future__ import annotations
+
+import math
+from collections.abc import Iterable, Sequence
+
+
+def _rel_set(relevant: Iterable[str]) -> set[str]:
+    return set(relevant)
+
+
+def recall_at_k(retrieved: Sequence[str], relevant: Iterable[str], k: int) -> float:
+    rel = _rel_set(relevant)
+    if not rel:
+        return 0.0
+    hits = sum(1 for doc in retrieved[:k] if doc in rel)
+    return hits / len(rel)
+
+
+def precision_at_k(retrieved: Sequence[str], relevant: Iterable[str], k: int) -> float:
+    if k <= 0:
+        return 0.0
+    rel = _rel_set(relevant)
+    hits = sum(1 for doc in retrieved[:k] if doc in rel)
+    return hits / k
+
+
+def hit_rate(retrieved: Sequence[str], relevant: Iterable[str], k: int) -> float:
+    """1.0 if any relevant doc appears in the top-k, else 0.0."""
+    rel = _rel_set(relevant)
+    return 1.0 if any(doc in rel for doc in retrieved[:k]) else 0.0
+
+
+def mrr(retrieved: Sequence[str], relevant: Iterable[str]) -> float:
+    """Reciprocal rank of the first relevant result."""
+    rel = _rel_set(relevant)
+    for i, doc in enumerate(retrieved, start=1):
+        if doc in rel:
+            return 1.0 / i
+    return 0.0
+
+
+def ndcg_at_k(retrieved: Sequence[str], relevant: Iterable[str], k: int) -> float:
+    """Normalised discounted cumulative gain with binary relevance."""
+    rel = _rel_set(relevant)
+    dcg = sum(
+        1.0 / math.log2(i + 1)
+        for i, doc in enumerate(retrieved[:k], start=1)
+        if doc in rel
+    )
+    ideal_hits = min(len(rel), k)
+    idcg = sum(1.0 / math.log2(i + 1) for i in range(1, ideal_hits + 1))
+    return dcg / idcg if idcg > 0 else 0.0

lumen_rag/ingestion/__init__.py

@@ -0,0 +1,5 @@
+from .chunker import chunk_text
+from .loaders import load_file, load_files
+from .pipeline import ingest_documents, ingest_paths
+
+__all__ = ["chunk_text", "ingest_documents", "ingest_paths", "load_file", "load_files"]

lumen_rag/ingestion/chunker.py

@@ -0,0 +1,44 @@
+"""Sentence-aware text chunking with overlap.
+
+Splits on sentence boundaries and packs sentences into ~`chunk_size`-word
+windows with `overlap` words carried into the next chunk. Overlap preserves
+context across boundaries so an answer that straddles two chunks is still
+retrievable.
+"""
+from __future__ import annotations
+
+import re
+
+_SENT_RE = re.compile(r"(?<=[.!?])\s+")
+
+
+def _sentences(text: str) -> list[str]:
+    text = re.sub(r"\s+", " ", text).strip()
+    return [s for s in _SENT_RE.split(text) if s]
+
+
+def chunk_text(text: str, chunk_size: int = 120, overlap: int = 20) -> list[str]:
+    """Return a list of overlapping chunks, each up to ~chunk_size words."""
+    if chunk_size <= 0:
+        raise ValueError("chunk_size must be positive")
+    if overlap >= chunk_size:
+        raise ValueError("overlap must be smaller than chunk_size")
+
+    chunks: list[str] = []
+    current: list[str] = []
+    count = 0
+
+    for sentence in _sentences(text):
+        words = sentence.split()
+        if count + len(words) > chunk_size and current:
+            chunks.append(" ".join(current))
+            # carry the last `overlap` words into the next window
+            carry = current[-overlap:] if overlap else []
+            current = list(carry)
+            count = len(carry)
+        current.extend(words)
+        count += len(words)
+
+    if current:
+        chunks.append(" ".join(current))
+    return chunks

lumen_rag/ingestion/loaders.py

@@ -0,0 +1,111 @@
+"""Document loaders: extract plain text from PDF, DOCX, and HTML files.
+
+All loaders return a ``{"id", "text", "metadata"}`` dict ready for
+``ingest_documents``. Each loader raises ``ImportError`` with a clear install
+hint if its optional dependency is missing so users only pay for what they use.
+
+Supported:
+  - ``.pdf``   → requires ``pypdf``  (``pip install lumen-rag[pdf]``)
+  - ``.docx``  → requires ``python-docx`` (``pip install lumen-rag[docx]``)
+  - ``.html`` / ``.htm`` → stdlib ``html.parser``, no extra deps
+  - ``.txt`` / ``.md``  → plain read, no extra deps
+"""
+from __future__ import annotations
+
+from pathlib import Path
+
+
+def _doc(path: Path, text: str) -> dict:
+    return {"id": path.stem, "text": text, "metadata": {"source": str(path)}}
+
+
+def load_txt(path: Path) -> dict:
+    return _doc(path, path.read_text(encoding="utf-8", errors="replace"))
+
+
+def load_pdf(path: Path) -> dict:
+    try:
+        from pypdf import PdfReader
+    except ImportError as e:
+        raise ImportError(
+            "pypdf is required to load PDF files. "
+            "Install it with: pip install 'lumen-rag[pdf]'"
+        ) from e
+
+    reader = PdfReader(str(path))
+    pages = [page.extract_text() or "" for page in reader.pages]
+    return _doc(path, "\n\n".join(pages))
+
+
+def load_docx(path: Path) -> dict:
+    try:
+        import docx
+    except ImportError as e:
+        raise ImportError(
+            "python-docx is required to load DOCX files. "
+            "Install it with: pip install 'lumen-rag[docx]'"
+        ) from e
+
+    doc = docx.Document(str(path))
+    text = "\n".join(para.text for para in doc.paragraphs if para.text.strip())
+    return _doc(path, text)
+
+
+def load_html(path: Path) -> dict:
+    from html.parser import HTMLParser
+
+    class _TextExtractor(HTMLParser):
+        SKIP_TAGS = {"script", "style", "head", "meta", "link"}
+
+        def __init__(self):
+            super().__init__()
+            self._parts: list[str] = []
+            self._skip = 0
+
+        def handle_starttag(self, tag, attrs):
+            if tag in self.SKIP_TAGS:
+                self._skip += 1
+
+        def handle_endtag(self, tag):
+            if tag in self.SKIP_TAGS and self._skip > 0:
+                self._skip -= 1
+
+        def handle_data(self, data):
+            if self._skip == 0:
+                stripped = data.strip()
+                if stripped:
+                    self._parts.append(stripped)
+
+        def text(self) -> str:
+            return " ".join(self._parts)
+
+    extractor = _TextExtractor()
+    extractor.feed(path.read_text(encoding="utf-8", errors="replace"))
+    return _doc(path, extractor.text())
+
+
+_LOADERS = {
+    ".txt": load_txt,
+    ".md": load_txt,
+    ".pdf": load_pdf,
+    ".docx": load_docx,
+    ".html": load_html,
+    ".htm": load_html,
+}
+
+
+def load_file(path: str | Path) -> dict:
+    """Dispatch to the right loader based on file extension."""
+    p = Path(path)
+    ext = p.suffix.lower()
+    loader = _LOADERS.get(ext)
+    if loader is None:
+        raise ValueError(
+            f"Unsupported file type '{ext}'. Supported: {sorted(_LOADERS)}"
+        )
+    return loader(p)
+
+
+def load_files(paths: list[str | Path]) -> list[dict]:
+    """Load multiple files, returning a list of document dicts."""
+    return [load_file(p) for p in paths]

lumen_rag/ingestion/pipeline.py

@@ -0,0 +1,49 @@
+"""Document → chunks → embeddings → vector store."""
+from __future__ import annotations
+
+import uuid
+from pathlib import Path
+
+from ..embeddings import Embedder, get_embedder
+from ..store import Chunk, VectorStore
+from .chunker import chunk_text
+from .loaders import _LOADERS, load_file
+
+
+def ingest_documents(
+    documents: list[dict],
+    store: VectorStore | None = None,
+    embedder: Embedder | None = None,
+    chunk_size: int = 120,
+    overlap: int = 20,
+) -> VectorStore:
+    """Ingest a list of ``{"id", "text", "metadata"?}`` dicts into a store."""
+    embedder = embedder or get_embedder()
+    # NB: use an explicit None check — an empty VectorStore is falsy (__len__ == 0),
+    # so `store or VectorStore(...)` would discard a passed-in empty store.
+    if store is None:
+        store = VectorStore(dim=embedder.dim)
+
+    chunks: list[Chunk] = []
+    for doc in documents:
+        doc_id = str(doc.get("id") or uuid.uuid4())
+        for piece in chunk_text(doc["text"], chunk_size, overlap):
+            chunks.append(
+                Chunk(
+                    id=str(uuid.uuid4()),
+                    text=piece,
+                    doc_id=doc_id,
+                    metadata=dict(doc.get("metadata", {})),
+                )
+            )
+
+    if chunks:
+        vectors = embedder.embed([c.text for c in chunks])
+        store.add(chunks, vectors)
+    return store
+
+
+def ingest_paths(paths: list[str | Path], **kwargs) -> VectorStore:
+    """Ingest files from disk. Supports .txt, .md, .pdf, .docx, .html, .htm."""
+    docs = [load_file(p) for p in paths]
+    return ingest_documents(docs, **kwargs)

lumen_rag/llm.py

@@ -0,0 +1,111 @@
+"""Answer synthesis from retrieved context, with inline citations.
+
+Offline (no API key), ``answer`` returns an extractive answer: the top chunks
+stitched together with citation markers. This keeps the full RAG loop runnable
+and testable end-to-end without a model. With a key, it calls the chat model
+using a grounded, citation-enforcing prompt.
+"""
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+from .config import settings
+from .store import ScoredChunk
+
+SYSTEM_PROMPT = (
+    "You are a precise assistant. Answer the question using ONLY the numbered "
+    "context passages. Cite sources inline as [1], [2]. If the context does not "
+    "contain the answer, say you don't know."
+)
+
+
+@dataclass
+class Answer:
+    text: str
+    citations: list[dict]  # [{"n": 1, "doc_id": ..., "source": ..., "score": ...}]
+
+
+def _format_context(chunks: list[ScoredChunk]) -> str:
+    return "\n\n".join(f"[{i + 1}] {c.chunk.text}" for i, c in enumerate(chunks))
+
+
+def _citations(chunks: list[ScoredChunk]) -> list[dict]:
+    return [
+        {
+            "n": i + 1,
+            "doc_id": c.chunk.doc_id,
+            "source": c.chunk.metadata.get("source"),
+            "score": round(c.score, 4),
+        }
+        for i, c in enumerate(chunks)
+    ]
+
+
+def answer(question: str, chunks: list[ScoredChunk]) -> Answer:
+    if not chunks:
+        return Answer("I don't have any indexed context to answer that.", [])
+
+    citations = _citations(chunks)
+
+    if settings.offline:
+        # Extractive fallback: surface the best passages with markers.
+        body = " ".join(f"{c.chunk.text} [{i + 1}]" for i, c in enumerate(chunks[:2]))
+        return Answer(f"(offline extractive answer) {body}", citations)
+
+    from openai import OpenAI
+
+    client = OpenAI(api_key=settings.openai_api_key, base_url=settings.openai_base_url)
+    resp = client.chat.completions.create(
+        model=settings.chat_model,
+        messages=[
+            {"role": "system", "content": SYSTEM_PROMPT},
+            {
+                "role": "user",
+                "content": f"Context:\n{_format_context(chunks)}\n\nQuestion: {question}",
+            },
+        ],
+        temperature=0.1,
+    )
+    return Answer(resp.choices[0].message.content or "", citations)
+
+
+def answer_stream(question: str, chunks: list[ScoredChunk]):
+    """Yield (token: str, citations: list | None) tuples.
+
+    The final tuple carries citations; all prior tuples have citations=None.
+    Callers can use this to build an SSE stream without buffering the full answer.
+    """
+    if not chunks:
+        yield ("I don't have any indexed context to answer that.", _citations(chunks))
+        return
+
+    citations = _citations(chunks)
+
+    if settings.offline:
+        body = " ".join(f"{c.chunk.text} [{i + 1}]" for i, c in enumerate(chunks[:2]))
+        full = f"(offline extractive answer) {body}"
+        for word in full.split(" "):
+            yield (word + " ", None)
+        yield ("", citations)
+        return
+
+    from openai import OpenAI
+
+    client = OpenAI(api_key=settings.openai_api_key, base_url=settings.openai_base_url)
+    stream = client.chat.completions.create(
+        model=settings.chat_model,
+        messages=[
+            {"role": "system", "content": SYSTEM_PROMPT},
+            {
+                "role": "user",
+                "content": f"Context:\n{_format_context(chunks)}\n\nQuestion: {question}",
+            },
+        ],
+        temperature=0.1,
+        stream=True,
+    )
+    for chunk in stream:
+        delta = chunk.choices[0].delta.content or ""
+        if delta:
+            yield (delta, None)
+    yield ("", citations)

lumen_rag/retrieval/__init__.py

@@ -0,0 +1,4 @@
+from .bm25 import BM25Index, reciprocal_rank_fusion
+from .retriever import Retriever, RetrievalMode
+
+__all__ = ["BM25Index", "RetrievalMode", "Retriever", "reciprocal_rank_fusion"]

lumen_rag/retrieval/bm25.py

@@ -0,0 +1,130 @@
+"""BM25 sparse retriever with Reciprocal Rank Fusion blending.
+
+BM25 rewards term frequency while penalising documents that are much longer
+than average — it consistently outperforms raw TF-IDF and complements dense
+vector search on exact-match, acronym, and entity queries.
+
+Reciprocal Rank Fusion (RRF) is the fusion strategy: each candidate is scored
+as Σ 1/(k+rank_i) across its rank in each sub-ranker.  RRF is parameter-robust
+and consistently beats linear score blending (Cormack, Clarke, Buettcher 2009).
+"""
+from __future__ import annotations
+
+import math
+import re
+from collections import Counter
+from dataclasses import dataclass
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+    from ..store import Chunk, ScoredChunk, VectorStore
+
+_TOKEN_RE = re.compile(r"[a-z0-9]+")
+
+
+def _tokenize(text: str) -> list[str]:
+    return _TOKEN_RE.findall(text.lower())
+
+
+@dataclass
+class BM25Index:
+    """Pre-computed BM25 index over a corpus of chunks.
+
+    Build once at ingest/load time; query many times with ``score``.
+    """
+
+    chunks: list["Chunk"]
+    avgdl: float
+    doc_freqs: dict[str, int]   # term → number of docs containing it
+    term_freqs: list[dict[str, int]]  # per-chunk term frequencies
+    n: int                      # total documents
+
+    k1: float = 1.5
+    b: float = 0.75
+
+    @classmethod
+    def build(cls, chunks: list["Chunk"], k1: float = 1.5, b: float = 0.75) -> "BM25Index":
+        term_freqs: list[dict[str, int]] = []
+        doc_freqs: dict[str, int] = Counter()
+        total_len = 0
+
+        for chunk in chunks:
+            tokens = _tokenize(chunk.text)
+            tf = Counter(tokens)
+            term_freqs.append(tf)
+            for tok in tf:
+                doc_freqs[tok] += 1
+            total_len += len(tokens)
+
+        avgdl = total_len / max(1, len(chunks))
+        idx = cls(
+            chunks=list(chunks),
+            avgdl=avgdl,
+            doc_freqs=dict(doc_freqs),
+            term_freqs=term_freqs,
+            n=len(chunks),
+            k1=k1,
+            b=b,
+        )
+        return idx
+
+    def score(self, query: str) -> list[float]:
+        """Return a BM25 score for every chunk in the index."""
+        q_tokens = _tokenize(query)
+        scores = [0.0] * self.n
+        for tok in q_tokens:
+            df = self.doc_freqs.get(tok, 0)
+            if df == 0:
+                continue
+            idf = math.log((self.n - df + 0.5) / (df + 0.5) + 1)
+            for i, tf in enumerate(self.term_freqs):
+                f = tf.get(tok, 0)
+                if f == 0:
+                    continue
+                dl = sum(tf.values())
+                numer = f * (self.k1 + 1)
+                denom = f + self.k1 * (1 - self.b + self.b * dl / self.avgdl)
+                scores[i] += idf * numer / denom
+        return scores
+
+    def search(self, query: str, k: int) -> "list[ScoredChunk]":
+        from ..store import ScoredChunk
+
+        scores = self.score(query)
+        if not scores:
+            return []
+        k = min(k, len(scores))
+        top = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)[:k]
+        return [ScoredChunk(self.chunks[i], scores[i]) for i in top]
+
+
+# ---------------------------------------------------------------------------
+# Reciprocal Rank Fusion
+# ---------------------------------------------------------------------------
+
+RRF_K = 60  # standard constant from the original paper
+
+
+def reciprocal_rank_fusion(
+    ranked_lists: list[list["ScoredChunk"]],
+    k: int,
+    rrf_k: int = RRF_K,
+) -> "list[ScoredChunk]":
+    """Fuse multiple ranked lists of ScoredChunk via RRF.
+
+    Each chunk_id accumulates 1/(rrf_k + rank) from every list it appears in.
+    Returns the top-k by fused score.
+    """
+    from ..store import ScoredChunk
+
+    fused: dict[str, float] = {}
+    chunk_map: dict[str, "Chunk"] = {}
+
+    for ranked in ranked_lists:
+        for rank, sc in enumerate(ranked, start=1):
+            cid = sc.chunk.id
+            fused[cid] = fused.get(cid, 0.0) + 1.0 / (rrf_k + rank)
+            chunk_map[cid] = sc.chunk
+
+    top = sorted(fused.items(), key=lambda x: x[1], reverse=True)[:k]
+    return [ScoredChunk(chunk_map[cid], score) for cid, score in top]

lumen_rag/retrieval/retriever.py

@@ -0,0 +1,87 @@
+"""Query-time retrieval: embed the query, search the store, optionally rerank.
+
+Three retrieval modes:
+  ``vector``  — dense cosine search only (original behaviour).
+  ``bm25``    — sparse BM25 keyword search only.
+  ``hybrid``  — Reciprocal Rank Fusion of both lists (default).
+
+The legacy ``rerank`` parameter is preserved for backwards compatibility: when
+mode is ``vector`` and rerank is True, the light lexical-overlap booster from
+v0.1 is applied on top of the vector hits.
+"""
+from __future__ import annotations
+
+import re
+from typing import Literal
+
+from ..embeddings import Embedder, get_embedder
+from ..store import ScoredChunk, VectorStore
+from .bm25 import BM25Index, reciprocal_rank_fusion
+
+_TOKEN_RE = re.compile(r"[a-z0-9]+")
+
+RetrievalMode = Literal["vector", "bm25", "hybrid"]
+
+
+def _lexical_overlap(query: str, text: str) -> float:
+    q = set(_TOKEN_RE.findall(query.lower()))
+    if not q:
+        return 0.0
+    t = set(_TOKEN_RE.findall(text.lower()))
+    return len(q & t) / len(q)
+
+
+class Retriever:
+    def __init__(self, store: VectorStore, embedder: Embedder | None = None) -> None:
+        self.store = store
+        self.embedder = embedder or get_embedder()
+        self._bm25: BM25Index | None = None
+
+    def _get_bm25(self) -> BM25Index:
+        """Build (or return cached) BM25 index from the current store chunks."""
+        if self._bm25 is None or len(self._bm25.chunks) != len(self.store._chunks):
+            self._bm25 = BM25Index.build(self.store._chunks)
+        return self._bm25
+
+    def retrieve(
+        self,
+        query: str,
+        k: int = 5,
+        *,
+        mode: RetrievalMode = "hybrid",
+        rerank: bool = True,
+        rerank_weight: float = 0.25,
+    ) -> list[ScoredChunk]:
+        """Retrieve top-k chunks for *query*.
+
+        Args:
+            query: Natural-language question.
+            k: Number of results to return.
+            mode: ``"vector"`` | ``"bm25"`` | ``"hybrid"`` (default).
+            rerank: When mode is ``"vector"``, blend lexical overlap into scores.
+            rerank_weight: Weight of the lexical component in vector+lexical blend.
+        """
+        if len(self.store) == 0:
+            return []
+
+        if mode == "bm25":
+            return self._get_bm25().search(query, k)
+
+        if mode == "hybrid":
+            fetch = k * 3
+            query_vec = self.embedder.embed([query])[0]
+            dense_list = self.store.search(query_vec, k=fetch)
+            sparse_list = self._get_bm25().search(query, fetch)
+            return reciprocal_rank_fusion([dense_list, sparse_list], k=k)
+
+        # --- vector (default legacy path) ---
+        query_vec = self.embedder.embed([query])[0]
+        pool = self.store.search(query_vec, k=k * 3 if rerank else k)
+        if not rerank:
+            return pool[:k]
+
+        for sc in pool:
+            lex = _lexical_overlap(query, sc.chunk.text)
+            sc.score = (1 - rerank_weight) * sc.score + rerank_weight * lex
+        pool.sort(key=lambda s: s.score, reverse=True)
+        return pool[:k]

lumen_rag/store.py

@@ -0,0 +1,74 @@
+"""A tiny, persistable in-memory vector store with exact cosine search.
+
+Deliberately not a full vector DB — the point is to make the retrieval maths
+visible and dependency-free. Vectors are L2-normalised on insert, so cosine
+similarity is a single matrix-vector dot product. Swap this module for
+pgvector/Qdrant/Chroma without touching the retriever.
+"""
+from __future__ import annotations
+
+import json
+from dataclasses import asdict, dataclass
+from pathlib import Path
+
+import numpy as np
+
+
+@dataclass
+class Chunk:
+    id: str
+    text: str
+    doc_id: str
+    metadata: dict
+
+
+@dataclass
+class ScoredChunk:
+    chunk: Chunk
+    score: float
+
+
+class VectorStore:
+    def __init__(self, dim: int) -> None:
+        self.dim = dim
+        self._vectors = np.zeros((0, dim), dtype=np.float32)
+        self._chunks: list[Chunk] = []
+
+    def __len__(self) -> int:
+        return len(self._chunks)
+
+    def add(self, chunks: list[Chunk], vectors: np.ndarray) -> None:
+        if vectors.shape[0] != len(chunks):
+            raise ValueError("chunks and vectors length mismatch")
+        if vectors.shape[1] != self.dim:
+            raise ValueError(f"expected dim {self.dim}, got {vectors.shape[1]}")
+        self._vectors = np.vstack([self._vectors, vectors.astype(np.float32)])
+        self._chunks.extend(chunks)
+
+    def search(self, query_vec: np.ndarray, k: int = 5) -> list[ScoredChunk]:
+        if len(self._chunks) == 0:
+            return []
+        scores = self._vectors @ query_vec.reshape(-1)  # cosine (already normalised)
+        k = min(k, len(self._chunks))
+        # argpartition for top-k, then sort just those.
+        top = np.argpartition(-scores, k - 1)[:k]
+        top = top[np.argsort(-scores[top])]
+        return [ScoredChunk(self._chunks[i], float(scores[i])) for i in top]
+
+    # --- persistence -------------------------------------------------------
+    def save(self, directory: str | Path) -> None:
+        d = Path(directory)
+        d.mkdir(parents=True, exist_ok=True)
+        np.save(d / "vectors.npy", self._vectors)
+        (d / "chunks.json").write_text(
+            json.dumps([asdict(c) for c in self._chunks], ensure_ascii=False)
+        )
+
+    @classmethod
+    def load(cls, directory: str | Path) -> "VectorStore":
+        d = Path(directory)
+        vectors = np.load(d / "vectors.npy")
+        store = cls(dim=int(vectors.shape[1]) if vectors.size else 0)
+        store._vectors = vectors.astype(np.float32)
+        store._chunks = [Chunk(**c) for c in json.loads((d / "chunks.json").read_text())]
+        return store

pyproject.toml

@@ -0,0 +1,43 @@
+[project]
+name = "lumen-rag"
+version = "0.1.0"
+description = "A transparent, evaluated RAG engine: ingest documents, retrieve with vector search, answer with citations, and measure retrieval quality."
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "MIT" }
+authors = [{ name = "Rishabh Verma" }]
+dependencies = [
+    "fastapi>=0.115",
+    "uvicorn[standard]>=0.32",
+    "pydantic>=2.9",
+    "numpy>=1.26",
+    "httpx>=0.27",
+    "python-dotenv>=1.0",
+    "typer>=0.13",
+    "python-multipart>=0.0.12",
+]
+
+[project.optional-dependencies]
+openai = ["openai>=1.55"]
+pdf = ["pypdf>=4.0"]
+docx = ["python-docx>=1.1"]
+loaders = ["pypdf>=4.0", "python-docx>=1.1"]
+dev = ["pytest>=8.3", "pytest-asyncio>=0.24", "ruff>=0.8"]
+
+[project.scripts]
+lumen = "lumen_rag.cli:app"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["lumen_rag"]
+
+[tool.pytest.ini_options]
+asyncio_mode = "auto"
+testpaths = ["tests"]
+
+[tool.ruff]
+line-length = 100
+target-version = "py310"

requirements.txt

@@ -0,0 +1 @@
+-e .

scripts/benchmark.py

@@ -0,0 +1,83 @@
+#!/usr/bin/env python
+"""Benchmark retrieval quality across pipeline configurations.
+
+Runs the same labelled eval set (data/eval.jsonl, 20 docs / 19 questions)
+through three configurations to quantify what chunking and hybrid retrieval
+actually buy you:
+
+  1. naive       — whole document as one chunk, vector search only
+  2. + chunking   — sentence-aware chunking (size=120, overlap=20), vector only
+  3. + hybrid     — same chunking, + BM25 and Reciprocal Rank Fusion
+
+Prints a markdown table (paste straight into the README) and writes
+benchmark_results.json alongside it.
+
+Usage:
+  python scripts/benchmark.py
+"""
+from __future__ import annotations
+
+import json
+import sys
+from pathlib import Path
+
+ROOT = Path(__file__).parent.parent
+sys.path.insert(0, str(ROOT))
+
+from lumen_rag.embeddings import get_embedder
+from lumen_rag.eval import evaluate
+from lumen_rag.eval.harness import load_cases
+from lumen_rag.ingestion.loaders import _LOADERS, load_file
+from lumen_rag.ingestion.pipeline import ingest_documents
+from lumen_rag.retrieval import Retriever
+from lumen_rag.store import VectorStore
+
+DOCS_DIR = ROOT / "data" / "docs"
+EVAL_PATH = ROOT / "data" / "eval.jsonl"
+K = 5
+
+CONFIGS = [
+    {"name": "naive (1 chunk/doc, vector-only)", "chunk_size": 100_000, "overlap": 0, "mode": "vector"},
+    {"name": "+ sentence chunking (vector-only)", "chunk_size": 120, "overlap": 20, "mode": "vector"},
+    {"name": "+ hybrid (BM25 + RRF)", "chunk_size": 120, "overlap": 20, "mode": "hybrid"},
+]
+
+
+def build_retriever(chunk_size: int, overlap: int) -> Retriever:
+    embedder = get_embedder()
+    store = VectorStore(dim=embedder.dim)
+    docs = [load_file(p) for p in sorted(DOCS_DIR.iterdir()) if p.suffix.lower() in _LOADERS]
+    ingest_documents(docs, store=store, embedder=embedder, chunk_size=chunk_size, overlap=overlap)
+    return Retriever(store, embedder)
+
+
+def main() -> None:
+    cases = load_cases(EVAL_PATH)
+    rows = []
+    for cfg in CONFIGS:
+        retriever = build_retriever(cfg["chunk_size"], cfg["overlap"])
+        report = evaluate(retriever, cases, k=K)
+        d = report.as_dict()
+        rows.append({"config": cfg["name"], **d})
+
+    header = ["Configuration", "recall@5", "precision@5", "MRR", "nDCG@5", "hit rate"]
+    lines = [
+        "| " + " | ".join(header) + " |",
+        "|" + "---|" * len(header),
+    ]
+    for r in rows:
+        lines.append(
+            "| {config} | {recall@k:.2f} | {precision@k:.2f} | {mrr:.2f} | {ndcg@k:.2f} | {hit_rate:.2f} |".format(
+                **r
+            )
+        )
+    table = "\n".join(lines)
+    print(table)
+
+    out = ROOT / "benchmark_results.json"
+    out.write_text(json.dumps({"k": K, "n_cases": len(cases), "results": rows}, indent=2))
+    print(f"\nWrote {out}")
+
+
+if __name__ == "__main__":
+    main()

scripts/check_eval.py

@@ -0,0 +1,80 @@
+#!/usr/bin/env python
+"""Eval regression guard: fail with a non-zero exit code if any metric drops below threshold.
+
+Usage:
+  python scripts/check_eval.py data/eval.jsonl --k 3
+
+The thresholds below represent a floor derived from the baseline offline run.
+If scores *drop* below them, this script exits 1 so CI fails. Raise the
+thresholds when you improve the pipeline; never lower them to pass CI.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+
+# Add project root so the script works without install when run from CI.
+sys.path.insert(0, str(Path(__file__).parent.parent))
+
+from lumen_rag.engine import RagEngine
+from lumen_rag.eval import evaluate
+from lumen_rag.eval.harness import load_cases
+from lumen_rag.retrieval import Retriever
+
+# Minimum acceptable scores. Adjust upward as the pipeline improves.
+THRESHOLDS: dict[str, float] = {
+    "recall@k": 0.80,
+    "hit_rate": 0.80,
+    "mrr": 0.70,
+}
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description="Lumen RAG eval regression guard")
+    parser.add_argument("dataset", help="Path to JSONL eval set")
+    parser.add_argument("--k", type=int, default=3, help="Top-k to evaluate")
+    parser.add_argument(
+        "--mode",
+        default="hybrid",
+        choices=["vector", "bm25", "hybrid"],
+        help="Retrieval mode",
+    )
+    parser.add_argument(
+        "--index-dir",
+        default=None,
+        help="Override index directory (defaults to LUMEN_INDEX_DIR or .lumen_index)",
+    )
+    args = parser.parse_args()
+
+    engine = RagEngine.load(args.index_dir) if args.index_dir else RagEngine.load()
+    cases = load_cases(args.dataset)
+    report = evaluate(Retriever(engine.store, engine.embedder), cases, k=args.k)
+
+    scores = report.as_dict()
+    print(f"\n  Retrieval eval — {scores['n_cases']} cases @ k={scores['k']}")
+    print("  " + "-" * 34)
+    for key in ("recall@k", "precision@k", "mrr", "ndcg@k", "hit_rate"):
+        threshold = THRESHOLDS.get(key)
+        status = ""
+        if threshold is not None:
+            status = " ✓" if scores[key] >= threshold else f" ✗  (threshold {threshold})"
+        print(f"  {key:<14} {scores[key]:.4f}{status}")
+
+    failures = [
+        f"{key}={scores[key]:.4f} < threshold {thr}"
+        for key, thr in THRESHOLDS.items()
+        if scores[key] < thr
+    ]
+
+    if failures:
+        print("\n  REGRESSION DETECTED:", ", ".join(failures), file=sys.stderr)
+        return 1
+
+    print("\n  All thresholds met.")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())