Moonley backend (HF Space build)

.dockerignore

@@ -0,0 +1,10 @@
+frontend
+**/node_modules
+.git
+**/__pycache__
+**/*.pyc
+**/logs
+*.log
+verifier/data
+.env
+.env.local

.env.example

@@ -0,0 +1,46 @@
+# Copy to .env and fill in. Used by the statute pipeline + backend.
+DEEPSEEK_API_KEY=your_deepseek_api_key_here
+
+# Clerk (required for the Moonley production API).
+# The publishable key is returned by the public /api/v2/auth/config endpoint;
+# the secret key must never be exposed in frontend code.
+CLERK_PUBLISHABLE_KEY=pk_test_your_publishable_key
+CLERK_SECRET_KEY=sk_test_your_secret_key
+# Optional but recommended: enables networkless JWT signature verification.
+CLERK_JWT_KEY="-----BEGIN PUBLIC KEY-----\nyour_public_key\n-----END PUBLIC KEY-----"
+# Exact comma-separated browser origins allowed to mint/carry session tokens.
+CLERK_AUTHORIZED_PARTIES=http://localhost:8000,http://localhost:5173,https://moonley-pilot.vercel.app
+
+# Optional: restrict CORS to your deployed frontend (comma-separated).
+# Defaults to "*" (open) for local development.
+# FRONTEND_ORIGIN=https://moonley-pilot.vercel.app
+
+# Private project knowledge. The live pilot can use its mounted persistent
+# volume immediately; these limits are enforced per authenticated Clerk user.
+MOONLEY_PROJECT_STORAGE_ROOT=/project-data
+MOONLEY_PROJECT_MAX_PROJECTS=20
+MOONLEY_PROJECT_MAX_DOCUMENTS=25
+MOONLEY_PROJECT_MAX_FILE_BYTES=10485760
+MOONLEY_PROJECT_MAX_BYTES=52428800
+MOONLEY_PROJECT_MAX_USER_BYTES=262144000
+
+# Private exact statute-text Chroma store. Point to the Chroma root containing
+# chroma.sqlite3 (or its UUID segment directory). Conversion remains exact-only.
+MOONLEY_STATUTE_CHROMA=/tmp/moonley_statutes
+MOONLEY_STATUTE_COLLECTION=indian_statutes
+
+# Optional scaled vector store. If omitted, private document vectors remain on
+# the mounted volume. Qdrant payloads contain identifiers/filter fields only;
+# source files and extracted text do not go into Qdrant.
+QDRANT_URL=https://your-cluster.region.cloud.qdrant.io
+QDRANT_API_KEY=your_qdrant_api_key
+QDRANT_KNOWLEDGE_COLLECTION=moonley_tenant_knowledge
+
+# Planned private Supabase source store. These are intentionally not used until
+# the private bucket and tenant-aware database policies have been provisioned.
+SUPABASE_URL=https://your-project.supabase.co
+SUPABASE_SERVICE_ROLE_KEY=your_server_only_service_role_key
+SUPABASE_STORAGE_BUCKET=moonley-private-documents
+
+# One compatibility release continues to read matching legacy THEMIS_* names.
+# Configure new and rotated environments with MOONLEY_* names only.

.gitignore

@@ -0,0 +1,51 @@
+# Python
+__pycache__/
+*.pyc
+*.pyo
+.venv/
+venv/
+env/
+
+# Secrets
+.env
+.env.local
+
+# Logs
+**/logs/
+*.log
+
+# Runtime PDF cache (pulled from the open registry on demand; not source)
+**/pdf_cache/
+
+# Frontend
+frontend/node_modules/
+frontend/dist/
+
+# OS
+.DS_Store
+
+# Private statute serving artifacts are downloaded by start_private_space.py
+# from a pinned, private Hugging Face dataset at container startup. Never add
+# raw statute records, generated indexes, vectors, or Chroma files to GitHub or
+# to the public Space image.
+statute corpus/all_statutes.json
+statute corpus/concordance.json
+statute corpus/statute_index.json
+statute corpus/statute_vectors.npy
+statute corpus/chroma_statutes/
+
+# Verifier Tier-2 cache is local-only:
+verifier/data/
+
+# Phase 1 pipeline raw data (reproducible from HF; do not commit)
+phase1/data/
+
+# Phase 1 DeepSeek generation cache (local-only; speeds up gold-set re-runs)
+phase1/eval/.gen_cache.json
+
+# Phase-1 derived citator/index outputs (large, reproducible)
+phase1/eval/good_law.jsonl
+phase1/eval/themis_bench_results.json
+phase1/eval/edges.jsonl
+phase1/scripts/edges.jsonl
+statute corpus/concordance_src/

DEPLOYMENT.md

@@ -0,0 +1,80 @@
+# Moonley deployment
+
+Moonley has one public user interface and one API:
+
+| Part | Host | Public role |
+|---|---|---|
+| React/Vite UI | Vercel | The only browser application: `https://moonley-pilot.vercel.app/` |
+| FastAPI backend | Hugging Face Spaces | Authenticated API, models and private corpus access only |
+
+The production source branch is `phase1.1`. The legacy Hugging Face frontend is not part of
+the deployment. The backend Space may keep its existing technical slug so its URL and secrets
+do not need to change; its root route returns JSON and never serves the UI.
+
+```mermaid
+flowchart LR
+  GH["GitHub · phase1.1"] -->|automatic build| VC["Vercel · Moonley React UI"]
+  GH -->|backend-only release| HF["HF Space · Moonley API"]
+  VC -->|Clerk session token + HTTPS| HF
+  HF -->|read-only token| JD["Private judgment release"]
+  HF -->|read-only token| SD["Private statute release"]
+```
+
+## Vercel UI
+
+- Project name: `moonley-pilot`
+- Production branch: `phase1.1`
+- Root directory: `vercel-frontend`
+- Framework: Vite
+- Production domain: `moonley-pilot.vercel.app`
+- Backend URL compiled into the current auth bridge: `https://vg15o2-themis.hf.space`
+
+Do not attach the previous `themis-*.vercel.app` aliases. The cutover intentionally has no
+redirect because Moonley is the canonical product and URL.
+
+## Hugging Face backend
+
+The Docker image starts `phase1/scripts/start_private_space.py`, downloads the pinned private
+judgment, statute and drafting-template artifacts, verifies the required files, removes the
+read token from the API process environment, and starts FastAPI on port 7860.
+
+Required secrets and variables:
+
+- `DEEPSEEK_API_KEY`
+- `HF_TOKEN` with read access to the private release datasets
+- Clerk keys used by `phase1/scripts/clerk_auth.py`
+- `CLERK_AUTHORIZED_PARTIES=https://moonley-pilot.vercel.app`
+- `FRONTEND_ORIGIN=https://moonley-pilot.vercel.app`
+
+Canonical application settings use `MOONLEY_*`. Matching `THEMIS_*` variables are accepted
+for one compatibility release so an existing Space can be rotated safely.
+
+Deploy the backend from a clean worktree:
+
+```bash
+git remote add space https://huggingface.co/spaces/<owner>/themis  # once
+bash scripts/deploy-space.sh
+```
+
+The script creates a disposable orphan branch, removes `frontend/`, `vercel-frontend/`,
+documentation and binary assets, adds the Docker Space frontmatter, and force-pushes only the
+backend package to the Space. It does not alter the production branch.
+
+## Smoke checks
+
+```text
+GET https://moonley-pilot.vercel.app/                 -> Moonley React UI
+GET https://vg15o2-themis.hf.space/                   -> Moonley API JSON
+GET https://vg15o2-themis.hf.space/api/v2/health      -> status: ok
+GET https://vg15o2-themis.hf.space/api/v2/auth/config -> Clerk public config
+```
+
+Then sign in at the Vercel URL and verify research, direct case lookup, judgment view,
+statute conversion, project documents, drafting, DOCX export and PDF export. Browser data
+stored under legacy `themis_*` keys is copied to canonical `moonley_*` keys on first load.
+
+## Rollback
+
+Redeploy the preceding successful `phase1.1` Vercel deployment and redeploy the preceding
+backend commit to the existing Space. Do not restore the old frontend alias; add a temporary
+maintenance page at the Moonley domain if the UI must be taken offline.

Dockerfile

@@ -0,0 +1,58 @@
+# Moonley schema-v5 backend — Hugging Face Space (Docker SDK), CPU-only.
+FROM python:3.11-slim
+
+ENV PYTHONUNBUFFERED=1 \
+    PIP_NO_CACHE_DIR=1 \
+    HF_HOME=/tmp/hf \
+    SENTENCE_TRANSFORMERS_HOME=/tmp/hf/sentence-transformers \
+    TRANSFORMERS_CACHE=/tmp/hf/transformers \
+    MOONLEY_DATA=/tmp/moonley_release \
+    MOONLEY_STATUTE="/app/statute corpus" \
+    MOONLEY_STATUTE_CHROMA=/tmp/moonley_statutes \
+    MOONLEY_QWEN_MODEL=/app/qwen_model \
+    MOONLEY_QWEN_DTYPE=bfloat16 \
+    MOONLEY_WARM_QUERY_MODEL=1 \
+    MOONLEY_DEEP=never \
+    MOONLEY_SKIM=0 \
+    MOONLEY_HELD_ARM=0 \
+    MOONLEY_CITECTX=0 \
+    MOONLEY_BUDGET_S=20 \
+    MOONLEY_DEVICE=cpu \
+    MOONLEY_LOG_DIR=/tmp/moonley_logs \
+    MOONLEY_PDF_CACHE=/tmp/pdf_cache \
+    MOONLEY_KEYWORD=0 \
+    OMP_NUM_THREADS=8 \
+    TOKENIZERS_PARALLELISM=false
+
+WORKDIR /app
+
+RUN apt-get update && apt-get install -y --no-install-recommends build-essential fonts-dejavu-core tesseract-ocr \
+    && rm -rf /var/lib/apt/lists/*
+
+# CPU torch in its own layer (heavy, rarely changes), then the serving deps.
+RUN pip install torch --index-url https://download.pytorch.org/whl/cpu
+COPY phase1/deploy/requirements.txt ./req.txt
+RUN pip install -r req.txt huggingface_hub
+
+# Fetch private artifacts at runtime so neither snapshot is embedded in this
+# public image. Revisions are immutable and every expected entrypoint is checked.
+ENV MOONLEY_RELEASE_REPO=vg15o2/themis-indian-kanoon-qwen-v1 \
+    MOONLEY_RELEASE_REVISION=12f58201987cc8ec7697010754ab75765c5f5a24 \
+    MOONLEY_STATUTE_REPO=vg15o2/themis-statutes-v1 \
+    MOONLEY_STATUTE_REVISION=ebf66528e417358a09903d95f6718ccfeb94a426
+
+ARG QWEN_MODEL_REVISION=5cf2132abc99cad020ac570b19d031efec650f2b
+RUN python -c "import os; from huggingface_hub import snapshot_download; snapshot_download(repo_id='Qwen/Qwen3-Embedding-4B', repo_type='model', revision=os.environ['QWEN_MODEL_REVISION'], local_dir='/app/qwen_model')"
+
+# Keep private release pointers after the large, cacheable model layer. Updating
+# a small template release must not force the Qwen model to download again.
+ENV MOONLEY_DRAFTING_TEMPLATE_REPO=vg15o2/themis-drafting-templates-v1 \
+    MOONLEY_DRAFTING_TEMPLATE_REVISION=2b036d4bef7ebe7a3b3bb8d0a094d7fefdd8c4d2
+
+COPY . .
+
+# HF runs the container as a non-root user; only /tmp is writable.
+RUN mkdir -p /tmp/hf /tmp/moonley_logs /tmp/pdf_cache && chmod -R 777 /tmp/hf /tmp/moonley_logs /tmp/pdf_cache
+
+EXPOSE 7860
+CMD ["python", "phase1/scripts/start_private_space.py"]

README.md

@@ -0,0 +1,315 @@
+---
+title: Moonley API
+sdk: docker
+app_port: 7860
+pinned: false
+---
+
+# Moonley
+
+**Grounded AI legal research for Indian law.** Moonley answers natural-language questions by
+retrieving the relevant **statutes** (IPC · BNS · CrPC · BNSS · IEA · BSA) and **Supreme
+Court judgments**, fusing them on a single cross-encoder, generating an answer with
+**DeepSeek**, and **verifying every cited section/case against the retrieved evidence** —
+streaming each reasoning step to the UI over Server-Sent Events.
+
+- **Live app:** [moonley-pilot.vercel.app](https://moonley-pilot.vercel.app/) (React/Vite SPA)
+- **Backend:** Hugging Face Spaces (FastAPI/Docker, API only)
+- **Production branch:** [`phase1.1`](../../tree/phase1.1) · **Repository:** `vg15o2/themis`
+
+---
+
+## Table of contents
+1. [Architecture](#1-architecture)
+2. [Tech stack](#2-tech-stack)
+3. [Repository layout](#3-repository-layout)
+4. [The retrieval pipeline (in depth)](#4-the-retrieval-pipeline-in-depth)
+5. [Data & indices](#5-data--indices)
+6. [Backend API contract (SSE)](#6-backend-api-contract-sse)
+7. [Frontend](#7-frontend)
+8. [Configuration](#8-configuration)
+9. [Local development](#9-local-development)
+10. [Deployment](#10-deployment)
+11. [Operational notes](#11-operational-notes)
+12. [Roadmap → V2](#12-roadmap--v2)
+13. [Documentation](#13-documentation)
+
+---
+
+## 1. Architecture
+
+```mermaid
+flowchart TB
+  UI["React SPA (Vercel)<br/>SSE reasoning UI"] -- "POST /ask (SSE)" --> API
+  subgraph Space["HF Spaces · Docker · CPU 16GB"]
+    API["FastAPI<br/>backend/app.py"] --> R["Unified router<br/>hybrid_rag/unified_legal_Rag.py"]
+    R --> SR["Statute pipeline<br/>statute_retrieval.py"]
+    R --> JR["Judgment pipeline<br/>llm_retriever.py"]
+    R --> VER["Verifier<br/>verifier/verified.py"]
+    SR --> CS[("Chroma<br/>indian_statutes")]
+    JR --> CJ[("Chroma<br/>sci_judgments_bge_v2")]
+    JR --> BM["BM25Okapi (in-memory)"]
+    SR --> M["BGE-small + ms-marco CE"]
+    JR --> M
+  end
+  R -- "stream" --> DS["DeepSeek<br/>deepseek-chat"]
+  DSset["HF Dataset<br/>vg15o2/themis-judgments"] -. "downloaded at build" .-> CJ
+```
+
+**Request lifecycle:** `route_query` decides statute / judgment / hybrid → each enabled
+pipeline retrieves candidates → `unified_rerank` re-scores **all** candidates on one
+cross-encoder (dedupe + per-source cap) → an intent-specific prompt + evidence is streamed
+through DeepSeek → `check_grounding` validates citations → `done`. Every stage is emitted
+as a typed SSE event.
+
+---
+
+## 2. Tech stack
+
+| Layer | Choice | Detail |
+|---|---|---|
+| Frontend | React 18 + Vite 5 | SSE via `fetch`+`ReadableStream`; `react-markdown`+`remark-gfm`; deployed on Vercel (root `frontend/`) |
+| Backend | FastAPI + Uvicorn | Single SSE endpoint; lazy in-process pipeline load |
+| Backend host | HF Spaces (Docker SDK) | 2 vCPU / 16 GB free tier; listens on `:7860` |
+| Vector store | ChromaDB `PersistentClient` | 2 collections, local on-disk |
+| Embeddings | `BAAI/bge-small-en-v1.5` | 384-dim, `normalize_embeddings=True`; query prefix `"Represent this sentence for searching relevant passages: "` |
+| Reranker | `cross-encoder/ms-marco-MiniLM-L-6-v2` | statute rerank, judgment child/parent rerank, **and** the unified cross-source rerank |
+| Lexical | `rank_bm25.BM25Okapi` | built in-memory over all judgment chunks at startup |
+| LLM | DeepSeek `deepseek-chat` | OpenAI-compatible client, `base_url=https://api.deepseek.com`, `stream=True`, `temperature=0.1`, `max_tokens=2500` |
+
+---
+
+## 3. Repository layout
+
+```
+backend/
+  app.py              FastAPI app · POST /ask (SSE) · citation links · lazy get_pipeline()
+  requirements.txt    fastapi, uvicorn, chromadb, sentence-transformers, openai, rank-bm25, huggingface_hub, python-dotenv
+hybrid_rag/
+  unified_legal_Rag.py  route_query · get_statute_evidence · get_judgment_evidence · unified_rerank · classify_intent · PROMPTS · Evidence
+statute corpus/
+  statute_retrieval.py  Chroma `indian_statutes` · parse_section_references · direct_lookup · expand_query · semantic_search · rerank_results
+  (serving data is fetched from a private, pinned Hugging Face dataset at runtime)
+llm_retriever.py        Chroma `sci_judgments_bge_v2` · dense_search · bm25_search · rrf_fusion · rerank_children · fetch_parent_chunks · rerank_parents · retrieve
+verifier/
+  verified.py           check_grounding (Tier-1/1.5) · verify_citations_live (Tier-2, lazy bharat_courts)
+  __init__.py
+frontend/
+  src/App.jsx           turns · ReasoningPanel · StepTimeline · Answer · CopyButton · intent dropdown · stop/new-chat
+  src/api.js            streamAsk(query, history, onEvent, signal, intent) — SSE parser
+  src/index.css         Harvey-inspired warm-light theme
+  public/icon.svg, favicon.ico
+Dockerfile              deps → COPY → private runtime artifact download → uvicorn
+render.yaml             (legacy) Render blueprint
+themis/                  current architecture, scaling and storyline documents
+```
+
+---
+
+## 4. The retrieval pipeline (in depth)
+
+### 4.1 Routing — `unified_legal_Rag.route_query(query) -> RouteDecision`
+```
+explicit INSC citation (judg_rag.extract_citations) AND not statute_signal -> judgment-only
+section reference (stat_rag.parse_section_references)  AND not judgment_signal -> statute-only
+otherwise -> hybrid (both)
+```
+Signal regexes: `JUDGMENT_SIGNAL_RE` (case/judgment/held/INSC/ratio/…), `STATUTE_SIGNAL_RE`
+(section/provision/IPC/BNS/…).
+
+### 4.2 Statute path — `get_statute_evidence(query)` → `statute_retrieval`
+1. `classify_query` → `{type: direct_lookup|semantic|hybrid, section_refs, acts_mentioned, is_comparative}`.
+2. `direct_lookup(act, sec)` for each parsed `(ACT, section)` → exact section (score `999.0`).
+3. `expand_query(query)` → DeepSeek rewrites colloquial → statutory language (2–3 variants).
+4. `semantic_search(query, top_k=40, act_filter)` over Chroma (+ per-expansion and per-act
+   searches, deduped by `(act_short, section_number)`).
+5. `rerank_results(rerank_query, candidates, top_k=15)` cross-encoder → wrapped as `Evidence`.
+
+### 4.3 Judgment path — `get_judgment_evidence(query)` → `llm_retriever.retrieve`
+Query is enriched with **severity context** (offence title from the referenced section) +
+an LLM **keyword expansion**. `retrieve` has three modes:
+- **citation** (1 INSC cite) → exact parent via `citation_search`.
+- **comparison** (≥2 cites + comparison words) → each parent.
+- **hybrid** → `dense_search(top 150)` + `bm25_search(top 150)` → `rrf_fusion(k=60)` →
+  `deduplicate_children(≤3/case)` → `rerank_children(top 30)` → `rank_cases_from_children`
+  → `fetch_parent_chunks(top 20)` → `rerank_parents(top 5)`.
+
+### 4.4 Unified rerank — `unified_rerank(query, evidence)`
+Exact hits (`score==999`) pinned on top; everything else scored on the **same**
+`ms-marco` cross-encoder so statute and judgment candidates are comparable. Then:
+**dedupe** by `("s", act, section)` / `("j", neutral_citation)`, **per-source cap**
+`MAX_SINGLE_TYPE_SHARE=6`, **final** `FINAL_EVIDENCE_N=8`. The rerank query is
+severity-enriched so "BNS 103" biases toward murder-class judgments.
+
+### 4.5 Generation
+`classify_intent(query)` (defaults to `LEGAL_RESEARCH`) selects one of five system prompts in
+`PROMPTS` (research citation-table / case summary / comprehensive study / comparison / story
+evaluation). All prompts enforce **evidence-grounding** (no model knowledge beyond evidence).
+The backend streams `stat_rag.llm_client.chat.completions.create(..., stream=True)` and emits
+each delta as a `token` event. A forced `intent` from the UI overrides classification.
+
+### 4.6 Verification — `verifier.check_grounding(answer, final_evidence, stat_rag, judg_rag)`
+Strips markdown bold, extracts cited sections + INSC citations, then:
+- cited section **in retrieved evidence** → grounded; **in full local DB but not retrieved**
+  → *retrieval miss* (real law); **not in DB** → hallucination (flagged).
+- cited case **not in evidence** → ungrounded citation (flagged).
+- `grounded = no hallucinated sections AND no ungrounded citations`.
+Tier-2 (`verify_citations_live`, live `bharat_courts` + 30-day cache) is **lazily imported**
+and currently deferred.
+
+---
+
+## 5. Data & indices
+
+| Corpus | Chroma collection | Count | On-disk | Provisioning |
+|---|---|---|---|---|
+| Statutes (6 acts) | `indian_statutes` | 2,353 sections | private artifact | **Downloaded at startup** from a pinned private Hugging Face dataset using the Space secret |
+| SCI judgments | schema-v5 FAISS + SQLite | release-defined | private artifact | **Downloaded at startup** from a pinned private Hugging Face dataset using the Space secret |
+
+- **Parent-child chunking**: parent = full judgment + metadata; children = ~512-token windows
+  (100 overlap). IDs: `<citation>__child_NNNN`, `<citation>__parent`.
+- **Judgment metadata** (per chunk): `case_name, neutral_citation, court, date, bench,
+  author_judge, acts, sections, issue, short_summary, full_headnote, outcome, source_url, …`.
+- **Statute record**: `{metadata: {act_short, act_name, section_number, title}, retrieval_text}`.
+- Neither statute nor judgment serving artifacts are committed to GitHub or baked into the
+  public Space image. `start_private_space.py` downloads both pinned snapshots with `HF_TOKEN`,
+  verifies their required entrypoints, removes the token from the API process environment, and
+  then starts FastAPI.
+
+---
+
+## 6. Backend API contract (SSE)
+
+### `POST /ask` → `text/event-stream`
+```jsonc
+// request
+{ "query": "string",
+  "history": [{"role": "user|assistant", "content": "..."}],
+  "intent": "AUTO | LEGAL_RESEARCH | CASE_SUMMARY | COMPREHENSIVE_CASE_STUDY | CASE_COMPARISON | STORY_EVALUATION" }
+```
+```jsonc
+// events — each emitted as `data: {json}\n\n`
+{ "type":"step",     "phase":"planning|retrieval|rerank|answer|verify", "title":"...", "detail":"..." }
+{ "type":"evidence", "items":[ /* statute or judgment items, see below */ ] }
+{ "type":"token",    "delta":"..." }
+{ "type":"verify",   "grounded":true, "hallucinated_sections":[], "retrieval_miss_sections":[], "unverified_citations":[] }
+{ "type":"done",     "answer":"...", "intent":"LEGAL_RESEARCH", "route":"...", "citations":[...], "elapsed_seconds":12.3 }
+{ "type":"error",    "message":"..." }
+```
+```jsonc
+// evidence items
+{ "kind":"statute",  "act":"BNS", "section":"103", "title":"Punishment for murder.", "score":1.23, "url":"https://indiankanoon.org/search/?formInput=..." }
+{ "kind":"judgment", "case":"Sanjay Kumar Sharma v. State of Bihar", "citation":"2026 INSC 223", "title":"<issue/summary>", "score":1.23, "url":"<source_url>" }
+```
+`GET /health` → `{status, service, pipeline_loaded}` · `GET /` → service info.
+
+Notes: an immediate `step:"Warming up"` is flushed **before** the lazy pipeline load so the
+SSE connection opens promptly; CORS origin via `FRONTEND_ORIGIN` (default `*`).
+
+---
+
+## 7. Frontend
+
+- `streamAsk(query, history, onEvent, signal, intent)` POSTs JSON and parses the SSE frame
+  stream (`\n\n`-delimited `data:` lines) off the `ReadableStream`.
+- Per-turn state `{query, steps[], evidence[], answer, citations[], verify, done, intent, elapsed_seconds}`.
+- **ReasoningPanel** — collapsed-by-default "Thinking…" disclosure; expands to the live
+  step timeline. **Answer** — markdown + top "Copy". **Copy answer + sources** — appends a
+  formatted `Sources:` block. **Stop** — `AbortController.abort()` (keeps partial answer).
+  **New chat** — clears turns + aborts. **Intent dropdown** — forces the answer style.
+- `VITE_API_URL` selects the backend (falls back to `/api`, proxied to `localhost:8000` in dev).
+
+---
+
+## 8. Configuration
+
+| Var | Where | Purpose |
+|---|---|---|
+| `DEEPSEEK_API_KEY` | backend env / HF secret | DeepSeek auth (required; checked before pipeline import) |
+| `CLERK_PUBLISHABLE_KEY` | backend env / HF variable | Public Clerk application key returned to both standalone frontends |
+| `CLERK_SECRET_KEY` | backend env / HF secret | Clerk backend API credential; never expose in frontend code |
+| `CLERK_JWT_KEY` | backend env / HF secret | Optional PEM public key for networkless session-token verification |
+| `CLERK_AUTHORIZED_PARTIES` | backend env / HF variable | Exact Moonley and local browser origins allowed by the API |
+| `JUDGMENTS_CHROMA_PATH` | backend env | Path to the judgments Chroma dir (Docker sets `/app/judgments_data/chroma_bge_v2`) |
+| `FRONTEND_ORIGIN` | backend env | CORS allowlist (comma-sep or `*`) |
+| `VITE_API_URL` | frontend build env | Backend base URL (set in Vercel) |
+| `HF_HOME`, `SENTENCE_TRANSFORMERS_HOME`, `TRANSFORMERS_CACHE` | Docker | model caches → `/tmp/hf` |
+
+---
+
+## 9. Local development
+
+```bash
+# Backend (needs private judgment/statute snapshots and DEEPSEEK_API_KEY).
+# Production downloads them automatically; new environments use MOONLEY_DATA and
+# MOONLEY_STATUTE_CHROMA. Legacy THEMIS_* names remain compatibility fallbacks.
+cp .env.example .env            # add DEEPSEEK_API_KEY
+pip install -r backend/requirements.txt
+uvicorn app:app --app-dir backend --host 0.0.0.0 --port 8000
+
+# Frontend (Vite dev server proxies /api -> http://localhost:8000)
+cd frontend && npm install && npm run dev
+```
+First `/ask` is slow (loads both indices + builds BM25 over 28,612 chunks + models), then warm.
+
+---
+
+## 10. Deployment
+
+```mermaid
+flowchart LR
+  GH["GitHub phase1.1"] -->|auto| VC["Vercel (React UI, root=vercel-frontend/)"]
+  GH -->|backend-only orphan push| SP["HF Space (FastAPI API)"]
+  SP -->|HF_TOKEN + snapshot_download| JD["Private judgment release"]
+  SP -->|HF_TOKEN + snapshot_download| SD["Private statute release"]
+```
+- **Frontend → Vercel:** the `phase1.1` production branch builds from
+  **Root Directory = `vercel-frontend`** and publishes only the React UI.
+- **Backend → HF Space (Docker SDK):** run the deploy script. It creates a clean orphan
+  branch, strips both frontend directories + `assets/` (HF rejects un-tracked binaries), and **injects the
+  HF config frontmatter into `README.md`** — which is kept *out* of the GitHub README (GitHub
+  renders frontmatter as an ugly table) — then force-pushes to the Space's `main`:
+  ```bash
+  git remote add space https://huggingface.co/spaces/<owner>/themis   # once
+  bash scripts/deploy-space.sh
+  ```
+- **Container startup:** download pinned private judgment and statute releases → verify required
+  files → remove `HF_TOKEN` from the API process environment → start `uvicorn` on `:7860`.
+- Set `DEEPSEEK_API_KEY` and a read-only `HF_TOKEN` as Space **secrets**; restart after rotation.
+
+---
+
+## 11. Operational notes
+
+- **Cold start:** `/health` is instant; the first `/ask` pays the full model+index+BM25 load
+  (~1–2 min on free CPU). Free Spaces sleep after ~48 h idle.
+- **Memory:** both Chroma collections + in-memory BM25 (28,612 chunks) + 2 transformer models
+  fit in 16 GB; this is the ceiling that motivates V2.
+- **Citations:** `parse_section_references` matches `BNS 103`, `BNS Section 103`,
+  `Section 103 of BNS`, and `Section 439 CrPC` (connector optional).
+- **Determinism:** `temperature=0.1` (answers), `0.0` (expansion/classification).
+
+---
+
+## 12. Roadmap → V2
+
+V1 is in-process and monolithic (index baked into the image, BM25 in RAM, models in the
+serving process) — fine for the pilot corpus, but it cannot reach the full Indian corpus. The scaling design
+re-platforms retrieval behind the *same* SSE contract: a managed, sharded **hybrid search
+engine** (server-side BM25 + dense ANN), **GPU embedding/rerank services**, **Postgres +
+object store**, and a real **ingestion ETL** with metadata pre-filtering and `court×year`
+sharding — so per-query work stays ~constant as the corpus grows.
+
+is kept with the production documentation on **[`phase1.1`](../../tree/phase1.1)**.
+
+---
+
+## 13. Documentation
+
+### Deployment
+- [Deployment guide — HF Spaces + Vercel](DEPLOYMENT.md)
+
+### Architecture and scaling — `phase1.1`
+- [Current architecture](themis/architecture.md)
+- [Scaling design](themis/scaling.md)
+- [Query lifecycle story](themis/storyline_themis.md)

backend/app.py

@@ -0,0 +1,381 @@
+"""
+themis — FastAPI backend (Phase 2: hybrid statutes + judgments)
+
+Wraps the unified legal RAG router (hybrid_rag/unified_legal_Rag.py) and streams
+every step over Server-Sent Events so the UI can render the reasoning live:
+
+    planning  -> route decision (statute / judgment / hybrid)
+    retrieval -> statute semantic search + SCI judgment dense+BM25+RRF
+    rerank    -> unified cross-encoder rerank across BOTH sources
+    answer    -> streamed DeepSeek tokens (intent-specific prompt)
+    verify    -> Tier-1 grounding for cited sections AND case citations
+    done      -> final answer + citation hyperlinks (sections + cases)
+
+Heavy modules (two Chroma indices, BM25 over 28k judgment chunks, embedding +
+cross-encoder models) load lazily on the first /ask, so the server boots
+instantly and the SSE stream emits a "warming up" step before the load.
+"""
+
+import asyncio
+import json
+import os
+import re
+import sys
+import time
+import logging
+from urllib.parse import quote_plus
+
+from fastapi import FastAPI
+from fastapi.middleware.cors import CORSMiddleware
+from fastapi.responses import StreamingResponse, JSONResponse
+from pydantic import BaseModel
+
+log = logging.getLogger("themis.backend")
+logging.basicConfig(level=logging.INFO, format="%(asctime)s  %(levelname)-8s  %(message)s", datefmt="%H:%M:%S")
+
+BACKEND_DIR = os.path.dirname(os.path.abspath(__file__))
+PROJECT_ROOT = os.path.dirname(BACKEND_DIR)
+
+try:
+    from dotenv import load_dotenv
+    load_dotenv(os.path.join(PROJECT_ROOT, ".env"))
+except Exception:
+    pass
+
+# The unified router lives in hybrid_rag/ and wires in both pipelines + verifier.
+sys.path.insert(0, os.path.join(PROJECT_ROOT, "hybrid_rag"))
+sys.path.insert(0, PROJECT_ROOT)
+
+_uni = None
+
+
+def get_pipeline():
+    """Import and cache the unified router (loads both Chroma indices + models)."""
+    global _uni
+    if _uni is None:
+        if not os.getenv("DEEPSEEK_API_KEY"):
+            raise RuntimeError("DEEPSEEK_API_KEY is not set in the environment.")
+        log.info("Loading unified pipeline (statutes + judgments + models)…")
+        import unified_legal_Rag as uni   # noqa: E402  (heavy import on purpose)
+        _uni = uni
+        log.info("Unified pipeline ready.")
+    return _uni
+
+
+def _norm_sec(section) -> str:
+    """'103(1)(a)' -> '103' — compare on the base section number."""
+    m = re.match(r"\s*([0-9]+[A-Za-z]?)", str(section))
+    return m.group(1) if m else str(section).strip()
+
+
+# ---------------------------------------------------------------------
+# Citation hyperlinks
+# ---------------------------------------------------------------------
+def _statute_url(act_full: str, act_short: str, section: str) -> str:
+    q = quote_plus(f"{act_full or act_short} Section {section}")
+    return f"https://indiankanoon.org/search/?formInput={q}"
+
+
+def _judgment_url(meta: dict) -> str:
+    url = meta.get("source_url") or ""
+    if url.startswith("http"):
+        return url
+    cite = meta.get("neutral_citation") or meta.get("case_name") or ""
+    return f"https://indiankanoon.org/search/?formInput={quote_plus(cite)}"
+
+
+def _evidence_item(e) -> dict:
+    """Normalise an Evidence object for the UI."""
+    m = e.meta or {}
+    score = round(float(getattr(e, "unified_score", 0.0)), 3)
+    if e.source_type == "judgment":
+        return {
+            "kind": "judgment",
+            "case": m.get("case_name", ""),
+            "citation": m.get("neutral_citation", ""),
+            "title": m.get("issue", "") or m.get("short_summary", ""),
+            "score": score,
+            "url": _judgment_url(m),
+        }
+    return {
+        "kind": "statute",
+        "act": m.get("act_short", ""),
+        "section": str(m.get("section_number", "")),
+        "title": m.get("title", ""),
+        "score": score,
+        "url": _statute_url(m.get("act_name", ""), m.get("act_short", ""), m.get("section_number", "")),
+    }
+
+
+def _sse(obj: dict) -> str:
+    return f"data: {json.dumps(obj, ensure_ascii=False)}\n\n"
+
+
+# =====================================================================
+# APP
+# =====================================================================
+app = FastAPI(title="themis", version="0.2.0")
+
+_origins_env = os.getenv("FRONTEND_ORIGIN", "*")
+_allow_origins = ["*"] if _origins_env.strip() == "*" else [o.strip() for o in _origins_env.split(",")]
+app.add_middleware(CORSMiddleware, allow_origins=_allow_origins, allow_methods=["*"], allow_headers=["*"])
+
+
+class Message(BaseModel):
+    role: str
+    content: str
+
+
+class AskRequest(BaseModel):
+    query: str
+    history: list[Message] = []
+    intent: str = "AUTO"   # "AUTO" -> classify; otherwise force a PROMPTS style
+
+
+@app.get("/health")
+def health():
+    return {"status": "ok", "service": "themis", "pipeline_loaded": _uni is not None}
+
+
+@app.get("/")
+def root():
+    return JSONResponse({"service": "themis", "version": "0.2.0", "ask": "POST /ask (SSE)"})
+
+
+@app.post("/ask")
+def ask(req: AskRequest):
+    def event_stream():
+        t_start = time.time()
+        query = req.query.strip()
+        if not query:
+            yield _sse({"type": "error", "message": "Empty query."})
+            return
+
+        # Flush an immediate event so the SSE connection opens before the
+        # (potentially slow) first-time model + index load.
+        yield _sse({"type": "step", "phase": "planning", "title": "Warming up",
+                    "detail": "Loading statute + judgment indices and models (first query only)…"})
+
+        try:
+            uni = get_pipeline()
+        except Exception as e:
+            yield _sse({"type": "error", "message": str(e)})
+            return
+
+        history = [m.model_dump() for m in req.history]
+
+        # ---- Route ----
+        decision = uni.route_query(query)
+        srcs = []
+        if decision.use_statute:
+            srcs.append("statutes")
+        if decision.use_judgment:
+            srcs.append("Supreme Court judgments")
+        yield _sse({"type": "step", "phase": "planning",
+                    "title": "Routing the query",
+                    "detail": f"Searching: {', '.join(srcs)}.\n{decision.reason}"})
+
+        all_ev = []
+
+        # ---- Statute retrieval ----
+        if decision.use_statute:
+            yield _sse({"type": "step", "phase": "retrieval",
+                        "title": "Searching statutes",
+                        "detail": "Query expansion → ChromaDB `indian_statutes` → cross-encoder rerank…"})
+            try:
+                sev = uni.get_statute_evidence(query)
+            except Exception as e:
+                log.warning("statute path failed: %s", e)
+                sev = []
+            all_ev.extend(sev)
+            yield _sse({"type": "step", "phase": "retrieval",
+                        "title": f"{len(sev)} statute candidate(s)", "detail": ""})
+
+        # ---- Judgment retrieval ----
+        if decision.use_judgment:
+            yield _sse({"type": "step", "phase": "retrieval",
+                        "title": "Searching Supreme Court judgments",
+                        "detail": "Dense + BM25 → RRF fusion → dedup → child & parent rerank…"})
+            try:
+                jev = uni.get_judgment_evidence(query)
+            except Exception as e:
+                log.warning("judgment path failed: %s", e)
+                jev = []
+            all_ev.extend(jev)
+            yield _sse({"type": "step", "phase": "retrieval",
+                        "title": f"{len(jev)} judgment candidate(s)", "detail": ""})
+
+        if not all_ev:
+            yield _sse({"type": "error", "message": "No relevant statutes or judgments found for this query."})
+            return
+
+        # ---- Unified rerank across both sources ----
+        yield _sse({"type": "step", "phase": "rerank",
+                    "title": "Reranking all evidence together",
+                    "detail": "Scoring statute + judgment candidates on one cross-encoder for a fair merge…"})
+        severity = uni._build_severity_context(query)
+        unified_query = f"{query} {severity}" if severity else query
+        final_ev = uni.unified_rerank(unified_query, all_ev)
+
+        yield _sse({"type": "evidence", "items": [_evidence_item(e) for e in final_ev]})
+
+        context = uni.build_unified_context(final_ev)
+
+        # ---- Intent (user-forced or auto-classified) + streamed answer ----
+        forced = (req.intent or "AUTO").upper()
+        if forced != "AUTO" and forced in uni.PROMPTS:
+            intent = forced
+        else:
+            intent = uni.classify_intent(query)
+        yield _sse({"type": "step", "phase": "answer",
+                    "title": f"Drafting the answer · {intent.replace('_', ' ').title()}",
+                    "detail": "Generating a grounded answer from the retrieved evidence…"})
+
+        system_prompt = uni.PROMPTS.get(intent, uni.PROMPTS["LEGAL_RESEARCH"])
+        messages = [{"role": "system", "content": system_prompt}]
+        if history:
+            messages.extend(history[-8:])
+        messages.append({"role": "user", "content": f"QUESTION:\n{query}\n\nEVIDENCE:\n{context}"})
+
+        full_answer = ""
+        try:
+            stream = uni.stat_rag.llm_client.chat.completions.create(
+                model=uni.LLM_MODEL,
+                messages=messages,
+                temperature=uni.LLM_TEMPERATURE,
+                max_tokens=uni.LLM_MAX_TOKENS,
+                stream=True,
+            )
+            for chunk in stream:
+                delta = chunk.choices[0].delta
+                if delta.content:
+                    full_answer += delta.content
+                    yield _sse({"type": "token", "delta": delta.content})
+        except Exception as e:
+            log.error("DeepSeek error: %s", e)
+            yield _sse({"type": "error", "message": f"Answer generation failed: {e}"})
+            return
+
+        # ---- Citation hyperlinks (sections + cases actually cited) ----
+        clean = full_answer.replace("**", "")
+        ev_by_cite = {
+            (e.meta.get("neutral_citation") or "").upper(): e.meta
+            for e in final_ev if e.source_type == "judgment"
+        }
+        citations = []
+        cited_cases = []          # (citation, case_name) cited in the answer
+        seen = set()
+        try:
+            for act, sec in (uni.stat_rag.parse_section_references(clean) or []):
+                key = ("s", act.upper(), str(sec))
+                if key in seen:
+                    continue
+                seen.add(key)
+                rec = uni.stat_rag.direct_lookup(act, sec)
+                meta = rec["metadata"] if rec else {}
+                citations.append({
+                    "kind": "statute",
+                    "label": f"{act.upper()} Section {sec}",
+                    "title": meta.get("title", ""),
+                    "url": _statute_url(meta.get("act_name", ""), act, sec),
+                })
+            for cite in (uni.judg_rag.extract_citations(clean) or []):
+                key = ("c", cite.upper())
+                if key in seen:
+                    continue
+                seen.add(key)
+                m = ev_by_cite.get(cite.upper(), {})
+                citations.append({
+                    "kind": "judgment",
+                    "label": (m.get("case_name") or cite),
+                    "title": cite if m.get("case_name") else "",
+                    "url": _judgment_url(m or {"neutral_citation": cite}),
+                })
+                cited_cases.append((cite, m.get("case_name", "")))
+        except Exception as e:
+            log.warning("citation links failed: %s", e)
+
+        # Emit the answer immediately; verification (which may hit slow external
+        # portals) runs afterwards and streams in as a follow-up `verify` event.
+        yield _sse({
+            "type": "done",
+            "answer": full_answer,
+            "intent": intent,
+            "route": decision.reason,
+            "citations": citations,
+            "elapsed_seconds": round(time.time() - t_start, 2),
+        })
+
+        # ---- Verification ----
+        yield _sse({"type": "step", "phase": "verify",
+                    "title": "Verifying citations",
+                    "detail": "Checking cited sections against the corpus; verifying cited cases "
+                              "against the SCI/eCourts portals + Indian Kanoon…"})
+
+        # Statute sections: in evidence / in DB (retrieval miss) / not in DB (hallucinated)
+        ev_sections = {
+            (e.meta.get("act_short", "").upper(), _norm_sec(e.meta.get("section_number", "")))
+            for e in final_ev if e.source_type == "statute"
+        }
+        flagged_sections, retrieval_miss = [], []
+        try:
+            sec_seen = set()
+            for act, sec in (uni.stat_rag.parse_section_references(clean) or []):
+                k = (act.upper(), _norm_sec(sec))
+                if k in sec_seen:
+                    continue
+                sec_seen.add(k)
+                if k in ev_sections:
+                    continue
+                in_db = (
+                    (act.upper(), str(sec)) in uni.stat_rag.section_db
+                    or (act.upper(), _norm_sec(sec)) in uni.stat_rag.section_db
+                )
+                (retrieval_miss if in_db else flagged_sections).append(f"{act.upper()} {sec}")
+        except Exception as e:
+            log.warning("section verify failed: %s", e)
+
+        # Case citations: in-corpus (real, instant) vs ungrounded (live-verify)
+        cases = []
+        to_live = []
+        case_seen = set()
+        for cite, _name in cited_cases:
+            if cite.upper() in case_seen:
+                continue
+            case_seen.add(cite.upper())
+            m = ev_by_cite.get(cite.upper())
+            if m:
+                cases.append({"citation": cite, "case": m.get("case_name", ""),
+                              "status": "IN_CORPUS", "url": _judgment_url(m),
+                              "note": "in retrieved corpus"})
+            else:
+                to_live.append(cite)
+
+        if to_live:
+            try:
+                from verifier import verify_citations
+                results = asyncio.run(verify_citations(to_live[:6], concurrency=2))
+                for cite, r in zip(to_live, results):
+                    link = r.ik_match_url or (next(iter(r.verify_links.values()), "") if r.verify_links else "")
+                    cases.append({"citation": cite, "case": (r.ik_match_title or ""),
+                                  "status": r.status.value, "url": link, "note": r.note or ""})
+            except Exception as e:
+                log.warning("live citation verify failed: %s", e)
+                for cite in to_live:
+                    cases.append({"citation": cite, "case": "", "status": "ERROR",
+                                  "url": "", "note": "live verification unavailable"})
+
+        grounded = (not flagged_sections) and all(c["status"] != "NOT_FOUND" for c in cases)
+        yield _sse({
+            "type": "verify",
+            "grounded": grounded,
+            "flagged_sections": flagged_sections,
+            "retrieval_miss_sections": retrieval_miss,
+            "cases": cases,
+        })
+
+    return StreamingResponse(
+        event_stream(),
+        media_type="text/event-stream",
+        headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
+    )

backend/build_index.py

@@ -0,0 +1,75 @@
+"""
+Build the `indian_statutes` ChromaDB collection from all_statutes.json.
+
+Run once at Docker build time so the 19 MB binary index never has to live in
+git (HF Spaces rejects >10 MB files without LFS, and rebuilding sidesteps any
+chromadb on-disk-format version drift). Reproduces the original embeddings
+faithfully: same model (BGE-small), same input (retrieval_text), same
+normalization — see "statute corpus/create_embedding.py".
+"""
+
+import json
+import os
+import sys
+
+import chromadb
+from sentence_transformers import SentenceTransformer
+
+THIS_DIR = os.path.dirname(os.path.abspath(__file__))
+PROJECT_ROOT = os.path.dirname(THIS_DIR)
+STATUTE_DIR = os.path.join(PROJECT_ROOT, "statute corpus")
+
+STATUTES_FILE = os.path.join(STATUTE_DIR, "all_statutes.json")
+CHROMA_PATH = os.path.join(STATUTE_DIR, "chroma_statutes")
+COLLECTION_NAME = "indian_statutes"
+MODEL_NAME = "BAAI/bge-small-en-v1.5"
+BATCH_SIZE = 100
+
+
+def main() -> None:
+    if not os.path.exists(STATUTES_FILE):
+        print(f"ERROR: {STATUTES_FILE} not found", file=sys.stderr)
+        sys.exit(1)
+
+    with open(STATUTES_FILE, "r", encoding="utf-8") as f:
+        statutes = json.load(f)
+    print(f"Loaded {len(statutes)} statute records")
+
+    model = SentenceTransformer(MODEL_NAME)
+    client = chromadb.PersistentClient(path=CHROMA_PATH)
+
+    try:
+        client.delete_collection(COLLECTION_NAME)
+    except Exception:
+        pass
+    collection = client.create_collection(COLLECTION_NAME)
+
+    ids, documents, metadatas = [], [], []
+    for i, record in enumerate(statutes):
+        meta = record["metadata"]
+        # Reconstructed JSON has no chunk_id — synthesize a unique, stable id.
+        ids.append(f"{meta.get('act_short','?')}__{meta.get('section_number','?')}__{i}")
+        documents.append(record["retrieval_text"])
+        metadatas.append({
+            "act_short": str(meta.get("act_short", "")),
+            "act_name": str(meta.get("act_name", "")),
+            "section_number": str(meta.get("section_number", "")),
+            "title": str(meta.get("title", "")),
+        })
+
+    for i in range(0, len(documents), BATCH_SIZE):
+        batch_docs = documents[i:i + BATCH_SIZE]
+        embeddings = model.encode(batch_docs, normalize_embeddings=True, show_progress_bar=False)
+        collection.add(
+            ids=ids[i:i + BATCH_SIZE],
+            documents=batch_docs,
+            metadatas=metadatas[i:i + BATCH_SIZE],
+            embeddings=embeddings.tolist(),
+        )
+        print(f"  embedded {min(i + BATCH_SIZE, len(documents))}/{len(documents)}")
+
+    print(f"DONE — {collection.count()} documents in '{COLLECTION_NAME}' at {CHROMA_PATH}")
+
+
+if __name__ == "__main__":
+    main()

backend/requirements.txt

@@ -0,0 +1,18 @@
+# themis backend — Phase 1 (statutes-only)
+fastapi>=0.110
+uvicorn[standard]>=0.29
+pydantic>=2.6
+
+# RAG pipeline (statutes + SCI judgments)
+chromadb>=0.4.24
+sentence-transformers>=2.6.0
+openai>=1.30
+python-dotenv>=1.0
+rank-bm25>=0.2.2
+huggingface_hub>=0.24
+
+# Live citation verification (verifier/citation_verifier.py)
+httpx>=0.27
+ddddocr>=1.4
+beautifulsoup4>=4.12
+rapidfuzz>=3.6

chunking.py

@@ -0,0 +1,417 @@
+"""
+Parent-Child Hierarchical Chunker
+=====================================================
+Parent = full judgment text (used for context retrieval)
+Child  = fixed-token sub-chunks of parent (used for embedding + search)
+
+Reads:  data/html/extracted_judgments.jsonl   (metadata)
+        data/pdfs/*.pdf                        (judgment text)
+Writes: data/chunks/parent_child/<neutral_citation>.json
+
+Architecture:
+  Query hits a child chunk (small, precise, embedded)
+      ↓
+  Child carries parent_chunk_id
+      ↓
+  Fetch parent for full context window
+      ↓
+  Send parent text to LLM for answer generation
+"""
+
+import json
+import os
+import re
+import fitz
+import logging
+import tiktoken
+import concurrent.futures
+from pathlib import Path
+from datetime import datetime
+
+# ---------------------------------------------------------------------------
+# Logging
+# ---------------------------------------------------------------------------
+logging.basicConfig(
+    level=logging.INFO,
+    format="%(asctime)s [%(levelname)s] %(message)s",
+    datefmt="%H:%M:%S",
+    handlers=[
+        logging.StreamHandler(),
+        logging.FileHandler("data/parent_child_chunker.log", encoding="utf-8"),
+    ]
+)
+log = logging.getLogger(__name__)
+
+# ---------------------------------------------------------------------------
+# Config
+# ---------------------------------------------------------------------------
+METADATA_FILE    = os.path.join("data", "html", "extracted_judgments.jsonl")
+PDF_DIR          = Path("data", "pdfs")
+OUTPUT_DIR       = Path("data", "chunks", "parent_child")
+ERROR_LOG        = Path("data", "chunks", "parent_child_errors.jsonl")
+
+CHILD_CHUNK_SIZE    = 512    # tokens per child chunk
+CHILD_CHUNK_OVERLAP = 100    # token overlap between children
+TOKENIZER_MODEL     = "cl100k_base"
+MAX_WORKERS         = min(8, (os.cpu_count() or 4))
+
+# Parent size threshold — if judgment is smaller than this, skip children
+MIN_TOKENS_FOR_CHILDREN = CHILD_CHUNK_SIZE
+
+OUTPUT_DIR.mkdir(parents=True, exist_ok=True)
+
+# ---------------------------------------------------------------------------
+# Globals (shared across threads — all read-only after init)
+# ---------------------------------------------------------------------------
+TOKENIZER = tiktoken.get_encoding(TOKENIZER_MODEL)
+
+# Pre-compiled regex
+RE_BACKSPACE    = re.compile(r"\x08")
+RE_AUTHOR       = re.compile(r"\n\*\s*Author\n\d+\n")
+RE_PAGE_NUMS    = re.compile(r"\n\s*\d{1,3}\s*\n")
+RE_FOOTER       = re.compile(r"\nJudgment\s*/\s*Order of the Supreme Court\n?", re.I)
+RE_HEADER       = re.compile(r"\n(Supreme Court of India|IN THE SUPREME COURT OF INDIA)\n", re.I)
+RE_NEWLINES     = re.compile(r"\n{3,}")
+RE_SPACES       = re.compile(r"[ \t]+")
+
+
+# ---------------------------------------------------------------------------
+# Step 1: Load metadata index keyed by PDF filename
+# ---------------------------------------------------------------------------
+def load_metadata_index(jsonl_path: str) -> dict:
+    """
+    Returns:
+    {
+      "2026_INSC_479.pdf": { ...full metadata record... },
+      ...
+    }
+    """
+    index = {}
+    missing = 0
+
+    with open(jsonl_path, "r", encoding="utf-8") as f:
+        for line in f:
+            line = line.strip()
+            if not line:
+                continue
+            try:
+                record = json.loads(line)
+            except json.JSONDecodeError:
+                continue
+
+            pdf_path = record.get("pdf_path", "")
+            if pdf_path:
+                filename = Path(pdf_path).name        # "2026_INSC_479.pdf"
+                index[filename] = record
+            else:
+                # Fallback: derive from neutral citation
+                nc = record.get("neutral_citation", "").strip()
+                if nc:
+                    filename = nc.replace(" ", "_") + ".pdf"
+                    index[filename] = record
+                    missing += 1
+
+    log.info(f"Loaded {len(index)} metadata records "
+             f"({missing} used neutral_citation fallback).")
+    return index
+
+
+# ---------------------------------------------------------------------------
+# Step 2: PDF text extraction
+# ---------------------------------------------------------------------------
+def extract_pdf_text(pdf_path: Path) -> str:
+    """Extract full text from PDF using pymupdf."""
+    parts = []
+    try:
+        doc = fitz.open(str(pdf_path))
+        for page in doc:
+            parts.append(page.get_text())
+        doc.close()
+    except Exception as e:
+        log.error(f"PDF read failed [{pdf_path.name}]: {e}")
+    return "\n".join(parts)
+
+
+# ---------------------------------------------------------------------------
+# Step 3: Text cleaning
+# ---------------------------------------------------------------------------
+def clean_text(text: str) -> str:
+    """Remove PDF artifacts, headers, footers, page numbers."""
+    if not text:
+        return ""
+    text = RE_BACKSPACE.sub("", text)
+    text = RE_AUTHOR.sub("\n", text)
+    text = RE_PAGE_NUMS.sub("\n\n", text)
+    text = RE_FOOTER.sub("\n", text)
+    text = RE_HEADER.sub("\n", text)
+    text = RE_NEWLINES.sub("\n\n", text)
+    text = RE_SPACES.sub(" ", text)
+    return text.strip()
+
+
+# ---------------------------------------------------------------------------
+# Step 4: Build lean metadata (for child chunks)
+# ---------------------------------------------------------------------------
+def build_lean_metadata(record: dict) -> dict:
+    """
+    Child chunks carry only the fields needed for Qdrant payload filtering.
+    Full metadata lives on the parent — fetched at answer-generation time.
+    """
+    return {
+        "case_name":        record.get("case_name", ""),
+        "neutral_citation": record.get("neutral_citation", ""),
+        "date":             record.get("date", ""),
+        "court":            record.get("court", "Supreme Court"),
+        "case_type":        record.get("case_type", ""),
+        "outcome":          record.get("outcome", ""),
+        "acts":             record.get("acts", []),
+        "keywords":         record.get("keywords", []),
+    }
+
+
+# ---------------------------------------------------------------------------
+# Step 5: Build full metadata (for parent chunk)
+# ---------------------------------------------------------------------------
+def build_full_metadata(record: dict) -> dict:
+    return {
+        "case_name":        record.get("case_name", ""),
+        "neutral_citation": record.get("neutral_citation", ""),
+        "appeal_no":        record.get("appeal_no", ""),
+        "citation":         record.get("citation", ""),
+        "date":             record.get("date", ""),
+        "court":            record.get("court", "Supreme Court"),
+        "lower_court":      record.get("lower_court", ""),
+        "jurisdiction":     record.get("jurisdiction", "India"),
+        "bench":            record.get("bench", []),
+        "author_judge":     record.get("author_judge", ""),
+        "outcome":          record.get("outcome", ""),
+        "case_type":        record.get("case_type", ""),
+        "acts":             record.get("acts", []),
+        "sections":         record.get("sections", []),
+        "cases_cited":      record.get("cases_cited", []),
+        "keywords":         record.get("keywords", []),
+        "issue":            record.get("issue", ""),
+        "short_summary":    record.get("short_summary", ""),
+        "full_headnote":    record.get("full_headnote", ""),
+        "source_url":       record.get("source_url", ""),
+        "scraped_at":       record.get("scraped_at", ""),
+    }
+
+
+# ---------------------------------------------------------------------------
+# Step 6: Core parent-child chunking logic
+# ---------------------------------------------------------------------------
+def build_parent_child(
+    neutral_citation: str,
+    cleaned_text: str,
+    full_metadata: dict,
+    lean_metadata: dict,
+) -> dict:
+    """
+    Returns:
+    {
+      "parent":   { single parent chunk with full text + full metadata },
+      "children": [ child chunks with sub-text + lean metadata ]
+    }
+    """
+    safe_nc      = neutral_citation.replace(" ", "_")
+    parent_id    = f"{safe_nc}__parent"
+
+    tokens       = TOKENIZER.encode(cleaned_text)
+    num_tokens   = len(tokens)
+
+    # --- Parent chunk ---
+    parent = {
+        "chunk_id":    parent_id,
+        "chunk_type":  "parent",
+        "document_id": neutral_citation,
+        "text":        cleaned_text,
+        "token_count": num_tokens,
+        "char_count":  len(cleaned_text),
+        "metadata":    full_metadata,
+        "chunked_at":  datetime.utcnow().isoformat(),
+    }
+
+    # --- Skip children if text is too small ---
+    if num_tokens <= MIN_TOKENS_FOR_CHILDREN:
+        parent["child_count"] = 0
+        return {"parent": parent, "children": []}
+
+    # --- Child chunks ---
+    step     = CHILD_CHUNK_SIZE - CHILD_CHUNK_OVERLAP
+    children = []
+    idx      = 0
+
+    for start in range(0, num_tokens, step):
+        end          = min(start + CHILD_CHUNK_SIZE, num_tokens)
+        chunk_tokens = tokens[start:end]
+        chunk_text   = TOKENIZER.decode(chunk_tokens)
+
+        children.append({
+            "chunk_id":       f"{safe_nc}__child_{idx:04d}",
+            "chunk_type":     "child",
+            "parent_chunk_id": parent_id,
+            "document_id":    neutral_citation,
+            "child_index":    idx,
+            "token_count":    len(chunk_tokens),
+            "char_count":     len(chunk_text),
+            "start_token":    start,
+            "end_token":      end,
+            "text":           chunk_text,
+            "metadata":       lean_metadata,
+        })
+        idx += 1
+
+        if end == num_tokens:
+            break
+
+    parent["child_count"] = len(children)
+    return {"parent": parent, "children": children}
+
+
+# ---------------------------------------------------------------------------
+# Step 7: Process single PDF (called by thread pool)
+# ---------------------------------------------------------------------------
+def process_pdf(pdf_path: Path, metadata_index: dict) -> dict:
+    """
+    Returns a result dict:
+    {
+      "status":  "success" | "skipped" | "error",
+      "file":    pdf filename,
+      "message": description,
+      "chunks":  { parent, children } or None
+    }
+    """
+    filename = pdf_path.name
+
+    # --- Idempotency: skip if already processed ---
+    record           = metadata_index.get(filename)
+    if not record:
+        return {"status": "unmatched", "file": filename,
+                "message": f"No metadata found for {filename}"}
+
+    neutral_citation = record.get("neutral_citation", "")
+    safe_nc          = neutral_citation.replace(" ", "_")
+    output_file      = OUTPUT_DIR / f"{safe_nc}.json"
+
+    if output_file.exists():
+        return {"status": "skipped", "file": filename,
+                "message": f"Already processed: {output_file.name}"}
+
+    # --- Extract + clean text ---
+    raw_text     = extract_pdf_text(pdf_path)
+    cleaned      = clean_text(raw_text)
+
+    if not cleaned:
+        return {"status": "error", "file": filename,
+                "message": "Empty text after cleaning"}
+
+    # --- Build metadata ---
+    full_meta = build_full_metadata(record)
+    lean_meta = build_lean_metadata(record)
+
+    # --- Build parent-child structure ---
+    result = build_parent_child(neutral_citation, cleaned, full_meta, lean_meta)
+
+    # --- Write output atomically ---
+    # Write to temp file first, then rename — prevents corrupt files on crash
+    temp_file = output_file.with_suffix(".tmp")
+    try:
+        with open(temp_file, "w", encoding="utf-8") as f:
+            json.dump(result, f, ensure_ascii=False, indent=2)
+        temp_file.rename(output_file)
+    except Exception as e:
+        if temp_file.exists():
+            temp_file.unlink()
+        return {"status": "error", "file": filename, "message": str(e)}
+
+    return {
+        "status":   "success",
+        "file":     filename,
+        "message":  f"{result['parent']['child_count']} children created",
+        "children": result["parent"]["child_count"],
+    }
+
+
+# ---------------------------------------------------------------------------
+# Step 8: Main pipeline
+# ---------------------------------------------------------------------------
+def main():
+    log.info("=" * 60)
+    log.info("Parent-Child Chunker — Starting")
+    log.info("=" * 60)
+
+    # Load metadata
+    metadata_index = load_metadata_index(METADATA_FILE)
+
+    # Discover PDFs
+    pdf_files = sorted(PDF_DIR.glob("*.pdf"))
+    log.info(f"Found {len(pdf_files)} PDFs in {PDF_DIR}")
+
+    if not pdf_files:
+        log.error("No PDFs found. Check PDF_DIR path.")
+        return
+
+    # Counters
+    counts = {"success": 0, "skipped": 0, "unmatched": 0, "error": 0}
+    total_children = 0
+    errors = []
+
+    # Process concurrently
+    log.info(f"Processing with {MAX_WORKERS} workers...")
+
+    with concurrent.futures.ThreadPoolExecutor(max_workers=MAX_WORKERS) as executor:
+        futures = {
+            executor.submit(process_pdf, pdf_path, metadata_index): pdf_path
+            for pdf_path in pdf_files
+        }
+
+        for i, future in enumerate(concurrent.futures.as_completed(futures), 1):
+            pdf_path = futures[future]
+            try:
+                result = future.result()
+                status = result["status"]
+                counts[status] = counts.get(status, 0) + 1
+
+                if status == "success":
+                    total_children += result.get("children", 0)
+                    if i % 50 == 0:
+                        log.info(
+                            f"Progress: {i}/{len(pdf_files)} | "
+                            f"Success: {counts['success']} | "
+                            f"Skipped: {counts['skipped']} | "
+                            f"Errors: {counts['error']}"
+                        )
+                elif status == "error":
+                    log.warning(f"[ERROR] {result['file']}: {result['message']}")
+                    errors.append(result)
+                elif status == "unmatched":
+                    log.warning(f"[UNMATCHED] {result['file']}")
+                    errors.append(result)
+
+            except Exception as exc:
+                counts["error"] += 1
+                log.error(f"[EXCEPTION] {pdf_path.name}: {exc}")
+                errors.append({"file": pdf_path.name, "message": str(exc)})
+
+    # Write error log
+    if errors:
+        with open(ERROR_LOG, "w", encoding="utf-8") as f:
+            for e in errors:
+                f.write(json.dumps(e, ensure_ascii=False) + "\n")
+        log.info(f"Error details → {ERROR_LOG}")
+
+    # Final summary
+    log.info("=" * 60)
+    log.info("PIPELINE COMPLETE")
+    log.info(f"  Successful  : {counts['success']}")
+    log.info(f"  Skipped     : {counts['skipped']} (already processed)")
+    log.info(f"  Unmatched   : {counts['unmatched']} (no metadata)")
+    log.info(f"  Errors      : {counts['error']}")
+    log.info(f"  Total children created : {total_children}")
+    log.info(f"  Output dir  : {OUTPUT_DIR.resolve()}")
+    log.info("=" * 60)
+
+
+if __name__ == "__main__":
+    main()

embedding.py

@@ -0,0 +1,106 @@
+#%%
+import os
+import json
+import chromadb
+from sentence_transformers import SentenceTransformer
+from tqdm import tqdm
+
+CHUNKS_DIR = "/content/drive/MyDrive/updatedparentchunk/chunks/parent_child"
+
+client = chromadb.PersistentClient(
+    path="/content/drive/MyDrive/chroma_bge_v2"
+)
+
+collection = client.get_or_create_collection(
+    "sci_judgments_bge_v2"
+)
+
+model = SentenceTransformer(
+    "BAAI/bge-small-en-v1.5"
+)
+
+def flatten_metadata(meta):
+    flat = {}
+
+    for k, v in meta.items():
+
+        if isinstance(v, (str, int, float, bool)):
+            flat[k] = v
+
+        elif isinstance(v, (list, dict)):
+            flat[k] = json.dumps(v)
+
+        else:
+            flat[k] = str(v)
+
+    return flat
+
+
+files = [
+    f for f in os.listdir(CHUNKS_DIR)
+    if f.endswith(".json")
+]
+
+print("Files:", len(files))
+
+all_ids = []
+all_docs = []
+all_meta = []
+
+for filename in tqdm(files):
+
+    with open(
+        os.path.join(CHUNKS_DIR, filename),
+        encoding="utf-8"
+    ) as f:
+
+        data = json.load(f)
+
+    parent = data["parent"]
+
+    all_ids.append(parent["chunk_id"])
+    all_docs.append(parent["text"])
+    all_meta.append(
+        flatten_metadata(parent["metadata"])
+    )
+
+    for child in data["children"]:
+
+        all_ids.append(child["chunk_id"])
+        all_docs.append(child["text"])
+        all_meta.append(
+            flatten_metadata(child["metadata"])
+        )
+
+print("Total chunks:", len(all_ids))
+
+BATCH_SIZE = 512
+
+for start in tqdm(
+    range(0, len(all_ids), BATCH_SIZE),
+    desc="Embedding"
+):
+
+    end = min(
+        start + BATCH_SIZE,
+        len(all_ids)
+    )
+
+    batch_docs = all_docs[start:end]
+
+    embeddings = model.encode(
+        batch_docs,
+        batch_size=128,
+        normalize_embeddings=True,
+        show_progress_bar=False
+    )
+
+    collection.add(
+        ids=all_ids[start:end],
+        embeddings=embeddings.tolist(),
+        documents=batch_docs,
+        metadatas=all_meta[start:end]
+    )
+
+print("\nDONE")
+print("Collection count:", collection.count())

hybrid_rag/unified_legal_Rag.py

@@ -0,0 +1,973 @@
+"""
+unified_legal_rag.py
+=====================
+Unified Legal RAG Router — LegalAIapex
+
+Connects two existing, independently-built retrieval pipelines:
+  1. Statute retrieval   (statute_retrieval.py)  — IPC/BNS/CrPC/BNSS/IEA/BSA
+  2. Judgment retrieval  (judgment_retrieval.py)  — SCI judgment corpus
+
+Architecture:
+
+    Query
+      |
+      v
+  Query Router  (decides: statute | judgment | hybrid)
+      |
+      +----------------+----------------+
+      |                |                |
+  Statute Path    Judgment Path     Hybrid Path
+  (direct lookup  (citation /       (BOTH retrieved
+   + semantic)     comparison /      in parallel)
+                    semantic)
+      |                |                |
+      +----------------+----------------+
+                       |
+                       v
+              Unified Cross-Encoder Rerank
+              (statute + judgment candidates
+               scored together, top-N kept)
+                       |
+                       v
+                Build Combined Context
+                       |
+                       v
+                    DeepSeek
+                       |
+                       v
+                     Answer
+
+Design notes:
+  - Both source pipelines are imported as modules, NOT rewritten.
+    This file only adds a routing + fusion layer on top.
+  - Each source's own internal logic (direct lookup, citation search,
+    comparison mode, query expansion) is preserved and called as-is.
+  - The unified reranker re-scores statute and judgment candidates
+    on the SAME scale so they can be merged fairly — this is the
+    critical step that makes "best evidence overall" meaningful.
+"""
+
+import logging
+import os
+import re
+import sys
+import time
+from dataclasses import dataclass, field
+from typing import Optional
+
+from dotenv import load_dotenv
+
+# ---------------------------------------------------------------------
+# Path setup — both retrieval pipelines live in sibling folders relative
+# to this file (hybrid rag/), not in this folder. We add both to
+# sys.path BEFORE importing them, and load the shared .env from the
+# project root so DEEPSEEK_API_KEY is available to both pipelines.
+# ---------------------------------------------------------------------
+BASE_DIR = os.path.dirname(os.path.abspath(__file__))
+PROJECT_ROOT = os.path.join(BASE_DIR, "..")
+
+load_dotenv(os.path.join(PROJECT_ROOT, ".env"))
+
+sys.path.insert(0, os.path.join(PROJECT_ROOT, "statute corpus"))
+sys.path.insert(0, PROJECT_ROOT)  # so the verifier package + llm_retriever are importable
+
+# ---------------------------------------------------------------------
+# Import both existing pipelines as modules.
+#   statute pipeline -> "statute corpus/statute_retrieval.py"
+#   judgment pipeline -> "llm_retriever.py" (project root)
+# ---------------------------------------------------------------------
+import statute_retrieval as stat_rag
+import llm_retriever as judg_rag
+
+# NOTE: verification is handled in the backend (backend/app.py) via the live
+# citation verifier (verifier/citation_verifier.py). The old grounding-based
+# verify_answer was removed; this module's CLI ask() no longer verifies.
+
+# =====================================================================
+# LOGGING
+# =====================================================================
+
+log = logging.getLogger("unified_rag")
+logging.basicConfig(
+    level=logging.INFO,
+    format="%(asctime)s  %(levelname)-8s  %(message)s",
+    datefmt="%H:%M:%S",
+    handlers=[logging.StreamHandler(sys.stdout)],
+)
+
+# =====================================================================
+# CONFIG
+# =====================================================================
+
+# How many candidates to pull from EACH source before unified rerank
+STATUTE_CANDIDATES_N  = 15
+JUDGMENT_CANDIDATES_N = 15
+
+# Final number of evidence blocks sent to DeepSeek after fusion
+FINAL_EVIDENCE_N = 8
+
+# Keep at most this many of one type if the other type is starved
+# (prevents one source from completely drowning out the other)
+MAX_SINGLE_TYPE_SHARE = 6
+
+LLM_MODEL       = "deepseek-chat"
+LLM_TEMPERATURE = 0.1
+LLM_MAX_TOKENS  = 2500
+
+# =====================================================================
+# QUERY ROUTING
+# =====================================================================
+
+# Words that signal the user wants case-law / judgment material
+JUDGMENT_SIGNAL_RE = re.compile(
+    r"\b(case|judgment|judgement|held|ruling|precedent|"
+    r"supreme court|high court|bench|justice|"
+    r"insc|scc|s\.c\.r|s\.c\.c|cited|"
+    r"per\s+incuriam|ratio|obiter|overrul)\b",
+    re.IGNORECASE,
+)
+
+# Words that signal the user wants statute / bare-act text
+STATUTE_SIGNAL_RE = re.compile(
+    r"\b(section|provision|act\b|sanhita|adhiniyam|"
+    r"punishment for|definition of|what is the law on|"
+    r"ipc|bns\b|crpc|bnss|iea|bsa)\b",
+    re.IGNORECASE,
+)
+
+
+@dataclass
+class RouteDecision:
+    use_statute:  bool
+    use_judgment: bool
+    reason:       str
+
+
+def route_query(query: str) -> RouteDecision:
+    """
+    Decide which retriever(s) to call.
+
+    Default behaviour is HYBRID — both retrievers run, because most
+    real legal questions benefit from both statute text and case law
+    interpreting it. We only skip a source when the query is clearly
+    and exclusively about the other domain.
+    """
+    has_citation = bool(judg_rag.extract_citations(query))
+    judgment_signal = bool(JUDGMENT_SIGNAL_RE.search(query))
+    statute_signal  = bool(STATUTE_SIGNAL_RE.search(query))
+    section_refs    = stat_rag.parse_section_references(query)
+
+    # Explicit citation (e.g. "2025 INSC 337") → judgment-only,
+    # this is an exact lookup, statute search would add noise
+    if has_citation and not statute_signal:
+        return RouteDecision(
+            use_statute=False, use_judgment=True,
+            reason="explicit case citation detected",
+        )
+
+    # Pure section reference with no case-law language → statute-only
+    if section_refs and not judgment_signal:
+        return RouteDecision(
+            use_statute=True, use_judgment=False,
+            reason="pure section reference, no case-law signal",
+        )
+
+    # Both signals present, or neither (ambiguous) → hybrid
+    return RouteDecision(
+        use_statute=True, use_judgment=True,
+        reason="hybrid: both or neither signal strongly present",
+    )
+
+
+# =====================================================================
+# UNIFIED CANDIDATE TYPE
+# =====================================================================
+
+@dataclass
+class Evidence:
+    """A single piece of evidence, normalised across both sources."""
+    source_type:  str             # "statute" | "judgment"
+    score:        float           # raw score from source-specific rerank
+    unified_score: float = 0.0    # score after unified rerank — set later
+    rerank_text:  str = ""        # text used for unified reranking
+    display_block: str = ""       # pre-formatted text for LLM context
+    meta:         dict = field(default_factory=dict)
+
+
+# =====================================================================
+# STATUTE PATH
+# =====================================================================
+
+def get_statute_evidence(query: str) -> list[Evidence]:
+    """
+    Runs the statute pipeline's own classification + direct lookup +
+    semantic search + its own rerank, then wraps results as Evidence.
+    Direct-lookup hits are kept separately so they always survive
+    into the unified rerank with a strong prior.
+    """
+    intent = stat_rag.classify_query(query)
+
+    direct_records = []
+    for act, sec in intent["section_refs"]:
+        rec = stat_rag.direct_lookup(act, sec)
+        if rec:
+            direct_records.append(rec)
+
+    expanded = (
+        stat_rag.expand_query(query)
+        if intent["type"] in ("semantic", "hybrid")
+        else [query]
+    )
+
+    rerank_query = query
+    if expanded and expanded[0] != query:
+        rerank_query = f"{query}. {' '.join(expanded[:2])}"
+
+    candidates = stat_rag.semantic_search(query)
+
+    existing_keys = {
+        (m.get("act_short", ""), str(m.get("section_number", "")))
+        for _, _, m in candidates
+    }
+
+    for exp_q in expanded:
+        if exp_q == query:
+            continue
+        for item in stat_rag.semantic_search(exp_q, top_k=20):
+            key = (item[2].get("act_short", ""), str(item[2].get("section_number", "")))
+            if key not in existing_keys:
+                candidates.append(item)
+                existing_keys.add(key)
+
+    if intent["acts_mentioned"]:
+        for act in intent["acts_mentioned"]:
+            for item in stat_rag.semantic_search(query, top_k=15, act_filter=act):
+                key = (item[2].get("act_short", ""), str(item[2].get("section_number", "")))
+                if key not in existing_keys:
+                    candidates.append(item)
+                    existing_keys.add(key)
+
+    if intent["is_comparative"]:
+        for act in list(intent["acts_mentioned"]):
+            equiv = stat_rag.ACT_EQUIVALENTS.get(act.upper())
+            if equiv:
+                for item in stat_rag.semantic_search(query, top_k=15, act_filter=equiv):
+                    key = (item[2].get("act_short", ""), str(item[2].get("section_number", "")))
+                    if key not in existing_keys:
+                        candidates.append(item)
+                        existing_keys.add(key)
+
+    ranked = stat_rag.rerank_results(
+        rerank_query, candidates, top_k=STATUTE_CANDIDATES_N
+    )
+
+    evidence: list[Evidence] = []
+
+    # Direct lookups — always included, marked with strong prior score
+    for rec in direct_records:
+        meta = rec["metadata"]
+        text = rec.get(
+            "retrieval_text",
+            rec.get("content_payload", {}).get("text", ""),
+        )
+        evidence.append(Evidence(
+            source_type="statute",
+            score=999.0,   # exact match — always wins ties pre-rerank
+            rerank_text=text,
+            display_block=(
+                f"--- STATUTE [DIRECT MATCH] ---\n"
+                f"Act: {meta['act_name']} ({meta['act_short']})\n"
+                f"Section: {meta['section_number']}\n"
+                f"Title: {meta.get('title','')}\n\n{text}\n"
+            ),
+            meta=meta,
+        ))
+
+    for score, doc, meta in ranked:
+        evidence.append(Evidence(
+            source_type="statute",
+            score=score,
+            rerank_text=doc,
+            display_block=(
+                f"--- STATUTE [score {score:.3f}] ---\n"
+                f"Act: {meta.get('act_name','')} ({meta.get('act_short','')})\n"
+                f"Section: {meta.get('section_number','')}\n"
+                f"Title: {meta.get('title','')}\n\n{doc}\n"
+            ),
+            meta=meta,
+        ))
+
+    log.info("Statute path: %d evidence items (%d direct)", len(evidence), len(direct_records))
+    return evidence
+
+
+# =====================================================================
+# JUDGMENT PATH
+# =====================================================================
+
+def _build_severity_context(query: str) -> str:
+    """
+    If the query references a specific statute section, fetch that
+    section's title (e.g. "Punishment for murder") and append it to
+    the judgment search query.
+
+    Why this matters: generic queries like "bail in cases under BNS 103"
+    share a lot of surface vocabulary ("bail", "BNSS 480", "section")
+    with judgments about completely different, less severe offences
+    (e.g. a tenancy-dispute cheating case). The cross-encoder reranker
+    scores surface/topical similarity, so without this enrichment a
+    legally irrelevant but vocabulary-similar judgment can outscore the
+    actually relevant one. Appending the offence title biases retrieval
+    and reranking toward judgments discussing that same severity class.
+    """
+    section_refs = stat_rag.parse_section_references(query)
+    if not section_refs:
+        return ""
+
+    titles = []
+    for act, sec in section_refs:
+        record = stat_rag.direct_lookup(act, sec)
+        if record:
+            title = record["metadata"].get("title", "")
+            if title:
+                titles.append(title)
+
+    return " ".join(titles)
+
+
+JUDGMENT_EXPANSION_PROMPT = """\
+You are an expert Indian Supreme Court legal researcher.
+Given a user's question, extract the core legal issues and rewrite it into a dense string of search terms likely to be found in Supreme Court headnotes, ratio decidendi, and legal doctrines.
+Do not write sentences. Just output a space-separated list of highly relevant legal keywords, maxims, and statutory references.
+For example, if the user asks "Can a dying declaration alone be the basis for conviction without corroboration?", output: "dying declaration corroboration sole basis of conviction evidentiary value section 32 indian evidence act"
+Output ONLY the keywords, nothing else.\
+"""
+
+def _expand_judgment_query(query: str) -> str:
+    """
+    Uses the DeepSeek LLM to translate a natural language question into
+    dense legal search terms optimized for Supreme Court judgment retrieval.
+    """
+    # Don't expand if it's just a raw citation (e.g. "2025 INSC 337")
+    if bool(judg_rag.extract_citations(query)) and len(query.split()) < 5:
+        return ""
+
+    try:
+        response = stat_rag.llm_client.chat.completions.create(
+            model=LLM_MODEL,
+            messages=[
+                {"role": "system", "content": JUDGMENT_EXPANSION_PROMPT},
+                {"role": "user", "content": query},
+            ],
+            temperature=0.0,
+            max_tokens=100,
+        )
+        expanded = response.choices[0].message.content.strip()
+
+        # Strip out any chatty prefix if the model ignored instructions
+        if ":" in expanded[:20]:
+            expanded = expanded.split(":", 1)[1].strip()
+
+        log.info("Agentic judgment expansion: '%s'", expanded)
+        return expanded
+    except Exception as e:
+        log.warning("Agentic judgment expansion failed: %s", e)
+        return ""
+
+
+def get_judgment_evidence(query: str) -> list[Evidence]:
+    """
+    Runs the judgment pipeline's own retrieve() — which internally
+    handles citation lookup, comparison mode, and hybrid semantic
+    search with dense+BM25+RRF+parent fetch — then wraps the
+    resulting parent documents as Evidence.
+
+    The query is enriched in two ways before search:
+    1. Agentic Expansion: LLM translates query to legal keywords/doctrines.
+    2. Severity Context: If a statute is referenced, its title is appended.
+    """
+    severity_context = _build_severity_context(query)
+    agent_expansion = _expand_judgment_query(query)
+
+    parts = [query]
+    if severity_context:
+        parts.append(severity_context)
+    if agent_expansion:
+        parts.append(agent_expansion)
+
+    search_query = " ".join(parts)
+
+    if len(parts) > 1:
+        log.info("Judgment query enriched to: '%s'", search_query)
+
+    result = judg_rag.retrieve(search_query)
+
+    if result is None:
+        log.info("Judgment path: no results")
+        return []
+
+    rtype = result.get("type", "")
+    evidence: list[Evidence] = []
+
+    # Single citation lookup → one parent doc, treat as a direct hit
+    if rtype in ("citation", "case"):
+        meta = result["metadata"]
+        text = result["document"]
+        rerank_text = " ".join(filter(None, [
+            meta.get("short_summary", ""),
+            meta.get("issue", ""),
+            text[:1000],
+        ]))
+        evidence.append(Evidence(
+            source_type="judgment",
+            score=999.0,  # exact citation match — always wins ties
+            rerank_text=rerank_text,
+            display_block=_format_judgment_block(meta, text, exact=True),
+            meta=meta,
+        ))
+        return evidence
+
+    # Comparison or semantic — multiple parent docs already reranked
+    # by the judgment pipeline's own rerank_parents()
+    for case in result.get("parents", []):
+        meta = case["metadata"]
+        text = case["document"]
+        rerank_text = " ".join(filter(None, [
+            meta.get("short_summary", ""),
+            meta.get("issue", ""),
+            text[:1000],
+        ]))
+        evidence.append(Evidence(
+            source_type="judgment",
+            score=0.0,  # will be re-scored by unified reranker
+            rerank_text=rerank_text,
+            display_block=_format_judgment_block(meta, text, exact=False),
+            meta=meta,
+        ))
+
+    log.info("Judgment path: %d evidence items (mode=%s)", len(evidence), rtype)
+    return evidence[:JUDGMENT_CANDIDATES_N]
+
+
+def _format_judgment_block(meta: dict, text: str, exact: bool) -> str:
+    tag = "DIRECT CITATION MATCH" if exact else "RELEVANT JUDGMENT"
+    return (
+        f"--- JUDGMENT [{tag}] ---\n"
+        f"Case: {meta.get('case_name','')}\n"
+        f"Citation: {meta.get('neutral_citation','')}\n"
+        f"Issue: {meta.get('issue','')}\n"
+        f"Summary: {meta.get('short_summary','')}\n"
+        f"Headnote: {meta.get('full_headnote','')[:1500]}\n"
+        f"Document: {text[:1500]}\n"
+    )
+
+
+# =====================================================================
+# UNIFIED RERANKING
+# =====================================================================
+
+def unified_rerank(query: str, evidence: list[Evidence]) -> list[Evidence]:
+    """
+    Re-scores ALL evidence (statute + judgment) on the same cross-encoder
+    so they are directly comparable. This is what makes "best evidence
+    overall" meaningful rather than just concatenating two top-5 lists.
+
+    Direct/exact matches (score == 999.0) are pulled out, reranked
+    separately to preserve their relative order, then placed first.
+    Everything else is reranked together and capped by source-share
+    limits so neither source can completely starve the other.
+    """
+    if not evidence:
+        return []
+
+    exact_hits = [e for e in evidence if e.score == 999.0]
+    semantic_ev = [e for e in evidence if e.score != 999.0]
+
+    # Rerank exact hits against each other (rare to have many, but
+    # if user asks about 2 sections + cites a case, order matters)
+    if len(exact_hits) > 1:
+        pairs = [(query, e.rerank_text) for e in exact_hits]
+        scores = stat_rag.reranker.predict(pairs)
+        for e, s in zip(exact_hits, scores):
+            e.unified_score = 1000.0 + float(s)  # keep them above all semantic
+        exact_hits.sort(key=lambda e: e.unified_score, reverse=True)
+    else:
+        for e in exact_hits:
+            e.unified_score = 1000.0
+
+    # Rerank everything else together
+    if semantic_ev:
+        pairs = [(query, e.rerank_text) for e in semantic_ev]
+        scores = stat_rag.reranker.predict(pairs)
+        for e, s in zip(semantic_ev, scores):
+            e.unified_score = float(s)
+        semantic_ev.sort(key=lambda e: e.unified_score, reverse=True)
+
+    merged = exact_hits + semantic_ev
+
+    # Enforce source-share cap so one source can't drown out the other
+    # in the final cut, while still respecting overall rank order
+    final: list[Evidence] = []
+    type_counts = {"statute": 0, "judgment": 0}
+    seen_keys: set = set()
+
+    for e in merged:
+        if len(final) >= FINAL_EVIDENCE_N:
+            break
+        # Dedupe: same statute section (or same case citation) can arrive as both
+        # a direct/exact hit and a semantic hit — keep only the first (best) one.
+        if e.source_type == "statute":
+            dkey = ("s", str(e.meta.get("act_short", "")).upper(), str(e.meta.get("section_number", "")))
+        else:
+            dkey = ("j", str(e.meta.get("neutral_citation", "")).upper())
+        if dkey in seen_keys:
+            continue
+        if type_counts[e.source_type] >= MAX_SINGLE_TYPE_SHARE:
+            continue
+        seen_keys.add(dkey)
+        final.append(e)
+        type_counts[e.source_type] += 1
+
+    log.info(
+        "Unified rerank: %d total -> %d final (statute=%d, judgment=%d)",
+        len(merged), len(final), type_counts["statute"], type_counts["judgment"],
+    )
+
+    for i, e in enumerate(final, 1):
+        log.info(
+            "  #%d [%s] score=%.3f  %s",
+            i, e.source_type, e.unified_score,
+            (e.meta.get("title") or e.meta.get("case_name") or "")[:50],
+        )
+
+    return final
+
+
+# =====================================================================
+# CONTEXT ASSEMBLY
+# =====================================================================
+
+def build_unified_context(evidence: list[Evidence]) -> str:
+    return "\n".join(e.display_block for e in evidence)
+
+
+# =====================================================================
+# LLM GENERATION
+# =====================================================================
+
+PROMPTS = {
+    "LEGAL_RESEARCH": """\
+You are an expert Indian Legal Assistant with deep knowledge of both \
+statutory law (IPC, BNS, CrPC, BNSS, IEA, BSA) and Supreme Court \
+jurisprudence interpreting that law.
+
+EVIDENCE-GROUNDING RULES — THESE OVERRIDE YOUR GENERAL LEGAL KNOWLEDGE:
+
+1. Every factual or legal claim you make must be traceable to a specific \
+block in the EVIDENCE section below. Before writing any sentence that \
+states what a court held, what a statute requires, or what factors a \
+court considered, locate the exact evidence block that supports it.
+
+2. You have broad general knowledge of Indian law from training. \
+DO NOT use that general knowledge to fill gaps in the evidence, even if \
+you are confident it is legally correct. If the evidence does not contain \
+a point, omit the point — do not add it from memory. This applies even to \
+well-known legal maxims (e.g. "bail is the rule, jail is the exception") \
+unless that maxim is explicitly discussed in the evidence provided.
+
+3. If you are tempted to write a claim and cannot point to which evidence \
+block supports it, do not write it. Stop and either omit it or explicitly \
+flag it as your own general knowledge using the marker \
+"[Note: general principle, not found in retrieved evidence]" — use this \
+marker sparingly and only when the point is necessary for a complete answer.
+
+4. Do not blend evidence from a judgment with evidence from a statute (or \
+vice versa) into a single unmarked sentence that implies one source said \
+both things.
+
+5. Only attribute a holding to a case if the holding text actually \
+appears in that judgment's evidence block. Only cite a statute section \
+if it appears in a statute evidence block.
+
+6. If multiple evidence blocks are provided but only some are relevant to \
+the question, silently drop the irrelevant ones — do not force them into \
+the table or the answer.
+
+7. If the evidence is insufficient to answer part of the question, say so \
+explicitly rather than guessing.
+
+OUTPUT FORMAT — THIS IS FOR DIRECT PRESENTATION, USE EXACTLY THIS SHAPE:
+
+8. Start with a 1-2 sentence direct answer to the question. No heading.
+
+9. Then give ONE markdown table, with these columns: Citation | Parties | \
+Section/Issue | Holding.
+   - Citation: case name + neutral citation (e.g. "DDA v. Corporation \
+Bank, 2025 INSC 1161"), or statute + section (e.g. "BNSS Sec. 480").
+   - Parties: who the dispute was between, in 3-5 words.
+   - Section/Issue: the specific statutory provision or legal question \
+at stake, in one short phrase.
+   - Holding: what the court/statute actually decided or requires, in \
+1 sentence, plain language.
+   Include one row per relevant case or statute provision in the \
+evidence. Do not repeat the same case across multiple rows.
+
+10. After the table, add at most 2-3 sentences of synthesis ONLY if \
+needed to connect the rows to the user's specific question (e.g. how \
+the precedent maps onto their fact pattern). If the table is self- \
+explanatory, skip this entirely.
+
+11. Do NOT use any other headings (no #, ##, ###), no "Summary" or \
+"Conclusion" section, and no prose paragraphs walking through each case \
+one by one outside the table. The table IS the structure — don't \
+duplicate its content in prose above or below it.
+
+12. If a legally material qualification doesn't fit in one table cell \
+(e.g. a key exception), add it as a single short sentence after the \
+table, not a new row or section.\
+""",
+
+    "CASE_SUMMARY": """\
+You are an expert Indian Legal Assistant. The user has asked for a summary of a specific case or statute.
+
+EVIDENCE-GROUNDING RULES:
+1. You must ONLY use the provided EVIDENCE to generate the summary. Do not invent facts or holdings.
+2. If the case/statute requested is not in the EVIDENCE, state that you cannot provide a summary based on the retrieved documents.
+
+OUTPUT FORMAT:
+Provide a structured summary in Markdown using the following headings (omit any that are irrelevant or lack evidence):
+**Facts**: Brief background of the case.
+**Issues**: The core legal questions the court had to decide.
+**Reasoning**: The court's rationale and legal analysis.
+**Holding / Rule of Law**: The final decision or the legal principle established.
+
+Do NOT include a citation table.\
+""",
+
+    "CASE_COMPARISON": """\
+You are an expert Indian Legal Assistant. The user has asked to compare multiple cases, statutes, or legal concepts.
+
+EVIDENCE-GROUNDING RULES:
+1. Every comparison point must be traceable to the EVIDENCE section below.
+2. Do not hallucinate differences or similarities. Only use the retrieved text.
+3. If the evidence does not provide enough information for a fair comparison, state the limitations explicitly.
+
+OUTPUT FORMAT:
+1. Start with a brief 1-2 sentence overview of the comparison.
+2. Provide a Markdown table contrasting the entities. The columns should represent the entities (e.g., 'Section 302 IPC' | 'Section 103 BNS', or 'Case A' | 'Case B') and the rows should be the points of comparison (e.g., 'Punishment', 'Definition', 'Core Issue').
+3. After the table, write a brief synthesis explaining the key differences or similarities based on the table.
+Do NOT use the standard citation table format.\
+""",
+
+    "COMPREHENSIVE_CASE_STUDY": """\
+You are an expert Indian Legal Scholar. The user has requested a comprehensive, in-depth case study or detailed legal research report on a specific case or topic.
+
+EVIDENCE-GROUNDING RULES:
+1. You must ONLY use the provided EVIDENCE. Do not invent arguments, facts, or rulings.
+2. If the evidence lacks certain details (like specific arguments of the appellant), omit that section rather than hallucinating.
+
+OUTPUT FORMAT:
+Provide an extensive, highly detailed academic case study in Markdown using the following headings:
+**1. Background & Context**: The factual matrix and history leading up to the Supreme Court.
+**2. Core Legal Issues**: A detailed breakdown of the exact questions of law the court had to decide.
+**3. Arguments Advanced**: What the Appellant and Respondent argued (if available in evidence).
+**4. Precedents & Statutes Relied Upon**: Key laws and past cases cited by the court.
+**5. Court's Analysis & Rationale**: An in-depth explanation of the court's reasoning, logical steps, and interpretation of the law.
+**6. Final Judgment & Holding**: The final verdict and rule of law established.
+**7. Legal Implications**: How this ruling impacts the broader legal landscape based on the court's dicta.
+
+Ensure the response is thorough, analytical, and highly detailed. Do NOT use a standard citation table.\
+""",
+
+    "STORY_EVALUATION": """\
+You are an expert Indian Legal Advisor. The user has provided a personal narrative or factual scenario. Your job is to analyze their situation using ONLY the provided EVIDENCE.
+
+EVIDENCE-GROUNDING RULES:
+1. You must base your legal analysis strictly on the retrieved EVIDENCE. 
+2. Do not invent applicable sections or case laws. If the evidence doesn't support a claim, do not make it.
+
+OUTPUT FORMAT:
+Provide a structured legal analysis in Markdown using the following headings:
+**1. Summary of Facts**: A very brief (1-2 sentences) summary of the user's situation.
+**2. Applicable Legal Provisions**: Identify the specific statutes and sections (e.g., from IPC, BNS, etc.) that could be framed based on the story. Explain exactly WHY they apply using the evidence.
+**3. Relevant Case Law**: Cite relevant Supreme Court judgments from the evidence that have similar fact patterns or deal with the same legal issues. Explain how the courts ruled in those similar situations.
+**4. Legal Assessment & Potential Outcomes**: Provide a practical legal assessment of the situation based strictly on the retrieved law and precedents.
+
+Do NOT include the standard citation table.\
+"""
+}
+
+
+def classify_intent(query: str) -> str:
+    """Classify the user's query into one of five intents."""
+    prompt = """
+    Classify the user's legal query into EXACTLY ONE intent.
+    DEFAULT to LEGAL_RESEARCH unless the query CLEARLY matches a more specific intent below.
+
+    - LEGAL_RESEARCH: the default for almost everything — any general legal question, what the
+      law says, what courts have held on a topic, applicable provisions, punishments,
+      definitions, conditions, procedure, etc.
+    - CASE_SUMMARY: ONLY when the user explicitly asks to summarize ONE specific named case or
+      neutral citation (e.g. "summarize 2025 INSC 337", "give me a summary of X v. Y").
+    - COMPREHENSIVE_CASE_STUDY: ONLY when the user explicitly asks for an in-depth/detailed case
+      study, report, or comprehensive analysis of a specific case.
+    - CASE_COMPARISON: ONLY when the user asks to compare/contrast two or more cases, statutes,
+      or doctrines.
+    - STORY_EVALUATION: ONLY when the user narrates a personal or hypothetical factual scenario
+      and asks what law applies to it.
+
+    Output ONLY the exact intent name. No other text.
+    """
+    try:
+        response = stat_rag.llm_client.chat.completions.create(
+            model=LLM_MODEL,
+            messages=[
+                {"role": "system", "content": prompt},
+                {"role": "user", "content": query}
+            ],
+            temperature=0.0,
+            max_tokens=10
+        )
+        intent = response.choices[0].message.content.strip().upper()
+        if intent in ["CASE_SUMMARY", "CASE_COMPARISON", "COMPREHENSIVE_CASE_STUDY", "STORY_EVALUATION"]:
+            return intent
+        return "LEGAL_RESEARCH"
+    except Exception as e:
+        log.warning("Intent classification failed: %s. Defaulting to LEGAL_RESEARCH.", e)
+        return "LEGAL_RESEARCH"
+
+
+def generate_answer(
+    query: str,
+    context: str,
+    intent: str,
+    conversation_history: Optional[list[dict]] = None,
+) -> str:
+    system_prompt = PROMPTS.get(intent, PROMPTS["LEGAL_RESEARCH"])
+    messages = [{"role": "system", "content": system_prompt}]
+
+    if conversation_history:
+        messages.extend(conversation_history[-8:])
+
+    messages.append({
+        "role": "user",
+        "content": f"QUESTION:\n{query}\n\nEVIDENCE:\n{context}",
+    })
+
+    try:
+        t0 = time.time()
+        response = stat_rag.llm_client.chat.completions.create(
+            model=LLM_MODEL,
+            messages=messages,
+            temperature=LLM_TEMPERATURE,
+            max_tokens=LLM_MAX_TOKENS,
+            stream=True,
+        )
+
+        full_response = ""
+        first_chunk = True
+        for chunk in response:
+            delta = chunk.choices[0].delta
+            if delta.content:
+                if first_chunk:
+                    log.info("First token in %.2fs", time.time() - t0)
+                    first_chunk = False
+                print(delta.content, end="", flush=True)
+                full_response += delta.content
+        print()
+
+        log.info("Generation complete in %.2fs", time.time() - t0)
+        return full_response
+
+    except Exception as e:
+        log.error("DeepSeek API error: %s", e)
+        return f"Error generating answer: {e}"
+
+
+# =====================================================================
+# MAIN UNIFIED PIPELINE
+# =====================================================================
+
+def ask(query: str, conversation_history: Optional[list[dict]] = None, force_intent: str = "AUTO") -> dict:
+    """
+    Full unified pipeline:
+      1. Route query (statute / judgment / hybrid)
+      2. Retrieve from chosen source(s) in their native pipelines
+      3. Normalise into Evidence objects
+      4. Unified rerank across both sources
+      5. Build combined context
+      6. Generate answer via DeepSeek
+
+    Returns a dict with routing decision, evidence summary, and answer.
+    """
+    t_start = time.time()
+
+    decision = route_query(query)
+    log.info("Route: statute=%s judgment=%s (%s)",
+             decision.use_statute, decision.use_judgment, decision.reason)
+
+    all_evidence: list[Evidence] = []
+
+    if decision.use_statute:
+        all_evidence.extend(get_statute_evidence(query))
+
+    if decision.use_judgment:
+        all_evidence.extend(get_judgment_evidence(query))
+
+    if not all_evidence:
+        # Before giving up, check if the user asked for a specific citation
+        # that exists on Bharat Courts but isn't in our local database.
+        explicit_citations = judg_rag.extract_citations(query)
+        if explicit_citations:
+            log.info("No local evidence found for %s (live lookup handled in backend).", explicit_citations)
+            live_res = {}
+
+            for cit in explicit_citations:
+                res = live_res.get(cit, {})
+                if res.get("verified") is True:
+                    title = res.get("matched_case_name") or "Unknown Case"
+                    all_evidence.append(Evidence(
+                        source_type="judgment",
+                        score=999.0,
+                        unified_score=999.0,
+                        rerank_text="",
+                        display_block=(
+                            f"--- LIVE WEB SEARCH ---\n"
+                            f"Citation: {cit}\n"
+                            f"Title: {title}\n"
+                            f"Note: This case exists on the Supreme Court website, but its full "
+                            f"text is NOT downloaded in the local database. You cannot provide "
+                            f"a deep legal analysis, but you MUST confirm to the user that it exists online."
+                        ),
+                        meta={"neutral_citation": cit, "case_name": title}
+                    ))
+                    log.info("Found %s on Bharat Courts. Injected as web-search evidence.", cit)
+
+        if not all_evidence:
+            return {
+                "query": query,
+                "route": decision.reason,
+                "evidence": [],
+                "answer": "No relevant statutes or judgments found for this query.",
+                "elapsed_seconds": round(time.time() - t_start, 2),
+            }
+
+    # Build the same severity-enriched query used inside get_judgment_evidence
+    # and reuse it here. Without this, judg_rag.retrieve() correctly ranks
+    # Darshan above Shuvendu Saha internally, but unified_rerank() would
+    # rescore everything from the BARE query and undo that improvement —
+    # the unified reranker has no other way to know "BNS 103" means murder.
+    severity_context = _build_severity_context(query)
+    unified_query = f"{query} {severity_context}" if severity_context else query
+    if severity_context:
+        log.info("Unified rerank query enriched with: '%s'", severity_context)
+
+    final_evidence = unified_rerank(unified_query, all_evidence)
+    context = build_unified_context(final_evidence)
+
+    if force_intent == "AUTO":
+        intent = classify_intent(query)
+        log.info("Detected Intent: %s", intent)
+    else:
+        intent = force_intent
+        log.info("Forced Intent by User: %s", intent)
+
+    print()
+    print("=" * 70)
+    print(f"  UNIFIED LEGAL ANSWER ({intent})")
+    print("=" * 70)
+    print()
+
+    answer = generate_answer(query, context, intent, conversation_history)
+
+    # Verification is performed in the backend (live citation verifier), not here.
+    verification = {}
+
+    elapsed = time.time() - t_start
+
+    return {
+        "query": query,
+        "route": decision.reason,
+        "intent": intent,
+        "evidence": [
+            {
+                "type":  e.source_type,
+                "score": round(e.unified_score, 3),
+                "label": e.meta.get("title") or e.meta.get("case_name") or "",
+            }
+            for e in final_evidence
+        ],
+        "answer": answer,
+        "verification": verification,
+        "elapsed_seconds": round(elapsed, 2),
+    }
+
+
+# =====================================================================
+# CLI
+# =====================================================================
+
+def display_banner():
+    print()
+    print("=" * 60)
+    print("  LegalAIapex — Unified Statute + Judgment RAG")
+    print("=" * 60)
+    print("  Type a legal question, or 'exit' to quit.")
+    print("=" * 60)
+    print()
+
+
+def main():
+    display_banner()
+    conversation_history: list[dict] = []
+
+    while True:
+        try:
+            query = input("Ask a legal question: ").strip()
+        except (EOFError, KeyboardInterrupt):
+            print("\nGoodbye.")
+            break
+
+        if not query:
+            continue
+        if query.lower() in ("exit", "quit", "q"):
+            print("Goodbye.")
+            break
+        if query.lower() == "clear":
+            conversation_history.clear()
+            print("Conversation history cleared.")
+            continue
+
+        result = ask(query, conversation_history)
+
+        conversation_history.append({"role": "user", "content": query})
+        conversation_history.append({"role": "assistant", "content": result["answer"]})
+
+        print()
+        print("-" * 60)
+        print(f"Route: {result['route']}")
+        print("Evidence used:")
+        for ev in result["evidence"]:
+            print(f"  [{ev['type']:8s}] score={ev['score']:+.3f}  {ev['label'][:50]}")
+
+        # --- Verification summary ---
+        v = result.get("verification", {})
+        if v:
+            grounded = v.get("grounded", None)
+            status = "✅ FULLY GROUNDED" if grounded else "⚠️  UNGROUNDED CITATIONS DETECTED"
+            print(f"\nVerification: {status}")
+            if v.get("flagged_sections"):
+                print(f"  Unverified statute sections:  {', '.join(v['flagged_sections'])}")
+            if v.get("citations_confirmed_via_live_lookup"):
+                print(f"  Citations confirmed (live):   {', '.join(v['citations_confirmed_via_live_lookup'])}")
+            if v.get("citations_likely_fabricated"):
+                print(f"  ❌ Likely fabricated:          {', '.join(v['citations_likely_fabricated'])}")
+            if v.get("citations_could_not_verify"):
+                print(f"  ❓ Could not verify:           {', '.join(v['citations_could_not_verify'])}")
+
+        print(f"Time: {result['elapsed_seconds']}s")
+        print("-" * 60)
+
+
+if __name__ == "__main__":
+    main()

llm_retriever.py

@@ -0,0 +1,645 @@
+import os
+import re
+import chromadb
+from sentence_transformers import SentenceTransformer, CrossEncoder
+from openai import OpenAI
+from rank_bm25 import BM25Okapi
+
+# =====================================================
+# CONFIG
+# =====================================================
+
+DEEPSEEK_API_KEY = os.getenv("DEEPSEEK_API_KEY", "")
+
+# Path to the SCI judgments Chroma index. In the container this is set to the
+# location the dataset is downloaded to (see Dockerfile); locally it can point
+# at a copy on disk. No more hardcoded Windows path.
+CHROMA_PATH = os.getenv(
+    "JUDGMENTS_CHROMA_PATH",
+    os.path.join(os.path.dirname(os.path.abspath(__file__)), "chroma_bge_v2"),
+)
+COLLECTION_NAME = "sci_judgments_bge_v2"
+
+# Retrieval tuning
+DENSE_TOP_K       = 150   # dense candidates before fusion
+BM25_TOP_K        = 150   # BM25 candidates before fusion
+DEDUP_MAX_PER_CASE = 3    # max child chunks per case after dedup
+RERANK_TOP_N      = 30    # top-N children to rerank after fusion
+PARENT_FETCH_N    = 20    # how many parent docs to fetch
+PARENT_FINAL_N    = 5     # how many parents to pass to LLM
+RRF_K             = 60    # RRF constant (standard value)
+
+# =====================================================
+# LOAD CHROMA
+# =====================================================
+
+print("Connecting to ChromaDB...")
+
+db_client = chromadb.PersistentClient(path=CHROMA_PATH)
+collection = db_client.get_collection(COLLECTION_NAME)
+
+print(f"Collection loaded: {collection.count()} chunks")
+
+# =====================================================
+# BUILD BM25 INDEX
+# =====================================================
+
+print("Loading full corpus for BM25...")
+
+all_docs = collection.get(include=["documents", "metadatas"])
+# Note: IDs are always returned automatically by ChromaDB — no need to include them
+
+bm25_corpus = [doc.lower().split() for doc in all_docs["documents"]]
+bm25_index  = BM25Okapi(bm25_corpus)
+
+print(f"BM25 indexed {len(bm25_corpus)} chunks")
+
+# =====================================================
+# LOAD EMBEDDING MODEL
+# =====================================================
+
+print("Loading embedding model...")
+embedder = SentenceTransformer("BAAI/bge-small-en-v1.5")
+
+# =====================================================
+# LOAD RERANKER
+# =====================================================
+
+print("Loading reranker...")
+reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
+# TODO: swap to Cohere Rerank v3 before production
+
+# =====================================================
+# LOAD DEEPSEEK
+# =====================================================
+
+llm = OpenAI(
+    api_key=DEEPSEEK_API_KEY,
+    base_url="https://api.deepseek.com"
+)
+
+print("Pipeline ready.\n")
+
+# =====================================================
+# QUERY EXPANSION
+# =====================================================
+
+EXPANSION_MAP = {
+    "settlement": """
+        compromise amicable settlement lok adalat
+        one time settlement quashing after settlement
+        compounding of offence
+    """,
+    "quash": """
+        section 482 crpc abuse of process inherent powers
+        criminal proceedings high court quashing
+    """,
+    "bail": """
+        anticipatory bail regular bail section 439 crpc
+        bail conditions personal liberty article 21
+    """,
+    "lease": """
+        agreement to lease lease deed lessee lessor
+        leasehold rights transfer of property act
+        nazul land unearned income
+    """,
+    "auction": """
+        liquidation proceedings winding up company court
+        official liquidator auction sale confirmed
+        as is where is basis
+    """,
+    "arbitration": """
+        section 11 arbitration conciliation act
+        appointment of arbitrator section 34 award
+        enforcement challenge
+    """,
+    "contempt": """
+        contempt of court wilful disobedience
+        section 2 contempt of courts act
+        civil contempt criminal contempt
+    """,
+}
+
+def expand_query(query: str) -> str:
+    q_lower = query.lower()
+    expansion = query
+    for keyword, extra_terms in EXPANSION_MAP.items():
+        if keyword in q_lower:
+            expansion += " " + extra_terms.strip()
+    return expansion
+
+# =====================================================
+# QUERY TYPE DETECTION
+# =====================================================
+
+COMPARISON_KEYWORDS = [
+    "compare", "difference", "distinguish",
+    "contrast", "similarity", "similarities",
+    "common principle", "approach", "versus", "vs"
+]
+
+def is_comparison_query(query: str) -> bool:
+    q = query.lower()
+    return any(k in q for k in COMPARISON_KEYWORDS)
+
+def extract_citations(text: str) -> list[str]:
+    """Extract all INSC citations from a string."""
+    return list(dict.fromkeys(
+        re.findall(r"\d{4}\s+INSC\s+\d+", text.upper())
+    ))
+
+# =====================================================
+# DENSE SEARCH
+# =====================================================
+
+def dense_search(query: str, top_k: int = DENSE_TOP_K) -> dict:
+    embedding = embedder.encode(
+        query,
+        normalize_embeddings=True
+    ).tolist()
+
+    results = collection.query(
+        query_embeddings=[embedding],
+        n_results=top_k,
+        include=["documents", "metadatas", "distances"]
+    )
+
+    print(f"\nDense search returned {len(results['documents'][0])} chunks")
+    return results
+
+# =====================================================
+# BM25 SEARCH
+# =====================================================
+
+def bm25_search(query: str, top_k: int = BM25_TOP_K) -> list[tuple[int, float]]:
+    scores = bm25_index.get_scores(query.lower().split())
+    ranked = sorted(enumerate(scores), key=lambda x: x[1], reverse=True)
+    nonzero = [(idx, score) for idx, score in ranked[:top_k] if score > 0]
+    print(f"BM25 search returned {len(nonzero)} non-zero chunks")
+    return nonzero
+
+# =====================================================
+# RRF FUSION
+# =====================================================
+
+def rrf_fusion(
+    dense_results: dict,
+    bm25_ranked: list[tuple[int, float]],
+    k: int = RRF_K
+) -> dict:
+    """
+    Reciprocal Rank Fusion of dense vector results and BM25 results.
+    Returns a unified results dict sorted by fused RRF score.
+    """
+    scores: dict[str, dict] = {}
+
+    # --- Dense contribution ---
+    for rank, (doc, meta, dist) in enumerate(zip(
+        dense_results["documents"][0],
+        dense_results["metadatas"][0],
+        dense_results["distances"][0]
+    )):
+        # Use first 80 chars of doc as part of key to handle same-citation chunks
+        chunk_key = meta.get("neutral_citation", "") + "|" + doc[:80]
+        if chunk_key not in scores:
+            scores[chunk_key] = {
+                "doc": doc, "meta": meta, "dist": dist, "score": 0.0
+            }
+        scores[chunk_key]["score"] += 1.0 / (k + rank + 1)
+
+    # --- BM25 contribution ---
+    for rank, (idx, bm25_score) in enumerate(bm25_ranked):
+        doc  = all_docs["documents"][idx]
+        meta = all_docs["metadatas"][idx]
+        chunk_key = meta.get("neutral_citation", "") + "|" + doc[:80]
+        if chunk_key not in scores:
+            scores[chunk_key] = {
+                "doc": doc, "meta": meta, "dist": 0.0, "score": 0.0
+            }
+        scores[chunk_key]["score"] += 1.0 / (k + rank + 1)
+
+    # Sort by fused score descending
+    sorted_chunks = sorted(
+        scores.values(), key=lambda x: x["score"], reverse=True
+    )
+
+    print(f"RRF fusion produced {len(sorted_chunks)} unique chunks")
+
+    return {
+        "documents": [[c["doc"]  for c in sorted_chunks]],
+        "metadatas": [[c["meta"] for c in sorted_chunks]],
+        "distances": [[c["dist"] for c in sorted_chunks]]
+    }
+
+# =====================================================
+# DEDUPLICATION  (max N child chunks per case)
+# =====================================================
+
+def deduplicate_children(results: dict, max_per_case: int = DEDUP_MAX_PER_CASE) -> dict:
+    seen: dict[str, int] = {}
+    docs, metas, dists = [], [], []
+
+    for doc, meta, dist in zip(
+        results["documents"][0],
+        results["metadatas"][0],
+        results["distances"][0]
+    ):
+        citation = meta.get("neutral_citation", "")
+        count = seen.get(citation, 0)
+        if count >= max_per_case:
+            continue
+        seen[citation] = count + 1
+        docs.append(doc)
+        metas.append(meta)
+        dists.append(dist)
+
+    print(f"After dedup: {len(docs)} child chunks across {len(seen)} cases")
+    return {"documents": [docs], "metadatas": [metas], "distances": [dists]}
+
+# =====================================================
+# CHILD RERANKING
+# =====================================================
+
+def rerank_children(query: str, results: dict, top_n: int = RERANK_TOP_N) -> dict:
+    docs  = results["documents"][0]
+    metas = results["metadatas"][0]
+    dists = results["distances"][0]
+
+    if not docs:
+        return results
+
+    pairs  = [(query, doc) for doc in docs]
+    scores = reranker.predict(pairs)
+
+    combined = sorted(
+        zip(scores, docs, metas, dists),
+        key=lambda x: x[0],
+        reverse=True
+    )[:top_n]
+
+    print(f"Child reranking kept top {len(combined)} chunks")
+
+    return {
+        "documents": [[x[1] for x in combined]],
+        "metadatas": [[x[2] for x in combined]],
+        "distances": [[x[3] for x in combined]]
+    }
+
+# =====================================================
+# CASE RANKING FROM CHILDREN
+# =====================================================
+
+def rank_cases_from_children(child_results: dict) -> list[str]:
+    """
+    Score each case by the sum of positional scores of its children.
+    Higher-ranked children contribute more to their case's score.
+    """
+    case_scores: dict[str, float] = {}
+
+    for rank, meta in enumerate(child_results["metadatas"][0]):
+        citation = meta.get("neutral_citation", "")
+        if not citation:
+            continue
+        # Positional score: rank 0 = highest
+        case_scores[citation] = (
+            case_scores.get(citation, 0.0) + 1.0 / (rank + 1)
+        )
+
+    ranked = sorted(case_scores.items(), key=lambda x: x[1], reverse=True)
+    return [citation for citation, _ in ranked]
+
+# =====================================================
+# CITATION SEARCH  (exact case lookup)
+# =====================================================
+
+def citation_search(citation: str) -> dict | None:
+    results = collection.get(where={"neutral_citation": citation})
+
+    for i, chunk_id in enumerate(results["ids"]):
+        if chunk_id.endswith("__parent"):
+            return {
+                "type":     "citation",
+                "id":       results["ids"][i],
+                "document": results["documents"][i],
+                "metadata": results["metadatas"][i]
+            }
+
+    # Fallback: return first chunk if no __parent marker
+    if results["ids"]:
+        return {
+            "type":     "citation",
+            "id":       results["ids"][0],
+            "document": results["documents"][0],
+            "metadata": results["metadatas"][0]
+        }
+
+    return None
+
+# =====================================================
+# PARENT FETCH
+# =====================================================
+
+def fetch_parent_chunks(citations: list[str]) -> list[dict]:
+    parents = []
+
+    for citation in citations:
+        data = collection.get(where={"neutral_citation": citation})
+
+        parent_found = False
+        for i, chunk_id in enumerate(data["ids"]):
+            if chunk_id.endswith("__parent"):
+                parents.append({
+                    "id":       data["ids"][i],
+                    "document": data["documents"][i],
+                    "metadata": data["metadatas"][i]
+                })
+                parent_found = True
+                break
+
+        # Fallback: use first chunk if no __parent
+        if not parent_found and data["ids"]:
+            parents.append({
+                "id":       data["ids"][0],
+                "document": data["documents"][0],
+                "metadata": data["metadatas"][0]
+            })
+
+    print(f"Fetched {len(parents)} parent documents")
+    return parents
+
+# =====================================================
+# PARENT RERANKING
+# =====================================================
+
+def rerank_parents(query: str, parents: list[dict], top_n: int = PARENT_FINAL_N) -> list[dict]:
+    if not parents:
+        return parents
+
+    # Rerank on headnote + summary + document text combined
+    texts = []
+    for p in parents:
+        meta = p["metadata"]
+        combined_text = " ".join(filter(None, [
+            meta.get("full_headnote", "")[:2000],
+            meta.get("short_summary", ""),
+            meta.get("issue", ""),
+            p["document"][:1000]
+        ]))
+        texts.append(combined_text)
+
+    pairs  = [(query, t) for t in texts]
+    scores = reranker.predict(pairs)
+
+    ranked = sorted(
+        zip(scores, parents),
+        key=lambda x: x[0],
+        reverse=True
+    )[:top_n]
+
+    print(f"Parent reranking kept top {len(ranked)} parents")
+    return [p for _, p in ranked]
+
+# =====================================================
+# MAIN RETRIEVER
+# =====================================================
+
+def retrieve(query: str) -> dict | None:
+    query_expanded = expand_query(query)
+    citations      = extract_citations(query)
+
+    # --------------------------------------------------
+    # MODE 1: Comparison — two or more explicit citations
+    # --------------------------------------------------
+    if len(citations) >= 2 and is_comparison_query(query):
+        print("\nMODE: COMPARISON SEARCH")
+        parents = [r for c in citations if (r := citation_search(c))]
+        return {"type": "comparison", "parents": parents}
+
+    # --------------------------------------------------
+    # MODE 2: Single explicit citation lookup
+    # --------------------------------------------------
+    if len(citations) == 1:
+        print("\nMODE: CITATION SEARCH")
+        return citation_search(citations[0])
+
+    # --------------------------------------------------
+    # MODE 3: Hybrid semantic search
+    # --------------------------------------------------
+    print("\nMODE: HYBRID SEARCH  (dense + BM25 + RRF + rerank)")
+
+    # Step 1: Dense + BM25 retrieval
+    dense_results = dense_search(query_expanded, top_k=DENSE_TOP_K)
+    bm25_ranked   = bm25_search(query_expanded,  top_k=BM25_TOP_K)
+
+    # Step 2: RRF fusion
+    fused = rrf_fusion(dense_results, bm25_ranked)
+
+    # Step 3: Deduplicate (max 3 child chunks per case)
+    fused = deduplicate_children(fused)
+
+    # Step 4: Rerank fused children
+    fused = rerank_children(query, fused, top_n=RERANK_TOP_N)
+
+    # Step 5: Log top-10 children after reranking
+    print("\nTOP 10 CHILDREN AFTER RERANKING:")
+    for i, meta in enumerate(fused["metadatas"][0][:10]):
+        print(f"  {i+1}. {meta.get('neutral_citation','')} | {meta.get('case_name','')}")
+
+    # Step 6: Rank cases by child scores
+    ranked_citations = rank_cases_from_children(fused)
+
+    print("\nCASE SCORES (top 10):")
+    for c in ranked_citations[:10]:
+        print(f"  {c}")
+
+    # Step 7: Fetch parent documents
+    parents = fetch_parent_chunks(ranked_citations[:PARENT_FETCH_N])
+    if not parents:
+        print("No parent documents found.")
+        return None
+
+    # Step 8: Rerank parents directly against original query
+    parents = rerank_parents(query, parents, top_n=PARENT_FINAL_N)
+
+    print("\nFINAL RETRIEVED CASES:")
+    for p in parents:
+        meta = p["metadata"]
+        print(f"  {meta.get('neutral_citation','')} | {meta.get('case_name','')}")
+
+    return {"type": "semantic", "parents": parents}
+
+# =====================================================
+# CONTEXT BUILDER
+# =====================================================
+
+def build_context(result: dict) -> str:
+    if result is None:
+        return ""
+
+    rtype = result.get("type", "")
+
+    # Single citation lookup
+    if rtype in ("citation", "case"):
+        meta = result["metadata"]
+        return f"""
+CASE NAME:    {meta.get('case_name', '')}
+CITATION:     {meta.get('neutral_citation', '')}
+ISSUE:        {meta.get('issue', '')}
+SUMMARY:      {meta.get('short_summary', '')}
+DOCUMENT:
+{result['document']}
+"""
+
+    # Comparison or semantic — multiple parents
+    if rtype in ("comparison", "semantic"):
+        blocks = []
+        for case in result["parents"]:
+            meta = case["metadata"]
+            blocks.append(f"""
+{'='*60}
+CASE NAME:  {meta.get('case_name', '')}
+CITATION:   {meta.get('neutral_citation', '')}
+ISSUE:      {meta.get('issue', '')}
+SUMMARY:    {meta.get('short_summary', '')}
+HEADNOTE:
+{meta.get('full_headnote', '')[:5000]}
+DOCUMENT:
+{case['document'][:3000]}
+{'='*60}
+""")
+        return "\n".join(blocks)
+
+    return ""
+
+# =====================================================
+# LLM ANSWER GENERATION
+# =====================================================
+
+SYSTEM_PROMPT = """You are an expert Indian legal research assistant with deep knowledge
+of Supreme Court jurisprudence. Answer questions accurately using only the supplied
+legal material. Always cite the case name and neutral citation."""
+
+RESEARCH_PROMPT = """
+You are an expert Indian legal research assistant.
+
+Answer the USER QUESTION using ONLY the supplied legal material.
+
+Rules:
+1. Answer the question directly and precisely.
+2. State the legal principle clearly.
+3. Cite the relevant case name and neutral citation (e.g. 2025 INSC 337).
+4. Reference specific paragraph numbers where relevant.
+5. Do not summarize the entire judgment — focus on the user's question.
+6. Do not invent facts, principles, or citations.
+7. If the answer is not in the supplied material, say so clearly.
+
+LEGAL MATERIAL:
+{context}
+
+USER QUESTION:
+{query}
+"""
+
+COMPARISON_PROMPT = """
+You are an expert Indian legal research assistant.
+
+Compare the supplied cases on the USER QUESTION.
+
+For each case identify:
+1. Material facts relevant to the question
+2. Legal issue decided
+3. Holding / ratio decidendi
+4. Key paragraphs / observations
+
+Then provide:
+5. Similarities between the cases
+6. Key differences / distinctions
+7. Evolution or development of the legal principle
+8. Practical rule a lawyer should apply
+
+Use a structured format. Use tables where helpful.
+Use ONLY the supplied material. Cite case names and citations throughout.
+
+LEGAL MATERIAL:
+{context}
+
+USER QUESTION:
+{query}
+"""
+
+def generate_answer(query: str, context: str, comparison: bool = False) -> str:
+    prompt_template = COMPARISON_PROMPT if comparison else RESEARCH_PROMPT
+    prompt = prompt_template.format(context=context, query=query)
+
+    response = llm.chat.completions.create(
+        model="deepseek-chat",
+        messages=[
+            {"role": "system", "content": SYSTEM_PROMPT},
+            {"role": "user",   "content": prompt}
+        ],
+        temperature=0,
+        max_tokens=2000
+    )
+
+    return response.choices[0].message.content
+
+# =====================================================
+# MAIN LOOP
+# =====================================================
+
+def main():
+    print("\n" + "="*60)
+    print("  LEGAL AI — SCI JUDGMENT RETRIEVAL SYSTEM")
+    print("="*60)
+    print("Type a legal question, a citation (e.g. 2025 INSC 337),")
+    print("or 'exit' to quit.\n")
+
+    while True:
+        query = input("Ask a legal question: ").strip()
+
+        if not query:
+            continue
+
+        if query.lower() == "exit":
+            print("Goodbye.")
+            break
+
+        # Retrieve
+        retrieved = retrieve(query)
+
+        if retrieved is None:
+            print("\nNo relevant cases found.")
+            continue
+
+        # Print retrieved cases summary
+        rtype = retrieved.get("type", "")
+
+        if rtype == "comparison":
+            print("\nCOMPARISON CASES:")
+            for case in retrieved["parents"]:
+                meta = case["metadata"]
+                print(f"  {meta.get('neutral_citation','')} | {meta.get('case_name','')}")
+
+        elif rtype == "semantic":
+            print("\nRETRIEVED CASES:")
+            for parent in retrieved["parents"]:
+                meta = parent["metadata"]
+                print(f"  {meta.get('neutral_citation','')} | {meta.get('case_name','')}")
+
+        # Build context and generate answer
+        context    = build_context(retrieved)
+        comparison = (
+            rtype == "comparison"
+            or is_comparison_query(query)
+        )
+        answer = generate_answer(query, context, comparison=comparison)
+
+        print("\n" + "="*80)
+        print("ANSWER")
+        print("="*80)
+        print(answer)
+        print()
+
+
+if __name__ == "__main__":
+    main()

metadata_matching_bharatlibrary.py

@@ -0,0 +1,88 @@
+import asyncio
+import json
+from pathlib import Path
+from bharat_courts import JudgmentSearchClient
+from bharat_courts.captcha.ocr import OCRCaptchaSolver
+
+async def main():
+    jsonl_path = Path("data/html/extracted_judgments.jsonl")
+    pdf_dir = Path("data/pdfs")
+    state_path = Path("data/pdf_download_state.txt")
+    failed_path = Path("data/pdf_failed.jsonl")
+    pdf_dir.mkdir(parents=True, exist_ok=True)
+
+    # Load all records
+    records = []
+    with open(jsonl_path, "r", encoding="utf-8") as f:
+        for line in f:
+            if line.strip():
+                records.append(json.loads(line))
+
+    print(f"Loaded {len(records)} records")
+
+    # Resume — skip already downloaded
+    completed = set()
+    if state_path.exists():
+        completed = {line.strip() for line in state_path.read_text().splitlines() if line.strip()}
+    print(f"Already downloaded: {len(completed)} | Remaining: {len(records) - len(completed)}")
+
+    success, failed = 0, 0
+
+    async with JudgmentSearchClient(captcha_solver=OCRCaptchaSolver()) as client:
+        for i, record in enumerate(records, 1):
+            query = record.get("neutral_citation", "").strip()
+            if not query:
+                query = record.get("case_name", "").strip()
+
+            # Skip already downloaded
+            if query in completed:
+                print(f"[{i}/{len(records)}] Skipping: {query}")
+                continue
+
+            print(f"\n[{i}/{len(records)}] Searching: {query}")
+
+            try:
+                results = await client.search(query, court_type="3", page_size=5)
+
+                if not results.items:
+                    print(f"  ✗ No results found")
+                    failed += 1
+                    with open(failed_path, "a") as ef:
+                        ef.write(json.dumps({"query": query, "reason": "no_results"}) + "\n")
+                    continue
+
+                judgment = results.items[0]
+                await client.download_pdf(judgment, court_type="3")
+
+                if judgment.pdf_bytes:
+                    safe_name = query.replace(" ", "_").replace("/", "-")
+                    pdf_path = pdf_dir / f"{safe_name}.pdf"
+                    pdf_path.write_bytes(judgment.pdf_bytes)
+                    record["pdf_path"] = str(pdf_path)
+                    # Save progress
+                    with open(state_path, "a") as sf:
+                        sf.write(query + "\n")
+                    completed.add(query)
+                    print(f"  ✓ {pdf_path.name} ({len(judgment.pdf_bytes) // 1024} KB)")
+                    success += 1
+                else:
+                    print(f"  ✗ PDF bytes empty")
+                    failed += 1
+                    with open(failed_path, "a") as ef:
+                        ef.write(json.dumps({"query": query, "reason": "empty_pdf"}) + "\n")
+
+            except Exception as e:
+                print(f"  ✗ Error: {e}")
+                failed += 1
+                with open(failed_path, "a") as ef:
+                    ef.write(json.dumps({"query": query, "reason": str(e)}) + "\n")
+
+    # Save updated metadata with pdf_path
+    with open(jsonl_path, "w", encoding="utf-8") as f:
+        for record in records:
+            f.write(json.dumps(record, ensure_ascii=False) + "\n")
+
+    print(f"\n{'='*50}")
+    print(f"✓ Downloaded: {success} | ✗ Failed: {failed}")
+
+asyncio.run(main())

metadata_retrieval.py

@@ -0,0 +1,896 @@
+"""
+SCR (Supreme Court Reports) Judgment Scraper
+scraper with complete metadata extraction for RAG pipelines.
+"""
+
+import os
+import sys
+import requests
+from bs4 import BeautifulSoup
+import json
+import re
+import logging
+import random
+import time
+from pathlib import Path
+from datetime import datetime
+from dataclasses import dataclass, field, asdict
+from typing import Optional
+
+# ---------------------------------------------------------------------------
+# Logging
+# ---------------------------------------------------------------------------
+Path("data").mkdir(exist_ok=True)
+logging.basicConfig(
+    level=logging.INFO,
+    format="%(asctime)s [%(levelname)s] %(message)s",
+    handlers=[
+        logging.StreamHandler(),
+        logging.FileHandler("data/scraper.log", encoding="utf-8"),
+    ],
+)
+log = logging.getLogger(__name__)
+
+
+# ---------------------------------------------------------------------------
+# Data model
+# ---------------------------------------------------------------------------
+@dataclass
+class CaseCited:
+    name: str
+    citation: str
+    treatment: str  # "relied on" | "referred to" | "overruled" | "distinguished"
+
+
+@dataclass
+class SectionRef:
+    act: str
+    provision: str   # e.g. "s.61(2)", "r.22"
+    number: str      # e.g. "61(2)", "22"
+
+
+@dataclass
+class JudgmentMetadata:
+    # --- identifiers ---
+    case_name: str = ""
+    appeal_no: str = ""
+    citation: str = ""
+    neutral_citation: str = ""
+
+    # --- court info ---
+    court: str = "Supreme Court"
+    lower_court: str = ""
+    jurisdiction: str = "India"
+    state: Optional[str] = None
+
+    # --- date ---
+    date: Optional[str] = None           # ISO-8601 YYYY-MM-DD
+
+    # --- bench ---
+    bench: list = field(default_factory=list)   # ["Sanjay Kumar, J", ...]
+    author_judge: str = ""
+
+    # --- outcome ---
+    outcome: str = ""
+    case_type: str = ""
+
+    # --- statutes ---
+    acts: list = field(default_factory=list)
+    sections: list = field(default_factory=list)
+
+    # --- case law ---
+    cases_cited: list = field(default_factory=list)
+
+    # --- text fields ---
+    keywords: list = field(default_factory=list)
+    issue: str = ""
+    short_summary: str = ""     # headnote-derived, for metadata filtering
+    full_headnote: str = ""     # full headnote text, used as RAG chunk content
+
+    # --- source ---
+    source_url: str = ""
+    scraped_at: str = field(default_factory=lambda: datetime.utcnow().isoformat())
+    pdf_path: str = ""
+
+
+# ---------------------------------------------------------------------------
+# Constants
+# ---------------------------------------------------------------------------
+MONTHS = {
+    "jan": 1, "feb": 2, "mar": 3, "apr": 4, "may": 5, "jun": 6,
+    "jul": 7, "aug": 8, "sep": 9, "oct": 10, "nov": 11, "dec": 12,
+    "january": 1, "february": 2, "march": 3, "april": 4, "june": 6,
+    "july": 7, "august": 8, "september": 9, "october": 10, "november": 11,
+    "december": 12,
+}
+
+# Canonical act names (keyed by lowercase fragment).
+# All variants of the same act must map to the SAME canonical string —
+# this prevents duplicate section entries under different name spellings.
+ACT_ALIASES = {
+    "insolvency and bankruptcy code": "Insolvency and Bankruptcy Code, 2016",
+    "ibc": "Insolvency and Bankruptcy Code, 2016",
+    # Both the full name and short name resolve to the same canonical string
+    "national company law appellate tribunal rules": "NCLAT Rules, 2016",
+    "nclat rules": "NCLAT Rules, 2016",
+    "nclat rule": "NCLAT Rules, 2016",
+    "constitution of india": "Constitution of India",
+    "code of criminal procedure": "Code of Criminal Procedure, 1973",
+    "crpc": "Code of Criminal Procedure, 1973",
+    "indian penal code": "Indian Penal Code, 1860",
+    "ipc": "Indian Penal Code, 1860",
+    "civil procedure code": "Code of Civil Procedure, 1908",
+    "cpc": "Code of Civil Procedure, 1908",
+    "arbitration and conciliation": "Arbitration and Conciliation Act, 1996",
+    "right of children": "Right of Children to Free and Compulsory Education Act, 2009",
+    "companies act": "Companies Act, 2013",
+}
+
+# Acts list normaliser — applied after scraping meta.acts so that
+# "National Company Law Appellate Tribunal Rules, 2016" and
+# "NCLAT Rules, 2016" are unified before section attribution.
+def _normalise_acts(acts: list[str]) -> list[str]:
+    """Deduplicate acts list by resolving all entries through ACT_ALIASES."""
+    seen: set[str] = set()
+    result: list[str] = []
+    for act in acts:
+        canonical = act  # default: keep as-is
+        act_lower = act.lower()
+        for key, canon in ACT_ALIASES.items():
+            if key in act_lower:
+                canonical = canon
+                break
+        if canonical not in seen:
+            seen.add(canonical)
+            result.append(canonical)
+    return result
+
+# Prefix type → which kind of act it belongs to
+# "section" prefixes → statutory acts (IBC, IPC, CPC…)
+# "rule" prefixes    → rules/regulations (NCLAT Rules, etc.)
+SECTION_PREFIXES = {"s", "ss", "sec", "section"}
+RULE_PREFIXES    = {"r", "rr", "rule"}
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+def parse_date(raw: str) -> Optional[str]:
+    """Return ISO-8601 date or None."""
+    if not raw:
+        return None
+    raw = raw.strip()
+    for fmt in ("%d %B %Y", "%d %b %Y", "%d-%m-%Y", "%Y-%m-%d"):
+        try:
+            return datetime.strptime(raw, fmt).strftime("%Y-%m-%d")
+        except ValueError:
+            pass
+    m = re.search(r"(\d{1,2})\s+([A-Za-z]+)\s+(\d{4})", raw)
+    if m:
+        day, mon, year = m.groups()
+        month_num = MONTHS.get(mon.lower())
+        if month_num:
+            return f"{int(year):04d}-{month_num:02d}-{int(day):02d}"
+    return None
+
+
+def clean_text(text: str) -> str:
+    return re.sub(r"\s+", " ", text).strip()
+
+
+def _resolve_act(prefix: str, sentence_act: Optional[str], known_acts: list[str]) -> str:
+    """
+    Determine the most appropriate act for a provision reference.
+    Priority: sentence-level act mention > prefix-type hint > first known act.
+    """
+    if sentence_act:
+        return sentence_act
+
+    p = prefix.lower().rstrip(".")
+    if p in RULE_PREFIXES:
+        # Find the first rules/regulations act in known_acts
+        for a in known_acts:
+            if "rules" in a.lower() or "regulations" in a.lower():
+                return a
+        return "NCLAT Rules, 2016"  # safe default for SCR judgments
+
+    if p in SECTION_PREFIXES:
+        # Find the first non-rules act in known_acts
+        for a in known_acts:
+            if "rules" not in a.lower() and "regulations" not in a.lower():
+                return a
+        return known_acts[0] if known_acts else "Unknown Act"
+
+    return known_acts[0] if known_acts else "Unknown Act"
+
+
+# ---------------------------------------------------------------------------
+# Core parser
+# ---------------------------------------------------------------------------
+def parse_judgment_html(html_content: str, source_url: str = "") -> dict:
+    """
+    Parse a judgment HTML fragment (from the SCR splitview endpoint)
+    and return a fully-populated JudgmentMetadata dict.
+    """
+    soup = BeautifulSoup(html_content, "html.parser")
+    meta = JudgmentMetadata(source_url=source_url)
+    full_text = soup.get_text(separator="\n")
+
+    # ------------------------------------------------------------------
+    # 1. Case name  (FIX: was returning file path slug)
+    # ------------------------------------------------------------------
+    # Try known CSS classes first
+    for cls in ["Case-Title", "CaseTitle", "case-title", "Parties", "Party-Name"]:
+        el = soup.find(class_=cls)
+        if el:
+            meta.case_name = clean_text(el.get_text())
+            break
+
+    # Fallback: extract "Appellant v. Respondent" pattern from full text
+    if not meta.case_name:
+        m = re.search(
+            r"([A-Z][A-Za-z\s,\.&]+)\s+[vV][sS]?\.?\s+([A-Z][A-Za-z\s,\.&]+)",
+            full_text,
+        )
+        if m:
+            meta.case_name = clean_text(m.group(0))
+
+    # Fallback: use citation-derived name (strip underscores, page range)
+    if not meta.case_name and source_url:
+        path_part = source_url.split("path=")[-1]
+        # e.g. "2026_5_577_583" → not a useful name, skip
+        if not re.match(r"^\d{4}_\d+_\d+", path_part):
+            meta.case_name = path_part.replace("_", " ").strip()
+
+    # ------------------------------------------------------------------
+    # 2. Appeal / case number  (FIX: broader regex patterns)
+    # ------------------------------------------------------------------
+    for cls in ["Appeal-No", "AppealNo", "CaseNo", "case-no", "Appeal-Number"]:
+        el = soup.find(class_=cls)
+        if el:
+            meta.appeal_no = clean_text(el.get_text())
+            break
+
+    if not meta.appeal_no:
+        # Covers all variants:
+        #   "Civil Appeal No. 7458 of 2026"
+        #   "Civil Appeal No(s). 14439-14440 of 2025"
+        #   "Criminal Appeal Nos. 123-124 of 2025"
+        #   "Special Leave Petition No. 456 of 2024"
+        m = re.search(
+            r"((?:Civil|Criminal|Special Leave|Writ)\s+(?:Appeal|Petition)\s+"
+            r"No(?:s|\(s\))?\.?\s*[\d\-]+(?:\s*(?:and|&)\s*[\d\-]+)?\s*of\s*\d{4})",
+            full_text,
+            re.I,
+        )
+        if m:
+            meta.appeal_no = clean_text(m.group(1))
+
+    # ------------------------------------------------------------------
+    # 3. Citation + neutral citation
+    # ------------------------------------------------------------------
+    cit_el = soup.find(class_="Citation")
+    if cit_el:
+        raw_cit = clean_text(cit_el.get_text(separator=" "))
+        meta.citation = raw_cit
+        nc_m = re.search(r"\d{4}\s+INSC\s+\d+", raw_cit)
+        if nc_m:
+            meta.neutral_citation = nc_m.group(0)
+
+    # ------------------------------------------------------------------
+    # 4. Date
+    # ------------------------------------------------------------------
+    date_el = soup.find(class_="Date-of-Decision")
+    meta.date = parse_date(date_el.get_text(strip=True) if date_el else "")
+
+    # ------------------------------------------------------------------
+    # 5. Bench & author judge  (FIX: "JJ." suffix leaking as a judge name)
+    # ------------------------------------------------------------------
+    coram_el = soup.find(class_="Coram")
+    if coram_el:
+        raw_bench = clean_text(coram_el.get_text()).strip("[]")
+
+        # Remove trailing "JJ." / "J." designations before splitting
+        # e.g. "Sanjay Kumar* and K. Vinod Chandran, JJ."
+        raw_bench = re.sub(r",?\s*JJ?\.$", "", raw_bench, flags=re.I).strip()
+
+        # Split on " and " or ", "
+        judges_raw = re.split(r"\s+and\s+|,\s*", raw_bench, flags=re.I)
+        judges_raw = [j.strip() for j in judges_raw if j.strip()]
+
+        bench_clean = []
+        for j in judges_raw:
+            is_author = "*" in j
+            name = j.replace("*", "").strip()
+            if not name:
+                continue
+            # Append ", J" suffix if not already present
+            if not re.search(r",?\s*J\.?$", name, re.I):
+                name = name + ", J"
+            bench_clean.append(name)
+            if is_author and not meta.author_judge:
+                meta.author_judge = name
+
+        meta.bench = bench_clean
+
+        # If author not marked by asterisk, use first judge as author
+        if not meta.author_judge and bench_clean:
+            meta.author_judge = bench_clean[0]
+
+    # ------------------------------------------------------------------
+    # 6. Acts  — normalised to canonical names to prevent duplicates
+    # ------------------------------------------------------------------
+    acts_el = soup.find(class_="Acts")
+    if acts_el:
+        raw_acts = acts_el.get_text(strip=True)
+        raw_list = [a.strip().rstrip(".") for a in raw_acts.split(";") if a.strip()]
+        meta.acts = _normalise_acts(raw_list)
+
+    # ------------------------------------------------------------------
+    # 7. Sections (structured, act-aware)  (FIX: wrong act attribution)
+    # ------------------------------------------------------------------
+    meta.sections = _extract_sections_structured(soup, meta.acts)
+
+    # ------------------------------------------------------------------
+    # 8. Lower court
+    # ------------------------------------------------------------------
+    m = re.search(
+        r"From the (?:Judgment and )?Order dated[^o]+of the\s+(.+?)\s+in\s+",
+        full_text,
+        re.I | re.S,
+    )
+    if m:
+        meta.lower_court = clean_text(m.group(1))
+    elif "NCLAT" in full_text:
+        nm = re.search(r"(National Company Law Appellate Tribunal[^,\n]*)", full_text)
+        if nm:
+            meta.lower_court = clean_text(nm.group(1))
+    elif "High Court" in full_text:
+        hm = re.search(r"(High Court of[^,\n]+)", full_text)
+        if hm:
+            meta.lower_court = clean_text(hm.group(1))
+
+    # ------------------------------------------------------------------
+    # 9. Case type
+    # ------------------------------------------------------------------
+    acts_lower = " ".join(meta.acts).lower()
+    if "insolvency" in acts_lower or "ibc" in acts_lower:
+        meta.case_type = "Insolvency / IBC"
+    elif "constitution" in acts_lower:
+        meta.case_type = "Constitutional"
+    elif "criminal procedure" in acts_lower or "ipc" in acts_lower:
+        meta.case_type = "Criminal"
+    elif "civil procedure" in acts_lower:
+        meta.case_type = "Civil"
+    elif "arbitration" in acts_lower:
+        meta.case_type = "Arbitration"
+    elif "tax" in acts_lower or "income" in acts_lower:
+        meta.case_type = "Tax"
+    elif "labour" in acts_lower or "industrial" in acts_lower:
+        meta.case_type = "Labour"
+
+    # ------------------------------------------------------------------
+    # 10. Outcome
+    # ------------------------------------------------------------------
+    result_el = soup.find(class_="Result")
+    if result_el:
+        meta.outcome = clean_text(result_el.get_text())
+    else:
+        tail = full_text[-600:]
+        for phrase in [
+            "appeals allowed", "appeal allowed",
+            "appeals dismissed", "appeal dismissed",
+            "petition allowed", "petition dismissed",
+            "partly allowed", "disposed of",
+            "remanded back", "set aside",
+        ]:
+            if phrase in tail.lower():
+                meta.outcome = phrase.title()
+                break
+
+    # ------------------------------------------------------------------
+    # 11. Cases cited  (FIX: citations were empty)
+    # ------------------------------------------------------------------
+    meta.cases_cited = _extract_cases_cited(soup)
+
+    # ------------------------------------------------------------------
+    # 12. Keywords
+    # ------------------------------------------------------------------
+    kw_el = soup.find(class_="Keywords")
+    if kw_el:
+        raw_kw = kw_el.get_text(strip=True)
+        meta.keywords = [k.strip().rstrip(".") for k in raw_kw.split(";") if k.strip()]
+
+    # ------------------------------------------------------------------
+    # 13. Issue + headnote + short summary  (FIX: summary was same as issue)
+    # ------------------------------------------------------------------
+    issue_el = soup.find(class_="Issues-for-Consideration")
+    if issue_el:
+        meta.issue = clean_text(issue_el.get_text())
+
+    headnote_els = soup.find_all(class_="Headnote")
+    meta.full_headnote = " ".join(
+        clean_text(h.get_text(separator=" ")) for h in headnote_els
+    )
+
+    # short_summary = first 2 sentences of headnote "Held:" portion
+    meta.short_summary = _make_short_summary(meta.full_headnote, meta.issue)
+
+    return asdict(meta)
+
+
+# ---------------------------------------------------------------------------
+# Section extraction  (FIX: s.61(2) was being attributed to NCLAT Rules)
+# ---------------------------------------------------------------------------
+_PROVISION_RE = re.compile(
+    r"\b(section|sec|ss?|rule|rr?)\s*\.?\s*([0-9]+(?:\([0-9A-Za-z]+\))?[A-Za-z]?)",
+    re.I,
+)
+
+_ACT_MENTION_RE = re.compile(
+    r"(insolvency\s+and\s+bankruptcy\s+code"
+    r"|nclat\s+rules?"
+    r"|national\s+company\s+law\s+appellate\s+tribunal\s+rules?"
+    r"|constitution\s+of\s+india"
+    r"|code\s+of\s+criminal\s+procedure"
+    r"|indian\s+penal\s+code"
+    r"|civil\s+procedure\s+code"
+    r"|arbitration\s+and\s+conciliation)",
+    re.I,
+)
+
+
+def _extract_sections_structured(soup: BeautifulSoup, known_acts: list[str]) -> list[dict]:
+    """
+    Extract section/rule references with correct act attribution.
+
+    Strategy:
+    - Scan headnote + keywords text sentence by sentence.
+    - If a sentence explicitly names an act, attribute all provisions in
+      that sentence to that act.
+    - Otherwise use prefix type (s/sec → statutory act, r/rule → rules act)
+      to pick the right act from known_acts.
+    - Deduplicate by (act, number) pair.
+    """
+    search_els = (
+        soup.find_all(class_="Headnote")
+        + soup.find_all(class_="Keywords")
+        + soup.find_all(class_="Judgment-Body")
+    )
+    text = " ".join(el.get_text(separator=" ") for el in search_els) if search_els else soup.get_text()
+
+    sentences = re.split(r"[.;–]\s+", text)
+    seen: set[tuple] = set()
+    results: list[dict] = []
+
+    for sentence in sentences:
+        # Resolve act context for this sentence
+        act_m = _ACT_MENTION_RE.search(sentence)
+        sentence_act: Optional[str] = None
+        if act_m:
+            alias_key = re.sub(r"\s+", " ", act_m.group(0).lower())
+            for key, canonical in ACT_ALIASES.items():
+                if key in alias_key:
+                    sentence_act = canonical
+                    break
+
+        for m in _PROVISION_RE.finditer(sentence):
+            prefix, number = m.group(1), m.group(2)
+            act = _resolve_act(prefix, sentence_act, known_acts)
+            provision = f"{prefix.lower().rstrip('.')}.{number}"
+            key = (act, number)
+            if key not in seen:
+                seen.add(key)
+                results.append(asdict(SectionRef(act=act, provision=provision, number=number)))
+
+    return results
+
+
+# ---------------------------------------------------------------------------
+# Cases cited  (FIX: citation regex wasn't matching inline SCR citations)
+# ---------------------------------------------------------------------------
+_TREATMENT_RE = re.compile(
+    r"\b(relied\s+on|referred\s+to|overruled|distinguished|followed|approved|dissented)\b",
+    re.I,
+)
+
+# Matches: (2022) 2 SCC 244  |  [2021] 14 SCR 736  |  2026 INSC 479
+_CITATION_RE = re.compile(
+    r"(?:\((\d{4})\)\s*\d+\s+SCC\s+\d+"
+    r"|\[(\d{4})\]\s*\d+\s+SCR\s+\d+"
+    r"|\d{4}\s+INSC\s+\d+)",
+    re.I,
+)
+
+# Matches the full citation string for capture
+_CITATION_FULL_RE = re.compile(
+    r"(?:\(\d{4}\)\s*\d+\s+SCC\s+\d+"
+    r"|\[\d{4}\]\s*\d+\s+SCR\s+\d+"
+    r"|\d{4}\s+INSC\s+\d+)",
+    re.I,
+)
+
+
+def _extract_cases_cited(soup: BeautifulSoup) -> list[dict]:
+    """
+    Parse the 'Case Law Cited' section.
+    Handles bold/italic case names followed by citations and treatment labels.
+    """
+    results: list[dict] = []
+    seen: set[str] = set()
+
+    # Strategy 1: find by CSS class
+    case_law_el = soup.find(class_=re.compile(r"Case.?Law|CaseLaw|Cases.?Cited", re.I))
+    if case_law_el:
+        raw_text = case_law_el.get_text(separator="\n")
+    else:
+        # Strategy 2: heuristic heading search
+        full = soup.get_text(separator="\n")
+        m = re.search(
+            r"Case Law Cited\s*\n(.*?)(?:\n(?:List of Acts|List of Keywords|Appearances|Judgment)\s*\n|\Z)",
+            full,
+            re.S | re.I,
+        )
+        raw_text = m.group(1) if m else ""
+
+    if not raw_text:
+        return results
+
+    # Each case entry is typically on 1-2 lines; split on blank lines or clear separators
+    blocks = re.split(r"\n{2,}|\n(?=[A-Z])", raw_text.strip())
+
+    for block in blocks:
+        block = clean_text(block)
+        if not block or len(block) < 10:
+            continue
+
+        # Skip section/list headers
+        if re.match(r"^(List of|Case Law|Appearances|Judgment)", block, re.I):
+            continue
+
+        # Find all citations in the block
+        all_citations = _CITATION_FULL_RE.findall(block)
+        citation_str = " : ".join(all_citations) if all_citations else ""
+
+        # Treatment
+        treatment_m = _TREATMENT_RE.search(block)
+        treatment = clean_text(treatment_m.group(0)).lower() if treatment_m else "cited"
+
+        # Case name: text before the first citation, or before " – relied on" etc.
+        name = block
+        first_cit_m = _CITATION_FULL_RE.search(block)
+        if first_cit_m:
+            name = block[: first_cit_m.start()]
+        # Also cut at treatment label
+        treatment_pos = _TREATMENT_RE.search(name)
+        if treatment_pos:
+            name = name[: treatment_pos.start()]
+
+        # Clean up trailing punctuation / dashes / SCR refs
+        name = re.sub(r"\s*[–\-:]+\s*$", "", name)
+        name = re.sub(r"\s*\[?\d{4}\]?\s*\d*\s*S[CR]{2}.*", "", name)
+        name = clean_text(name)
+
+        if not name or name in seen or len(name) < 5:
+            continue
+        seen.add(name)
+
+        results.append(asdict(CaseCited(name=name, citation=citation_str, treatment=treatment)))
+
+    return results
+
+
+# ---------------------------------------------------------------------------
+# Short summary  (FIX: was identical to issue; now derived from headnote)
+# ---------------------------------------------------------------------------
+def _make_short_summary(headnote: str, issue: str) -> str:
+    """
+    Extract a 2-3 sentence summary from the 'Held:' portion of the headnote.
+    Falls back to the first 2 sentences of the headnote, then to the issue.
+    """
+    if headnote:
+        # Prefer the "Held:" conclusion
+        held_m = re.search(r"\bHeld\s*:\s*(.+?)(?:\[Para|\Z)", headnote, re.S | re.I)
+        base = held_m.group(1) if held_m else headnote
+
+        sentences = re.split(r"(?<=[.!?])\s+–?\s*", base.strip())
+        summary = " ".join(sentences[:3]).strip()
+
+        if len(summary) > 500:
+            summary = summary[:500].rsplit(" ", 1)[0] + "…"
+        if summary:
+            return summary
+
+    # Final fallback: issue
+    return issue[:400] + "…" if len(issue) > 400 else issue
+
+
+# ---------------------------------------------------------------------------
+# HTTP layer
+# ---------------------------------------------------------------------------
+BASE_URL = "https://scr.sci.gov.in/scrsearch/"
+HEADERS = {
+    "User-Agent": (
+        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
+        "AppleWebKit/537.36 (KHTML, like Gecko) "
+        "Chrome/124.0.0.0 Safari/537.36"
+    ),
+    "Accept-Language": "en-US,en;q=0.9",
+}
+
+
+def build_session() -> requests.Session:
+    s = requests.Session()
+    s.headers.update(HEADERS)
+    return s
+
+
+def fetch_homepage(session: requests.Session) -> BeautifulSoup:
+    log.info("Fetching SCR homepage...")
+    r = session.get(BASE_URL, timeout=30)
+    r.raise_for_status()
+    return BeautifulSoup(r.text, "html.parser")
+
+
+def download_captcha(session: requests.Session, soup: BeautifulSoup, out_path: Path) -> None:
+    captcha_img = soup.find(id="captcha_image")
+    if not captcha_img:
+        raise RuntimeError("CAPTCHA image element not found on homepage.")
+    url = f"https://scr.sci.gov.in{captcha_img.get('src', '')}"
+    r = session.get(url, timeout=15)
+    r.raise_for_status()
+    out_path.write_bytes(r.content)
+    log.info(f"CAPTCHA saved → {out_path.resolve()}")
+
+
+def verify_captcha(session: requests.Session, captcha_code: str, search_text: str) -> str:
+    payload = {
+        "captcha": captcha_code, "search_text": search_text,
+        "search_opt": "PHRASE", "escr_flag": "", "proximity": "",
+        "sel_lang": "", "neu_cit_year": "", "neu_no": "", "ncn": "",
+        "citation_vol": "", "citation_year": "", "citation_supl": "",
+        "citation_page": "", "ajax_req": "true", "app_token": "",
+    }
+    r = session.post(
+        f"{BASE_URL}?p=pdf_search/checkCaptcha", data=payload,
+        headers={"X-Requested-With": "XMLHttpRequest"}, timeout=30,
+    )
+    r.raise_for_status()
+    data = r.json()
+    if data.get("captcha_status") != "Y":
+        raise ValueError("CAPTCHA verification failed.")
+    return data.get("app_token", "")
+
+
+def init_search_session(session, search_text, captcha_code, app_token):
+    params = {
+        "p": "pdf_search/home", "text": search_text, "captcha": captcha_code,
+        "search_opt": "PHRASE", "fcourt_type": "3", "escr_flag": "", "app_token": app_token,
+    }
+    session.get(BASE_URL, params=params, timeout=30).raise_for_status()
+    log.info("Search session initialized.")
+
+
+def fetch_results_list(session, app_token, start=0, length=50):
+    """Fetch a specific paginated batch of results."""
+    payload = {
+        "p": "pdf_search/home/", "sEcho": "1", "iColumns": "2", "sColumns": ",",
+        "iDisplayStart": str(start), "iDisplayLength": str(length),
+        "fcourt_type": "3", "search_opt": "PHRASE", "ajax_req": "true", "app_token": app_token,
+    }
+    r = session.post(
+        f"{BASE_URL}?p=pdf_search/home/", data=payload,
+        headers={"X-Requested-With": "XMLHttpRequest"}, timeout=30,
+    )
+    r.raise_for_status()
+    return r.json().get("reportrow", {}).get("aaData", [])
+
+
+def fetch_splitview(session, args, app_token):
+    payload = {
+        "val": args[0], "citation_year": args[1], "path": args[2],
+        "fcourt_type": "3", "nc_display": args[3], "flag": args[4],
+        "ajax_req": "true", "app_token": app_token,
+    }
+    r = session.post(
+        f"{BASE_URL}?p=pdf_search/splitview", data=payload,
+        headers={"X-Requested-With": "XMLHttpRequest"}, timeout=30,
+    )
+    r.raise_for_status()
+    return r.json().get("outputfile", "")
+
+
+def parse_splitview_args(row_html: str) -> Optional[list]:
+    soup = BeautifulSoup(row_html, "html.parser")
+    btn = soup.find("a", onclick=lambda x: x and "open_splitview" in x and "'H'" in x)
+    if not btn:
+        return None
+    m = re.search(r"open_splitview\(([^)]+)\)", btn.get("onclick", ""))
+    if not m:
+        return None
+    args = [a.strip().strip("'\"") for a in m.group(1).split(",")]
+    return args if len(args) >= 5 else None
+
+
+def fetch_pdf_from_splitview(session, html_content, path, app_token, pdf_dir):
+    from pathlib import Path
+    import re
+
+    pdf_dir = Path(pdf_dir)
+    pdf_dir.mkdir(parents=True, exist_ok=True)
+
+    # Extract hidden field values from splitview HTML
+    year_m   = re.search(r"name='year'[^>]*value='(\d+)'", html_content)
+    vol_m    = re.search(r"name='volume'[^>]*value='(\d+)'", html_content)
+    part_m   = re.search(r"name='partno'[^>]*value='(\d+)'", html_content)
+
+    year   = year_m.group(1)   if year_m   else path.split("_")[0]
+    volume = vol_m.group(1)    if vol_m    else path.split("_")[1]
+    part   = part_m.group(1)   if part_m   else path.split("_")[2]
+
+    # path = "2026_5_577_583" → pages = "577_583"
+    parts = path.split("_")
+    pages = f"{parts[2]}_{parts[3]}" if len(parts) >= 4 else path
+
+    candidate_urls = [
+        f"https://scr.sci.gov.in/scrsearch/pdfs/{year}/{volume}/{pages}.pdf",
+        f"https://scr.sci.gov.in/scrsearch/pdfs/{path}.pdf",
+        f"https://scr.sci.gov.in/scrsearch/pdfs/{year}_{volume}_{pages}.pdf",
+        f"https://scr.sci.gov.in/scrsearch/?p=pdf_search/viewpdf&year={year}&volume={volume}&partno={part}&app_token={app_token}",
+    ]
+
+    for url in candidate_urls:
+        try:
+            r = session.get(url, timeout=30)
+            if r.status_code == 200 and r.content[:4] == b"%PDF":
+                pdf_path = pdf_dir / f"{path}.pdf"
+                pdf_path.write_bytes(r.content)
+                log.info(f"  ✓ PDF saved ({len(r.content) // 1024} KB) from: {url}")
+                return str(pdf_path)
+            else:
+                log.info(f"  Not a PDF at: {url} (status={r.status_code})")
+        except Exception as e:
+            log.info(f"  Failed: {url} → {e}")
+
+    log.warning(f"  No PDF found for: {path}")
+    return ""
+# ---------------------------------------------------------------------------
+# Rewritten Main Function
+# ---------------------------------------------------------------------------
+def main():
+    Path("data/html").mkdir(parents=True, exist_ok=True)
+    session = build_session()
+
+    # Switch to JSON Lines (.jsonl) for incremental, crash-proof saving
+    out_path = Path("data/html/extracted_judgments.jsonl")
+    err_path = Path("data/html/errors.jsonl")
+
+    # 1. Auto-Resume Check
+    start_offset = 0
+    if out_path.exists():
+        with open(out_path, "r", encoding="utf-8") as f:
+            start_offset = sum(1 for _ in f)
+        if start_offset > 0:
+            log.info(f"Found {start_offset} existing records. Resuming from there.")
+
+    # 2. Homepage + CAPTCHA
+    try:
+        homepage_soup = fetch_homepage(session)
+    except Exception as e:
+        log.error(f"Could not reach SCR homepage: {e}")
+        return
+
+    captcha_path = Path("data/html/captcha.png")
+    try:
+        download_captcha(session, homepage_soup, captcha_path)
+        if sys.platform == "win32":
+            os.startfile(str(captcha_path.resolve()))
+    except Exception as e:
+        log.error(f"CAPTCHA download failed: {e}")
+        return
+
+    # 3. Collect inputs
+    search_text = input("Enter search keyword [insolvency]: ").strip() or "insolvency"
+    max_results_raw = input("How many total judgments to scrape? [1000]: ").strip()
+    max_results = int(max_results_raw) if max_results_raw.isdigit() else 1000
+
+    if start_offset >= max_results:
+        log.info("Target number of judgments already reached in previous runs. Exiting.")
+        return
+
+    # 4. CAPTCHA verification loop
+    app_token = None
+    for attempt in range(1, 4):
+        captcha_code = input(f"Enter CAPTCHA code (attempt {attempt}/3): ").strip()
+        if not captcha_code:
+            continue
+        try:
+            app_token = verify_captcha(session, captcha_code, search_text)
+            log.info("CAPTCHA verified successfully.")
+            break
+        except ValueError as e:
+            log.warning(f"Attempt {attempt} failed: {e}")
+            if attempt < 3:
+                try:
+                    homepage_soup = fetch_homepage(session)
+                    download_captcha(session, homepage_soup, captcha_path)
+                    if sys.platform == "win32":
+                        os.startfile(str(captcha_path.resolve()))
+                except Exception as ref_e:
+                    log.error(f"CAPTCHA refresh failed: {ref_e}")
+
+    if app_token is None:
+        log.error("All CAPTCHA attempts failed. Exiting.")
+        return
+
+    # 5. Init search session
+    try:
+        init_search_session(session, search_text, captcha_code, app_token)
+    except Exception as e:
+        log.error(f"Search session init failed: {e}")
+        return
+
+    # 6. Paginated Fetch & Incremental Save
+    batch_size = 50
+    records_fetched = start_offset
+
+    log.info(f"Targeting {max_results} judgments. Starting from index {records_fetched}...")
+
+    while records_fetched < max_results:
+        fetch_count = min(batch_size, max_results - records_fetched)
+        log.info(f"\n--- Fetching batch: {records_fetched} to {records_fetched + fetch_count - 1} ---")
+
+        # Fetch the page chunk
+        try:
+            rows = fetch_results_list(session, app_token, start=records_fetched, length=fetch_count)
+        except Exception as e:
+            log.error(f"Failed to fetch results batch at offset {records_fetched}: {e}")
+            log.info("Session may have timed out. Restart the script; it will auto-resume where it left off.")
+            break
+
+        if not rows:
+            log.info("No more results returned by the server. Search exhausted.")
+            break
+
+        # Process the chunk
+        for idx, row in enumerate(rows, 1):
+            row_html = row[1] if len(row) > 1 else ""
+            args = parse_splitview_args(row_html)
+            current_global_idx = records_fetched + idx
+
+            if not args:
+                log.warning(f"[{current_global_idx}] Could not parse splitview args. Skipping.")
+                with open(err_path, "a", encoding="utf-8") as ef:
+                    ef.write(json.dumps({"index": current_global_idx, "reason": "no splitview args"}) + "\n")
+                continue
+
+            citation_path = args[2]
+            log.info(f"[{current_global_idx}/{max_results}] Parsing: {citation_path}")
+
+            try:
+                html_content = fetch_splitview(session, args, app_token)
+                source_url = f"{BASE_URL}?p=pdf_search/splitview&path={citation_path}"
+                metadata = parse_judgment_html(html_content, source_url=source_url)
+
+                metadata["pdf_path"] = fetch_pdf_from_splitview(
+                    session, html_content, citation_path, app_token, pdf_dir="data/pdfs"
+                )
+                # Incremental Save: Append to JSONL immediately
+                with open(out_path, "a", encoding="utf-8") as f:
+                    f.write(json.dumps(metadata, ensure_ascii=False) + "\n")
+
+            except Exception as e:
+                log.error(f"[{current_global_idx}] Failed: {e}")
+                with open(err_path, "a", encoding="utf-8") as ef:
+                    ef.write(json.dumps({"index": current_global_idx, "path": citation_path, "reason": str(e)}) + "\n")
+
+            # Dynamic human-like delay to prevent IP blocking
+            time.sleep(random.uniform(1.0, 2.5))
+
+        records_fetched += len(rows)
+
+    log.info(f"\n✓ Process stopped. Data securely saved to {out_path.resolve()}")
+
+
+if __name__ == "__main__":
+    main()

phase1/.gitignore

@@ -0,0 +1,16 @@
+# derived eval run artifacts (regenerable; not source)
+eval/auth_*.tsv
+eval/agent_*.tsv
+eval/authority_baseline.tsv
+eval/authority_spine.tsv
+eval/silver_baseline.tsv
+eval/silver_spine.tsv
+eval/run.tsv
+eval/run_baseline.tsv
+eval/plan_*.json
+eval/last_score.json
+__pycache__/
+scripts/__pycache__/
+eval/pagerank.json
+phase1/COSTING.md
+phase1/Themis_Costing.pdf

phase1/AGENTIC_SEARCH_SPEC.md

@@ -0,0 +1,234 @@
+# Themis — Agentic Search Spec (the "controller" front door)
+
+Status: **DRAFT for founder verification** (2026-06-25). Once George signs off, build proceeds
+stage-by-stage, each behind an eval gate, then to Hitin's pilot.
+
+Companion docs: `HANDOFF.md` (current system + autoresearch method), `eval/` (the measurement rig).
+This spec describes what *replaces* the split fast/deep endpoints with one adaptive, self-correcting loop.
+
+---
+
+## 0. The idea in one paragraph
+
+Search today is a **fixed pipeline**: retrieve → rerank → verify → (maybe one requery) → answer. It commits
+to its first retrieval and can't notice when it missed the controlling authority. We replace the front door
+with a **bounded control loop** that keeps a working "brief," *inspects its own results after each round
+against a legal-quality signal*, and takes a corrective action when the brief is weak — until it has a
+high-authority, good-law, grounded answer or hits its round/time budget. Easy queries stop in ~2s; hard ones
+spend up to ~12s with **visible progress**. The proven authority prior (+0.072 nDCG@10 on doctrinal queries,
+§7b of HANDOFF) and the good-law filter are *tools the loop wields*, query-routed, not a global setting.
+
+**Latency budget:** soft 12s, hard 15s wall-clock, with a progress event per action. (Founder call, 2026-06-25.)
+
+---
+
+## 1. What we KEEP (proven primitives — do not rebuild)
+
+All of these exist in `scripts/serve.py` and are reused verbatim as the loop's tools:
+
+| Primitive | Function | Role in the loop |
+|---|---|---|
+| Hybrid retrieval | `candidates()` (dense BGE + BM25, RRF) | base `search` action |
+| Cross-encoder rerank | `rerank()` | scores any candidate set |
+| Relevance screen | `verify()` (DeepSeek paralegal) | gates what enters the brief |
+| Identity lookup | `identity_hits()` / `name_search()` / `id_card()` | the known-item fast exit |
+| Plan | `_plan()` → sub-issues + expected authority names | seeds the brief's `doctrine` + `expected_authorities` |
+| Citation graph | `cites_docs()` / `cited_by_docs()` + `edge_meta` (treatment) | the `walk_citations` action |
+| Authority signal | `cite_indeg` (CITE in-degree) | the validated ranking prior |
+| Good-law | `goodlaw[doc]` (status + provenance + treatment) | filter / flag / pivot |
+| Card for a specific doc | `card_for_doc()` | scores graph/authority additions |
+| Grounding gate | `grounded_answer()` / `verify_claims()` (verbatim substring) | the answer invariant — unchanged |
+| Progress protocol | SSE `{t:"step", k, s, label}` + `answer_delta`/`claims`/`dropped_claims` | the live UI |
+
+**The grounding invariant is non-negotiable and already correct:** the loop may *add* candidates freely, but the
+user-facing answer is rendered only from claims whose ≥4-word quote is a verbatim substring of a loaded case.
+Self-correction therefore cannot amplify a hallucination — the worst a bad round does is waste budget.
+
+---
+
+## 2. The working brief (loop state)
+
+A single dict accumulated across rounds — the "layered understanding":
+
+```
+brief = {
+  "q":  original query,
+  "intent": one of {known_item, doctrinal, factual, mixed},   # set once, round 0 (see §3)
+  "doctrine": [issue phrases],            # from _plan
+  "statute_refs": [section ids],          # detected + concordance-resolved (Stage 2)
+  "expected_authorities": [case names],   # from _plan; each grounded via name_search
+  "pool": {doc_id: card},                 # everything retrieved so far, de-duped
+  "confirmed": [doc_id, ...],             # relevant ∧ good-law ∧ on-point  → the answer set
+  "gaps": [gap-tokens],                   # what's missing; drives the next action
+  "rounds": int,
+  "spent_ms": int,
+}
+```
+
+`gaps` is the engine of self-correction — each round recomputes it, and the controller picks the action that
+closes the biggest gap.
+
+---
+
+## 3. Round 0 — intent (the "router," reframed)
+
+The router you flagged as too dumb becomes a **state-setter, not a gate**. One cheap DeepSeek call (or a
+rule+embedding shortcut) labels `intent` and extracts `statute_refs`. It does **not** decide the whole strategy
+— it just initializes the brief and selects defaults:
+
+- `known_item` (a name/citation) → exact-lookup exit (`identity_hits`), no loop. Already works; success@1 guardrail.
+- `doctrinal` / `mixed` → authority prior **ON** (α=0.3), full loop eligible.
+- `factual` (narrow fact pattern, no landmark expected) → authority prior **OFF** (it's −0.29 on general silver),
+  loop still eligible for coverage but not authority-boosted.
+
+Why route, not globally apply: the prior is **+0.072 on doctrinal, −0.29 on general** — it must be conditional.
+The intent label is the cheapest correct place to make that switch.
+
+---
+
+## 4. The control loop
+
+```
+brief = init(q)                       # round 0: intent, _plan → doctrine + expected_authorities, statute_refs
+if brief.intent == known_item: return identity_exit(brief)
+
+round 1: brief.pool += search(q); verify(); brief.confirmed = keep(); rank(brief)
+while not STOP(brief):                # rounds 2..MAX
+    action = choose(brief)            # the self-correction policy (§5)
+    if action is None: break          # nothing would close the gap → stop (don't spin)
+    emit_step(action.label)           # visible progress
+    brief.pool += action.run(brief); verify(new); brief.confirmed = keep(); rank(brief)
+answer = ground(brief.confirmed)      # the verbatim gate, unchanged
+```
+
+**STOP** (any one fires):
+- coverage ok: ≥3 relevant good-law cases **and** an authority is present (top-of-list cite_indeg ≥ doctrine
+  threshold, or an `expected_authority` resolved into `confirmed`) **and** the grounding gate verified ≥1 holding;
+- `rounds ≥ MAX` (default 3);
+- `spent_ms ≥ 12000` (soft) / hard cut at 15000;
+- `choose()` returns None (no remaining gap is addressable).
+
+**rank(brief):** `score = sigmoid(rr) + (α·log1p(cite_indeg[d]) if intent∈{doctrinal,mixed} else 0)`, then drop
+or sink any `good_law_status ∈ {overruled, partly_overruled, per_incuriam}` unless the query is *about* that case.
+This is exactly the §7b commit config (authority prior + good-law filter), now applied every round.
+
+---
+
+## 5. The self-correction policy — `choose(brief)`
+
+After each round, compute the gap signal (all cheap / mechanical) and pick the action addressing the biggest gap.
+Each maps a *detected deficiency in the previous output* to a corrective tool — this is the "self-corrects based
+on its previous outputs" behaviour, made concrete and legally meaningful:
+
+| Detected gap (signal) | Action | Tool(s) |
+|---|---|---|
+| No high-authority case in `confirmed` (top cite_indeg below doctrine threshold) → likely missed the landmark | pull expected authorities + draft-the-holding requery | `name_search(expected_authorities)`, **`hyde(q)`** → `search` |
+| An `expected_authority` was named but never retrieved | targeted name lookup + walk in from its nearest neighbour | `name_search(name)`, `walk_citations(near, in)` |
+| Top result flagged overruled, no live replacement | pivot to current law | `walk_citations(overruling_edge)`, `good_law` |
+| `statute_refs` present but no statute/section pulled | statute pivot | **`statute_lookup(section)`** (Stage 2) |
+| `confirmed` splits into ≥2 disjoint citation clusters (conflicting lines) | fetch the resolver | larger-bench / later case via `walk_citations` |
+| Thin coverage (<3 kept) but authority present | one rephrase (the existing move) | `requery` (LLM rewrite) → `search` |
+| None of the above, gap remains | **stop** (return None) | — |
+
+`hyde(q)` and `statute_lookup()` are the only genuinely new retrieval tools; everything else is an existing
+function called from a new policy. Each action is tried **at most once** per request (no loops on the same move).
+
+---
+
+## 6. The eval harness (every stage clears a gate — autoresearch discipline)
+
+The current rig scores a *static* run.tsv. To measure a *loop*, we add **`eval/agentic_run.py`**: imports the
+controller, runs the full loop per query (DeepSeek + tools live), emits the final ranked `confirmed` as run.tsv →
+scored by the existing `score_qrels.py`. Same frozen qrels, no LLM at score time, judge family ≠ serving family.
+
+**Slices:**
+- `authority_*` (150 doctrinal) — where the loop should shine; primary nDCG@10.
+- `queries.tsv` (800 silver) — general; must **not** regress.
+- **NEW `recall_recovery_*`** (~40 queries whose gold landmark is *outside* dense top-100) — the loop's whole
+  reason to exist; measures recall the one-shot spine *cannot* achieve. Built by scanning doctrinal queries for
+  gold docs absent from the dense pool (e.g. the Maneka Gandhi / Royappa / Indra Sawhney pool-misses).
+
+**Metrics & guardrails:**
+- primary: nDCG@10 (authority slice ↑, silver flat), recall@10 on recall-recovery (↑).
+- guardrails: bad-law@10 (good-law precision, must stay ≤ baseline), success@1 on known-item (identity route),
+  grounding rate (% queries with ≥1 verified holding), latency p50/p95.
+- **agentic-specific: marginal lift per round** — nDCG of `confirmed` after round 1 vs round 2 vs final, computed
+  only over queries where each round actually fired. If round N doesn't beat round N−1 on its own fired set, that
+  self-correction move is **cut**. This is how we prove the loop earns its latency rather than assuming it.
+
+---
+
+## 7. Stagewise build plan
+
+Each stage is independently shippable and measured. Stages 0–1 are the **thin pilot cut for Hitin**.
+
+### Stage 0 — fold the proven levers into one measured spine  *(foundation; ~2 days)*
+- Bake the authority prior into `rank` (α=0.3) **query-routed** by a round-0 intent label.
+- Make the good-law filter **real in the fast path** (drop/flag overruled — closes the F6 hole; deep mode already does it).
+- Ship `eval/agentic_run.py` so everything downstream is measured end-to-end.
+- **Gate:** authority nDCG@10 ≥ 0.35 (proved 0.351), silver no regression, bad-law@10 ≤ baseline, success@1 held.
+- *Already a shippable improvement; no loop yet.*
+
+### Stage 1 — the controller + ONE self-correction round + progress  *(the pilot cut; ~3–4 days)*
+- Introduce `scripts/agent.py`: the brief, the bounded loop, STOP rules, the intent state-setter.
+- Implement one corrective move first: **missed-landmark / thin-coverage → `name_search(expected_authorities)` +
+  `hyde(q)` requery**. (Reuses `_plan`; `hyde` is new and small.)
+- Collapse fast/deep into one adaptive endpoint: round-1-only for easy queries (~2s), escalate for hard (~12s).
+- Emit a progress step per action — the self-correction is *visible* (the trust-builder and the demo magic).
+- **Gate:** on authority + recall-recovery slices, loop final nDCG ≥ Stage-0 spine **and** recall-recovery ↑;
+  latency p95 ≤ 15s; marginal lift of round-2 positive on its fired queries. → **hand to Hitin.**
+
+### Stage 2 — the rest of the self-correction moves (layered understanding)  *(~1 week)*
+- **Statute layer:** embed `statute corpus/all_statutes.json` (2,353 sections — cheap) + a BNS↔IPC/BNSS↔CrPC/
+  BSA↔Evidence concordance table; wire `statute_lookup()`. Add the statute-pivot move.
+- Add overruled-pivot and conflict-resolution moves. Each added **behind its own marginal-lift gate** — a move
+  that doesn't move its target subset is cut, not kept "because it's principled."
+
+### Stage 3 — ranking & answer polish  *(measured, post-pilot)*
+- PageRank / Personalized-PageRank authority (stronger than raw cite_indeg; gated on clean Tier-1/2 edges).
+- Headnote-fed `verify` (down-weight, don't drop, high-authority); cross-family entailment gate after the
+  verbatim gate.
+
+### Stage 4 — promote eval to gold + retune
+- Hitin audits a stratified sample of authority+silver+recall-recovery → kappa → gold. Retune α, STOP thresholds,
+  round budget, doctrine cite_indeg threshold against gold rather than silver.
+
+---
+
+## 8. Where a self-correcting loop could make legal answers WORSE (panel concerns + mitigations)
+
+- **Hypothesis lock-in** — a wrong round-0 `_plan` (hallucinated authority names) steers every later round.
+  *Mitigation:* names are candidates only; each must resolve via `name_search` against the corpus or it's dropped
+  (already how deep mode works). `choose()` reads the *result* pool, not the plan, for its gap signal.
+- **Authority over-surfacing dead law** — the prior boosts high-cite_indeg cases, and overruled ex-landmarks are
+  high-cite_indeg. *Mitigation:* prior and good-law filter ship as one unit (proved: bad-law 0.173→0.000); never apart.
+- **Budget spiral / spinner fatigue** — each move runs at most once; hard 15s cut; STOP returns on "no addressable gap."
+- **Latency tax on easy queries** — round-1 STOP keeps clean doctrinal/known-item queries at ~2s; the loop only
+  spends rounds when the gap signal is real.
+- **Self-grading creep** — eval judge family stays ≠ serving family; no LLM at score time; frozen qrels.
+- **The loop can't ground** — if `confirmed` is off-topic the verbatim gate yields nothing; we render "couldn't
+  ground — review the cases" rather than a fluent hallucination (already the behaviour).
+
+---
+
+## 9. Open founder decisions (verify before build)
+
+1. **Latency:** soft 12s / hard 15s with progress — confirm. (Affects MAX rounds = 3.)
+2. **Posture for the pilot:** conservative (precision-first: smaller `confirmed`, drop on any doubt) vs aggressive
+   (surface more authority, flag uncertainty). Recommend **conservative** for a lawyer's first impression.
+3. **Concordance sourcing** (Stage 2): license a statute concordance vs hand-curate the BNS↔IPC core map.
+4. **Endpoint cutover:** ship the agentic loop as the new default `search_stream` and keep old fast/deep as
+   fallbacks for one pilot cycle, or replace outright. Recommend **keep as fallback** through Hitin's pilot.
+
+---
+
+## 10. Code touch-points (for the build)
+
+- **NEW** `scripts/agent.py` — brief, tools (thin wrappers over serve.py fns), `choose()` policy, the loop, STOP.
+- `scripts/serve.py` — new adaptive `search_stream` driving `agent.py`; keep `deep_search_stream`/old fast as fallback.
+- **NEW** `scripts/hyde.py` (or in agent.py) — draft-the-holding requery.
+- **NEW** `scripts/build_statute_index.py` + `statute_lookup()` — Stage 2.
+- **NEW** `eval/agentic_run.py` — runs the loop per query → run.tsv (the measurement substrate for every stage).
+- **NEW** `eval/recall_recovery_{queries,qrels,badlaw}.tsv` — the recall-recovery slice.
+</content>
+</invoke>

phase1/CITATOR_DESIGN.md

@@ -0,0 +1,43 @@
+# CP5 — good-law citator design (LOCKED)
+
+*From the legal-agent debate (senior advocate · paralegal · citator architect). `good_law_status` is a **derived, cached projection** of inbound treatment edges — never an extraction, never inferred from silence. Default **`unknown`**. Asymmetric precision: near-abstention on `good_law` and on definitive `overruled`; recall of negatives routed into the conservative **`doubted`** sink. Every published badge carries provenance + a deep link a non-lawyer can clerically verify. No DeepSeek required for the verdict; the LLM (a **local Thor model**) only widens recall in Layer 2 and never certifies death.*
+
+## Pipeline (recompute on ingest)
+- **Layer 0 — edge resolution + nodes (deterministic).** Build the graph from `cases_cited`. Resolve each edge on **both** join keys — `neutral_citation` AND `equivalent_citations` (reporter cites), since pre-2023 cited-lists are reporter-only. **Legacy-only landmarks with no neutral citation (ADM Jabalpur, A.K. Gopalan, M.P. Sharma class) must become first-class reporter-cite-keyed vertices**, else their overruling edge dangles and the dead case silently reads `unknown`. Attach per edge: verbatim passage, pin-cite, citing+target date and bench_strength. Run the **self/reversal gate** here.
+- **Layer 1 — express-declaration extraction (high-precision, GOLD).** Regex/(cheap-LLM) hunt for operative phrases naming a target: *"is/stands overruled · we overrule · no longer good law · does not lay down the correct law · overruled to the extent · declared per incuriam · hereby recalled."* The **only** thing allowed to auto-publish a clean negative. Replaces proprietary editorial flags (which are barred as IP liability).
+- **Layer 2 — edge classification + deterministic gated aggregation (recall spine).** A **local Thor model** classifies each inbound edge → `{relied_on, referred_to, followed, distinguished, reaffirmed/approved, doubted/disapproved, overruled, per_incuriam-flag, referred_to_larger_bench}` + confidence + verbatim quote + pin-cite (default fail-safe `cited`). A **deterministic, auditable aggregator (never the LLM)** rolls edges through the gates. **Layer 2 alone never auto-publishes a clean `overruled`** — an uncorroborated model `overruled` is capped at `doubted` pending Layer-1 corroboration or review. Its job is to *manufacture doubt, not certify death.*
+- **Layer 3 — statute supersession (Approach D, Phase-2, DARK).** `superseded_by_statute` only from a curated repeal/amendment+commencement table keyed on `sections` (never an LLM). Phase 1 ships it dark and instead shows a non-destructive **IPC/CrPC/IEA ↔ BNS/BNSS/BSA crosswalk banner** (BNS-class is prospective; pre-01-07-2024 offences keep the IPC line live; interpretive principles survive re-enactment).
+- **Aggregation.** Final badge = highest-precedence non-null layer. Conflict = **worst-valid-status-wins** (doubt dominates affirmation; larger bench controls smaller). Approach C (holistic per-case LLM) is **barred from writing any state** — only a triage sort-key in the review queue. On every new judgment, recompute the status of every case it cites (staleness via `scraped_at`/`content_hash`).
+
+## Gates (all deterministic)
+- **Bench-strength** (both ends, fail-safe): `overruled/partly` valid only if `strength(citing) ≥ strength(target)` (single 1 < division 2 < full 3 < constitution 5 < larger 7). Either bench unknown → **downgrade to `doubted` + `bench-unverified`**, never `overruled`.
+- **Temporal**: overruling/doubting edge must be strictly later than the target (tie-break on INSC number).
+- **Court-hierarchy**: only SC may write negative status on an SC node; an `HC:<state>` edge can never overrule/suppress an SC target.
+- **Ratio-vs-dicta / sub-silentio**: a negative counts only against the ratio the target is relied on for; disapproval of mere dicta or a silent conflict does not suppress.
+- **Self/reversal**: *"reversed the impugned/HC judgment"* (appellate disposition, acts on parties) ≠ *"overruled"* (acts on a precedent) — never writes negative on cited precedents.
+- **Confidence**: per-edge confidence below threshold → treated as neutral `cited`, contributes to no negative state.
+- **Resolution/quorum**: conflicting inbound edges → worst valid status; larger-bench edge controls.
+
+## Enum (trigger → guardrail)
+- **`unknown`** (default) — no inbound negative, or negatives failed a gate, or low confidence, or thin coverage. Renders: *"No negative treatment found in N citing judgments — not yet human-reviewed; absence of flags is not a clearance,"* with provenance. **Code path for `good_law` must be structurally incapable of firing on silence** (unit test: a zero-edge node can only be `unknown`).
+- **`good_law`** (narrow, Phase-1) — **positive assertion only**: ≥1 affirming edge (followed-and-approved/reaffirmed) by an equal-or-larger bench post-dating the case, zero unresolved negatives, bench known; names the affirming authority. Lights the 11 positive controls (Kesavananda, Maneka Gandhi, Minerva Mills, Puttaswamy). Bare `relied_on/followed` does **not** upgrade `unknown→good_law`.
+- **`overruled`** — whole ratio displaced; Layer-1 express OR Layer-2 passing both gates AND corroborated (Layer-1 or dual-model). Auto-publishes only via Layer-1 express + gates + clean resolution; Layer-2-only → review queue, case holds prior state meanwhile.
+- **`partly_overruled`** — severable part; scoped language; badge must carry surviving-vs-displaced text; **always human-reviewed** before it suppresses.
+- **`doubted`** — the conservative fail-safe sink, deliberately over-inclusive (later doubt/disapproval without overruling; reference to a larger bench = `referred_to_larger_bench` sub-flag). **Auto-publishes freely** (only adds caution). `distinguished` does **not** land here (distinguished = still good law).
+- **`per_incuriam`** — express-declaration ONLY (Layer-1 literal token naming the target). The classifier may detect but **never conclude** per incuriam by reasoning; guessed → `doubted` + "possible per incuriam" note.
+
+## Auto-publish vs review
+- **Tier-0 auto-publish (safe by construction):** `unknown`; `doubted`/`partly`-as-suppressor; and a negative with a **Layer-1 express verbatim sentence + named target + both gates + clean resolution** (the quote is its own auditor), or an edge in the human-signed gold set.
+- **Everything else → review queue**, case holds its prior conservative state.
+- Acceptance bar before any badge ships: `goodlaw_goldset.json` (21 scored + 11 positive controls) is both the auto-publish whitelist and the calibration target.
+
+## Phase 1 vs later
+- **Phase 1:** Layers 0–2 writing `{good_law (affirm-only, narrow), overruled, partly_overruled, doubted, unknown}` + all gates; `per_incuriam` via Layer-1 literal only; Tier-2 classifier = **local Thor model**.
+- **Phase 2:** `superseded_by_statute` (curated table); proposition-scoped good-law; expanded review tooling.
+
+## Open questions (need a qualified lawyer)
+- per_incuriam competence rule (may a coordinate/smaller bench declare it?).
+- Exact edge-pattern + bench conditions distinguishing per_incuriam vs superseded vs partly_overruled.
+- Bounding surviving-vs-displaced text for `partly_overruled`.
+- Final sign-off that narrow automated `good_law` is acceptable at launch.
+- Confidence-cutoff calibration against the gold set.

phase1/HANDOFF.md

@@ -0,0 +1,331 @@
+# Themis — Search-Relevance Handoff
+
+> Owner handoff for whoever drives Themis search quality next. Covers: what the product is, **exactly how
+> search works today (fast + deep), step by step**, the known weaknesses, and — most importantly — **the
+> autoresearch methodology, the eval rig, the compute setup, and the experiment discipline** so you can run
+> the propose→measure→commit loop yourself from day one. Read §4–§7 before touching any code.
+
+---
+
+## 1. What Themis is & the goal
+
+Grounded AI legal research over **37,898 reportable Indian Supreme Court judgments** (the open AWS registry
+`indian-supreme-court-judgments`, ap-south-1). It retrieves on-point cases for a lawyer's query and produces a
+grounded summary (every asserted point backed by a verbatim quote from a retrieved case).
+
+**North star right now:** make *search quality* good enough for a live pilot with practising lawyers (via Hitin,
+the lawyer co-founder). A blind benchmark vs CaseMine put us at nDCG@5 0.77 vs 0.64 — but it was a **3–3 split**
+and **partly self-graded** (see §4), and the real, repeatedly-confirmed gap is **foundational-authority recall on
+doctrinal queries** (landmark cases like Khushal Rao, Indra Sawhney rank too low or don't surface).
+
+---
+
+## 2. How search works TODAY — step by step
+
+### 2.0 Serving stack
+`phase1/scripts/serve.py` (FastAPI) + `frontend.html` (vanilla JS). The corpus is **chunked**: ~1,299,748
+passages. Each chunk has a **BGE-small-en-v1.5** embedding (384-d, float32 matrix `M = escr_vectors.npy`) and
+sits in a **BM25** index. The serving LLM is **DeepSeek** (`deepseek-chat`) via `~/.../.env` `DEEPSEEK_API_KEY`.
+Artifacts: `escr_chunks.jsonl` (chunk text), `escr_vectors.npy`, `escr_meta.jsonl` (per-judgment metadata incl.
+`issue`/`held` headnotes), `edges.jsonl` (citation graph, 86,702 edges), `good_law.jsonl` (citator), `escr_pdfmap.jsonl`.
+
+### Shared retrieval primitives
+- **`dense(q)`** — embed `"Represent this sentence for searching relevant passages: " + q` with BGE-small, cosine
+  `M @ qv`, take **top CAND=40 chunks**.
+- **`bm25_top(q)`** — BM25 over tokenized chunks, top 40. ⚠ `rank_bm25.get_scores` scans 1.3M postings in pure
+  Python = **~68 s/query** — the reason the eval can't use the full serve path (see §6).
+- **`candidates(q)`** — **RRF-fuse** dense+BM25 (k=60), top ~48 chunk candidates.
+- **`rerank(q,cand,k)`** — cross-encoder **ms-marco-MiniLM-L-6-v2** scores `(q, chunk)`, dedups to **best chunk per
+  judgment**, returns top-k docs (`rr` = cross-encoder score).
+- **`verify(q,results)`** — a DeepSeek "paralegal" labels each result `relevant/partial/not`, seeing only a
+  ~280-char `passage_snippet` (NOT the holding). On JSON parse failure → everything defaults to `partial`.
+- **Grounding gate** (`grounded_answer`/`verify_claims`) — DeepSeek emits `{claim, n, quote}` triples; the gate drops
+  any claim whose `quote` isn't a ≥4-word verbatim substring of cited case `n`'s chunk; renders only survivors.
+  (Checks provenance, **not entailment** — a real quote can be misread; that's a known residual.)
+- **Good-law mask** is "dark": only confirmed-overruled is flagged; unknown shows nothing.
+
+### FAST mode — `GET /api/search_stream` (SSE, stepwise)
+- **F0 ROUTER** `identity_hits(q)` — if q is a bare **citation** (regex) or **"X v Y" name** (fuzzy `name_search`,
+  difflib + `cite_indeg` salience tiebreak), it's a *lookup* → metadata `id_card`, **bypass** the pipeline.
+- **F1 RETRIEVE** `candidates(q)` → ~48 hybrid chunks.
+- **F2 RERANK** cross-encoder → top 12 judgments.
+- **F3 REVIEW** `verify` → keep relevant+partial, drop `not`.
+- **F4 THIN-GUARD** if kept<4 → one DeepSeek query rewrite → re-retrieve/rerank/verify, add new.
+- **F5 ORDER** sort relevant-first, then by `rr`; top 8.
+- **F6 GOOD-LAW** dark mask. ⚠ **In fast mode this is currently a no-op** — the SSE label says "Checked which
+  results are still good law" but NO fast-path code filters/demotes overruled cases (deep mode does). Fix pending.
+- **F7 ANSWER** grounded summary (claims + verbatim-quote gate).
+
+### DEEP mode — `GET /api/deep_search_stream`
+- **D0 ROUTER** same identity bypass.
+- **D1 PLAN** DeepSeek decomposes the issue into 1–3 sub-issues + **names ≤5 leading authorities** a lawyer expects
+  (names only; each grounded via `name_search` — a hallucinated name simply fails to resolve).
+- **D2 SEED** `retrieve(q,12)` then `verify` → relevant/partial seed set.
+- **D3 EXPAND** (the only agentic step, one batch): (a) ground each named authority via `name_search`; (b) **citation
+  neighbours** — from top-6 seed docs, pull what THEY cite (`out_edges`), take 6 most-common; (c) `card_for_doc` per
+  added doc (reranks **only its first 6 chunks — `cis[:6]`, a real bug**: a landmark's holding is often deeper → it
+  gets mis-scored and dropped) → verify → keep relevant/partial AND not-overruled.
+- **D4 AUTHORITY-CHECK** report which named authorities landed (conflates "found but off-point" vs "not in corpus").
+- **D5 RANK** a reviewer-confirmed PLAN authority that's `relevant` gets a top slot; then relevant by `rr`; then
+  partials; top 10. ⚠ authority bonus is **binary + AND-gated on the snippet-derived `relevant` label**.
+- **D6 GOOD-LAW + ANSWER** same as fast.
+
+### Other endpoints
+`/api/judgment?id=` (full judgment view — metadata, issue/held headnote, citator, dark good-law, cleaned text, +
+**`/api/pdf?id=`** which pulls the official SCR PDF from the open registry and caches ≤20 locally, embedded inline),
+`/api/search` (non-stream fallback). The "ask this judgment" feature was removed.
+
+---
+
+## 3. Known weaknesses (panel-reviewed, grounded in serve.py) — prioritized
+
+The foundational-recall gap is **created in F1 (shallow 48-chunk pool)**, never recovered because **ranking ignores
+authority (F5/D5 use raw `rr`; `cite_indeg`/`bench`/`date` sit on every card unused)**, and actively *worsened* by
+**F3 deleting weakly-phrased landmarks on a 280-char snippet** and **D3's `cis[:6]` bug**. Plus the **F6 fast-mode
+good-law no-op** (a trust defect: an overruled case can rank #1 under a false "checked" label; only ~40 of 43,175
+docs are ever flaggable, `partly_overruled` in the D3 filter is dead code, `doubted` is wrongly omitted). Both
+models (BGE-small embedder, ms-marco reranker) are **general web English, not legal/India-tuned**.
+
+Full per-step verdicts + the prioritized sequence are in the memory file index and the panel transcripts (§9).
+
+---
+
+## 4. The AUTORESEARCH approach (the methodology — read this)
+
+Modeled on **Karpathy's AutoResearch** (Mar 2026): an agent runs experiments in a loop — *read code → propose ONE
+change → run a short job → measure ONE mechanical metric → `git commit` if it improved / `git revert` if not →
+repeat.* Three pillars: **(1) a hard constraint, (2) one mechanical metric, (3) autonomous propose/score/commit.**
+Plus his older "Recipe" discipline: become one with the data first, dumb baseline, **change ONE thing at a time,
+never add unverified complexity.**
+
+How we adapt it (the rules — do not break these):
+- **The metric is a FROZEN qrels file scored by pure arithmetic — NO LLM at score time.** This makes it
+  millisecond-cheap to recompute and **non-self-gradeable by construction**.
+- **The judge must NEVER be the serving model family.** Our live `verify` gate is DeepSeek, so DeepSeek is *banned*
+  from labeling the eval (that's why the old 0.77 number is inadmissible — `20_score_benchmark.py` graded
+  DeepSeek-with-DeepSeek). Labels came from **Claude** (a different family); **Codex** is the cross-family second
+  judge; the citation graph + known-item are model-free anchors.
+- **Commit a change only if its metric delta clears the paired-bootstrap 95% CI AND no guardrail regresses.**
+- **Cheap vs expensive experiments:** anything that only re-ranks (pool depth, authority α, reranker swap, the
+  ordering rule) is a ~65 s offline re-score → run hundreds. Anything that **re-embeds the 1.3M corpus** (swapping
+  the *embedder*) is expensive and gated behind a proven "landmarks enter the pool but rank low" signal.
+- **Precision-first:** a wrong overruled-case shown to a lawyer is the catastrophic error → the **bad-law@10**
+  guardrail and **known-item success@1** can never regress for a commit to count.
+
+---
+
+## 5. The EVAL RIG (the measurement substrate) — `phase1/eval/`
+
+**Frozen set: 800 queries** (`queries.tsv` + `qrels.tsv` + `bad_law_docids.txt`):
+- **500 SILVER** doctrinal / fact-pattern / vague — generated by **Claude subagents** (a Workflow, 25 agents) that
+  read a judgment's `held`/`issue` headnote and wrote 2 natural, **anti-leak** queries it answers (NEVER naming the
+  case/citation); gold = that source case (grade 3). `gen_sample.json` = the 250 sampled judgments. This is
+  **silver** → to be **audited by Hitin** (double-label a stratified ~200 sample, compute Cohen's κ, promote to
+  gold). The Claude+Codex dual-judge for graded multi-relevant labels (pooling top-k from fast/deep/BM25) is the
+  next eval upgrade.
+- **300 known-item** (150 neutral-citation + 150 case-name) — gold = own doc; the **success@1 control**.
+- **104 bad-law deny-list** (overruled/doubted/per_incuriam doc_ids) — the model-free **precision guardrail**.
+
+> ⚠ **Critical caveat:** the silver gold = a *random* source case, not a landmark → **this set measures GENERAL
+> retrieval, NOT foundational-authority recall**. The authority prior (PageRank/`cite_indeg`) must be tested on a
+> **landmark set** (extend `eval/gold_foundational.json`), where gold IS a landmark — on the silver set it
+> *catastrophically hurts* (see §7). Building a bigger landmark/foundational gold set is a top eval to-do.
+
+**Metric suite** (`score_qrels.py`, pure numpy, frozen, no network):
+- **PRIMARY: nDCG@10** (graded, 2^g−1 gain) + nDCG@5.
+- **GUARDRAILS: known-item success@1** (must stay ~1 once the router is in the harness) and **bad-law@10** (lower is
+  better; a commit that raises it is auto-rejected).
+- **DIAGNOSTICS:** recall@10, MAP@20, MRR, per-intent breakdown.
+- All with **bootstrap-over-queries 95% CIs**.
+
+**Harnesses:**
+- `lean_run.py` — serial, dense+CE, drops BM25 (the 68s killer), env knobs. ~0.74 s/q.
+- `batched_run.py` — **GPU-batched** (encode all queries → one dense matmul → ONE batched cross-encoder pass) →
+  **full 800-query run in ~65 s** (~9× the serial). This is the loop harness.
+- `sweep.py` — loads the corpus once and scores many (reranker × CAND × ALPHA) configs with paired bootstrap vs
+  baseline. The autonomous experiment driver.
+- `embed_chunks.py` — re-embed the corpus on GPU (exists but **too slow** ~80 min — transfer original vectors instead).
+
+---
+
+## 6. COMPUTE — where & how to run
+
+### The Windows GPU box (the experiment machine)
+`ssh admin@100.81.98.43` (tailnet). **RTX 5060 Ti 16 GB (Blackwell sm_120), 62 GB RAM, Python 3.11, default shell
+PowerShell.** App dir `C:\Users\admin\themis`. Setup gotchas you WILL hit:
+- My key is in `C:\ProgramData\ssh\administrators_authorized_keys` with an `icacls` perms lock — **required** or
+  sshd silently ignores it (admin accounts).
+- **torch must be `+cu128`** for Blackwell. The venv's old pip backtracks to the CPU wheel — `pip install -U pip`
+  first, then `pip install torch --index-url https://download.pytorch.org/whl/cu128` (NOT `--extra-index-url`,
+  which re-picks the CPU build). Current: `torch 2.11.0+cu128`, CUDA True.
+- **`PYTHONUTF8=1` is MANDATORY** — Windows `open()` defaults to cp1252 → `UnicodeDecodeError` on the legal text.
+- Box is set to never-sleep + `tailscale up --unattended` so it stays reachable logged-out.
+
+**Run an experiment (the loop in practice):**
+```powershell
+cd $env:USERPROFILE\themis
+$env:PYTHONUTF8="1"; $env:THEMIS_DATA="."; $env:THEMIS_EVAL="."; $env:THEMIS_DEVICE="cuda"
+# one config:
+$env:CAND="40"; $env:ALPHA="0"; $env:THEMIS_RERANKER="cross-encoder/ms-marco-MiniLM-L-6-v2"
+.\venv\Scripts\python.exe batched_run.py        # -> run.tsv  (~65s)
+.\venv\Scripts\python.exe score_qrels.py run.tsv
+# or sweep many at once:
+.\venv\Scripts\python.exe sweep.py
+```
+
+### Getting artifacts onto the box (it was painful — documented so you don't repeat it)
+Mac→box tailnet is DERP-relayed (~0.8 MB/s, useless). Mac can't reach Thor from its network. **Thor** (the old
+Jetson, now `100.99.130.27` tailnet, **flaky**) shares a LAN with the box → the box pulls Thor:`~/backup` over LAN
+(`192.168.1.76`, ~3–5 MB/s). We added the box's own SSH key to Thor (from the Mac, retrying through Thor's
+flakiness). **Re-embedding on the 5060 Ti is too slow (~80 min, CPU-tokenization-bound) — transferring the original
+`escr_vectors.npy` (~10 min) beats it and is exact.**
+
+### Other places it runs
+- **Local serve (the live app, for demoing/QA):** on the Mac, `.venv` (py3.12) + torch CPU, launched from the
+  artifacts dir: `THEMIS_LOG_DIR=… .venv/bin/uvicorn --app-dir phase1/scripts serve:app --host 127.0.0.1 --port 8000`
+  (artifacts opened CWD-relative; `scripts/.env` holds the DeepSeek key; no passcode on localhost).
+- **Pilot deploy:** `phase1/deploy/` has a ready Caddy + systemd + runbook for a Hetzner CPX41 (Caddy auto-TLS →
+  uvicorn + a passcode gate, `themis.apexflo.ai`). Serving is CPU-only — no GPU needed to host. Not yet provisioned.
+
+---
+
+## 7. Results so far (the experiment log — keep appending to this)
+
+| Config | nDCG@10 | recall@10 | succ@1 (known) | bad-law@10 | Δ vs baseline (paired bootstrap) | Verdict |
+|---|---|---|---|---|---|---|
+| **ms-marco, CAND=40, α=0** (baseline) | **0.517** | 0.595 | 0.31 | 0.037 | — | baseline |
+| ms-marco, CAND=100, α=0 | 0.532 | 0.611 | 0.34 | 0.036 | **+0.015 [+0.005, +0.026] ✓sig** | **COMMIT** (deeper pool helps) |
+| ms-marco, CAND=40, α=0.5 | 0.231 | 0.495 | 0.02 | 0.102 | −0.286 [−0.309, −0.262] | **REJECT** (authority prior wrong on silver) |
+| ms-marco, CAND=100, α=0.5 | 0.131 | 0.299 | 0.00 | 0.179 | −0.386 | **REJECT** |
+
+### 7b. The AUTHORITY slice — `authority_{queries,qrels,badlaw}` (150 landmark doctrinal queries)
+
+Built to make the rig *see* the foundational-authority gap the silver set hides (silver gold is the random
+source-case, not the landmark; so it actively penalizes authority). Gold = the doctrine's landmark (grade 3) +
+strong-citation progeny (grade 2). Run with `THEMIS_QFILE=authority_queries.tsv THEMIS_QRELS=authority_qrels.tsv
+THEMIS_QUERIES=authority_queries.tsv THEMIS_BADLAW=authority_badlaw.txt`.
+
+| Config | nDCG@10 | nDCG@5 | MRR | recall@10 | bad-law@10 | Verdict |
+|---|---|---|---|---|---|---|
+| **ms-marco, α=0** (baseline) | **0.282** [.253,.312] | — | 0.551 | 0.204 | 0.113 | baseline — the gap, quantified (vs 0.517 silver) |
+| bge-reranker-base, α=0 | 0.283 [.251,.313] | 0.315 | 0.553 | 0.195 | 0.127 | **REJECT** — Δ+0.001, a better topical reranker does NOT find foundational law |
+| ms-marco, α=0.3 | 0.354 [.324,.385] | — | 0.692 | 0.195 | 0.173 | win on nDCG but **bad-law regresses** (boosts overruled ex-landmarks) |
+| ms-marco, α=0.6 | 0.353 | — | 0.686 | 0.194 | 0.180 | α plateaus past 0.3 |
+| **ms-marco, α=0.3 + good-law filter** | **0.351** [.320,.384] | **0.416** | **0.693** | 0.193 | **0.000** | **COMMIT** — full nDCG gain held, bad-law → 0 |
+
+Reading — three decisive results:
+1. **Reranker swap is dead.** bge ≈ ms-marco (+0.001) on the exact slice it was meant to fix. Topical rerankers
+   under-score old-language landmarks regardless of model. Don't ship the 1.1GB model / 10× slower CPU pass.
+2. **The authority prior is the lever:** +0.072 nDCG@10 (+24% rel), MRR 0.551→0.692; recall@10 flat → it's a pure
+   **ranking** fix (landmarks were always in-pool, just low) — exactly the diagnosis the pool-probe gave.
+3. **It ships gated by good-law:** authority alone lifts bad-law (0.113→0.173); the denylist filter holds the gain
+   (0.351) and drops bad-law to 0.000. **Authority prior + good-law filter are one unit, never shipped apart.**
+
+The architecture consequence: the prior is **−0.29 on general silver, +0.072 on doctrinal** → it must be
+**query-routed** (ON for doctrinal/principle-seeking, OFF for known-item/fact lookup). That swing is the empirical
+mandate for a cheap LLM query-classifier at the front. Caveats: silver labels (Hitin audit pending); mild
+circularity (slice gold & prior both key off `cite_indeg` → magnitude may inflate, direction is sound); bad-law→0
+is only as real as the denylist, so it depends on F6 good-law being real in production.
+
+### 7c. STAGE 0 of the agentic build — routed authority prior + good-law spine (committed 2026-06-25)
+
+First stage of `AGENTIC_SEARCH_SPEC.md`. Folds the §7b lever into one *query-routed* spine and measures it
+end-to-end. Round-0 intent classifier (`classify_intent.py`, DeepSeek: AUTHORITY vs SPECIFIC) gates the prior;
+good-law filter drops the 43 confirmed-bad docs (`goodlaw_badlaw.txt`, the real product signal — not the ad-hoc
+`authority_badlaw.txt`). Harness: `spine_run.py` (loads corpus once, CE once/slice, derives baseline + spine).
+Routing fires on **69%** of authority queries, **6%** of silver — aggressive-OFF on general, as designed.
+
+| Slice | paired ΔnDCG@10 vs baseline | Δsucc@1 | bad-law@10 |
+|---|---|---|---|
+| **AUTHORITY (150)** | **+0.048 [+0.028, +0.070] SIG** | **+0.12 SIG** | 0.033 → **0.000** |
+| SILVER all-800 | −0.013 [−0.021, −0.006] SIG | −0.020 | 0.013 → **0.000** |
+
+The silver regression is **confined to `silver_doctrinal` (−0.036 SIG); factpattern, vague, known-item are all
+flat/ns.** It is an eval artifact: silver_doctrinal gold is the *arbitrary source case*, not the doctrine's
+landmark, so it penalizes us for correctly surfacing leading authority — the **same query type scores +0.048 on
+the authority slice where the gold is the landmark.** The real (correctly-specified) guardrails — known-item
+success@1, factpattern, vague, bad-law — all hold. **Verdict: PASS**; proceed to Stage 1.
+
+Open caveats logged for Hitin's audit: (a) classifier precision — it over-fires AUTHORITY on a few *narrow*
+doctrinal queries ("frustration in a statutory tenancy") where a specific case may beat the landmark; correct
+gold is needed to tune this. (b) good-law **coverage** — only 43 docs flagged bad in 38k; the filter is perfect
+within what's labeled but labeling is thin (the F6 dependency). (c) silver_doctrinal needs landmark gold to be a
+valid instrument for authority features.
+
+Files: `classify_intent.py`, `spine_run.py`, `goodlaw_badlaw.txt`, `intent_{authority,silver}.json`,
+`{authority,silver}_{baseline,spine}.tsv`.
+
+### 7d. STAGE 1 — the agentic controller (two-turn parallel) — PASS (2026-06-26)
+
+Tool-rich ReAct agent (founder-chosen over the binary router), shaped by the panel as **two-turn parallel**,
+not serial N-step ReAct: 1 plan LLM call (intent + expected authorities + statute refs + HyDE) → ALL retrieval
+tools fan out at once (`vector` + `authority` + `name_lookup`(plan authorities) + `statute`/`cases_on_section`
++ `hyde` + citation-`graph`) → merge → one batched CE rerank → authority-prior rank → good-law **flag-don't-drop
+-for-authority**. Files: `scripts/tools.py` (14-tool registry), `scripts/agent.py` (controller), `eval/agentic_run.py`.
+
+| Authority slice (150) | nDCG@10 | MRR | succ@1 | recall@10 | bad-law@10 |
+|---|---|---|---|---|---|
+| baseline | 0.282 | 0.551 | 0.380 | 0.204 | 0.033 |
+| spine (Stage 0) | 0.330 | 0.638 | 0.500 | 0.194 | 0.000 |
+| **agent** | **0.389** | **0.788** | **0.687** | 0.181 | 0.160 |
+
+- **agent vs spine paired ΔnDCG@10 = +0.0585 [+0.028,+0.089] SIG**; Δrecall@10 ns (no real loss). The controlling
+  authority is #1 in **69%** of queries (succ@1 0.500→0.687).
+- **Recall-recovery slice (8 landmarks OUTSIDE the dense pool): recall@10 = 0.75** vs spine ≈ 0 — the agent
+  (name_lookup of LLM-named authorities + graph + hyde) solves what one-shot retrieval structurally cannot.
+- **bad-law@10 0.160 is 88% artifact:** 23/26 hits are high-authority FALSE-POSITIVE landmarks (Maneka-type)
+  that flag-don't-drop keeps WITH A WARNING; only 3 are genuine dead law (≈ spine's leak). The headline rise is
+  the good-law data's mislabels, not a precision regression. **→ Hitin's good-law audit is now the #1 data item.**
+
+Latency: the Mac eval is ~11s/query (CPU, multiple sequential CE passes) — production (GPU box, async parallel
+tools) hits the panel's 15s budget. `name_lookup` indexed (token postings) so it's no longer O(corpus).
+NEXT: product path (2nd LLM turn = judge + streamed grounded answer + SSE steps), wire as serve.py adaptive
+endpoint (keep fast/deep as fallback), per-tool ablation (cut tools that don't earn latency), good-law audit.
+
+---
+
+## 8. The stepwise plan (roadmap — panel-prioritized)
+
+- **Step 0 — eval rig** ✅ (this doc / §5).
+- **Step 1 — cheap honesty/correctness** (ship together, low risk): make F6 good-law **real in fast mode** (demote/
+  tag the ~40 confirmed-overruled), fix the D3 good-law token set (drop `partly_overruled`, add `doubted`), log the
+  `verify` parse-failure no-op, sigmoid-normalize `rr`, share ONE F0/D0 router fn, emit D4 got/miss to the usage log.
+- **Step 2 — pool depth** (CAND↑ + diversity) — first signal positive (+0.015).
+- **Step 3 — ranking core (now data-backed, §7b):** ship the **authority prior** `sigmoid(rr)+α·log1p(cite_indeg)`
+  at **α=0.3**, **query-routed** (ON for doctrinal/principle queries only — it's −0.29 on general, +0.072 on
+  doctrinal) and **gated by the good-law filter** (the two are one unit). Needs (a) a cheap LLM query-classifier at
+  the front, (b) F6 good-law real. headnote-fed `verify` (down-weight not drop high-authority), fix `card_for_doc
+  cis[:6]`. **NOTE: the bge-reranker swap is REJECTED (§7b) — do not pursue; the reranker is not the lever.**
+- **Step 4 — router guards** (question-word before bypass; bare-name lookup) + A/B-and-maybe-cut the F4 rewrite.
+- **Step 5 — answer:** deep synthesis over top-8 + force-include confirmed authorities + a Qwen/cross-family
+  **entailment** gate after the verbatim gate.
+- **Stage 2 (post-pilot, measured):** swap to a **legal/Indic embedder** (needs a re-embed of the corpus —
+  expensive; note the **bge-reranker is already ruled out by §7b**, so this is an *embedder* bet, not a reranker one); the **citation-resolution rebuild** (it's a MISSING-KEYS problem — SCC resolves at 0% — not
+  name-matching; tiered exact-lookup spine + trust firewall); the **statute layer (BNS↔IPC / BNSS↔CrPC / BSA↔Evidence)
+  + unified intent→plan→tools** agentic search (collapse fast/deep into one adaptive-effort pipeline);
+  **PageRank / Personalized PageRank** for authority (gated on clean Tier-1/2 edges).
+
+---
+
+## 9. Rigor principles (the culture to keep)
+
+- **Verify against the real code/data — the panels repeatedly caught wrong assumptions** (e.g., "registry has SCC
+  cites" was false; "re-scrape the registry" buys 0 keys; the "76 overruled" figure was wrong — it's 40). Read the
+  file before you claim.
+- **Never self-grade.** Judge family ≠ serving family. Anchor in model-free truth (citation graph, known-item) where
+  possible; reserve LLMs for the gaps and audit them.
+- **One change at a time → paired-bootstrap CI → commit or revert.** No unverified complexity.
+- **Precision-first guardrails are non-negotiable** (bad-law@10, known-item succ@1).
+- **Reproducibility/backups:** artifacts live on the Mac (`phase1/data/thor_artifacts/`, byte-verified), Thor
+  `~/backup`, and the GPU box; code on GitHub. The eval set + qrels are frozen and versioned.
+
+---
+
+## 10. Pointers
+
+- Code: `phase1/scripts/serve.py`, `frontend.html`; eval: `phase1/eval/{build_qrels,score_qrels,lean_run,batched_run,sweep,embed_chunks}.py` + `queries.tsv`/`qrels.tsv`/`bad_law_docids.txt`; deploy: `phase1/deploy/`.
+- Design records (in the user's auto-memory): **themis-eval-rig**, **themis-citation-resolution**, **themis-agentic-search-design**, **themis-benchmark-v1**, **thor-gpu-embedding** (the box saga + gotchas).
+- The panel reviews (per-step search critique, eval-metric design, citation-resolution) ran as multi-agent Workflows; their full transcripts are in the session's `subagents/workflows/` dirs.
+
+**First thing to do when you pick this up:** ssh the GPU box, run `sweep.py`, read the bge-reranker-base row, and
+either commit it (if it clears the CI and guardrails hold) or move to Step 1. The loop is live — turn the crank.

phase1/INDIAN_KANOON_MIGRATION.md

@@ -0,0 +1,41 @@
+# Indian Kanoon corpus migration contract
+
+The case page and case chat are source-agnostic. They consume stable artifact contracts rather than
+AWS or Indian Kanoon URLs, so the frontend does not need another rewrite when the corpus changes.
+
+## Extraction output
+
+Keep `doc_id` stable across every artifact. In addition to metadata, paragraph-aware judgment text,
+source provenance, and embeddings, produce:
+
+`judgment_summaries.jsonl`
+
+```json
+{
+  "doc_id": "2024 INSC 123",
+  "summary": "A neutral 150–400 word case summary grounded only in the extracted judgment.",
+  "provider": "indian_kanoon",
+  "version": "ik-extraction-v1",
+  "generated": true
+}
+```
+
+`summary` may alternatively be a structured object with `facts`, `issues`, `holding`, `reasoning`,
+and `outcome`. The serving layer normalizes either form. Case chat receives only this normalized
+summary, never the full judgment or retrieval corpus.
+
+## Corpus-repair parity gate
+
+Before cutover, rebuild and validate every repair described in `themis-audits/corpus_repair.html`:
+
+- identity ledger, decision-year correction, text health, and sibling canonicals;
+- famous-name aliases with the doctrinal-query residue guard;
+- body-text citation graph and guarded parallel-citation crosswalk;
+- citation-context and HELD/headnote retrieval representations;
+- synthetic headnotes only where extracted summaries/headnotes remain missing;
+- edge treatment classification and rolled-up good-law status;
+- paragraph-aware chunks, pin-cite identifiers, FTS, dense vectors, and evaluation qrels.
+
+Run the same known-item, doctrine, factual, statute, bad-law, PDF/source, and lawyer-gold evaluation
+gates before switching the backend artifact revision. The UI will automatically prefer the new
+extraction summary and disclose its provenance.

phase1/METADATA_SCHEMA_CP2.md

@@ -0,0 +1,110 @@
+# CP2 — Themis canonical per-judgment metadata schema (LOCKED)
+
+*Converged via a 3-persona debate (paralegal / senior advocate / data-feasibility) + adversarial cross-examination + presiding synthesis. Claims were cross-checked against the on-disk 3k SC corpus. Anchored on the existing `JudgmentMetadata` (themisV2) + the V2 Postgres `case` table.*
+
+> LOCKED: Themis CP2 canonical per-judgment schema — 27 fields. The schema's job is threefold: pin the RIGHT case, weigh its AUTHORITY, and assert GOOD LAW only when an edge proves it. Two non-obvious but decisive locks, both confirmed against the on-disk 3k corpus: (1) equivalent_citations is a P0 LIST and the citator's second join key — INSC appears in only 0.9% of corpus and the good-law gold set is 100% reporter-keyed, so a neutral-citation-only graph cannot resolve its own ground truth or any pre-2023 landmark; (2) good_law_status is a derived, cached projection that DEFAULTS to 'unknown' (never 'good_law') with a 7-state enum, and bench_strength is a correctness gate read on both edge ends that FAILS SAFE — verified bench parses on only 54.5% of corpus, so a missing-coram overruling is downgraded to doubted/unknown, never validated. disposition is split from good-law as a closed enum (corpus shows real 'disposed'/'disposed of' free-text noise). HELD and issue are P1, verbatim-where-present, null-never-fabricated (HELD marker 3.8%, issue 0.2% off-SCR). advocates and topic_path taxonomy are deferred — zero extraction budget while the treatment-classification moat is unfinished.
+
+
+## P0 fields
+
+| field | roles | extractability | improves | source | notes |
+|---|---|---|---|---|---|
+| `acts (normalized list of canonical act IDs)` | filter/retrieval/citator | reliable | filter UX + known-item retrieval | SCR 'List of Acts' + body via act gazetteer; maintained alias table + IPC/CrPC/IEA<->BNS/BNSS/BSA crosswalk | Top-3 daily statute-anchored intent and the bridge to the statute layer. MUST normalize to canonical IDs and carry the old<->new crosswalk (a 'BNS 103' query must retrieve the IPC-302 line) or filters split across spelling variants. Reliable for marquee statutes via dictionary; long-tail/amendment acts best-effort + cheap-LLM stragglers. The crosswalk is a deterministic lookup the product owns. |
+| `appeal_no / case_number (LIST, with case-type prefix)` | filter/retrieval/dedup/display | moderate | known-item retrieval + dedup + filter UX | SCR/eCourts case-no field + cause-title; regex on type prefix + number + year | Essential for known-item lookup (client gives a diary/appeal number) and linking connected matters. MUST be a LIST — one judgment disposes of many connected numbers. Case-type prefix (Crl.A./C.A./W.P.(C)/SLP) seeds case_type. Verified noisy on legacy corpus ('Appeal (crl.) 197 of 198 197_198 -') — normalize best-effort, display/lookup-grade, don't over-trust as a clean filter. |
+| `bench / coram (list of judges, normalized)` | filter/retrieval/citator/display | moderate | filter UX + known-item retrieval + citator soundness | SCR CORAM/BENCH line; regex split + judge-authority list for canonical IDs | Serves judge-based retrieval and is the raw material for bench_strength. Verified: BENCH/CORAM line parses on 54.5% of legacy corpus (near-100% on SCR). Raw list extraction reliable; canonical judge-ID normalization (honorifics, initials, 'D.Y.' vs 'Dhananjaya Y') is the moderate part and needs a maintained authority list or the facet fragments. |
+| `bench_strength (int + bucket: single/division-2/full-3/constitution-5/larger-7+)` | filter/citator/display | reliable | citator soundness + filter UX + reranking | derived from len(bench); static size->label lookup | DECISIVE LOCK as a CORRECTNESS GATE, not just a 'Constitution Bench only' filter. Free to derive. Read on BOTH ends of every treatment edge: a smaller bench cannot overrule a larger one. FAILS SAFE: where bench is null (verified ~46% of legacy corpus) on either edge end, a treatment='overruled' is DOWNGRADED to doubted/unknown with a provenance flag — never validated as settled law. Reliability inherited from bench parse. |
+| `case_name (raw petitioner + respondent split + canonicalized display/fuzzy string)` | retrieval/display/dedup/filter | reliable | known-item retrieval + filter UX + dedup | SCR cause-title / pet+res structured fields; deterministic normalizer (abbrev/LRs/&Ors stripping) | Most-used human handle and the headline of every results row. Store BOTH raw pet/res AND a canonical fuzzy-match string ('Maneka Gandhi' <-> 'Maneka Gandhi v. Union of India'). Raw extraction reliable; the alias/normalized form is the work. |
+| `cases_cited (edge list of {to_neutral_citation, name, equivalent_citation, treatment, paras/pin-cite})` | citator/retrieval/filter/display | needs-LLM | citator soundness + known-item retrieval | SCR 'Case Law Cited' section for structure (regex); treatment via DeepSeek with conservative 'unknown' default | THE crown jewel and the citator spine — the extraction budget belongs HERE. Two hard rules: (1) every edge resolves to a neutral_citation (else graph unjoinable) — use equivalent_citations as the bridge for pre-2023 cited cases; (2) treatment in {relied_on, referred_to, followed, distinguished, doubted, overruled} defaults to 'unknown' rather than a guessed 'overruled' — get it wrong and you tell a lawyer dead law is live. Names+cites regexable; treatment is needs-LLM. Off-SCR the whole field drops to needs-LLM over body text. Pin-cite hook per edge for grounding. |
+| `court (SC / HC:<state>, specific forum)` | filter/retrieval/citator/display | reliable | filter UX + citator soundness + reranking | connector/provenance (NOT text) — constant 'SC' for Phase 1; eCourts court-code for HC phase | Highest-leverage filter AND an authority guard. Connector-set, not NLP-extracted, hence reliable at scale. MUST encode the specific HC (HC:KA vs HC:DEL), never bare 'HC' — the citator must never let an HC ruling read as overruling an SC one. Model now to avoid Phase-4 migration. |
+| `date (ISO YYYY-MM-DD)` | filter/retrieval/citator/display | reliable | filter UX + known-item retrieval + citator soundness | SCR 'DATE OF JUDGMENT' preamble line; deterministic format normalizer | Powers year/range filters AND the temporal-direction check in the citator (an overruling case must be LATER — auto-rejects impossible edges). Work is normalizing Indian formats (DD-MM-YYYY, '31st January 2000') to ISO; anchor on the labeled line to avoid grabbing reserved/registration date. Validate year against neutral_citation. |
+| `disposition (CLOSED enum: allowed/dismissed/partly_allowed/set_aside/remanded/acquitted/convicted/disposed)` | filter/retrieval/display | moderate | filter UX + reranking + citator soundness | regex over operative ORDER paragraph -> closed vocabulary; coarse reliable, nuance via LLM | REDEFINED from free-text 'outcome'. Verified noise on disk: 'disposed of' (447) and 'disposed' (67) are separate raw values — exactly why a closed enum is mandatory. STRICTLY SEPARATE from good_law_status (an 'allowed' case can now be bad law). Coarse result reliable from stock verbs; partly-allowed / per-party / split-outcome nuance is the LLM part. |
+| `equivalent_citations (LIST of {reporter, volume, page, year})` | filter/retrieval/citator/dedup | moderate | citator soundness + known-item retrieval + dedup | SCR 'Equivalent Citations' table / headnote; reporter regex + LLM fallback for free-text strings | DECISIVE LOCK. The citator's SECOND join key. Verified: good-law gold set is 100% reporter-cite-keyed (SCC/AIR) with 0% INSC; reporter cites appear in 44.2% of legacy corpus vs 0.9% INSC. A single citation STRING is a silent-lookup-failure; must be a LIST so cited-list strings resolve regardless of which reporter was pleaded. Era-dependent nulls (recent INSC-only judgments) are normal coverage. |
+| `good_law_status (DERIVED cached: enum {good_law, overruled, partly_overruled, doubted, per_incuriam, superseded_by_statute, unknown})` | filter/citator/display | needs-LLM | citator soundness + filter UX | computed projection of inbound cases_cited treatment edges + bench_strength validity; recomputed on ingest | DECISIVE LOCK and the product's headline differentiator. DERIVED, not extracted; cached projection refreshed on ingest. MUST DEFAULT TO 'unknown', never 'good_law' — absence of an overruling edge is not proof of soundness. 7-state enum mandatory because the gold set on disk contains a per-incuriam row and an amendment-reversal row a binary model would mis-weight. P0 because it is what the product is sold on; extractability needs-LLM because it depends on the treatment edges feeding it. |
+| `neutral_citation (YYYY INSC N)` | filter/retrieval/citator/dedup | reliable | known-item retrieval + dedup + citator soundness | SCR header/digiSCR; regex /(\d{4})\s+INSC\s+(\d+)/ | Canonical primary key and to-node of every citation edge. Verified present in only 0.9% of legacy 3k corpus — that is a coverage gap (pre-2023 predates INSC), NOT an extraction error. Reliable wherever it exists. NOT the sole join key (see equivalent_citations). |
+| `sections (list of {act, provision, number})` | filter/retrieval/citator | moderate | filter UX + known-item retrieval + citator soundness | SCR headnote/body regex; act-binding via proximity heuristic / cheap LLM | Finest-grained statute filter and a grounding check. Verified ~77% section coverage cited; structured in SCR. Keep {act, number} shape — 's.302' is meaningless without 'IPC'. Bare section list reliable; BINDING the section to the right act when context is ambiguous is the moderate part. Pairs with the BNS crosswalk. |
+
+## P1 fields
+
+| field | roles | extractability | improves | source | notes |
+|---|---|---|---|---|---|
+| `author_judge` | filter/retrieval/display | moderate | known-item retrieval + filter UX | first all-caps name heading after 'JUDGMENT:' marker; signature block; LLM for multi-opinion attribution | Distinct from coram; matters for weight and judge-search. Moderate because per curiam, 'by the Court', and separate concurring/dissenting opinions complicate 'the' author. Cheap, do not block schema on it. |
+| `case_type / nature (closed taxonomy: civil/criminal/writ/SLP/review/curative/...)` | filter/retrieval/display | moderate | filter UX + reranking | derived from appeal_no prefix + SCR subject; fixed lookup + small classifier | Useful coarse filter ('criminal matters only') and rerank signal. Keep strictly PROCEDURAL (from the case-number prefix); subject-matter/practice-area belongs under keywords, not here. Reliable when appeal_no is clean; small 'other' bucket for composite filings is fine. |
+| `cited_by_count + cites_count (graph degree)` | filter/retrieval/citator/display | reliable | reranking + filter UX | derived from corpus citation graph once cases_cited loaded | Free once cases_cited exists. Strong leading-case/authority proxy and at-a-glance badge (Indian Kanoon's 'Cited by 53834'). Excellent reranking feature. P1 — a signal, not a correctness requirement. |
+| `full_headnote (verbatim)` | retrieval/display/citator | reliable | known-item retrieval + citator soundness | SCR headnote block (raw HTML parse); store pointer per V2 object-store design | Rawest, most reliable artifact and the GROUND TRUTH every derived field is parsed from (provenance + grounding safety net). Bulk text — store the pointer, not in the hot metadata row. Structurally null off-SCR (HC/eCourts have no official headnote); product degrades to body-text chunking there. |
+| `held / ratio (operative holding)` | retrieval/display/citator | moderate | known-item retrieval + reranking + citator soundness | SCR headnote 'Held' VERBATIM where present; LLM grounded against headnote/operative para and FLAGGED as generated where absent | DECISIVE LOCK: its OWN field (not folded into short_summary) but P1 (senior's tier held). The sentence a lawyer quotes as 'what the case decided'. Verified HELD marker in only 3.8% of legacy full texts — off-SCR it is needs-LLM and MUST be grounded against source, clearly flagged generated, or left null. A fabricated holding is the exact liability a grounded citator exists to eliminate. |
+| `issue (issues for consideration)` | retrieval/display | needs-LLM | known-item retrieval + reranking | SCR 'Issues for consideration' header (reframe via LLM into atomic proposition); needs-LLM everywhere off-SCR | DECISIVE LOCK at P1 (paralegal conceded down from P0). Best single field to EMBED for fact-pattern retrieval but an embedding input, not a filter/authority signal — must not block schema. Verified 'issues for consideration' appears in only 0.2% of legacy corpus — needs-LLM everywhere off-SCR. NEVER FABRICATE: where absent, field is null and the row shows held/summary; do not let a thin data region become a hallucination surface in the results row. |
+| `reportable_flag (reportable / non-reportable)` | filter/citator/display | reliable | filter UX + reranking + citator soundness | SCR set-membership provenance (default True) + first-page REPORTABLE stamp regex for HC/eCourts | The corpus's defining authority boundary (Phase 1 IS the reportable set). Near-free from SCR provenance. Verified REPORTABLE stamp in 38.1% of legacy text — OCR/older-convention gaps make text-extraction moderate, but provenance default is reliable for SCR. P1 not P0: a near-constant in Phase 1, a real discriminator (and weaker citator node) only in the HC phase. |
+| `short_summary` | retrieval/display | moderate | filter UX + reranking | SCR case summary, or LLM from issue+held grounded on headnote | 2-3 line gist that makes the 20-row list scannable. Convenience layer — must NOT absorb the structured issue/held fields. Generated version must be grounded against headnote to avoid hallucinated holdings. |
+| `source_url / pdf_path / object_key / scraped_at / content_hash (provenance + dedup)` | display/dedup/citator | reliable | citator soundness (verifiability) + dedup | scraper output (bharat_courts/SCR portal); content_hash = SHA over canonical text; pipeline timestamp | Ship-blocking trust infrastructure: every surfaced proposition must deep-link to the official SCR page/PDF or the answer isn't verifiable. content_hash ADDED — the V2 idempotent/incremental ingest dedups by neutral_citation + hash and catches silently-revised judgments; without an explicit hash the contract is unenforceable. scraped_at drives re-good-law-check staleness. Not filters, but non-negotiable for a grounded citing product. |
+
+## P2 fields
+
+| field | roles | extractability | improves | source | notes |
+|---|---|---|---|---|---|
+| `advocates / appearances` | display | unreliable | filter UX (niche) | SCR 'Appearance' block where cleanly delimited — display-only, ZERO LLM extraction in Phase 1 | DECISIVE LOCK: DEFERRED. Capture as raw display-only text ONLY where it falls out of a preamble parse for free; build NO LLM extraction pass. Verified ~57% appearance-block coverage but parsing (senior vs AOR vs briefing counsel, multi-line, honorifics) is unreliable via regex. Changes no legal conclusion and competes for the SAME DeepSeek budget as treatment classification — the moat. Field kept forward-compat, never gates retrieval. |
+| `jurisdiction / state` | filter | moderate | filter UX (HC phase only) | derived from appeal_no/court for SC (near-constant); eCourts connector for HC state | DECISIVE LOCK at P2 / forward-compat only. For the national SC, jurisdiction is a near-constant and state is null — surfacing them as live filters now would imply meaning they lack. Define the columns so the HC shard (court x year x state) needs no migration; do NOT render as meaningful Phase-1 filters. |
+| `keywords (court's OWN catchwords only)` | retrieval/filter | moderate | known-item retrieval | SCR 'List of Keywords'/catchwords ONLY — never LLM-generated | DECISIVE LOCK: keep ONLY the court's own catchwords as a cheap BM25/tag-recall layer; LLM-GENERATED keyword lists are CUT — they are drift that competes with the full_headnote embedding. Coverage inconsistent. Demoted to P2. |
+| `lower_court / case_arising_from` | filter/retrieval/display | unreliable | known-item retrieval (appellate chain) | SCR preamble 'arises out of...' narrative; best-effort regex, NO dedicated LLM extraction | DECISIVE LOCK at P2. Genuine appellate-chain use (find the HC judgment under appeal) but lives in free-text prose, so reliable extraction needs LLM — and that budget belongs to treatment classification. Capture where it falls out of the preamble for free; null otherwise. Becomes a real filter in the HC phase. |
+
+## Recommended P0 filter set (UI)
+
+- court (SC vs HC, and which specific HC — the highest-leverage filter; binding vs persuasive turns on it)
+- year / date range (derived from neutral_citation, free; serves the 'approximate year' vague-recall intent)
+- judge / coram (judge-based retrieval) + bench_strength bucket (Constitution Bench / authority filter)
+- acts + sections (statute-anchored, normalized, carrying the IPC/CrPC/IEA <-> BNS/BNSS/BSA crosswalk)
+- disposition (closed enum: allowed/dismissed/set_aside/... — the 'who won' filter)
+- good_law_status (good/doubted/overruled/... defaulting to 'unknown' — the 'only good law' filter, the product's headline)
+- reportable_flag (the authority axis; near-constant in Phase 1, a real discriminator in the HC phase)
+- case_type (procedural: civil/criminal/writ/SLP) — coarse scoping facet
+
+## Resolved debates
+
+**Q: Is the clean INSC neutral_citation enough as the citator's join key, or must reporter cites (SCC/SCR/AIR) be a P0 first-class LIST field?**  
+→ LOCK equivalent_citations as a P0 LIST of {reporter, volume, page, year} — the citator's SECOND join key; neutral_citation stays the canonical primary key and to-node but is explicitly NOT the sole join key.  
+*Verified on disk: INSC appears in only 0.9% of the 3k corpus while the good-law gold set is 100% reporter-cite-keyed with 0% neutral — a neutral-only graph cannot resolve its own ground truth or any pre-2023 landmark, the exact cases the product exists to weigh. A single citation string is a silent-lookup-failure.*
+
+**Q: Should good_law_status be a stored filterable field now, and must court-action disposition be split from good-law status?**  
+→ LOCK both: a closed disposition enum (court action) kept STRICTLY SEPARATE from a derived, cached good_law_status that DEFAULTS to 'unknown' (never 'good_law') with a 7-state enum {good_law, overruled, partly_overruled, doubted, per_incuriam, superseded_by_statute, unknown}.  
+*An 'allowed' case can now be bad law — conflating them tells a lawyer dead law is live. Default 'unknown' because absence of an overruling edge is not proof of soundness. 7 states because the gold set on disk contains per-incuriam and amendment-reversal rows a binary model mis-weights. On-disk 'disposed'/'disposed of' duplication proves the free-text outcome must become a closed enum.*
+
+**Q: Is bench_strength a P0 correctness requirement or a cheap nice-to-have filter, and does the validity gate fail open or closed?**  
+→ LOCK P0 as a correctness GATE read on BOTH edge ends; it FAILS SAFE — a missing/smaller bench_strength on either end downgrades a treatment='overruled' to doubted/unknown with a provenance flag, never validating it.  
+*A smaller bench cannot overrule a larger one; without strength on both ends the citator emits legally void 'overrulings' as settled law. Free to derive (len(bench)). Verified bench parses on only 54.5% of legacy corpus — the gate must hold back, never wave through, an unverifiable small-bench overruling of a Constitution Bench.*
+
+**Q: Must HELD be a separately extracted field, and is it P0 or P1?**  
+→ LOCK HELD as its OWN field (not folded into short_summary) at P1; verbatim from the court's headnote where present, LLM-generated only when grounded against source text and flagged, null where it cannot be grounded.  
+*HELD is the sentence quoted as 'what the case decided' — burying it in a generic summary makes it unquotable. P1 (not P0) because it is a retrieval/display input, not a filter/authority signal, and must not block the schema. Verified HELD marker in only 3.8% of legacy texts — off-SCR it is needs-LLM; a fabricated holding is the precise liability a grounded citator exists to eliminate.*
+
+**Q: Is 'issue' a reliable P0 extract or a needs-LLM P1 field?**  
+→ LOCK P1, extractability needs-LLM, embed-where-present, NEVER fabricate — null off-SCR with the row falling back to held/summary.  
+*Best field to embed for fact-pattern retrieval but an embedding input, not a filter. Verified 'issues for consideration' appears in only 0.2% of legacy corpus — treating it as reliable P0 display would fill the results row (the surface a lawyer trusts at a glance) with empty or hallucinated issues exactly where data is thinnest.*
+
+**Q: Collect advocates in Phase 1, or cut as vanity?**  
+→ LOCK DEFER: P2, display-only, captured ONLY where it falls out of a preamble parse for free, with ZERO dedicated LLM extraction in Phase 1.  
+*Advocate names change no legal conclusion and 'cases argued by X' is a niche BD query, not a precedent-finding intent. Verified ~57% block coverage but parsing is unreliable via regex. It competes for the SAME DeepSeek budget as cases_cited treatment — the moat and highest-liability field. Spend the budget on treatment, not advocate-name parsing.*
+
+**Q: Keep keywords at all, and is the normalized topic_path taxonomy in Phase-1 scope?**  
+→ LOCK keywords at P2 — the court's OWN catchwords ONLY, never LLM-generated. DEFER the topic_path taxonomy out of Phase 1.  
+*An LLM-generated keyword list is drift that competes with the full_headnote embedding. A hand-maintained hierarchical taxonomy is real ongoing cost for a 'browse by area of law' intent that semantic search over the headnote already serves passably; build it only when browse demand is demonstrated.*
+
+**Q: Keep lower_court / jurisdiction / state in Phase 1, or defer to the HC phase?**  
+→ LOCK differentiated: lower_court P2 (free preamble parse only, no LLM); jurisdiction + state P2 / forward-compat columns only, NOT rendered as meaningful Phase-1 filters.  
+*lower_court's appellate-chain use is genuine but lives in free-text needing LLM budget owed to treatment. For the national SC, jurisdiction is near-constant and state is null — surfacing them as live filters implies meaning they lack until HC sharding (court x year x state) in Phase 4. Define the columns to avoid migration; extract nothing on spec.*
+
+**Q: Should appeal_no / case_number be P0 (paralegal/feasibility) or P1 (senior)?**  
+→ LOCK P0, stored as a LIST, with the case-type prefix retained as the source of case_type.  
+*Essential for known-item lookup before a neutral citation exists and for linking connected matters; one judgment disposes of many numbers so a scalar loses lookups. Verified noisy on the legacy corpus — normalize best-effort, treat as lookup/display-grade not a clean filter.*
+
+**Q: Is content_hash needed given it was only implied by the V2 design?**  
+→ LOCK ADD content_hash (SHA over canonical text) as a P1 dedup field alongside source_url/pdf_path/scraped_at.  
+*The V2 ingestion DAG dedups by neutral_citation + content hash and only re-embeds changed docs; without an explicit hash field the idempotent/incremental contract is unenforceable, and it catches silently-revised judgments.*
+
+
+## Open questions (need a human / lawyer call before the citator writes states)
+
+- good_law_status edge-case modeling: the gold set contains a 'per incuriam' row and a 'reversed mainly by constitutional amendment, not a clean SC overrule' row. The 7-state enum names these, but the DERIVATION logic (what edge pattern + bench-strength condition yields per_incuriam vs superseded_by_statute vs partly_overruled) is unspecified and needs a human-authored rule set with a lawyer in the loop before the citator writes these states.
+- Citator join across the era boundary: cases_cited edges from pre-2023 judgments reference cited cases by reporter cite only. Resolving those to a neutral_citation to-node depends on a complete equivalent_citations index — needs a decision on what happens when a cited case has NO neutral citation at all (legacy-only landmark): does the edge resolve to a reporter-cite-keyed node, and is that node a first-class citator vertex?
+- Fail-safe-closed gate calibration: downgrading every missing-coram overruling to doubted/unknown is safe but will suppress real overrulings on ~46% of the legacy corpus where bench doesn't parse. Worth a human decision on whether to prioritize a bench-line backfill pass (re-parse / LLM) for high-inbound-degree cases so landmark overrulings aren't silently withheld.
+- HELD/issue off-SCR fallback UX: the rule is null-never-fabricate, but the product still needs a defined results-row fallback order (held -> short_summary -> first headnote sentence -> ??) for the majority HC/eCourts corpus where none of issue/held/headnote exist. Needs a UX + grounding decision.
+- Whether disposition's per-party and split-outcome nuance ('allowed for appellant A, dismissed for B; remanded on one issue') is in Phase-1 scope or deferred — the closed enum captures the headline result but multi-appeal judgments carry split outcomes the single enum value cannot represent.
+- Treatment-classification confidence threshold and human review: cases_cited treatment is the highest-liability field. Needs a decision on the DeepSeek confidence cutoff below which an edge stays 'unknown', and whether overruled/doubted classifications get a mandatory human-review queue before they flip a downstream good_law_status.

phase1/SESSION1_CHANGE_PLAN.md

@@ -0,0 +1,121 @@
+# THEMIS — DECISION-GRADE CHANGE PLAN
+(Panel chair synthesis. All 15 expected cases are in corpus; every failure below is pipeline, not coverage.)
+
+---
+
+## 1. WHAT THE 11 QUERIES PROVE
+
+**Root cause A — Query-side vocabulary gap → pool-recall failure (dominant: 5/11 queries).**
+The lawyer writes fact-narratives ("summons never served", "document discovered after evidence closed"); the controlling judgments speak statute (O9R13, O41R27, s.14 Limitation, s.43 TPA). BGE-small + BM25 cannot bridge this, so the gold case never enters the pool: **Q4** (Nortel), **Q5** (Ibrahim Uddin), **Q7** (Parimal), **Q3** (Avitel), **Q11** (Lala Durga Prasad). Two sub-forms:
+- *Enumerable procedural postures* (Q4/Q5/Q6/Q7): a fixed, small set of posture→controlling-authority mappings every junior lawyer knows. Q6 is the ranking variant of the same gap (Revajeetu in pool but 4th).
+- *Doctrinal issue statements* (Q9 s.43 ostensible-owner, Q11 s.44 co-owner alienation): the issue phrase appears nowhere in the query; raw-text vectors match scenery, not ratio.
+Corollary: **the Q4/Q5 abstentions were honest-given-pool but fired on a repairable pool** — abstain is downstream of retrieval with no repair loop. Plus one deterministic recall hole: **Q5's "Ibrahimuddin" ≠ "Ibrahim Uddin"** name-index bug (no LLM involved).
+
+**Root cause B — Authority signal starvation (3-4/11 queries, ranking mode).**
+63k edges over 37,898 judgments ≈ 1.7 edges/judgment vs a true ~10-20 cites/judgment (~500k-1M edges): <15% of the graph extracted. The lawyer's Q1 rubric — "most COMMONLY USED first, weighted by how many later orders relied on it" — is literally usage-weighted in-degree with treatment labels, and it is currently **uncomputable**. This is why Revajeetu sits 4th (**Q6**), Thomson Press falls to the extended tier (**Q10**), Q1/Q8 ordering disappoints, and cited-by expansion from Ayyasamy/World Sport cannot pull in Avitel (**Q3**). It also starves the ranker of deterministic tiebreaks, feeding root cause C.
+
+**Root cause C — Lawyer-visible nondeterminism (Q2: rank 1 → rank 4 on identical re-run).**
+Final order currently emerges from an LLM judge plus tie-breaking on noisy rerank scores. Where the variance enters is **unverified** — the panel's bet is frame() naming different landmarks per run, not the judge — so it must be localized before it's fixed. The litigator's line is the product truth: rank-4-instead-of-1 is forgivable; a different answer to the same question is a slot machine, and trust never recovers.
+
+---
+
+## 2. THE CHANGES (value-per-effort order; ⟂ = runs in parallel)
+
+**C0 — Instrumentation gate + frozen lawyer-gold slice.** *Days 1-2. Nothing else ships before this exists.*
+- Freeze the 11 queries + 15 expected authorities as named must-pass assertions (not an aggregate — n=11 has a 0.28-0.79 CI; one flip = 9 points).
+- Log per stage: frame() output, per-lane pools, **PoolRecall-pre-judge per expected case**, judge order. Metrics: Expected@3, nDCG@5 against the lawyer's stated Q1 ordering, Stability@5 (5× runs, ~$0.50, ~20 min; report top-3 Jaccard + max rank displacement), abstain-rate conditional on PoolRecall.
+- Run nightly. Baseline today: ~9/15 expected cases reach the pool. Guard the 78q benchmark with paired per-query McNemar over 3 averaged runs (±2-3pt variance swamps small deltas otherwise).
+- *Fixes:* nothing directly; prevents misattributing every other fix. *Cost:* 1-2 eng-days, ~$0.50/run.
+
+**C1 ⟂ — Name-variant normalizer.** *Half a day.*
+- Known-item index: whitespace/punct-collapse, honorific stripping (Shri/Smt/M/s), -uddin/-ud-din collapse, Mohammad/Md variants, v./vs./versus; edit-distance ≤2 fallback. Alias table later enriched by C4's observed citation strings.
+- *Fixes:* Q5 lookup bug. *Measured:* variant unit tests (both spellings exact-resolve), zero regression risk. Both spellings added to the frozen slice.
+
+**C2 — Posture→controlling-authority spine + lookup-before-abstain.** *Week 1; ~2-3 hrs founder + 2-3 eng-days.*
+- ~30 YAML rows, founder-authored: {posture phrase, provision, controlling docid, 1-line ratio}. Rows enter by PR; CI acceptance = docid exact-resolves AND one deep-read returns verdict=controls on a canonical posture query. Every future session failure adds a row.
+- frame() gains a `procedural_posture` enum field (same single DeepSeek call — zero marginal cost/latency). Spine hit injects the mapped case as a **protected candidate that must still earn rank via deep-read verdict=controls** — never a hard slot-1 pin (a misclassified posture must not override the grounding gate).
+- **Reorder the abstain path:** weak-pool signal (all reads background/irrelevant OR rerank margin < threshold) → posture lookup + one hint re-retrieval + deep-read of injected case → only then abstain. Repair path costs +$0.01-0.02, +5-10s, abstain-path only.
+- *Fixes:* Q4, Q5, Q7 (recall), Q6 (ranking). *Measured:* PoolRecall 4/4 on Q4/5/6/7; Expected@3 ≥9/11; predicted hit@1 0.71→~0.78. Additive, 78q-safe.
+
+**C3 — Determinism package.** *Week 1, overlapping C2. Details in §4.*
+- *Fixes:* Q2. *Measured:* Stability@5 max displacement ≤1. **No ranking change lands before this passes** — you cannot attribute ranking gains while run swings exceed effect sizes.
+
+**C4 ⟂ — Single full-corpus batch pass (launch the overnight job NOW; integration gated).** *~$125, 6-10 hrs at 60-80 concurrent; zero query-time cost.*
+- Step 1 (free, same day): regex skeleton over SCC/AIR/SCR patterns — validates the density hypothesis and serves as a cross-check, **not** the extraction (Indian SC judgments cite half their authorities by bare name; regex gives no treatment labels, no aliases).
+- Step 2: one DeepSeek read of all 37,898 full texts, **three outputs per judgment**: (a) every cited case, raw string + resolved docid + treatment label {followed/relied/distinguished/overruled/referred}; (b) raw citation strings as observed aliases → feeds C1's table systematically; (c) posture + issue tags (O6R17, O9R13, O41R27, s.14 Limitation, s.43/s.44 TPA…) + 1-line holding.
+- **Integration gates before anything touches lawyer-visible ranking:** in-edge sanity on 20 known landmarks; treatment-label precision ≥0.9 on a 100-edge hand audit.
+- *Fixes:* enabler for Q1/Q8/Q10 ordering, Q3-Avitel via cited-by expansion, Q9/Q11 via issue tags, Q5-class aliases. *Measured:* standalone sanity metrics first, then C5's gates.
+
+**C5 — Authority prior + the ranking objective (§3) + cited-by expansion in the doctrine lane.** *Week 2, after C3 passes and C4 audits clear.*
+- *Fixes:* Q1, Q6, Q10 ordering; Q3 Avitel recall. *Measured:* 11-slice must-passes + McNemar'd 78q over 3 runs.
+
+**C6 ⟂ — "Deeper explanation" card for the top pick.** *~1 day.*
+- Surface what deep-read already produced: the test laid down, paragraph pinpoint, one line on later-bench application, why it controls THIS posture. Plus a "cite this in court for X" tag per pick.
+- *Fixes:* Q3 feedback ("no deeper explanation"). *Measured:* qualitative; founder review.
+
+**C7 — CONDITIONAL: Workstream B issue-vector arm (~$200).** *Only if Q9/Q11 still fail after C4's issue tags are embedded as a retrieval arm and C5's graph expansion lands.* The batch pass already extracts issue/holding text — embed that first; don't pay for a second full read.
+
+**Expected end state:** session slice ~5-6/11 clean → ~9-10/11; hit@1 ~0.71 → ~0.78-0.80; zero lawyer-visible rank flips.
+
+---
+
+## 3. THE RANKING OBJECTIVE (the lawyer's rubric, made implementable)
+
+The Q1 rubric = **usage-weighted, coverage-diverse, minimal ordering with courtroom-use tags**. It lives in a deterministic assembly step AFTER the judge — never in the judge prompt (with a 63k-edge graph the LLM cannot estimate "most used" and will hallucinate it).
+
+**Division of labor:** the judge (and deep-reads) emit *verdicts and facet annotations only*. Final order is a pure function of the pool:
+
+```
+sort key (descending priority):
+  1. verdict tier          controls > supports > background   (from deep-read)
+  2. spine/seminal flag    lawyer-authored spine hit           (from frame() lookup)
+  3. authority prior       Phase 1: raw recency-weighted in-degree (post 20-landmark sanity)
+                           Phase 2: treatment-weighted in-degree per doctrine cluster,
+                                    relied/followed only (post ≥0.9 label audit)
+  4. year (desc), then docid   — guarantees a total order, no residual ties
+```
+
+**Slot 2 = maximum marginal coverage:** MMR over ratio embeddings among controls/supports — the heavily-used case answering the facet slot 1 does not (Wakf-Board-after-Lakshmi-Reddy behavior, deterministic).
+**Stop rule:** cut at 3-4 picks once frame()'s issue facets are covered; do not pad to 6. Extended tier absorbs the rest.
+**Per-pick tag:** "cite this in court for X" — ratio + pinpoint from the deep-read verdict card.
+**Explicitly deferred:** "usage-for-THIS-proposition" attribution (litigator's key 1 in full). It stacks unvalidated proposition-alignment on unaudited treatment labels. Raw in-degree ships first; treatment weighting graduates after the 100-edge audit; proposition-level attribution only if a measured gap remains.
+
+---
+
+## 4. STABILITY FIX (Q2 must never happen again)
+
+Ordering must be a **pure function of the query** — LLM stays where it is invisible and cacheable; determinism owns everything the lawyer sees.
+
+1. **Localize first (one afternoon, ~$0.50):** run the 11 queries 5×, diff frame() output / lane pools / judge order per run. Panel's bet: frame() names different landmarks run-to-run, changing the pool itself — if true, judge-side fixes alone change nothing.
+2. **Upstream:** frame() at temperature 0 AND cached per normalized query (post-C1 normalization). Same query ⇒ byte-identical frame ⇒ identical pool.
+3. **Downstream:** judge emits verdicts/slots only; final rank from the §3 sort key, terminating in docid — total order by construction, ties impossible.
+4. **Middle:** cache deep-read verdicts per (query-hash, doc-id).
+5. **Verify, don't assume:** Stability@5 nightly on the frozen slice; targets: max displacement of expected cases ≤1 (Q2's swing was 3), top-3 Jaccard ≈ 1.0.
+6. **Only if residual verdict flips still break Stability@5:** add majority-of-3 on verdict disagreement (parallel, ~+$0.01/q). Not before — three panelists independently flagged it as premature spend on an unlocalized hypothesis.
+
+---
+
+## 5. WHAT NOT TO DO
+
+- **No majority-of-3 judging now** — deterministic sort removes ordering variance; voting is a contingency, not a default.
+- **No hard slot-1 spine pinning** — spine rows are protected candidates that must earn verdict=controls; the grounding gate stays the arbiter. A misclassified posture must degrade gracefully, not inject a wrong "controlling" case.
+- **No regex-only citation graph as final extraction** — misses name-only cites ("in *Ayyasamy*"), carries no treatment labels; ranking by raw untreated in-degree promotes heavily-*distinguished* cases, and citing a distinguished authority as controlling loses the lawyer credibility in court. Regex = skeleton + sanity check only.
+- **No graph feature in ranking before audits pass** (20-landmark in-edge sanity; treatment precision ≥0.9 on 100 hand-labeled edges).
+- **No "most used" estimation inside the judge prompt** — the LLM will confabulate usage the graph can't support.
+- **No proposition-level usage ranking yet** (see §3).
+- **Don't fund Workstream B's separate $200 summary pass yet** — C4's issue tags ride the same read for free; check Q9/Q11 against them first. Never pay for the same full-corpus read twice.
+- **No bigger index, no new embedding model, no re-embedding** — coverage is proven not to be the problem; every dollar there is misdirected.
+- **Don't treat n=11 as a statistical gate** — named must-pass assertions only; never edit frozen golds (freeze weekly snapshots as sessions accrue); don't trust the n=7 statute slice; McNemar-guard the 78q.
+- **Don't book unmeasured wins** — "+10-15 recall@20 for summary arms" is a literature prior, not a Themis measurement.
+
+---
+
+## 6. THE ONE FOUNDER DECISION
+
+**Commit Themis to a founder-curated editorial layer: the posture→controlling-authority spine, with you as its named editor.**
+
+Everything else in this plan is engineering the panel already converged on ($125 batch job, determinism, normalizer — none founder-level). The spine is different in kind: each row is Themis asserting, on your professional authority, "*this* is THE controlling case for *this* posture." That makes Themis a hybrid curated-plus-retrieval product, not a pure algorithm — with your ongoing obligation attached (~2-3 hours now for 30 rows, ~15 min/week as session failures add rows via PR, and ownership of any row that's wrong in front of a lawyer).
+
+Decide yes/no on that commitment. If yes, it converts 4 of 11 session failures this week at near-zero cost, and the curated spine + treatment-labeled graph together become exactly the citator-style moat the competitive analysis said Themis needs. If no, Q4/Q5/Q6/Q7-class queries wait on the slower, less certain representation track.
+
+The panel's recommendation is **yes** — with the safeguard already built in: spine entries are protected candidates gated by deep-read verdicts, so your editorial judgment is always checked against the text of the judgment itself before a lawyer sees it.

phase1/TWO_LAYER_PLAN.md

@@ -0,0 +1,83 @@
+# THEMIS — DECISION PLAN: THE TWO-LAYER ITERATION
+
+## 1. THE CORE INSIGHT
+
+The lawyer's observation diagnoses where the pipeline is blind, not what it lacks: retrieval already finds the right cases (hit@any 0.85), but every decision that matters — ranking, relevance verdicts, abstention — is made by a judge reading 700 characters. Moving comprehension to full judgment text at decision time (Workstream A) attacks the 0.85→0.55 gap, the query-4 false-authority failure, and quote quality simultaneously. But deep-read's ceiling is exactly hit@any: it cannot rank what retrieval never surfaced, so Workstream B separately owns the 15% recall gap (query-3 class). Fund both; never expect one to do the other's job.
+
+## 2. WORKSTREAM A — THE DEEP-READ AGENTIC LOOP
+
+**Trigger policy.** Shallow path stays default. Go deep when: doctrine/statute lanes dominate the candidate pool, cross-encoder top-1 margin is flat, or judge confidence < threshold. Never deep-read when the known-item lane fires (it's at 1.00). "~40% of queries go deep" is a hypothesis to measure, not a design input.
+
+**Which docs.** Top-8 post-cross-encoder candidates, union across lanes, dedup by case.
+
+**What is read (chair ruling on tiered-vs-full).** Full judgment text up to a 25k-token cap — this covers the large majority at avg ~17k tokens. Legal-researcher is right that in multi-issue SC judgments the controlling passage sits mid-judgment, exactly where head/tail packs go blind, and the cost delta ($0.013 vs $0.037) is noise. Latency-systems is right that *output* tokens are the latency tail — so read full, extract lean. Only above-cap monsters (Kesavananda-class) get the tiered pack: HELD + first 2k + last 3k + ±1.5k around every hit chunk, with escalation hard-capped at 2 concurrent, picked by reranker score.
+
+**Extraction (one DeepSeek call per doc, all 8 parallel, strict JSON, prompt ordered `[judgment][query]` for prefix caching — bake this in now, retrofitting invalidates the cache):**
+```
+{verdict: controls|supports|background|irrelevant, confidence,
+ ratio_one_liner, exact_passage (verbatim), what_it_does_NOT_decide,
+ missing_doctrine_hint}
+```
+No treatment extraction (followed/distinguished/overruled) in the query path — that's an offline batch job.
+
+**Feeds judge + grounding.** judge() ranks from cards, not chunks. `exact_passage` passes through the existing substring gate unchanged — the one hard rule survives, and pass rates should rise because the reader saw real text.
+
+**Cost/latency.** Worst case ~136k input ≈ $0.037/query; wall clock ≈ slowest parallel call. Deep mode lands ~30–35s total; SLO is p95 ≤ 60s, not the mean. SSE progress mandatory: candidates at ~5s, each read as it completes, answer last.
+
+**Honesty mechanism (query 4).** Three-tier output: **controlling** / **persuasive-only** (weak authority, flagged as weak) / **no SC authority** — the last emits a fixed template: "No strong Supreme Court authority found — this area has developed principally in High Courts." Initial bar: abstain when no card reaches supports/confidence ≥ 0.6 — but the threshold is *swept* on the 15 no-answer benchmark queries and reported as a FAR-vs-hit@3 frontier; pick the operating point from the curve, never tune one side alone. `what_it_does_NOT_decide` is what makes "the bail case is irrelevant" detectable.
+
+**Query-3 fix (missed controlling doctrine).** Primary, deterministic: add a relief-sought/procedural-posture field to frame() plus a ~30-row remedy→controlling-authority spine curated by Hitin (posture = quashing → Bhajan Lal / Pepsi Foods / Neeharika enter the doctrine lane as known-items). Validated on *held-out* posture queries, not the 4 we've seen. Fallback for uncovered postures: exactly ONE bounded `missing_doctrine_hint` re-retrieval round, then stop.
+
+## 3. WORKSTREAM B — REPRESENTATION UPGRADE
+
+**What to embed.** Chunks stay (grounding gate needs them; known-item/fact already 1.00/0.82). Add ~3 doc-level vectors per judgment — issues / holding / facts — from DeepSeek-generated structured JSON, anchored on HELD where present (56% checkable). ~114k new vectors beside 1.3M; minutes to embed on the 5060 Ti. Field-routed from frame(): `doctrine_issue`→issues vector, `fact_query`→facts vector, added as an RRF arm, never a replacement. **Summaries are retrieval keys only, never evidence** — judge and grounding operate on original text, so a hallucinated summary can only inject a candidate that downstream filters kill. Share one schema between deep-read cards and summary JSON so query-time reads back-fill the index. ISSUE field (3%) is dead; ignore it.
+
+**Generation plan + cost.** Full corpus ≈ 430–650M input tokens ≈ $175–200 (off-peak batch discount can roughly halve it); monsters map-reduced at a 60k cap.
+
+**Ablation-first protocol.** Stage 0 (today, $0): HELD-vector RRF arm in the doctrine lane; run the frozen rig. Stage 1 (~$27): 5k-judgment pilot — all gold docs for the rig subset + random distractors — one variable at a time: baseline (0.517) vs +HELD vs +summary vs both. Stage 2: corpus-wide spend only on pilot pass.
+
+**Expected lift / kill criteria.** Hypothesis: +3–6 nDCG on doctrine queries, hit@1 0.55→~0.62. Kill if: <+0.02 nDCG@10 on the 800q rig, OR any lane regresses (known-item must hold 1.00), OR hit@3 not non-inferior on the 128q set. No re-embedding of 1.3M chunks and no embedder swap until added-lane gains plateau.
+
+## 4. WHAT NOT TO DO
+
+- **Per-query treatment extraction** (followed/distinguished/overruled cards) — a corpus-annotation project smuggled into the query path; +5–8s of generation per call. Do it offline later; it repairs the citation graph as a byproduct.
+- **The (case_id, issue_hash) extraction cache and three-layer cache plumbing** — pilot queries are near-unique; hit rate ~0. Keep only DeepSeek prefix caching. Revisit at real traffic.
+- **The 50-entry HC-dominant-doctrine list** before measuring whether reader-abstention alone hits FAR ≤5% — curated lists are maintenance debt and tuning-on-the-test-set; buy them only if the model can't earn the gate, and evaluate on held-out queries.
+- **Corpus-wide summary spend before the $27 pilot gate.**
+- **Open-ended agent loops** — one bounded re-retrieval round, then stop.
+- **PageRank on the 63k-edge graph** (already rejected; the graph is under-extracted, not under-ranked).
+- **Deep-reading the known-item lane** (it's at 1.00; only downside risk).
+
+## 5. BUILD ORDER
+
+1. **Freeze the 128q benchmark + FAR metric** (week 1; Hitin authors 25 real queries, 15 no-answer, 10 controlling-doctrine; 78q absorbed). Blocking for every ship gate below. *Runs parallel with step 2.*
+2. **Free wins, $0** (week 1, *parallel with 1*): widen judge window to HELD + ~3k chars per candidate (+$0.004/query, +2s); add the HELD-vector RRF arm to the doctrine lane. Validate on the frozen 800q rig; re-baseline on 128q once frozen. This is experiment zero — if hit@1 moves to ~0.60 on context alone, the deep-read mechanism is confirmed before any architecture is built.
+3. **Posture field + remedy→authority spine** (week 2, *parallel with 4*): Hitin curates ~30 rows over a weekend; gate on held-out posture queries.
+4. **Deep-read mode** (weeks 2–3): three-arm A/B on all 128q for <$10 — 700-char vs widened vs deep-read. Ship gates: hit@1 ≥ +0.05 over the *widened* arm, pick-change rate ≥15%, p95 ≤60s, plus 40 blind lawyer-judged why-line pairs (never DeepSeek grading DeepSeek).
+5. **Abstention operating point** (week 3, rides on 4): sweep the threshold, publish the FAR-vs-hit@3 frontier, ship the three-tier template. Gate: FAR ≤5% AND hit@3 non-inferior on the same frozen run.
+6. **Summary pilot → corpus** (weeks 3–4, *parallel with 4–5*): $27 5k-judgment pilot; ~$200 corpus-wide off-peak run only on ≥+0.02 nDCG@10 with known-item = 1.00 guard.
+
+## 6. THE ONE FOUNDER DECISION
+
+**Adopt FAR ≤5% (jointly with hit@3 non-inferiority) as the pilot go/no-go gate — committing Themis to answering "no strong SC authority, likely a High Court matter" rather than ever confidently citing a weak case, even at the cost of a few points of answer rate.** This single call fixes the abstention operating point, defines pilot success, and commits Hitin's ~3 days of scarce time (benchmark authoring, spine curation, blind grading) that the entire measurement discipline depends on. Everything else in this plan is reversible engineering; this is the product's identity — a lawyer forgives a miss, never a confident wrong citation.
+---
+
+## AMENDMENT (post-Hitin call): the eval-set reality
+
+Hitin's input: **building an eval set is very hard because even lawyers don't know the not-so-famous
+precedents** — settled law and famous precedents anyone can find; the tail is beyond recall. This is
+simultaneously the product's value proposition and the reason gold labels can't be authored directly.
+
+**Recalibrated gold strategy — recall is hard, verification is easy:**
+1. **Settled/famous** — Hitin labels directly (fast; also the 30-row posture→authority spine is
+   settled doctrine, squarely inside what lawyers know cold).
+2. **Long tail — gold by construction:** reverse-generate queries FROM known cases (the silver-set /
+   78q methodology): the source case IS the answer, no recall required. Primary tail-gold source;
+   generate harder variants (multi-issue, posture-heavy) the same way.
+3. **Real queries — pooled verification (TREC-style):** collect real lawyer queries (cheap), run
+   Themis (+ competitor side-by-side), Hitin BLIND-GRADES pooled results relevant/not — each session
+   incrementally builds qrels. Relative judgment (A vs B) is easier still.
+
+**Build-order change:** Step 1 becomes "Hitin supplies queries + blind-verifies pooled results"
+(not "authors gold answers"). His scarce hours go to verification, the only method that scales to
+the tail. The FAR/no-answer set is unaffected (knowing "this is HC territory" is what lawyers DO know).

phase1/deploy/Caddyfile

@@ -0,0 +1,11 @@
+themis.apexflo.ai {
+	# Auto Let's Encrypt TLS for the subdomain (needs the DNS A record + ports 80/443 open).
+	# uvicorn listens only on 127.0.0.1:8000; Caddy is the public edge.
+	reverse_proxy 127.0.0.1:8000 {
+		flush_interval -1   # stream SSE (search_stream / deep_search_stream) without buffering
+	}
+	encode gzip
+	request_body {
+		max_size 5MB
+	}
+}

phase1/deploy/DEPLOY.md

@@ -0,0 +1,68 @@
+# Themis cloud deploy runbook (Hetzner CPX41 + Caddy + themis.apexflo.ai)
+
+Serving is **CPU-only** — no GPU. The box just loads the prebuilt artifacts and serves.
+
+## Prereqs (George)
+- Hetzner CPX41 (8 vCPU / 16 GB / 240 GB), Ubuntu 24.04.
+- My key in `~/.ssh/authorized_keys`: `ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKHKQSaIlbdEE3b+46UWMn6E2+8kplUvOuseq7+ikQYx claude-thor-access`
+- DNS: `themis.apexflo.ai` A record → box IP.
+- Hetzner cloud firewall: allow inbound 22, 80, 443. (8000 stays internal.)
+
+## What gets uploaded from the Mac (5.8 GB)
+From `phase1/data/thor_artifacts/`: escr_vectors.npy, escr_chunks.jsonl, escr_corpus_full.jsonl, escr_meta.jsonl, edges.jsonl, good_law.jsonl, escr_pdfmap.jsonl
+From `phase1/scripts/`: serve.py, frontend.html
+Secret: `themis/.env` with `DEEPSEEK_API_KEY`, `CLERK_SECRET_KEY`, and the
+Clerk public/configuration values documented in
+[`documentation/15_CLERK_AUTHENTICATION.md`](../../documentation/15_CLERK_AUTHENTICATION.md).
+Use mode 600. Never commit it or paste the secret key into chat.
+
+## Steps (Claude, once IP is known — APP_DIR=/opt/themis)
+```bash
+# 0. base packages + Caddy
+apt-get update && apt-get install -y python3 python3-venv python3-pip rsync curl debian-keyring debian-archive-keyring apt-transport-https
+curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
+curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | tee /etc/apt/sources.list.d/caddy-stable.list
+apt-get update && apt-get install -y caddy
+
+# 1. app dir + venv + deps (CPU torch)
+mkdir -p /opt/themis && cd /opt/themis
+python3 -m venv venv
+./venv/bin/pip install -U pip
+./venv/bin/pip install torch --index-url https://download.pytorch.org/whl/cpu
+./venv/bin/pip install numpy requests fastapi "uvicorn[standard]" clerk-backend-api sentence-transformers rank-bm25
+
+# 2. upload from the Mac (run FROM the Mac; rsync is resumable over slow links)
+#    rsync -avP phase1/data/thor_artifacts/{escr_vectors.npy,escr_chunks.jsonl,escr_corpus_full.jsonl,escr_meta.jsonl,edges.jsonl,good_law.jsonl,escr_pdfmap.jsonl} USER@IP:/opt/themis/
+#    rsync -avP phase1/scripts/{serve.py,clerk_auth.py,frontend.html} USER@IP:/opt/themis/
+#    (.env handled separately)
+
+# 3. secrets — copy the complete Clerk-enabled .env, then lock it down
+#    chmod 600 /opt/themis/.env
+
+# 4. systemd service
+cp /opt/themis/themis.service /etc/systemd/system/themis.service   # (uploaded from deploy/)
+systemctl daemon-reload && systemctl enable --now themis
+#    first start loads 5.8GB → ~30-60s; watch:  journalctl -u themis -f   (wait for "READY — 37898 judgments")
+
+# 5. Caddy (auto-TLS for themis.apexflo.ai)
+cp /opt/themis/Caddyfile /etc/caddy/Caddyfile   # (uploaded from deploy/)
+systemctl reload caddy
+
+# 6. verify
+curl -s -o /dev/null -w '%{http_code}\n' https://themis.apexflo.ai/                     # expect 200 (sign-in shell)
+curl -s https://themis.apexflo.ai/api/v2/auth/config                                    # expect configured=true
+curl -s -o /dev/null -w '%{http_code}\n' https://themis.apexflo.ai/api/search?q=test          # expect 401 without bearer token
+```
+
+## Verify checklist
+- [ ] `journalctl -u themis` shows `pdfmap: 37898` + `READY — 37898 judgments, DeepSeek serving=yes`
+- [ ] public auth config reports `configured: true`; protected API returns 401 without a Clerk token
+- [ ] a search streams (SSE through Caddy `flush_interval -1`)
+- [ ] a judgment opens + Official PDF embeds (`/api/pdf` pulls from open registry, caches ≤20)
+- [ ] logs writing to `/opt/themis/logs/usage-*.jsonl` + `internal-*.jsonl`
+- [ ] reboot test: `systemctl reboot`, confirm themis + caddy come back
+
+## Rollback / notes
+- Old Thor is gone (reflashed). This box is now the source of truth for *serving*; the Mac holds the artifact backup + git holds the code.
+- To rotate Clerk keys: update `/opt/themis/.env`, then `systemctl restart themis`.
+- The citation-graph rebuild (edges.jsonl) is separate offline work; serving just consumes whatever edges.jsonl is present.

phase1/deploy/requirements.txt

@@ -0,0 +1,16 @@
+# Themis CPU serving deps (no GPU). Install torch from the CPU wheel index separately:
+#   pip install torch --index-url https://download.pytorch.org/whl/cpu
+numpy
+requests
+fastapi
+uvicorn[standard]
+clerk-backend-api>=6.0.1,<7
+sentence-transformers
+faiss-cpu
+rank-bm25
+bharat-courts[archive]==0.3.3
+PyMuPDF>=1.24,<2
+python-docx>=1.1,<2
+pytesseract>=0.3.13,<1
+Pillow>=10,<13
+chromadb>=1.5,<2

phase1/deploy/themis.service

@@ -0,0 +1,19 @@
+[Unit]
+Description=Themis legal-search API (uvicorn)
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=simple
+WorkingDirectory=/opt/themis
+# .env holds DEEPSEEK_API_KEY and Clerk backend configuration (chmod 600, never committed)
+EnvironmentFile=/opt/themis/.env
+ExecStart=/opt/themis/venv/bin/uvicorn serve:app --host 127.0.0.1 --port 8000
+Restart=always
+RestartSec=3
+# loading 5.8GB of artifacts takes ~30-60s; don't let systemd kill it as "not started"
+TimeoutStartSec=300
+LimitNOFILE=65536
+
+[Install]
+WantedBy=multi-user.target

phase1/drafting/templates.json

@@ -0,0 +1,53 @@
+{
+  "version": 1,
+  "templates": [
+    {
+      "id": "slp-civil-full",
+      "title": "Special Leave Petition — Civil",
+      "description": "Full civil SLP structure with facts, questions of law, grounds and prayers.",
+      "filename": "slp_civil_full.pdf",
+      "court": "Supreme Court of India",
+      "category": "SLP"
+    },
+    {
+      "id": "slp-criminal-full",
+      "title": "Special Leave Petition — Criminal",
+      "description": "Full criminal SLP structure with drafting notes and continuation sections.",
+      "filename": "slp_criminal_full.pdf",
+      "court": "Supreme Court of India",
+      "category": "SLP"
+    },
+    {
+      "id": "slp-outline",
+      "title": "Special Leave Petition — Outline",
+      "description": "A shorter SLP format for an initial working draft.",
+      "filename": "slp_outline.pdf",
+      "court": "Supreme Court of India",
+      "category": "SLP"
+    },
+    {
+      "id": "article-32",
+      "title": "Article 32 Petition",
+      "description": "Petition format for invoking the Supreme Court's writ jurisdiction.",
+      "filename": "article_32_petition.pdf",
+      "court": "Supreme Court of India",
+      "category": "Writ"
+    },
+    {
+      "id": "civil-appeal",
+      "title": "Civil Appeal",
+      "description": "A concise civil appeal format.",
+      "filename": "civil_appeal.pdf",
+      "court": "Supreme Court of India",
+      "category": "Appeal"
+    },
+    {
+      "id": "curative-petition",
+      "title": "Curative Petition",
+      "description": "Curative petition structure for the Supreme Court of India.",
+      "filename": "curative_petition.pdf",
+      "court": "Supreme Court of India",
+      "category": "Curative"
+    }
+  ]
+}

phase1/eval/.gitignore

@@ -0,0 +1,8 @@
+run.tsv
+run_baseline.tsv
+last_score.json
+sweep_out.txt
+sweep_err.txt
+escr_*.jsonl
+escr_*.jsonl.gz
+*.npy

phase1/eval/BENCHMARK.md

@@ -0,0 +1,24 @@
+# Themis ⇄ CaseMine search-quality benchmark harness
+
+Goal: a defensible, repeatable "Themis vs CaseMine" search-quality score so every change is measured.
+
+## Method
+- **Query set**: `bench_queries.json` — 15 realistic, era-neutral queries across intents (fact ×6, issue ×4, vague ×2, citation ×1, casename ×2). Free queries (no fixed gold), since the two systems have different corpora.
+- **Themis side**: `19_benchmark_themis.py` (runs on Thor) — loads the persisted eSCR index once, runs every query through dense(BGE bf16)+BM25+RRF+cross-encoder, saves top-10 per query → `themis_bench_results.json`.
+- **CaseMine side**: collected from `casemine.com/search/in/<query>` in-browser. Result cards = `.listing-card-container`, name link = `.jdlink`. Extractor (run in page context):
+  ```js
+  [...document.querySelectorAll('.listing-card-container')].slice(0,10).map(c=>({
+    case_name: c.querySelector('.jdlink')?.textContent?.trim(),
+    snippet: c.innerText.replace(/\s+/g,' ').trim().slice(0,300)}))
+  ```
+  → `casemine_bench_results.json`. (CaseMine search is slow, ~tens of s/query; collect patiently, one navigation per query. Their results are fixed, so collect once and reuse.)
+- **Scoring**: `20_score_benchmark.py` — the isolated relevance reviewer judges each result (query + case + snippet/passage) blind → relevant=1/partial=0.5/not=0 → **relevance@10** per query, per intent, and overall; reports the head-to-head + per-query win/loss.
+
+## Baseline data point (demo query, Themis on 8.8k recent-only corpus)
+Query: *"quashing of FIR due to criminal activity included in money diversion of an investor's money"*
+- **Themis relevance@10 = 60%** (2 relevant, 8 partial) — all results on-topic, recent (2021–2025).
+- **CaseMine**: top-10 all clearly on-point (Vikram Doshi, Sushil Suri, Iridium, Bikram Chatterji/Amrapali, B. Rama Raju, Jaypee/63 Moons…) — but they lead with **2010–2014 landmarks + HC cases that are pre-2015, i.e. outside our current corpus window**.
+- **Finding**: the gap on this query is **corpus coverage, not the ranker.** → fix is the full-corpus backfill (1950–2025), after which this becomes a fair algorithm comparison.
+
+## Next (post-backfill)
+Re-run `19_benchmark_themis.py` on the full ~41.5k index, collect CaseMine for all 15, run `20_score_benchmark.py` → the real per-intent + overall "Themis vs CaseMine, ±N points" scorecard.

phase1/eval/BENCHMARK_3WAY.md

@@ -0,0 +1,41 @@
+# Search-quality benchmark — Themis vs CaseMine (Niyam pending)
+
+**Date:** 2026-06-24 · **Queries:** 6 (multi-intent subset of bench_queries.json) · **Metric:** blind graded relevance of each system's **top-5 result list**.
+
+## Method
+- Each query run on Themis (`/api/search_stream`) and CaseMine (semantic CiteTEXT search), top-5 captured.
+- For each query, the two top-5 lists were **pooled, de-labelled, and shuffled**; a neutral legal-expert judge (one per query, blind to source) scored every candidate **0–3** (3 = leading on-point authority, 2 = relevant, 1 = marginal, 0 = irrelevant / wrong case).
+- nDCG@5 (gain = 2^rel−1), Precision@5 (rel ≥ 2 counts as relevant), mean relevance — computed deterministically. nDCG capped at 1.0.
+- **Fairness note:** I built Themis, so scoring was blind (judge never told which engine produced a result) and used Indian SC legal merit only.
+
+## Aggregate (6 queries)
+| Metric | **Themis** | CaseMine |
+|---|---|---|
+| nDCG@5 | **0.77** | 0.64 |
+| Precision@5 | **0.50** | 0.43 |
+| Mean relevance | **1.53** | 1.40 |
+| Query wins (by nDCG) | 3 (fact-1, vague-1, casename-2) | 3 (fact-4, issue-1, issue-3) |
+
+Themis is ahead on aggregate, but it's a **genuine 3–3 split**, not a blowout. The pattern is the real story.
+
+## Per-query (nDCG@5)
+| Query | intent | Themis | CaseMine | winner |
+|---|---|---|---|---|
+| fact-1 — quash 498A FIR on settlement | fact | **0.97** | 0.19 | **Themis (big)** |
+| fact-4 — anticipatory bail, economic offence | fact | 0.63 | **0.72** | CaseMine |
+| issue-1 — dying declaration sole basis | issue | 0.57 | **0.75** | CaseMine |
+| issue-3 — Art 14 arbitrariness, judicial review | issue | 0.51 | **0.85** | CaseMine |
+| vague-1 — privacy fundamental right / Aadhaar | vague | **1.00** | 0.72 | **Themis** |
+| casename-2 — Vishaka v State of Rajasthan | casename | **0.92** | 0.64 | **Themis** |
+
+## What the split means (actionable)
+- **Themis wins the "find the right case fast" queries.** fact-1: Themis put **Gian Singh** + **Jitendra Raghuvanshi** (the two leading authorities on quashing 498A on settlement) at #1–2; CaseMine missed both (top results scored 0–1). vague-1 + casename-2: the new **identity-lookup path** (case-name/citation → metadata) put the right judgment first — casename-2 went from **0 results to a win** after that fix.
+- **CaseMine wins the doctrinal "landmark" queries.** issue-1: CaseMine surfaced **Khushal Rao v State of Bombay** (the foundational dying-declaration authority) at #1 — Themis missed it. issue-3: CaseMine led with **Shrilekha Vidyarthi** + **Om Kumar** (the leading Art-14-arbitrariness authorities); Themis had Om Kumar only at #5. CaseMine's citation-network / CiteTEXT is better at pulling the *old foundational* case for a doctrine.
+
+→ **Themis's #1 improvement target: foundational-authority recall on doctrinal/issue queries.** Our dense+rerank favours recent, textually-similar judgments; it under-weights the seminal older case that everyone cites. Candidate fixes: citation-graph signal (boost high-cited-by authorities), or a "leading case" re-rank feature.
+
+## Caveats (don't over-read)
+- **2-way, not yet 3-way.** Niyam couldn't be driven in this headless browser — its React search input doesn't fire from synthetic events and keystrokes weren't landing reliably; no results captured. One prior data point: Niyam's exact-name search for "Achin Gupta v State of Haryana" returned a 1980 tax case as #1 (a clear miss). Niyam leg still owed.
+- n = 6 queries, single blind judge per query → **directional, not definitive**. Graded relevance is somewhat subjective.
+- Compares **result lists only** (the fair common denominator). Does not score Themis's grounded answer, CaseMine's AMICUS, or CiteTEXT passages.
+- vague-1: Themis returned multiple Puttaswamy judgments (Privacy + Aadhaar) — all relevant to an Aadhaar-privacy query, but worth a results **dedupe-by-case** pass so near-identical entries don't crowd the top-5.

phase1/eval/CP-B_results.md

@@ -0,0 +1,32 @@
+# CP-B — Citation graph + good-law accuracy (results, 2026-06-24)
+
+Artifacts (`good_law.jsonl`, `edges.jsonl`) are gitignored (large, derived; rebuilt by `21_citator.py`
+on Thor over `escr_corpus_full.jsonl`). This file is the committed, auditable record of the numbers.
+
+## Citation graph (`21_citator.py` → `edges.jsonl`)
+- **86,702 directed edges** (63.2k cite + 23.5k name) · **0 self-edges, 0 dangling targets** (every edge → a real corpus doc).
+- **cited_by nonzero coverage: 43%** (baseline before Stage 2 was ~10%).
+- Foundational cases recovered via name-edges: **Khushal Rao 0→37**, Indra Sawhney 90, Vishaka 66, Shrilekha Vidyarthi 18, Maneka Gandhi 247.
+- Build: expanded citation regex (bare/no-vol SCR, Supp., year-first AIR); cross-reporter union-find with **same-year guard**; **name-edges** require a citation cue + are disambiguated by cite-popularity + verified on the second party.
+
+## Name-edge precision audit (`37_audit_name_edges.py`)
+| | initial | after cue+popularity | after second-party check |
+|---|---|---|---|
+| name-edges | 57,566 | 29,773 | **23,502** |
+| LLM-judged real-reference | 50% | 53% | **77%** |
+| ambiguity rate (shared-key) | 53% | 45% | **37.5%** |
+
+Name-edge noise is **contained**: name-edges do NOT drive good-law (negatives resolve cite-anchored only), and the UI shows no treatment tags — so for search they are a recall booster the cross-encoder vets (a wrong edge costs a wasted re-rank, never a false answer).
+
+## Good-law accuracy (`34_eval_goodlaw.py` vs `goodlaw_goldset.json`: 23 overruled + 11 good-law)
+- **Good-law FALSE POSITIVES: 0/7 matched** (Maneka Gandhi, previously falsely "overruled", cleared by the local-Qwen confirmation pass `35_confirm_negatives.py`).
+- **Overruled recall: 2/14 matched** — precision-first per asymmetric cost; the ±170-char Qwen window is the documented recall limiter. (Eval denominator is biased low — single-token landmarks aren't matched; real recall is a smoke test, not calibrated.)
+- Status distribution: unknown 37,794 · overruled 76 (49 after Qwen confirm) · doubted 14→5 · per_incuriam 14→6.
+- **Positive "good law" derivation built (`36_*`) but NOT shipped** — eval showed it falsely cleared 9 known-overruled cases at current recall.
+
+## Decisions (founder)
+- **Good-law deferred** — keep only the 0-FP confirmed-negative flags; further good-law work (recall via Tier-2 whole-doc Qwen, positive assertion) is later.
+- **Display**: good-law status shown only for confirmed negatives; **nothing for unknown** (no "not-reviewed" disclaimer); **treatment tags removed** (no false health signal on possibly-overruled cases).
+- **Speed = tier knob** (fast vs deep search) confirmed.
+
+## CP-B panel verdict: PASS_WITH_FIXES (wf_383119c2-62d) — fixes applied: false-reassurance treatment tags removed; this evidence file commits the numbers. Deferred: higher overruled recall (Tier-2), hard-mapped eval gold, per_incuriam/partly_overruled coverage.

phase1/eval/CP2_extraction_findings.md

@@ -0,0 +1,39 @@
+# CP2 — metadata source + extraction findings (validated on a sample)
+
+## Source (settled)
+**AWS Open Data: `s3://indian-supreme-court-judgments`** (region `ap-south-1`, `--no-sign-request`).
+- eCourts / **eSCR** (electronic Supreme Court Reports = the *reportable* corpus), **1950–2025**, **~41,539 judgments**.
+- Open, **no CAPTCHA, no scraping, no login**; CC-BY-4.0; refreshed **bi-monthly**; maintained by Dattam Labs.
+- Layout: `metadata/json/year=YYYY/<path>.json` (one per judgment, has a structured `raw_html` eSCR card) · `metadata/parquet/year=YYYY/metadata.parquet` · `data/pdf/year=YYYY/english/<path>_EN.pdf` (the headnoted judgment) · `data/tar/` (bulk).
+- **Decommissioned/unavailable:** digiSCR (`digiscr.sci.gov.in`, NXDOMAIN — merged into SCR & retired) and `main.sci.gov.in`; `scr.sci.gov.in` is live but CAPTCHA-gated. The open S3 dataset replaces all of them.
+
+## Extraction (two deterministic layers + one LLM field)
+**Layer 1 — P0 fields from the `raw_html` card (regex/bs4, no LLM, no PDF).** Coverage on a 75-record sample across 2024/2005/1990:
+
+| field | overall | 2024 | 2005 | 1990 |
+|---|---|---|---|---|
+| case_name, neutral_citation, equivalent_citations, cnr, reportable, bench, bench_strength, date, case_number, court, year | **100%** | 100% | 100% | 100% |
+| author_judge | 92% | 90% | 100% | 87% |
+| disposition | 91% | 82% | 100% | 100% |
+
+> **neutral_citation = 100% across all eras** (eSCR backfilled `YYYY INSC N`). Contrast the HF `sinhal` dataset: 0.9%. This alone fixes the citator's join-key problem the CP2 debate flagged.
+
+**Layer 2 — deep/headnote fields by deterministic slice of the PDF headnote** (labeled sections: `Issue for Consideration`, `Held`, `Case Law Cited`, `List of Acts`). Coverage on 15 2024 PDFs:
+
+| field | coverage |
+|---|---|
+| issue | 100% |
+| held | 100% |
+| cases_cited (with equivalent citations) | **80%** (avg **5.7 edges/judgment**) |
+| acts (from headnote) | 87% |
+
+> `Case Law Cited` parses each precedent **with both its SCR and SCC cites**, e.g. *"State of Goa v. Sanjay Thakran [2007] 3 SCR 507 : (2007) 3 SCC 755"* — the citator's edge list, deterministically. (cases_cited <100% because not every judgment cites prior cases.)
+
+**Layer 3 — the only `needs-LLM` field: `treatment`** (relied-on / overruled / distinguished …) classified per cited case from the Held text. Demonstrated with DeepSeek; this is the one human-review-worthy step (feeds `good_law_status`).
+
+## What this means
+- Practically the **entire CP2 schema is deterministically extractable** from a free, open, era-robust dataset — only `treatment` needs an LLM.
+- **HF `sinhal` → retired** as the metadata source; eSCR S3 dataset is the production source.
+- Cost axes: P0 = cheap JSON GETs (~41.5k). Deep = PDF download+parse (~41.5k PDFs ≈ tens of GB; use `data/tar/`, ideally run on Thor — in-region to ap-south-1). Embedding/retrieval index is a separate GPU axis.
+
+Script: `phase1/scripts/15_extract_escr.py` · sample: `phase1/eval/escr_sample.jsonl`.

phase1/eval/CP4_scale_results.md

@@ -0,0 +1,49 @@
+# CP4 — per-intent retrieval scorecard AT SCALE (GPU)
+
+*255 validated gold queries over a 10,000-doc SC corpus (built on Thor GPU, sentence-transformers BGE-small @ 112 chunks/s). doc-level RRF; rerank = ms-marco cross-encoder over top-40. Retrieval-only (no mq/graded — those need the DeepSeek key, kept off this box).*
+
+## top-5 (gold case in first 5 results)
+
+| Intent | n | dense | bm25 | hybrid | hybrid+rerank |
+|---|---|---|---|---|---|
+| citation | 40 | 0% | 60% | 0% | 100% |
+| casename_exact | 40 | 57% | 48% | 85% | 100% |
+| casename_fuzzy | 37 | 30% | 19% | 32% | 92% |
+| fact | 44 | 68% | 77% | 75% | 77% |
+| issue | 39 | 51% | 74% | 69% | 72% |
+| vague | 30 | 67% | 90% | 87% | 87% |
+| judge | 25 | 32% | 48% | 48% | 60% |
+
+## top-1 / MRR (best pipeline per intent)
+
+| Intent | best | top-1 | top-5 | MRR |
+|---|---|---|---|---|
+| citation | **hybrid+rerank** | 90% | 100% | 0.95 |
+| casename_exact | **hybrid+rerank** | 95% | 100% | 0.97 |
+| casename_fuzzy | **hybrid+rerank** | 84% | 92% | 0.87 |
+| fact | **hybrid+rerank** | 64% | 77% | 0.70 |
+| issue | **bm25** | 36% | 74% | 0.52 |
+| vague | **bm25** | 43% | 90% | 0.63 |
+| judge | **hybrid+rerank** | 36% | 60% | 0.47 |
+
+## 300 → 10,000 docs: what held, what degraded (apples-to-apples `hybrid+rerank` top-5)
+
+| Intent | 300 (CP3) | 10k (CP4) | Δ | read |
+|---|---|---|---|---|
+| citation | 100% | 100% | 0 | **held** — BM25 keeps the exact cite in the top-40 pool, rerank promotes it; my "rerank-rescue collapses at scale" worry did **not** materialise at 10k |
+| casename_exact | 100% | 100% | 0 | **held** |
+| casename_fuzzy | 97% | 92% | −5 | mostly held |
+| fact | 98% | 77% | −21 | degraded |
+| issue | 85% | 72% | −13 | degraded |
+| vague | 97% | 87% | −10 | degraded |
+| judge | 88% | 60% | −28 | degraded most |
+
+**Verdict.** Known-item intents (citation / case-name) are **scale-robust** — the lexical+rerank path holds at 33× corpus. Semantic intents **degrade** as distractors grow (expected), with `judge` worst. Two notable scale effects:
+- At 10k, **plain BM25 beats hybrid+rerank** on top-5 for `issue` (74% vs 72%) and `vague` (90% vs 87%) — the cross-encoder slightly *hurts* these at scale. Reranker tuning / candidate-depth is now a real lever.
+- Dense-alone is weak at scale (citation 0%, issue 51%); the value is in the **fusion + rerank**, not dense by itself.
+
+### Caveats (don't over-read the semantic drop)
+1. **Retrieval-only run** — no multi-query rewrite and no graded relevance@1 (both need the DeepSeek key, kept off Thor). At 300, `mq` lifted fact/issue and graded relevant@1 was **95–100%** even where exact top-1 was ~74%. So the semantic top-1 here (fact 64%, issue 36%) **understates** real usefulness; the full pipeline would recover much of the gap.
+2. **Embedder differs** — 10k used sentence-transformers BGE-small (GPU); 300 used fastembed BGE-small (same weights, minor numerical differences).
+3. **GPU throughput 112 chunks/s** — works, but that's the *server* cu130 wheel on Thor's integrated Blackwell GPU (compatibility path, not Thor-native kernels). A JetPack-native torch / TensorRT build would be much faster; bulk re-indexing should use that.
+4. Exact-gold is still single-target; on common topics many cases are equally right, so the metric is a floor.

phase1/eval/agentic_run.py

@@ -0,0 +1,65 @@
+"""Run the agentic controller over a query file -> run.tsv (final ranked docs), for nDCG scoring.
+Two phases: (1) batch-plan all queries in parallel (DeepSeek, cached per qid so ablation runs reuse
+the SAME plans), (2) assemble each query (parallel fetch + rerank + rank) sequentially.
+
+Env: THEMIS_DATA, THEMIS_STATUTE, THEMIS_QFILE, OUT, PLANCACHE, THEMIS_ENABLED=all|vector,authority,...
+"""
+import os, sys, json, time
+import concurrent.futures as cf
+import requests
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "scripts"))
+from tools import Corpus
+import agent as A
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+def _load_env(p):
+    for l in open(p):
+        l = l.strip()
+        if l and not l.startswith("#") and "=" in l:
+            k, v = l.split("=", 1); os.environ.setdefault(k.strip(), v.strip().strip('"').strip("'"))
+_load_env(os.path.join(HERE, "..", "scripts", ".env"))
+HDR = {"Authorization": f"Bearer {os.environ['DEEPSEEK_API_KEY']}", "Content-Type": "application/json"}
+
+def llm_fn(msgs):
+    for _ in range(3):
+        try:
+            r = requests.post("https://api.deepseek.com/chat/completions", headers=HDR, timeout=60,
+                json={"model": "deepseek-chat", "temperature": 0, "max_tokens": 400, "messages": msgs})
+            if r.status_code == 200: return r.json()["choices"][0]["message"]["content"]
+        except Exception: time.sleep(2)
+    return "{}"
+
+QFILE = os.environ.get("THEMIS_QFILE", "authority_queries.tsv")
+OUT = os.environ.get("OUT", "agent_run.tsv")
+PLANCACHE = os.environ.get("PLANCACHE", "plan_" + os.path.basename(QFILE).split(".")[0] + ".json")
+ENABLED = A.ALL_TOOLS if os.environ.get("THEMIS_ENABLED", "all") == "all" else set(os.environ["THEMIS_ENABLED"].split(","))
+ALPHA = float(os.environ.get("ALPHA", "0.3"))
+
+rows = []
+for l in open(QFILE, encoding="utf-8"):
+    qid, it, t = l.rstrip("\n").split("\t", 2); rows.append((qid, t))
+
+# Phase 1: batch-plan (parallel, cached)
+plans = json.load(open(PLANCACHE)) if os.path.exists(PLANCACHE) else {}
+todo = [(qid, t) for qid, t in rows if qid not in plans]
+if todo:
+    print(f"planning {len(todo)} queries ...", flush=True); t0 = time.time()
+    with cf.ThreadPoolExecutor(max_workers=24) as ex:
+        futs = {ex.submit(A.plan, t, llm_fn): qid for qid, t in todo}
+        done = 0
+        for f in cf.as_completed(futs):
+            plans[futs[f]] = f.result(); done += 1
+            if done % 50 == 0: json.dump(plans, open(PLANCACHE, "w")); print(f"  {done}/{len(todo)} {time.time()-t0:.0f}s", flush=True)
+    json.dump(plans, open(PLANCACHE, "w"))
+    print(f"plans done {time.time()-t0:.0f}s", flush=True)
+
+# Phase 2: assemble (sequential; CE is CPU-bound)
+C = Corpus(os.environ.get("THEMIS_DATA", "."), os.environ.get("THEMIS_STATUTE", "."), device=os.environ.get("THEMIS_DEVICE", "cpu"))
+print(f"assembling {len(rows)} queries (enabled={sorted(ENABLED)}) ...", flush=True)
+t0 = time.time()
+with open(OUT, "w", encoding="utf-8") as f:
+    for i, (qid, t) in enumerate(rows):
+        ranked, info = A.assemble(C, t, plans[qid], enabled=ENABLED, alpha=ALPHA)
+        for rank, d in enumerate(ranked, 1): f.write(f"{qid}\t{rank}\t{d}\n")
+        if (i + 1) % 30 == 0: print(f"  {i+1}/{len(rows)} {time.time()-t0:.0f}s", flush=True)
+print(f"DONE {len(rows)} -> {OUT} in {time.time()-t0:.0f}s", flush=True)

phase1/eval/authority_badlaw.txt

@@ -0,0 +1,104 @@
+2019 INSC 372
+2015 INSC 793
+2015 INSC 52
+2015 INSC 160
+2015 INSC 235
+2015 INSC 257
+2016 INSC 375
+2016 INSC 491
+2016 INSC 526
+2017 INSC 1026
+2017 INSC 1092
+2017 INSC 121
+2017 INSC 102
+2017 INSC 899
+2018 INSC 896
+2018 INSC 732
+2018 INSC 84
+2018 INSC 405
+2018 INSC 714
+2019 INSC 1184
+2019 INSC 511
+2020 INSC 294
+2021 INSC 314
+2022 INSC 1312
+1960 INSC 107
+1960 INSC 123
+1960 INSC 195
+1960 INSC 200
+1961 INSC 177
+1962 INSC 247
+1962 INSC 389
+1962 INSC 328
+1964 INSC 7
+1964 INSC 27
+1964 INSC 203
+1964 INSC 206
+1965 INSC 154
+1966 INSC 155
+1967 INSC 45
+1967 INSC 87
+1967 INSC 122
+1967 INSC 173
+1968 INSC 72
+1969 INSC 8
+1969 INSC 20
+1969 INSC 77
+1969 INSC 99
+1969 INSC 87
+1970 INSC 18
+1970 INSC 190
+1971 INSC 20
+1974 INSC 256
+1975 INSC 212
+1976 INSC 270
+1976 INSC 231
+1976 INSC 250
+1976 INSC 272
+1977 INSC 28
+1977 INSC 75
+1977 INSC 155
+1978 INSC 16
+1981 INSC 175
+1981 INSC 211
+1981 INSC 209
+1983 INSC 10
+1984 INSC 67
+1984 INSC 152
+1985 INSC 101
+1987 INSC 259
+1988 INSC 61
+1989 INSC 54
+1996 INSC 90
+1996 INSC 800
+1997 INSC 441
+1998 INSC 185
+2001 INSC 158
+2002 INSC 66
+2002 INSC 123
+2004 INSC 34
+2004 INSC 182
+2005 INSC 146
+2007 INSC 1026
+2007 INSC 28
+2007 INSC 475
+2007 INSC 772
+2008 INSC 930
+2008 INSC 82
+2008 INSC 677
+2009 INSC 946
+2009 INSC 1045
+2009 INSC 1195
+2009 INSC 209
+2010 INSC 69
+2010 INSC 177
+2011 INSC 508
+2012 INSC 49
+2013 INSC 830
+2013 INSC 684
+2013 INSC 823
+2013 INSC 377
+2013 INSC 494
+2014 INSC 218
+2014 INSC 617
+2014 INSC 579

phase1/eval/authority_qrels.tsv

@@ -0,0 +1,1486 @@
+1	2006 INSC 216	3
+1	2015 INSC 792	2
+1	2016 INSC 1215	2
+1	2017 INSC 1101	2
+1	2017 INSC 81	2
+1	2018 INSC 1123	2
+1	2018 INSC 190	2
+1	2018 INSC 258	2
+1	2019 INSC 983	2
+1	2019 INSC 1373	2
+1	2019 INSC 1157	2
+2	1955 INSC 36	3
+2	2018 INSC 740	2
+2	2019 INSC 976	2
+2	2020 INSC 548	2
+2	2021 INSC 624	2
+2	2021 INSC 324	2
+2	2022 INSC 506	2
+2	2023 INSC 123	2
+2	1958 INSC 43	2
+2	1961 INSC 90	2
+2	1961 INSC 161	2
+3	2020 INSC 294	3
+3	2020 INSC 135	2
+3	2021 INSC 43	2
+3	2021 INSC 711	2
+3	2022 INSC 884	2
+3	2022 INSC 362	2
+3	2022 INSC 1190	2
+3	2022 INSC 1230	2
+3	2022 INSC 1231	2
+3	2022 INSC 1248	2
+3	2022 INSC 1245	2
+4	1952 INSC 1	3
+4	2017 INSC 801	2
+4	2020 INSC 432	2
+4	2021 INSC 28	2
+4	2021 INSC 777	2
+4	2022 INSC 1085	2
+4	2023 INSC 144	2
+4	2025 INSC 593	2
+4	1953 INSC 51	2
+4	1958 INSC 101	2
+4	1988 INSC 123	2
+5	2009 INSC 506	3
+5	2015 INSC 420	2
+5	2016 INSC 909	2
+5	2017 INSC 1150	2
+5	2017 INSC 1068	2
+5	2017 INSC 614	2
+5	2018 INSC 738	2
+5	2018 INSC 967	2
+5	2018 INSC 43	2
+5	2018 INSC 42	2
+5	2018 INSC 126	2
+6	1994 INSC 112	3
+6	2015 INSC 120	2
+6	2015 INSC 257	2
+6	2015 INSC 455	2
+6	2016 INSC 630	2
+6	2017 INSC 960	2
+6	2017 INSC 406	2
+6	2018 INSC 165	2
+6	2018 INSC 282	2
+6	2018 INSC 266	2
+6	2018 INSC 248	2
+7	2002 INSC 454	3
+7	2016 INSC 318	2
+7	2014 INSC 623	2
+7	2019 INSC 1081	2
+7	2020 INSC 557	2
+7	2021 INSC 551	2
+7	2022 INSC 1175	2
+7	2023 INSC 147	2
+7	2024_11_1647_2038	2
+7	2025 INSC 1063	2
+7	2007 INSC 198	2
+8	2014 INSC 53	3
+8	2015 INSC 179	2
+8	2016 INSC 1197	2
+8	2018 INSC 115	2
+8	2019 INSC 1184	2
+8	2022 INSC 362	2
+8	2022 INSC 1190	2
+8	2022 INSC 1230	2
+8	2022 INSC 1231	2
+8	2022 INSC 1248	2
+8	2022 INSC 1245	2
+9	2007 INSC 142	3
+9	2016 INSC 217	2
+9	2018 INSC 705	2
+9	2018 INSC 833	2
+9	2018 INSC 594	2
+9	2019 INSC 1363	2
+9	2019 INSC 84	2
+9	2019 INSC 481	2
+9	2020 INSC 675	2
+9	2020 INSC 563	2
+9	2021 INSC 60	2
+10	1994 INSC 283	3
+10	2016 INSC 273	2
+10	2016 INSC 516	2
+10	2018 INSC 115	2
+10	2020 INSC 399	2
+10	2021 INSC 326	2
+10	2021 INSC 492	2
+10	2022 INSC 1208	2
+10	2023 INSC 324	2
+10	2023 INSC 760	2
+10	2024 INSC 904	2
+11	1998 INSC 183	3
+11	2017 INSC 1269	2
+11	2018 INSC 862	2
+11	2018 INSC 248	2
+11	2019 INSC 1099	2
+11	2019 INSC 1102	2
+11	2019 INSC 1248	2
+11	2019 INSC 99	2
+11	2019 INSC 679	2
+11	2020 INSC 373	2
+11	2020 INSC 489	2
+12	2000 INSC 339	3
+12	2017_2_779_787	2
+12	2018 INSC 446	2
+12	2018 INSC 522	2
+12	2019 INSC 743	2
+12	2019 INSC 815	2
+12	2020 INSC 674	2
+12	2020 INSC 576	2
+12	2021 INSC 591	2
+12	2021 INSC 389	2
+12	2022 INSC 1024	2
+13	2006 INSC 711	3
+13	2016 INSC 1174	2
+13	2016 INSC 482	2
+13	2017 INSC 801	2
+13	2017 INSC 88	2
+13	2017 INSC 121	2
+13	2018 INSC 223	2
+13	2019 INSC 312	2
+13	2019 INSC 671	2
+13	2020 INSC 344	2
+13	2025 INSC 1101	2
+14	1997 INSC 288	3
+14	2015 INSC 217	2
+14	2017 INSC 768	2
+14	2018 INSC 282	2
+14	2018 INSC 456	2
+14	2019 INSC 823	2
+14	2018 INSC 880	2
+14	2019 INSC 1236	2
+14	2019 INSC 1353	2
+14	2019 INSC 1233	2
+14	2019 INSC 220	2
+15	1960 INSC 61	3
+15	2016 INSC 851	2
+15	2018 INSC 246	2
+15	2021 INSC 177	2
+15	2021 INSC 253	2
+15	2022 INSC 1294	2
+15	2022 INSC 827	2
+15	2023 INSC 189	2
+15	2024 INSC 897	2
+15	2025 INSC 596	2
+15	2025 INSC 767	2
+16	2008 INSC 853	3
+16	2015 INSC 886	2
+16	2017 INSC 1012	2
+16	2018 INSC 997	2
+16	2018 INSC 1194	2
+16	2019 INSC 1116	2
+16	2019 INSC 1107	2
+16	2019 INSC 851	2
+16	2019 INSC 518	2
+16	2019 INSC 196	2
+16	2020 INSC 624	2
+17	2005 INSC 526	3
+17	2016 INSC 454	2
+17	2019 INSC 1292	2
+17	2019 INSC 1299	2
+17	2019 INSC 511	2
+17	2021 INSC 12	2
+17	2023 INSC 423	2
+17	2024 INSC 155	2
+17	2024 INSC 710	2
+17	2007 INSC 463	2
+17	2007 INSC 826	2
+18	2019 INSC 95	3
+18	2019 INSC 889	2
+18	2019 INSC 1289	2
+18	2019 INSC 1256	2
+18	2020 INSC 490	2
+18	2020 INSC 699	2
+18	2020 INSC 264	2
+18	2020 INSC 227	2
+18	2021 INSC 590	2
+18	2021 INSC 828	2
+18	2021 INSC 133	2
+19	2017 INSC 1068	3
+19	2018 INSC 828	2
+19	2018 INSC 967	2
+19	2018 INSC 679	2
+19	2019 INSC 912	2
+19	2019 INSC 1341	2
+19	2019 INSC 1348	2
+19	2019 INSC 200	2
+19	2019 INSC 489	2
+19	2019 INSC 668	2
+19	2020 INSC 535	2
+20	2017 INSC 801	3
+20	2018 INSC 898	2
+20	2018 INSC 1201	2
+20	2018 INSC 223	2
+20	2019 INSC 855	2
+20	2018 INSC 880	2
+20	2019 INSC 915	2
+20	2014 INSC 623	2
+20	2019 INSC 1233	2
+20	2019 INSC 52	2
+20	2020 INSC 572	2
+21	2004 INSC 256	3
+21	2018 INSC 1153	2
+21	2019 INSC 724	2
+21	2020 INSC 549	2
+21	2020 INSC 652	2
+21	2021 INSC 642	2
+21	2021 INSC 133	2
+21	2021 INSC 430	2
+21	2022 INSC 807	2
+21	2011 INSC 162	2
+21	2013 INSC 811	2
+22	2014 INSC 229	3
+22	2015 INSC 218	2
+22	2018 INSC 820	2
+22	2018 INSC 248	2
+22	2019 INSC 1333	2
+22	2019 INSC 611	2
+22	2017 INSC 64	2
+22	2020 INSC 524	2
+22	2021 INSC 614	2
+22	2021 INSC 304	2
+22	2021 INSC 643	2
+23	1997 INSC 604	3
+23	2017 INSC 801	2
+23	2018 INSC 223	2
+23	2018 INSC 790	2
+23	2020 INSC 355	2
+23	2023 INSC 920	2
+23	2023 INSC 190	2
+23	2025 INSC 118	2
+23	2014 INSC 894	2
+23	2014 INSC 46	2
+23	2014 INSC 275	2
+24	1958 INSC 18	3
+24	2015 INSC 85	2
+24	2015 INSC 425	2
+24	2016 INSC 65	2
+24	2016 INSC 939	2
+24	2020 INSC 547	2
+24	2020 INSC 540	2
+24	2021 INSC 802	2
+24	2022 INSC 153	2
+24	2023 INSC 79	2
+24	1974 INSC 218	2
+25	1955 INSC 27	3
+25	2016 INSC 934	2
+25	2016 INSC 526	2
+25	2018 INSC 728	2
+25	2018 INSC 593	2
+25	2019 INSC 947	2
+25	2018 INSC 880	2
+25	2019 INSC 915	2
+25	2020 INSC 508	2
+25	2020 INSC 158	2
+25	2022 INSC 1085	2
+26	1957 INSC 38	3
+26	2017 INSC 1268	2
+26	2019 INSC 851	2
+26	2020 INSC 624	2
+26	2021 INSC 443	2
+26	2022 INSC 670	2
+26	2022 INSC 637	2
+26	2023 INSC 269	2
+26	2024 INSC 312	2
+26	2025 INSC 936	2
+26	2008 INSC 856	2
+27	1960 INSC 221	3
+27	2018 INSC 241	2
+27	2020 INSC 355	2
+27	2020 INSC 428	2
+27	2020 INSC 264	2
+27	2020 INSC 294	2
+27	2020 INSC 633	2
+27	2022 INSC 841	2
+27	2022 INSC 3	2
+27	2025 INSC 124	2
+27	1967 INSC 172	2
+28	2010 INSC 219	3
+28	2015 INSC 828	2
+28	2015 INSC 419	2
+28	2016 INSC 384	2
+28	2016 INSC 943	2
+28	2017 INSC 250	2
+28	2019 INSC 1346	2
+28	2019 INSC 1303	2
+28	2020 INSC 539	2
+28	2022 INSC 807	2
+28	2022 INSC 1177	2
+29	2014 INSC 21	3
+29	2016 INSC 401	2
+29	2017 INSC 999	2
+29	2017 INSC 1201	2
+29	2018 INSC 282	2
+29	2019 INSC 798	2
+29	2019 INSC 1161	2
+29	2019 INSC 1355	2
+29	2019 INSC 1146	2
+29	2019 INSC 1303	2
+29	2019 INSC 371	2
+30	1989 INSC 192	3
+30	2015 INSC 76	2
+30	2018 INSC 248	2
+30	2020 INSC 512	2
+30	2022 INSC 975	2
+30	2022 INSC 780	2
+30	2024 INSC 13	2
+30	2025 INSC 249	2
+30	1992 INSC 171	2
+30	1994 INSC 380	2
+30	1994 INSC 478	2
+31	1958 INSC 17	3
+31	2016 INSC 1019	2
+31	2019 INSC 734	2
+31	2021 INSC 659	2
+31	2022 INSC 331	2
+31	2023 INSC 817	2
+31	2025 INSC 757	2
+31	1960 INSC 190	2
+31	1961 INSC 120	2
+31	1971 INSC 280	2
+31	1977 INSC 177	2
+32	2007 INSC 28	3
+32	2017 INSC 801	2
+32	2017 INSC 768	2
+32	2018 INSC 881	2
+32	2018 INSC 880	2
+32	2019 INSC 1102	2
+32	2019 INSC 1007	2
+32	2020 INSC 512	2
+32	2020 INSC 344	2
+32	2021 INSC 434	2
+32	2021 INSC 340	2
+33	2018 INSC 115	3
+33	2019 INSC 678	2
+34	2009 INSC 808	3
+34	2018 INSC 1193	2
+34	2018 INSC 1112	2
+34	2018 INSC 1194	2
+34	2019 INSC 247	2
+34	2019 INSC 1107	2
+34	2019 INSC 851	2
+34	2019 INSC 518	2
+34	2019 INSC 196	2
+34	2020 INSC 624	2
+34	2022 INSC 52	2
+35	1956 INSC 28	3
+35	2019 INSC 1224	2
+35	2020 INSC 382	2
+35	2021 INSC 115	2
+35	2021 INSC 283	2
+35	2022 INSC 752	2
+35	2022 INSC 545	2
+35	2024 INSC 812	2
+35	1995 INSC 212	2
+35	1996 INSC 419	2
+35	1997 INSC 43	2
+36	1957 INSC 35	3
+36	2015 INSC 257	2
+36	2016 INSC 955	2
+36	2017 INSC 658	2
+36	2019 INSC 1236	2
+36	2021 INSC 92	2
+36	2021 INSC 340	2
+36	2022 INSC 331	2
+36	2023 INSC 81	2
+36	1963 INSC 202	2
+36	1982 INSC 58	2
+37	1993 INSC 316	3
+37	2019 INSC 529	2
+37	2019 INSC 764	2
+37	2020 INSC 93	2
+37	2023 INSC 975	2
+37	2023 INSC 11	2
+37	2025 INSC 555	2
+37	2025 INSC 742	2
+37	2025 INSC 997	2
+37	2003 INSC 442	2
+37	2010 INSC 90	2
+38	2001 INSC 80	3
+38	2017 INSC 802	2
+38	2018 INSC 850	2
+38	2020 INSC 185	2
+38	2020 INSC 173	2
+38	2020 INSC 511	2
+38	2021 INSC 862	2
+38	2022 INSC 642	2
+38	2022 INSC 433	2
+38	2022 INSC 997	2
+38	2007 INSC 932	2
+39	2013 INSC 748	3
+39	2018 INSC 820	2
+39	2018 INSC 1039	2
+39	2018 INSC 549	2
+39	2018 INSC 248	2
+39	2019 INSC 1102	2
+39	2019 INSC 1333	2
+39	2019 INSC 1242	2
+39	2019 INSC 611	2
+39	2020 INSC 682	2
+39	2020 INSC 432	2
+40	1996 INSC 419	3
+40	2015 INSC 912	2
+40	2017 INSC 1014	2
+40	2017 INSC 478	2
+40	2018 INSC 880	2
+40	2020 INSC 382	2
+40	2020 INSC 350	2
+40	2023 INSC 324	2
+40	2007 INSC 370	2
+41	1960 INSC 100	3
+41	2015 INSC 906	2
+41	2022 INSC 506	2
+41	1964 INSC 209	2
+41	1968 INSC 72	2
+41	1968 INSC 268	2
+41	1995 INSC 328	2
+41	2010 INSC 124	2
+41	2011 INSC 555	2
+41	2011 INSC 635	2
+41	2004 INSC 203	2
+42	2002 INSC 253	3
+42	2016 INSC 289	2
+42	2018 INSC 862	2
+42	2018 INSC 164	2
+42	2019 INSC 1103	2
+42	2019 INSC 1233	2
+42	2019 INSC 210	2
+42	2022 INSC 958	2
+42	2023 INSC 499	2
+42	2024 INSC 30	2
+42	2024 INSC 113	2
+43	1957 INSC 10	3
+43	2015 INSC 966	2
+43	2018 INSC 969	2
+43	2019 INSC 688	2
+43	2021 INSC 189	2
+43	2022 INSC 188	2
+43	2025 INSC 1024	2
+43	1960 INSC 86	2
+43	1960 INSC 87	2
+43	2010 INSC 730	2
+43	2013 INSC 670	2
+44	1952 INSC 2	3
+44	2015 INSC 912	2
+44	2016_2_65_70	2
+44	2019 INSC 1236	2
+44	2021 INSC 179	2
+44	2023 INSC 190	2
+44	1975 INSC 134	2
+44	1975 INSC 214	2
+44	1977 INSC 227	2
+44	1988 INSC 46	2
+44	1994 INSC 111	2
+45	2005 INSC 129	3
+45	2016 INSC 1056	2
+45	2017 INSC 1074	2
+45	2017 INSC 776	2
+45	2017 INSC 658	2
+45	2018 INSC 214	2
+45	2019 INSC 636	2
+45	2020 INSC 531	2
+45	2021 INSC 180	2
+45	2023 INSC 724	2
+45	2024 INSC 260	2
+46	2017 INSC 452	3
+46	2018 INSC 898	2
+46	2018 INSC 455	2
+46	2018 INSC 790	2
+46	2019 INSC 889	2
+46	2018 INSC 880	2
+46	2019 INSC 457	2
+46	2020 INSC 707	2
+46	2023 INSC 29	2
+46	2024 INSC 751	2
+46	2024 INSC 113	2
+47	2004 INSC 244	3
+47	2015 INSC 942	2
+47	2017 INSC 976	2
+47	2017 INSC 388	2
+47	2017 INSC 355	2
+47	2018 INSC 1034	2
+47	2018 INSC 200	2
+47	2018 INSC 241	2
+47	2019 INSC 889	2
+47	2019 INSC 148	2
+47	2019 INSC 707	2
+48	1999 INSC 282	3
+48	2015 INSC 341	2
+48	2018 INSC 426	2
+48	2019 INSC 1145	2
+48	2020 INSC 620	2
+48	2020 INSC 197	2
+48	2020 INSC 524	2
+48	2023 INSC 878	2
+48	2025 INSC 1045	2
+48	2025 INSC 1090	2
+48	2025 INSC 1111	2
+49	2012 INSC 428	3
+49	2018 INSC 1018	2
+49	2018 INSC 110	2
+49	2018 INSC 455	2
+49	2019 INSC 799	2
+49	2023 INSC 607	2
+49	2023 INSC 7	2
+49	2025 INSC 255	2
+49	2014 INSC 562	2
+49	2014 INSC 294	2
+50	2003 INSC 176	3
+50	2015 INSC 912	2
+50	2015 INSC 942	2
+50	2018 INSC 164	2
+50	2018 INSC 248	2
+50	2018 INSC 880	2
+50	2019 INSC 1237	2
+50	2019 INSC 1233	2
+50	2021 INSC 388	2
+50	2023 INSC 4	2
+50	2024 INSC 113	2
+51	1993 INSC 40	3
+51	2017 INSC 801	2
+51	2018 INSC 1201	2
+51	2018 INSC 880	2
+51	2023 INSC 190	2
+51	2025 INSC 1063	2
+51	1995 INSC 184	2
+51	1999 INSC 516	2
+51	2010 INSC 392	2
+52	2003 INSC 241	3
+52	2015 INSC 92	2
+52	2019 INSC 218	2
+52	2019 INSC 647	2
+52	2019 INSC 687	2
+52	2020 INSC 705	2
+52	2020 INSC 345	2
+52	2021 INSC 269	2
+52	2009 INSC 1278	2
+52	2014 INSC 102	2
+53	1953 INSC 89	3
+53	2020 INSC 23	2
+53	1954 INSC 90	2
+53	1964 INSC 48	2
+54	1950 INSC 14	3
+54	2015 INSC 257	2
+54	2018 INSC 880	2
+54	2019 INSC 505	2
+54	2019 INSC 517	2
+54	2020 INSC 572	2
+54	2023 INSC 4	2
+54	1952 INSC 1	2
+54	1985 INSC 238	2
+54	2014 INSC 347	2
+55	2008 INSC 473	3
+55	2017 INSC 668	2
+55	2018 INSC 881	2
+55	2018 INSC 880	2
+55	2019 INSC 680	2
+55	2021 INSC 194	2
+55	2022 INSC 1175	2
+55	2023 INSC 145	2
+55	2023 INSC 292	2
+55	2023 INSC 559	2
+55	2024 INSC 562	2
+56	1954 INSC 125	3
+56	2019 INSC 1146	2
+56	2019 INSC 1406	2
+56	2020 INSC 645	2
+56	2020 INSC 620	2
+56	2021 INSC 649	2
+56	2022 INSC 318	2
+56	2023 INSC 460	2
+56	2024 INSC 363	2
+56	1971 INSC 100	2
+56	2009 INSC 611	2
+57	2018 INSC 646	3
+57	2018 INSC 1213	2
+57	2019 INSC 1055	2
+57	2019 INSC 937	2
+57	2019 INSC 1067	2
+57	2019 INSC 1257	2
+57	2019 INSC 231	2
+57	2019 INSC 410	2
+57	2020 INSC 334	2
+57	2020 INSC 415	2
+57	2020 INSC 456	2
+58	2012 INSC 419	3
+58	2015 INSC 484	2
+58	2017 INSC 683	2
+58	2019 INSC 2	2
+58	2019 INSC 254	2
+58	2020 INSC 163	2
+58	2021 INSC 650	2
+58	2021 INSC 568	2
+58	2022 INSC 940	2
+58	2023 INSC 468	2
+58	2024 INSC 846	2
+59	1951 INSC 52	3
+59	2020 INSC 635	2
+59	2022 INSC 516	2
+59	2023 INSC 1032	2
+59	1999 INSC 407	2
+59	2010 INSC 371	2
+59	2011 INSC 113	2
+59	2011 INSC 348	2
+59	2011 INSC 366	2
+59	2013 INSC 580	2
+59	2013 INSC 94	2
+60	2002 INSC 148	3
+60	2017 INSC 1111	2
+60	2019 INSC 1325	2
+60	2020 INSC 665	2
+60	2021 INSC 195	2
+60	2021 INSC 919	2
+60	2022 INSC 467	2
+60	2022 INSC 57	2
+60	2022 INSC 297	2
+61	1996 INSC 952	3
+61	2018 INSC 981	2
+61	2018 INSC 804	2
+61	2021 INSC 624	2
+61	2024 INSC 178	2
+61	2013 INSC 840	2
+61	2013 INSC 834	2
+62	1958 INSC 5	3
+62	2015 INSC 485	2
+62	2018 INSC 282	2
+62	2024 INSC 812	2
+62	2024 INSC 266	2
+62	1962 INSC 348	2
+62	1963 INSC 205	2
+62	1977 INSC 211	2
+62	1978 INSC 187	2
+62	1979 INSC 244	2
+62	2001 INSC 555	2
+63	1960 INSC 211	3
+63	2016 INSC 301	2
+63	2021 INSC 659	2
+63	2022 INSC 331	2
+63	2023 INSC 81	2
+63	2012 INSC 305	2
+63	2018 INSC 244	2
+64	2005 INSC 186	3
+64	2017 INSC 896	2
+64	2017 INSC 957	2
+64	2018 INSC 648	2
+64	2019 INSC 595	2
+64	2020 INSC 274	2
+64	2020 INSC 345	2
+64	2021 INSC 754	2
+64	2022 INSC 841	2
+64	2024 INSC 627	2
+64	2025 INSC 478	2
+65	2002 INSC 165	3
+65	2018 INSC 921	2
+65	2018 INSC 288	2
+65	2018 INSC 291	2
+65	2019 INSC 1067	2
+65	2019 INSC 1260	2
+65	2020 INSC 376	2
+65	2017 INSC 64	2
+65	2022 INSC 304	2
+65	2023 INSC 532	2
+65	2023 INSC 564	2
+66	2004 INSC 4	3
+66	2018 INSC 781	2
+66	2018 INSC 36	2
+66	2018 INSC 206	2
+66	2018 INSC 285	2
+66	2018 INSC 192	2
+66	2018 INSC 311	2
+66	2018 INSC 531	2
+66	2020 INSC 106	2
+66	2018 INSC 700	2
+66	2018 INSC 678	2
+67	2006 INSC 452	3
+67	2018 INSC 1060	2
+67	2019 INSC 216	2
+67	2019 INSC 663	2
+67	2024 INSC 233	2
+67	2008 INSC 955	2
+67	2008 INSC 1438	2
+67	2009 INSC 456	2
+67	2010 INSC 532	2
+67	2014 INSC 754	2
+67	2022 INSC 1049	2
+68	2012 INSC 68	3
+68	2018 INSC 1018	2
+68	2018 INSC 110	2
+68	2018 INSC 455	2
+68	2018 INSC 880	2
+68	2022 INSC 255	2
+68	2023 INSC 7	2
+68	2014 INSC 962	2
+68	2014 INSC 562	2
+68	2019 INSC 1374	2
+68	2023 INSC 459	2
+69	2005 INSC 432	3
+69	2016 INSC 298	2
+69	2020 INSC 465	2
+69	2021 INSC 798	2
+69	2021 INSC 823	2
+69	2022 INSC 1212	2
+69	2022 INSC 775	2
+69	2022 INSC 970	2
+69	2025 INSC 1210	2
+69	2025 INSC 223	2
+69	2025 INSC 427	2
+70	1952 INSC 10	3
+70	2016 INSC 1019	2
+70	1989 INSC 396	2
+71	1998 INSC 400	3
+71	2019 INSC 378	2
+71	2025 INSC 25	2
+71	2025 INSC 697	2
+71	2005 INSC 418	2
+71	2009 INSC 361	2
+71	2011 INSC 772	2
+72	2005 INSC 358	3
+72	2020 INSC 557	2
+72	2020 INSC 3	2
+72	2022 INSC 1111	2
+72	2022 INSC 1183	2
+72	2008 INSC 585	2
+72	2012 INSC 363	2
+73	2004 INSC 34	3
+73	2016 INSC 1019	2
+73	2018 INSC 646	2
+73	2020 INSC 382	2
+73	2021 INSC 659	2
+73	2022 INSC 331	2
+73	2022 INSC 506	2
+73	2023 INSC 27	2
+73	2024 INSC 554	2
+73	2011 INSC 336	2
+74	2001 INSC 515	3
+74	2019 INSC 810	2
+74	2022 INSC 1073	2
+74	2024 INSC 261	2
+74	2024 INSC 519	2
+74	2025 INSC 1089	2
+74	2025 INSC 86	2
+74	2025 INSC 168	2
+74	2025 INSC 76	2
+74	2012 INSC 376	2
+74	2024 INSC 847	2
+75	2006 INSC 532	3
+75	2018 INSC 728	2
+75	2018 INSC 880	2
+75	2020 INSC 350	2
+75	2023 INSC 856	2
+76	1961 INSC 6	3
+76	2016 INSC 943	2
+76	2019 INSC 545	2
+76	2020 INSC 63	2
+76	2022 INSC 1173	2
+76	2023 INSC 771	2
+76	2023 INSC 990	2
+76	2023 INSC 426	2
+76	2010 INSC 159	2
+76	2013 INSC 657	2
+76	2013 INSC 101	2
+77	1960 INSC 255	3
+77	2021 INSC 227	2
+77	1965 INSC 197	2
+77	1974 INSC 220	2
+77	1975 INSC 65	2
+77	1989 INSC 304	2
+77	1997 INSC 490	2
+78	2002 INSC 189	3
+78	2015 INSC 765	2
+78	2015 INSC 485	2
+78	2016 INSC 630	2
+78	2017 INSC 1112	2
+78	2019 INSC 1248	2
+78	2019 INSC 1242	2
+78	2022 INSC 1056	2
+78	2024 INSC 145	2
+78	2025 INSC 1308	2
+78	2006 INSC 117	2
+79	2000 INSC 34	3
+79	2018 INSC 1060	2
+79	2021 INSC 132	2
+79	2021 INSC 675	2
+79	2022 INSC 326	2
+79	2022 INSC 825	2
+79	2008 INSC 1460	2
+80	2002 INSC 136	3
+80	2015 INSC 647	2
+80	2017 INSC 658	2
+80	2018 INSC 200	2
+80	2018 INSC 115	2
+80	2020 INSC 294	2
+80	2020 INSC 320	2
+80	2021 INSC 332	2
+80	2021 INSC 554	2
+80	2007 INSC 599	2
+80	2011 INSC 843	2
+81	2014 INSC 358	3
+81	2015 INSC 912	2
+81	2016 INSC 955	2
+81	2017 INSC 1030	2
+81	2017 INSC 143	2
+81	2019 INSC 860	2
+81	2019 INSC 889	2
+81	2019 INSC 1010	2
+81	2023 INSC 292	2
+81	2024 INSC 1003	2
+81	2024 INSC 41	2
+82	2008 INSC 785	3
+82	2018 INSC 78	2
+82	2018 INSC 248	2
+82	2018 INSC 714	2
+82	2020 INSC 620	2
+82	2020 INSC 524	2
+82	2022 INSC 164	2
+82	2023 INSC 634	2
+82	2024 INSC 511	2
+82	2010 INSC 495	2
+82	2011 INSC 109	2
+83	2003 INSC 391	3
+83	2016 INSC 332	2
+83	2004 INSC 234	2
+83	2011 INSC 388	2
+83	2012 INSC 428	2
+84	2017 INSC 1026	3
+84	2018 INSC 732	2
+84	2019 INSC 1008	2
+84	2019 INSC 1292	2
+84	2019 INSC 415	2
+84	2019 INSC 511	2
+84	2020 INSC 697	2
+84	2020 INSC 380	2
+84	2021 INSC 175	2
+84	2021 INSC 229	2
+84	2021 INSC 12	2
+85	1995 INSC 661	3
+85	2019 INSC 822	2
+85	2022 INSC 451	2
+85	2022 INSC 322	2
+85	2022 INSC 436	2
+85	2025 INSC 264	2
+85	1999 INSC 499	2
+85	2007 INSC 805	2
+85	2002 INSC 234	2
+85	2006 INSC 49	2
+86	2018 INSC 790	3
+86	2019 INSC 518	2
+86	2021 INSC 28	2
+86	2021 INSC 303	2
+86	2021 INSC 777	2
+86	2022 INSC 411	2
+86	2022 INSC 1085	2
+86	2023 INSC 920	2
+86	2023 INSC 115	2
+86	2023 INSC 144	2
+86	2023 INSC 99	2
+87	2014 INSC 568	3
+87	2015 INSC 726	2
+87	2015 INSC 912	2
+87	2016 INSC 526	2
+87	2017 INSC 314	2
+87	2017 INSC 892	2
+87	2018 INSC 862	2
+87	2018 INSC 728	2
+87	2019 INSC 194	2
+87	2018 INSC 790	2
+87	2019 INSC 1007	2
+88	1957 INSC 99	3
+88	2019 INSC 1231	2
+88	2021 INSC 284	2
+88	2022 INSC 1085	2
+88	2024 INSC 119	2
+88	1961 INSC 350	2
+88	1962 INSC 1	2
+88	1990 INSC 168	2
+88	1997 INSC 268	2
+88	2014 INSC 362	2
+89	1963 INSC 172	3
+89	2016 INSC 301	2
+89	2020 INSC 382	2
+89	2020 INSC 346	2
+89	2022 INSC 331	2
+89	1990 INSC 232	2
+89	1991 INSC 90	2
+89	1996 INSC 222	2
+89	2002 INSC 44	2
+89	2012 INSC 305	2
+89	2014 INSC 331	2
+90	1994 INSC 6	3
+90	2017 INSC 448	2
+90	2019 INSC 1114	2
+90	2019 INSC 851	2
+90	2020 INSC 624	2
+90	2022 INSC 164	2
+90	2005 INSC 282	2
+90	2008 INSC 997	2
+90	2012 INSC 565	2
+90	2013 INSC 281	2
+91	2003 INSC 258	3
+91	2019 INSC 1020	2
+91	2019 INSC 1048	2
+91	2021 INSC 326	2
+91	2023 INSC 96	2
+91	2007 INSC 822	2
+91	2008 INSC 334	2
+91	2008 INSC 517	2
+91	2011 INSC 651	2
+91	2014 INSC 374	2
+92	2001 INSC 251	3
+92	2017 INSC 1111	2
+92	2020 INSC 706	2
+92	2021 INSC 50	2
+92	2022 INSC 431	2
+92	2022 INSC 427	2
+92	2008 INSC 1201	2
+92	2009 INSC 140	2
+92	2012 INSC 477	2
+93	2011 INSC 554	3
+93	2018 INSC 880	2
+93	2020 INSC 492	2
+93	2020 INSC 653	2
+93	2020 INSC 482	2
+93	2020 INSC 688	2
+93	2020 INSC 23	2
+93	2021 INSC 115	2
+93	2022 INSC 430	2
+93	2023 INSC 123	2
+93	2023 INSC 81	2
+94	1950 INSC 36	3
+94	2018 INSC 790	2
+94	2022 INSC 378	2
+94	2011 INSC 304	2
+94	2012 INSC 45	2
+94	2012 INSC 234	2
+94	2013 INSC 510	2
+94	1959 INSC 2	2
+95	2001 INSC 487	3
+95	2016 INSC 1187	2
+95	2018 INSC 896	2
+95	2018 INSC 1158	2
+95	2019 INSC 1112	2
+95	2021 INSC 199	2
+95	2024 INSC 944	2
+95	2010 INSC 212	2
+95	2014 INSC 525	2
+96	2002 INSC 433	3
+96	2021 INSC 698	2
+96	2025 INSC 880	2
+96	2008 INSC 952	2
+96	2008 INSC 512	2
+96	2011 INSC 645	2
+96	2012 INSC 565	2
+96	2012 INSC 243	2
+96	2013 INSC 67	2
+96	2014 INSC 847	2
+97	2012 INSC 379	3
+97	2015 INSC 201	2
+97	2017 INSC 369	2
+97	2018 INSC 724	2
+97	2019 INSC 1349	2
+97	2019 INSC 817	2
+97	2020 INSC 659	2
+97	2020 INSC 294	2
+97	2020 INSC 284	2
+97	2021 INSC 264	2
+97	2022 INSC 1299	2
+98	1999 INSC 235	3
+98	2015 INSC 436	2
+98	2017 INSC 448	2
+98	2018 INSC 658	2
+98	2020 INSC 325	2
+98	2024 INSC 1007	2
+98	2025 INSC 1090	2
+98	2000 INSC 308	2
+98	2008 INSC 534	2
+98	2009 INSC 1019	2
+98	2010 INSC 499	2
+99	1996 INSC 612	3
+99	2008 INSC 1024	2
+99	2008 INSC 476	2
+99	2008 INSC 646	2
+99	2014 INSC 147	2
+100	2005 INSC 58	3
+100	2020 INSC 577	2
+100	2020 INSC 508	2
+100	2010 INSC 238	2
+100	2012 INSC 395	2
+100	2013 INSC 176	2
+100	2013 INSC 283	2
+100	2014 INSC 221	2
+100	1960 INSC 88	2
+100	1969 INSC 266	2
+100	1971 INSC 322	2
+101	2005 INSC 334	3
+101	2017 INSC 826	2
+101	2017 INSC 450	2
+101	2018 INSC 915	2
+101	2019 INSC 1378	2
+101	2019 INSC 1242	2
+101	2019 INSC 266	2
+101	2021 INSC 304	2
+101	2008 INSC 785	2
+101	2009 INSC 657	2
+101	2009 INSC 811	2
+102	2002 INSC 39	3
+102	2019 INSC 247	2
+102	2019 INSC 1114	2
+102	2019 INSC 196	2
+102	2022 INSC 606	2
+102	2007 INSC 901	2
+102	2008 INSC 880	2
+102	2008 INSC 512	2
+103	1960 INSC 15	3
+103	1975 INSC 240	2
+103	1978 INSC 41	2
+103	2000 INSC 459	2
+103	2009 INSC 1035	2
+103	2009 INSC 1037	2
+104	1963 INSC 173	3
+104	2015 INSC 647	2
+104	2019 INSC 895	2
+104	2019 INSC 215	2
+104	2021 INSC 703	2
+104	2022 INSC 779	2
+104	2011 INSC 814	2
+104	2011 INSC 842	2
+104	2012 INSC 122	2
+104	2013 INSC 663	2
+104	2014 INSC 494	2
+105	1964 INSC 17	3
+105	2020 INSC 344	2
+105	2023 INSC 81	2
+105	1980 INSC 218	2
+105	S_1992_2_454_1007	2
+105	2011 INSC 516	2
+105	2011 INSC 246	2
+105	2013 INSC 41	2
+106	2012 INSC 187	3
+106	2017 INSC 830	2
+106	2019 INSC 1283	2
+106	2019 INSC 53	2
+106	2020 INSC 634	2
+106	2021 INSC 699	2
+106	2021 INSC 133	2
+106	2022 INSC 1212	2
+106	2022 INSC 578	2
+106	2023 INSC 956	2
+106	2024 INSC 551	2
+107	2005 INSC 433	3
+107	2022 INSC 394	2
+107	2011 INSC 843	2
+107	2011 INSC 590	2
+107	2013 INSC 199	2
+107	2014 INSC 68	2
+107	2013 INSC 133	2
+108	2011 INSC 379	3
+108	2018 INSC 797	2
+108	2018 INSC 115	2
+108	2018 INSC 159	2
+108	2020 INSC 376	2
+108	2020 INSC 294	2
+108	2022 INSC 401	2
+108	2022 INSC 767	2
+108	2022 INSC 560	2
+108	2011 INSC 531	2
+108	2012 INSC 65	2
+109	2008 INSC 1234	3
+109	2016 INSC 1111	2
+109	2016 INSC 993	2
+109	2021 INSC 209	2
+109	2022 INSC 452	2
+109	2023 INSC 460	2
+109	2024 INSC 466	2
+109	2009 INSC 1010	2
+109	2009 INSC 1018	2
+109	2009 INSC 162	2
+109	2009 INSC 440	2
+110	1957 INSC 79	3
+110	2016 INSC 1049	2
+110	2019 INSC 871	2
+110	2022 INSC 1139	2
+110	2022 INSC 133	2
+110	2023 INSC 978	2
+110	2023 INSC 924	2
+110	1976 INSC 140	2
+110	1983 INSC 3	2
+110	1985 INSC 11	2
+110	2000 INSC 375	2
+111	1991 INSC 225	3
+111	2017 INSC 462	2
+111	2018 INSC 912	2
+111	2018 INSC 369	2
+111	2019 INSC 1007	2
+111	2020 INSC 373	2
+111	2024_11_1647_2038	2
+111	2023 INSC 190	2
+111	2025 INSC 694	2
+111	1995 INSC 179	2
+111	1998 INSC 382	2
+112	1994 INSC 371	3
+112	2017 INSC 754	2
+112	2020 INSC 589	2
+112	2022 INSC 1013	2
+112	2024 INSC 58	2
+112	2011 INSC 734	2
+112	2011 INSC 706	2
+112	2013 INSC 97	2
+113	2015 INSC 886	3
+113	2019 INSC 1007	2
+113	2019 INSC 1116	2
+113	2019 INSC 1107	2
+113	2019 INSC 574	2
+113	2019 INSC 545	2
+113	2019 INSC 518	2
+113	2021 INSC 223	2
+113	2022 INSC 164	2
+113	2022 INSC 565	2
+113	2022 INSC 939	2
+114	2006 INSC 691	3
+114	2016 INSC 464	2
+114	2018 INSC 985	2
+114	2019 INSC 420	2
+114	2020 INSC 649	2
+114	2021 INSC 136	2
+114	2022 INSC 1079	2
+114	2022 INSC 608	2
+114	2024 INSC 809	2
+114	2024 INSC 19	2
+114	2024 INSC 211	2
+115	2001 INSC 323	3
+115	2018 INSC 115	2
+115	2020 INSC 294	2
+115	2021 INSC 817	2
+115	2022 INSC 840	2
+115	2023 INSC 373	2
+115	2011 INSC 638	2
+116	1995 INSC 100	3
+116	2016_9_771_799	2
+116	2019 INSC 1099	2
+116	2019 INSC 1120	2
+116	2019 INSC 1387	2
+116	2021 INSC 731	2
+116	2022 INSC 647	2
+116	2006 INSC 352	2
+116	2008 INSC 304	2
+116	2009 INSC 254	2
+116	2012 INSC 54	2
+117	1960 INSC 163	3
+117	2016 INSC 1019	2
+117	2020 INSC 408	2
+117	2022 INSC 331	2
+117	2023 INSC 81	2
+117	2025 INSC 1154	2
+117	1997 INSC 305	2
+117	1999 INSC 230	2
+117	2004 INSC 352	2
+117	2013 INSC 707	2
+117	2007 INSC 22	2
+118	2007 INSC 241	3
+118	2017 INSC 467	2
+118	2020 INSC 563	2
+118	2021 INSC 145	2
+118	2021 INSC 159	2
+118	2021 INSC 285	2
+118	2007 INSC 1267	2
+118	2008 INSC 877	2
+118	2008 INSC 476	2
+118	2008 INSC 646	2
+118	2009 INSC 482	2
+119	2019 INSC 647	3
+119	2020 INSC 548	2
+119	2020 INSC 705	2
+119	2020 INSC 403	2
+119	2021 INSC 712	2
+119	2021 INSC 140	2
+119	2021 INSC 365	2
+119	2021 INSC 344	2
+119	2021 INSC 464	2
+119	2021 INSC 392	2
+119	2022 INSC 483	2
+120	2015 INSC 163	3
+120	2015 INSC 823	2
+120	2016 INSC 355	2
+120	2017 INSC 1042	2
+120	2017 INSC 1014	2
+120	2018 INSC 711	2
+120	2019 INSC 593	2
+120	2022 INSC 853	2
+120	2022 INSC 669	2
+120	2023 INSC 1030	2
+120	2024 INSC 900	2
+121	2000 INSC 405	3
+121	2015 INSC 396	2
+121	2017 INSC 293	2
+121	2020 INSC 321	2
+121	2021 INSC 599	2
+121	2021 INSC 45	2
+121	2023 INSC 87	2
+121	2024 INSC 213	2
+121	2004 INSC 373	2
+121	2013 INSC 6	2
+122	1960 INSC 256	3
+122	2021 INSC 200	2
+122	2022 INSC 19	2
+122	2024 INSC 104	2
+122	1963 INSC 151	2
+122	1997 INSC 609	2
+122	2011 INSC 565	2
+122	2013 INSC 225	2
+122	2008 INSC 888	2
+122	2008 INSC 887	2
+122	2008 INSC 1024	2
+123	1996 INSC 75	3
+123	2017 INSC 448	2
+123	2018 INSC 1192	2
+123	2018 INSC 1120	2
+123	2020 INSC 192	2
+123	2021 INSC 192	2
+123	2021 INSC 298	2
+123	2022 INSC 648	2
+123	2023 INSC 959	2
+123	2008 INSC 830	2
+123	2008 INSC 1474	2
+124	2005 INSC 190	3
+124	2019 INSC 149	2
+124	2019 INSC 456	2
+124	2021 INSC 654	2
+124	2021 INSC 688	2
+124	2022 INSC 1288	2
+124	2006 INSC 856	2
+124	2008 INSC 44	2
+124	2012 INSC 494	2
+124	2006 INSC 342	2
+124	2007 INSC 839	2
+125	1959 INSC 2	3
+125	2016_8_477_498	2
+125	2018 INSC 880	2
+125	2020 INSC 285	2
+125	2020 INSC 451	2
+125	2022 INSC 134	2
+125	2022 INSC 545	2
+125	2024 INSC 812	2
+125	1985 INSC 121	2
+125	1990 INSC 59	2
+125	2021 INSC 798	2
+126	2014 INSC 841	3
+126	2016 INSC 1061	2
+126	2017 INSC 283	2
+126	2019 INSC 1325	2
+126	2021 INSC 271	2
+126	2021 INSC 343	2
+126	2022 INSC 431	2
+126	2022 INSC 427	2
+126	2023 INSC 252	2
+126	2025 INSC 979	2
+126	2025 INSC 935	2
+127	2019 INSC 889	3
+127	2019 INSC 1289	2
+127	2020 INSC 701	2
+127	2020 INSC 625	2
+127	2020 INSC 227	2
+127	2021 INSC 133	2
+127	2021 INSC 59	2
+127	2021 INSC 51	2
+127	2021 INSC 28	2
+127	2021 INSC 206	2
+127	2021 INSC 233	2
+128	1962 INSC 279	3
+128	2020 INSC 344	2
+128	2022 INSC 73	2
+128	2022 INSC 1175	2
+128	2024 INSC 562	2
+128	1995 INSC 375	2
+128	2011 INSC 388	2
+129	2019 INSC 1256	3
+129	2021 INSC 468	2
+129	2021 INSC 206	2
+129	2021 INSC 187	2
+129	2021 INSC 227	2
+129	2021 INSC 296	2
+129	2021 INSC 395	2
+129	2021 INSC 923	2
+129	2022 INSC 957	2
+129	2023 INSC 625	2
+129	2023 INSC 232	2
+130	2002 INSC 138	3
+130	2015 INSC 201	2
+130	2016 INSC 1196	2
+130	2017 INSC 589	2
+130	2019 INSC 817	2
+130	2020 INSC 659	2
+130	2021 INSC 264	2
+130	2024 INSC 857	2
+130	2024 INSC 607	2
+130	2007 INSC 796	2
+130	2011 INSC 683	2
+131	1999 INSC 407	3
+131	2020 INSC 339	2
+131	2020 INSC 364	2
+131	2000 INSC 562	2
+131	2006 INSC 711	2
+131	2012 INSC 558	2
+132	2014 INSC 590	3
+132	2017 INSC 801	2
+132	2018 INSC 1031	2
+132	2018 INSC 1193	2
+132	2019 INSC 1107	2
+132	2019 INSC 1216	2
+132	2019 INSC 518	2
+132	2019 INSC 196	2
+132	2020 INSC 624	2
+132	2022 INSC 565	2
+132	2023 INSC 264	2
+133	2009 INSC 693	3
+133	2015 INSC 1044	2
+133	2016 INSC 1184	2
+133	2020 INSC 577	2
+133	2021 INSC 118	2
+133	2010 INSC 238	2
+133	2011 INSC 20	2
+133	2011 INSC 552	2
+133	2013 INSC 176	2
+133	1997 INSC 275	2
+133	2012 INSC 70	2
+134	2013 INSC 179	3
+134	2016 INSC 934	2
+134	2020 INSC 276	2
+134	2021 INSC 218	2
+134	2021 INSC 283	2
+134	2022 INSC 1252	2
+134	2023 INSC 971	2
+134	2023 INSC 998	2
+135	1954 INSC 5	3
+135	2021 INSC 374	2
+135	2023 INSC 499	2
+135	1955 INSC 35	2
+135	1984 INSC 203	2
+135	2008 INSC 1337	2
+135	2009 INSC 1160	2
+135	1999 INSC 271	2
+136	2010 INSC 146	3
+136	2018 INSC 629	2
+136	2019 INSC 1300	2
+136	2019 INSC 63	2
+136	2019 INSC 718	2
+136	2020 INSC 557	2
+136	2020 INSC 452	2
+136	2020 INSC 218	2
+136	2021 INSC 558	2
+136	2024 INSC 791	2
+136	2023 INSC 560	2
+137	1952 INSC 28	3
+137	2017 INSC 945	2
+137	2019 INSC 597	2
+137	2025 INSC 481	2
+137	2008 INSC 1017	2
+137	2010 INSC 589	2
+137	2012 INSC 564	2
+137	2012 INSC 305	2
+137	2014 INSC 864	2
+138	2014 INSC 463	3
+138	2016 INSC 441	2
+138	2018 INSC 820	2
+138	2018 INSC 248	2
+138	2020 INSC 517	2
+138	2021 INSC 295	2
+138	2022 INSC 736	2
+138	2022 INSC 163	2
+138	2023 INSC 660	2
+138	2023 INSC 677	2
+138	2023 INSC 380	2
+139	2004 INSC 585	3
+139	2017 INSC 1286	2
+139	2018 INSC 797	2
+139	2019 INSC 400	2
+139	2020 INSC 525	2
+139	2021 INSC 817	2
+139	2022 INSC 757	2
+139	2008 INSC 938	2
+139	2009 INSC 885	2
+139	2020 INSC 294	2
+140	1999 INSC 299	3
+140	2020 INSC 697	2
+140	2023 INSC 4	2
+140	2000 INSC 38	2
+140	2003 INSC 638	2
+140	2010 INSC 624	2
+141	1955 INSC 15	3
+141	2017 INSC 855	2
+141	2018 INSC 1140	2
+141	2018 INSC 221	2
+141	2019 INSC 823	2
+141	2019 INSC 1236	2
+141	2020 INSC 320	2
+141	2022 INSC 579	2
+141	2022 INSC 681	2
+141	2023 INSC 717	2
+141	2004 INSC 608	2
+142	2006 INSC 326	3
+142	2017 INSC 1281	2
+142	2018 INSC 53	2
+142	2019 INSC 218	2
+142	2020 INSC 392	2
+142	2022 INSC 1043	2
+143	1996 INSC 237	3
+143	2018 INSC 804	2
+143	2019 INSC 651	2
+143	2024 INSC 178	2
+143	2024 INSC 545	2
+143	2013 INSC 528	2
+143	2025 INSC 491	2
+143	2012 INSC 486	2
+143	2014 INSC 48	2
+144	2002 INSC 203	3
+144	2018 INSC 288	2
+144	2018 INSC 437	2
+144	2019 INSC 1007	2
+144	2021 INSC 650	2
+144	2021 INSC 332	2
+144	2024 INSC 150	2
+144	2004 INSC 502	2
+144	2011 INSC 626	2
+144	2012 INSC 342	2
+144	2012 INSC 382	2
+145	1994 INSC 348	3
+145	2017 INSC 591	2
+145	2023 INSC 249	2
+145	1995 INSC 272	2
+145	2005 INSC 416	2
+145	2008 INSC 867	2
+145	2011 INSC 737	2
+145	2011 INSC 788	2
+146	1962 INSC 289	3
+146	2017 INSC 26	2
+146	2025 INSC 684	2
+146	2025 INSC 1130	2
+146	2006 INSC 681	2
+146	2010 INSC 204	2
+147	2011 INSC 301	3
+147	2016 INSC 1133	2
+147	2016 INSC 948	2
+147	2016 INSC 608	2
+147	2017 INSC 1038	2
+147	2018 INSC 1184	2
+147	2020 INSC 498	2
+147	2020 INSC 497	2
+147	2020 INSC 711	2
+147	2020 INSC 697	2
+147	2021 INSC 216	2
+148	1961 INSC 196	3
+148	2022 INSC 105	2
+148	2023 INSC 613	2
+148	1970 INSC 256	2
+148	2008 INSC 876	2
+149	1997 INSC 622	3
+149	2019 INSC 597	2
+149	2020 INSC 656	2
+149	2022 INSC 322	2
+150	2001 INSC 294	3
+150	2019 INSC 420	2
+150	2019 INSC 706	2
+150	2020 INSC 682	2
+150	2020 INSC 400	2
+150	2021 INSC 160	2
+150	2021 INSC 200	2
+150	2024 INSC 324	2
+150	2013 INSC 224	2

phase1/eval/authority_queries.tsv

@@ -0,0 +1,150 @@
+1	authority	can daily wage or temporary employees claim a right to regularization or permanent absorption in government service
+2	authority	can the Supreme Court depart from or overrule its own previous decision when satisfied of error
+3	authority	whether 'or' in section 24(2) between possession not taken and compensation not paid is read conjunctively as 'nor' so both conditions must be unmet for deemed lapse of acquisition
+4	authority	conferring uncontrolled discretion on government to pick cases for special court procedure without classification violates Article 14 equality
+5	authority	selection of multiplier and deduction for personal living expenses in computing motor accident compensation for deceased
+6	authority	constitutional validity of TADA and whether vague terrorism offence provisions violate fundamental rights
+7	authority	extent of government regulation over admissions and administration of private unaided and minority educational institutions
+8	authority	whether deposit of compensation in government treasury instead of court amounts to compensation paid under section 24(2) deemed lapse
+9	authority	scope of High Court power to reverse an acquittal in appeal and when two views are possible the one favouring the accused
+10	authority	scope of judicial review of government tender and contract award decisions under Article 14
+11	authority	can the Supreme Court suspend an advocate's licence to practice while punishing him for criminal contempt of court under Article 129 and 142
+12	authority	doctrine of merger applicability when special leave petition is dismissed in limine versus dismissed by a speaking order after grant of leave
+13	authority	constitutional validity of reservation in promotion with consequential seniority and the requirement of quantifiable data on backwardness inadequacy and efficiency
+14	authority	whether power of judicial review of High Courts under Article 226 and Supreme Court under Article 32 is part of the basic structure and can tribunals exclude it
+15	authority	categories and scope of inherent power of the High Court to quash criminal proceedings and FIR to prevent abuse of process
+16	authority	whether courts can award imprisonment for life with no remission for full term as an alternative to death penalty in cases short of rarest of rare
+17	authority	nature of the power of the Chief Justice to appoint an arbitrator under section 11 whether administrative or judicial and what issues he can decide
+18	authority	constitutional validity of the Insolvency and Bankruptcy Code and whether classification between financial creditors and operational creditors violates Article 14
+19	authority	addition for future prospects to income of deceased and standardized percentages for permanent job self employed and fixed salary in motor accident compensation
+20	authority	whether the right to privacy is a fundamental right protected under Article 21 as part of the right to life and personal liberty
+21	authority	duty of court to take active participatory role in collecting evidence and ensure a fair trial when witnesses turn hostile
+22	authority	whether High Court judges elevated from the Bar are entitled to same full pension as judges drawn from judicial service by adding notional service period
+23	authority	binding guidelines to prevent sexual harassment of women at the workplace in absence of legislation
+24	authority	ingredients to be proved for murder under section 300 thirdly intention to inflict the particular bodily injury sufficient in ordinary course of nature to cause death
+25	authority	scope of executive power of the State whether prior legislation is required for executive to act under Articles 73 and 162
+26	authority	whether conviction can be based on uncorroborated testimony of a single witness and classification of witnesses as reliable or unreliable
+27	authority	extent to which pleasure doctrine of holding office under Article 310 is subject to procedural safeguards and statutory rules in dismissal of public servant
+28	authority	power of appellate court to reverse an order of acquittal and re-appreciate evidence and adverse inference from false answers under section 313
+29	authority	stage and scope of power to summon additional accused under section 319 CrPC meaning of inquiry and trial
+30	authority	whether a decision of a coordinate bench is binding on a later bench of equal strength and doctrine of binding precedent
+31	authority	whether a state sales tax law that imposes tax on inter-State sales is merely dormant until Parliament lifts the ban under Article 286(2) or wholly void
+32	authority	can laws placed in the Ninth Schedule after 24 April 1973 be challenged for violating the basic structure and fundamental rights under Articles 14, 19 and 21
+33	authority	does mere non-deposit of compensation in court under the 1894 Act cause land acquisition to lapse under section 24(2), or does 'paid' mean tender of payment
+34	authority	how must courts weigh aggravating and mitigating circumstances and apply the rarest of rare doctrine before imposing the death penalty
+35	authority	test for repugnancy under Article 254 when Parliament and a State legislature both legislate under a Concurrent List entry occupying different fields
+36	authority	whether prize competitions involving gambling are protected as trade or business under Article 19(1)(g) and severability of a statute partly invalid
+37	authority	is a delinquent employee entitled to a copy of the inquiry officer's report before the disciplinary authority imposes punishment
+38	authority	what constitutes a substantial question of law required for admission of a second appeal under section 100 CPC
+39	authority	is registration of an FIR under section 154 mandatory when information discloses a cognizable offence or can the police hold a preliminary inquiry first
+40	authority	grounds on which a statute can be struck down as arbitrary and whether arbitrariness alone violates Article 14
+41	authority	whether Article 31A protection for acquisition of estates is limited to agrarian reform legislation
+42	authority	power of Election Commission under Article 324 to direct candidates to disclose criminal antecedents and assets
+43	authority	right of appeal is a vested substantive right governed by the law in force on the date of institution of the suit
+44	authority	whether courts can interfere with the election process before declaration of result or only by election petition under Article 329(b)
+45	authority	bar under Section 195(1)(b)(ii) CrPC applies only to forgery of a document committed while it is in custodia legis
+46	authority	whether separate charges and trials can be framed for distinct offences in a single conspiracy spanning different years
+47	authority	constitutional validity of enforcement of security interest without court intervention under SARFAESI Section 13
+48	authority	mandatory compliance with Section 50 NDPS Act right to be searched before a Gazetted Officer or Magistrate
+49	authority	whether auction is the only constitutionally permissible method for alienation of natural resources by the State
+50	authority	voter's right to know antecedents of candidates as a facet of Article 19(1)(a) and whether legislature can nullify it
+51	authority	is the right to education a fundamental right flowing from the right to life under article 21
+52	authority	can an arbitral award be set aside as against public policy of india for patent illegality
+53	authority	distinction between regulation of property under article 19(1)(f) and deprivation of property under article 31
+54	authority	can freedom of speech be restricted only to protect security of the state and not general public order
+55	authority	constitutional validity of 27 percent OBC reservation in central educational institutions and exclusion of creamy layer
+56	authority	does an illegal investigation in breach of mandatory provisions vitiate the trial without miscarriage of justice
+57	authority	strict interpretation of tax exemption notification and whether ambiguity benefits revenue or assessee
+58	authority	power of high court under section 482 to quash non-compoundable criminal proceedings on settlement between parties
+59	authority	validity of an administrative order judged only by reasons stated in the order itself
+60	authority	principles for grant and cancellation of bail and need for reasoned order exercising judicial discretion
+61	authority	precautionary principle and polluter pays principle as part of Indian environmental law sustainable development burden of proof on industry
+62	authority	scope of certiorari and judicial review over decisions of highest statutory appellate authority error within jurisdiction natural justice
+63	authority	distinction between tax and fee quid pro quo element of service rendered constitutional validity of cess
+64	authority	whether time limit to file written statement under Order VIII Rule 1 CPC is mandatory or directory power of court to extend time
+65	authority	service conditions and pay scales of subordinate judiciary parity of judicial officers with executive judicial pay commission
+66	authority	insurer's liability third party claim breach of policy condition driver without valid licence burden on insurer to prove breach contributed to accident
+67	authority	quashing criminal proceedings on ground of pending civil dispute breach of contract whether civil remedy bars criminal prosecution
+68	authority	allocation of scarce natural resources spectrum first come first served versus auction transparency arbitrariness Article 14
+69	authority	vicarious liability of director under Section 141 Negotiable Instruments Act necessity of specific averment in cheque dishonour complaint
+70	authority	reasonable classification under Article 14 permissible differentiation versus discrimination intelligible differentia special courts
+71	authority	can a writ petition under Article 226 be entertained to challenge a show cause notice when an alternative statutory remedy is available
+72	authority	extent of state regulation over admissions and fee structure in unaided minority and private professional educational institutions under Articles 29 and 30
+73	authority	power of states to levy tax on mineral bearing land under Entry 49 List II despite Centre's regulation of mines under the MMDR Act
+74	authority	what constitutes abetment of suicide under Section 306 IPC and whether mere harassment amounts to instigation
+75	authority	is residence in the state a constitutional requirement for election to the Rajya Sabha and validity of open ballot in Council of States elections
+76	authority	does a sentence of imprisonment for life mean imprisonment for the whole remaining life unless remitted or commuted by the appropriate government
+77	authority	whether a taxing statute imposing a flat rate of tax without classification and irrespective of income violates Article 14
+78	authority	maintainability of a curative petition to reconsider a final Supreme Court judgment after dismissal of review under inherent powers and Article 142
+79	authority	quashing of criminal proceedings under Section 482 CrPC where a civil dispute is given the cloak of a criminal offence
+80	authority	whether courts can supply a casus omissus and read words into a plain and unambiguous statutory provision
+81	authority	is requiring prior government sanction before CBI inquiry against senior officials a valid Article 14 classification under the Prevention of Corruption Act
+82	authority	constitutionality of reverse burden of proof and presumption provisions under the NDPS Act against presumption of innocence
+83	authority	right of private unaided professional colleges to fix their own fee structure and admission procedure and need for state regulatory committees
+84	authority	scope of court's power under section 11(6A) confined to examining existence of arbitration agreement at the appointment stage
+85	authority	when can a court or tribunal interfere with quantum of disciplinary penalty imposed by the disciplinary authority
+86	authority	is section 377 criminalising consensual same-sex acts between adults unconstitutional under Articles 14 and 21
+87	authority	scope of judicial review over Prime Minister's discretion to appoint persons with criminal antecedents as ministers
+88	authority	harmonising denominational temple management rights under Article 26(b) with public right of temple entry under Article 25(2)(b)
+89	authority	doctrine of repugnancy where a central law evinces intention to occupy the entire field overriding state legislation
+90	authority	rarest of rare doctrine and balancing crime against criminal in awarding death penalty
+91	authority	can employees who accept voluntary retirement scheme later claim revision of pay scale from a back date
+92	authority	grounds for cancellation of bail granted by sessions judge without reasons in dowry death case
+93	authority	scope of right to property under Article 300A and deprivation of property only by authority of law with public purpose
+94	authority	single shareholder locus standi to file writ petition under Article 32 challenging acquisition of company property
+95	authority	whether section 5 of Limitation Act applies to delay in filing application to set aside arbitral award beyond the prescribed period under section 34
+96	authority	credibility of interested or related witnesses and applicability of falsus in uno falsus in omnibus in criminal trial
+97	authority	applicability of Part I of Arbitration Act to foreign seated international commercial arbitration and interim relief under section 9
+98	authority	evidentiary value of confession of co-accused recorded under TADA and its use against other accused
+99	authority	scope of High Court interference in acquittal appeal reappreciating evidence where trial court view is reasonable
+100	authority	relevant date for determining age of juvenile offender date of offence or date of production before court
+101	authority	standard of care for criminal negligence against a doctor under section 304A and the requirement of gross negligence
+102	authority	factors for choosing between death penalty and life imprisonment based on character antecedents and reformability of the offender
+103	authority	test for what constitutes an industry under the Industrial Disputes Act and application of noscitur a sociis to the definition
+104	authority	where a statute prescribes the manner of doing an act it must be done in that manner or not at all
+105	authority	whether Article 166 of the Constitution on authentication of government orders is directory or mandatory
+106	authority	whether a director or signatory can be prosecuted under section 138 cheque dishonour without the company being arraigned as accused
+107	authority	requirement of hearing and recording of satisfaction for invoking urgency clause and dispensing with enquiry in land acquisition
+108	authority	whether rules of pleading and burden of proof apply to public interest litigation
+109	authority	binding effect of precedent and duty of coordinate and smaller benches to follow larger bench decisions under stare decisis
+110	authority	whether a dying declaration can be the sole basis for conviction without corroboration
+111	authority	do High Courts and the Supreme Court as courts of record have inherent power to punish contempt of subordinate courts
+112	authority	what does conscious possession of unauthorised arms in a notified area require under TADA section 5 and is the presumption rebuttable
+113	authority	can courts impose a special category life sentence beyond remission for a fixed term exceeding fourteen years instead of death penalty
+114	authority	burden on accused to explain wife's unnatural death in matrimonial home under section 106 Evidence Act in circumstantial evidence cases
+115	authority	whether a beneficial amendment during pendency of appeal applies retrospectively to defeat a vested right that accrued on the trial court decree
+116	authority	scope of judicial review of viva voce interview marks and selection process in public service recruitment
+117	authority	whether freedom of trade commerce and intercourse under Article 301 includes freedom from tax laws restricting movement of goods
+118	authority	when can an appellate court reverse an acquittal in a case based on circumstantial evidence and the only-perverse-view standard
+119	authority	when can an arbitral award be set aside under section 34 for breach of natural justice or conflict with the fundamental policy of Indian law after the 2015 amendment
+120	authority	requirements for reconversion to claim scheduled caste status and acceptance by the community
+121	authority	if landowner fails to file objections under Section 5A can he later challenge the land acquisition declaration
+122	authority	powers and principles for appellate court to interfere with and reverse an order of acquittal
+123	authority	conviction in rape case on sole uncorroborated testimony of prosecutrix and justification for delay in filing FIR
+124	authority	twin conditions for grant of bail under MCOCA and meaning of reasonable grounds to believe accused not guilty
+125	authority	whether doctrine of eclipse applies to a post-constitution law violating fundamental rights
+126	authority	can bail be granted on ground of parity to a history-sheeter habitual offender
+127	authority	are homebuyers financial creditors under IBC and does the Code prevail over RERA
+128	authority	permissible ceiling on reservation and caste as sole basis under Article 15(4)
+129	authority	commercial wisdom of committee of creditors and limited scope of judicial review of resolution plan under IBC
+130	authority	applicability of Part I of Arbitration Act to international commercial arbitration held outside India
+131	authority	can roster point reserved category promotees count seniority over senior general candidates who reach the promotional level later; is reservation in promotion under Article 16(4A) a fundamental right or only an enabling provision
+132	authority	is an oral hearing in open court mandatory at the review petition stage in death sentence cases under Article 21
+133	authority	does the Juvenile Justice Act apply to determine juvenility on the date of commission of the offence even in pending or already concluded cases; how is age of juvenility determined
+134	authority	does life imprisonment mean imprisonment for the whole of the convict's natural life subject only to statutory remission and constitutional clemency powers
+135	authority	is non-joinder of a necessary party or non-compliance with Section 82 of the Representation of the People Act fatal to an election petition, or is such a provision directory rather than mandatory
+136	authority	can the validity of regulations framed by a statutory regulator under delegated legislative power be challenged before the appellate tribunal or only by judicial review; distinction between an order and a regulation
+137	authority	is a land reform statute placed in the Ninth Schedule immune from challenge for failure to provide compensation for acquisition of property under Article 31
+138	authority	when offence is punishable up to seven years can police arrest automatically; mandatory safeguards and recording of reasons under Section 41 and 41A CrPC before arrest in Section 498A cases
+139	authority	do declaratory and explanatory amending statutes operate retrospectively; does substitution of a statutory provision amount to repeal and re-enactment for retrospective effect
+140	authority	is the exercise of a minister's discretionary quota in allotment subject to judicial review for arbitrariness and abuse of discretion in public law
+141	authority	if a party who was set ex parte appears at a later adjourned hearing of a suit, can the court let him participate and from what stage do proceedings continue under Order 9 Rule 7 CPC
+142	authority	is an interim or partial award under the Arbitration Act final and challengeable under Section 34, and can the arbitrator's interpretation of the contract including questions of law be interfered with
+143	authority	polluter pays principle and absolute liability of hazardous chemical industries to bear the cost of remediation, and power of Central Government to recover remedial costs under the Environment Protection Act
+144	authority	can courts fix outer time limits or bars of limitation for conclusion of criminal trials to enforce the right to speedy trial, or is that impermissible judicial legislation
+145	authority	procedure and guidelines for issuance, scrutiny and verification of caste or scheduled tribe status certificates and consequences of admissions obtained on a false caste certificate
+146	authority	meaning of manufacture for levy of excise duty and whether processing that does not bring into existence a new and distinct marketable commodity amounts to manufacture
+147	authority	which categories of disputes are non-arbitrable and when will a court refuse a Section 8 reference to arbitration because the subject matter is reserved for a public forum
+148	authority	does the reservation power under Article 16(4) extend to reservation in promotions to selection posts and not merely initial appointments
+149	authority	scope of judicial review of administrative action on grounds of unreasonableness and irrationality and applicability of the doctrine of proportionality in India
+150	authority	whether a second FIR can be registered for the same cognizable offence arising out of the same transaction under Section 154 CrPC

phase1/eval/bad_law_docids.txt

@@ -0,0 +1,104 @@
+2019 INSC 372
+2015 INSC 793
+2015 INSC 52
+2015 INSC 160
+2015 INSC 235
+2015 INSC 257
+2016 INSC 375
+2016 INSC 491
+2016 INSC 526
+2017 INSC 1026
+2017 INSC 1092
+2017 INSC 121
+2017 INSC 102
+2017 INSC 899
+2018 INSC 896
+2018 INSC 732
+2018 INSC 84
+2018 INSC 405
+2018 INSC 714
+2019 INSC 1184
+2019 INSC 511
+2020 INSC 294
+2021 INSC 314
+2022 INSC 1312
+1960 INSC 107
+1960 INSC 123
+1960 INSC 195
+1960 INSC 200
+1961 INSC 177
+1962 INSC 247
+1962 INSC 389
+1962 INSC 328
+1964 INSC 7
+1964 INSC 27
+1964 INSC 203
+1964 INSC 206
+1965 INSC 154
+1966 INSC 155
+1967 INSC 45
+1967 INSC 87
+1967 INSC 122
+1967 INSC 173
+1968 INSC 72
+1969 INSC 8
+1969 INSC 20
+1969 INSC 77
+1969 INSC 99
+1969 INSC 87
+1970 INSC 18
+1970 INSC 190
+1971 INSC 20
+1974 INSC 256
+1975 INSC 212
+1976 INSC 270
+1976 INSC 231
+1976 INSC 250
+1976 INSC 272
+1977 INSC 28
+1977 INSC 75
+1977 INSC 155
+1978 INSC 16
+1981 INSC 175
+1981 INSC 211
+1981 INSC 209
+1983 INSC 10
+1984 INSC 67
+1984 INSC 152
+1985 INSC 101
+1987 INSC 259
+1988 INSC 61
+1989 INSC 54
+1996 INSC 90
+1996 INSC 800
+1997 INSC 441
+1998 INSC 185
+2001 INSC 158
+2002 INSC 66
+2002 INSC 123
+2004 INSC 34
+2004 INSC 182
+2005 INSC 146
+2007 INSC 1026
+2007 INSC 28
+2007 INSC 475
+2007 INSC 772
+2008 INSC 930
+2008 INSC 82
+2008 INSC 677
+2009 INSC 946
+2009 INSC 1045
+2009 INSC 1195
+2009 INSC 209
+2010 INSC 69
+2010 INSC 177
+2011 INSC 508
+2012 INSC 49
+2013 INSC 830
+2013 INSC 684
+2013 INSC 823
+2013 INSC 377
+2013 INSC 494
+2014 INSC 218
+2014 INSC 617
+2014 INSC 579

phase1/eval/batched_run.py

@@ -0,0 +1,71 @@
+"""GPU-BATCHED retrieval harness: encode ALL queries at once, one big dense matmul, and ONE batched
+cross-encoder pass over every (query, candidate) pair — so the GPU is actually saturated. Turns a
+~10min serial run into ~30s. Same knobs (CAND, ALPHA). Emits run.tsv."""
+import json, os, time
+import numpy as np
+from sentence_transformers import SentenceTransformer, CrossEncoder
+DATA = os.environ.get("THEMIS_DATA", ".")
+EVAL = os.environ.get("THEMIS_EVAL", ".")
+DEVICE = os.environ.get("THEMIS_DEVICE", "cuda")
+CAND  = int(os.environ.get("CAND", "40"))
+ALPHA = float(os.environ.get("ALPHA", "0"))
+CE_BATCH = int(os.environ.get("CE_BATCH", "512"))
+BGE_Q = "Represent this sentence for searching relevant passages: "
+
+print("loading chunks/vectors/models ...", flush=True)
+texts = []; chunk_doc = []
+with open(os.path.join(DATA, "escr_chunks.jsonl"), encoding="utf-8") as f:
+    for l in f:
+        c = json.loads(l); texts.append(c["text"]); chunk_doc.append(c["doc_id"])
+M = np.load(os.path.join(DATA, "escr_vectors.npy"))                 # (nchunks, 384) float32
+cite_indeg = {}
+if ALPHA:
+    from collections import Counter
+    ci = Counter()
+    with open(os.path.join(DATA, "edges.jsonl"), encoding="utf-8") as f:
+        for l in f:
+            e = json.loads(l)
+            if e.get("method") == "cite": ci[e["target"]] += 1
+    cite_indeg = ci
+st = SentenceTransformer("BAAI/bge-small-en-v1.5", device=DEVICE)
+ce = CrossEncoder(os.environ.get("THEMIS_RERANKER","cross-encoder/ms-marco-MiniLM-L-6-v2"), device=DEVICE)
+print(f"ready (CAND={CAND} ALPHA={ALPHA} dev={DEVICE})", flush=True)
+
+QFILE = os.environ.get("THEMIS_QFILE", os.path.join(EVAL, "queries.tsv"))
+OUT = os.environ.get("THEMIS_OUT", os.path.join(EVAL, "run.tsv"))
+qids = []; qtexts = []
+with open(QFILE, encoding="utf-8") as f:
+    for l in f:
+        qid, intent, text = l.rstrip("\n").split("\t", 2); qids.append(qid); qtexts.append(text)
+nq = len(qids)
+t0 = time.time()
+# 1) batch-encode all queries, 2) dense top-CAND per query (chunked over queries to bound RAM)
+Q = st.encode([BGE_Q + q for q in qtexts], batch_size=256, normalize_embeddings=True, convert_to_numpy=True).astype(np.float32)
+cand_idx = np.empty((nq, CAND), dtype=np.int64)
+STEP = 200
+for s in range(0, nq, STEP):
+    sims = M @ Q[s:s+STEP].T                                        # (nchunks, b)
+    for j in range(sims.shape[1]):
+        col = sims[:, j]; top = np.argpartition(-col, CAND)[:CAND]
+        cand_idx[s+j] = top[np.argsort(-col[top])]
+print(f"dense done {time.time()-t0:.0f}s", flush=True)
+# 3) ONE batched cross-encoder pass over all (query, candidate-chunk) pairs
+pairs = []; owner = []
+for i in range(nq):
+    for ci_ in cand_idx[i]:
+        pairs.append((qtexts[i], texts[ci_])); owner.append(i)
+rr = ce.predict(pairs, batch_size=CE_BATCH, show_progress_bar=True)
+print(f"rerank done {time.time()-t0:.0f}s", flush=True)
+# 4) best chunk per doc per query, blend authority, rank
+def _sig(x): return 1.0 / (1.0 + np.exp(-x))
+best = [dict() for _ in range(nq)]
+for k, s in enumerate(rr):
+    i = owner[k]; d = chunk_doc[cand_idx[i][k % CAND]]
+    if d not in best[i] or s > best[i][d]: best[i][d] = float(s)
+with open(OUT, "w", encoding="utf-8") as f:
+    for i in range(nq):
+        scored = [(_sig(s) + (ALPHA * np.log1p(cite_indeg.get(d, 0)) if ALPHA else 0.0), d) for d, s in best[i].items()]
+        scored.sort(reverse=True)
+        for rank, (_, d) in enumerate(scored[:20], 1):
+            f.write(f"{qids[i]}\t{rank}\t{d}\n")
+print(f"done {nq} queries in {time.time()-t0:.0f}s -> run.tsv", flush=True)

phase1/eval/bench_queries.json

@@ -0,0 +1,17 @@
+[
+  {"id": "fact-1", "intent": "fact", "query": "husband and his family harassing wife for dowry, can the FIR under section 498A be quashed if the parties reach a settlement"},
+  {"id": "fact-2", "intent": "fact", "query": "cheque issued on behalf of a company was dishonoured, is the director personally liable under section 138 of the Negotiable Instruments Act"},
+  {"id": "fact-3", "intent": "fact", "query": "employee dismissed from service without a departmental inquiry, whether the termination violates principles of natural justice"},
+  {"id": "fact-4", "intent": "fact", "query": "accused seeking anticipatory bail in an economic offence involving diversion of investor money"},
+  {"id": "fact-5", "intent": "fact", "query": "government acquired private land and the owner claims the compensation awarded is far below the market value"},
+  {"id": "fact-6", "intent": "fact", "query": "person contracted a second marriage while the divorce petition from the first marriage was still pending, validity and bigamy"},
+  {"id": "issue-1", "intent": "issue", "query": "whether a dying declaration alone, without corroboration, is sufficient to sustain a conviction"},
+  {"id": "issue-2", "intent": "issue", "query": "the tests for grant of a temporary injunction: prima facie case, balance of convenience and irreparable injury"},
+  {"id": "issue-3", "intent": "issue", "query": "scope of judicial review of administrative action on the ground of arbitrariness under Article 14"},
+  {"id": "issue-4", "intent": "issue", "query": "whether bail once granted can be cancelled merely on the basis of subsequent developments"},
+  {"id": "vague-1", "intent": "vague", "query": "the supreme court judgment holding that privacy is a fundamental right, connected with the aadhaar matter"},
+  {"id": "vague-2", "intent": "vague", "query": "constitution bench decision on reservation in promotion for scheduled caste and scheduled tribe employees"},
+  {"id": "citation-1", "intent": "citation", "query": "(2017) 10 SCC 1"},
+  {"id": "casename-1", "intent": "casename", "query": "K.S. Puttaswamy v Union of India"},
+  {"id": "casename-2", "intent": "casename", "query": "Vishaka v State of Rajasthan"}
+]

phase1/eval/build_held_vectors.py

@@ -0,0 +1,18 @@
+"""Build doc-level HELD embeddings (the $0 representation win): one vector per judgment that has a
+HELD headnote (~56%), same BGE-small space. Output: held_vectors.npy + held_docids.json."""
+import json, os, time
+import numpy as np
+from sentence_transformers import SentenceTransformer
+DATA = os.environ.get("THEMIS_DATA", ".")
+docs = []; texts = []
+for l in open(os.path.join(DATA, "escr_meta.jsonl"), encoding="utf-8"):
+    m = json.loads(l); h = (m.get("held") or "").strip()
+    if len(h) > 40:
+        docs.append(m["doc_id"]); texts.append(h[:1800])
+print(f"{len(docs)} judgments with HELD", flush=True)
+st = SentenceTransformer("BAAI/bge-small-en-v1.5", device="cpu")
+t0 = time.time()
+V = st.encode(texts, batch_size=128, normalize_embeddings=True, convert_to_numpy=True).astype(np.float32)
+np.save(os.path.join(DATA, "held_vectors.npy"), V)
+json.dump(docs, open(os.path.join(DATA, "held_docids.json"), "w"))
+print(f"DONE {V.shape} in {time.time()-t0:.0f}s -> held_vectors.npy", flush=True)

phase1/eval/build_qrels.py

@@ -0,0 +1,96 @@
+"""Build the v1 MECHANICAL eval set (zero model, zero human, leakage-controlled).
+Outputs (TREC-style, frozen):
+  queries.tsv        qid \\t intent \\t query_text
+  qrels.tsv          qid \\t doc_id \\t grade        (graded 0/1/2/3; here single grade-3 target per query)
+  bad_law_docids.txt confirmed-overruled/doubted/per_incuriam doc_ids (the precision guardrail deny-list)
+
+Slices:
+  citing_passage  ~500  query = a citing court's snippet where it RELIED ON / FOLLOWED a case,
+                        with the cited case's NAME + CITATIONS stripped out (anti-leak); gold = that case (grade 3).
+                        Ground truth authored by real SC benches (edges.jsonl strong-positive treatments).
+  known_item_cite ~150  query = a neutral citation; gold = its own doc (grade 3).  success@1 control.
+  known_item_name ~150  query = a case name; gold = its own doc (grade 3).         success@1 control.
+"""
+import json, re, random, os
+random.seed(13)
+DATA = "/Users/gongura/Code/themis/phase1/data/thor_artifacts"
+OUT  = "/Users/gongura/Code/themis/phase1/eval"
+os.makedirs(OUT, exist_ok=True)
+
+STOP = set("the of and v vs versus state union india ltd co anr ors etc rep by".split())
+FUNC = set("the of a an that which is are was were to in on for by with as has have had not no be been it this these those under where when whether while held holds court case law cases section right rule order appeal".split())
+def name_tokens(nm):
+    return [t for t in re.findall(r"[a-z]+", (nm or "").lower()) if len(t) >= 4 and t not in STOP]
+_BRACKET = re.compile(r"\[[^\]]{0,40}\]")            # [Para 23], [1187-C], [122-E- G; 123-8]
+CITE_PAT = re.compile(r"\[?\(?\d{4}\)?\]?(?:\s*\(?\d+\)?){0,2}\s*(?:supp\.?\s*)?(?:scc\s*online\s*sc|s\s?\.?\s?c\s?\.?\s?r\.?|s\s?\.?\s?c\s?\.?\s?c\.?|a\s?\.?\s?i\s?\.?\s?r\.?|insc)\s*\.?(?:\s*\d+)?", re.I)
+
+print("loading meta + edges + good_law ...")
+meta = {}
+for l in open(f"{DATA}/escr_meta.jsonl"):
+    r = json.loads(l); meta[r["doc_id"]] = r
+edges = [json.loads(l) for l in open(f"{DATA}/edges.jsonl")]
+bad = [json.loads(l) for l in open(f"{DATA}/good_law.jsonl")]
+bad_ids = [g["doc_id"] for g in bad if g.get("good_law_status") in ("overruled", "doubted", "per_incuriam")]
+
+queries = []   # (qid, intent, text)
+qrels   = []   # (qid, doc_id, grade)
+qid = 0
+
+# --- slice 1: citing-passage -> relied-on authority (grade 3) ---
+STRONG = {"relied_on", "followed", "approved", "affirmed"}
+pos = [e for e in edges if e.get("treatment") in STRONG and e.get("para") and e.get("target") in meta]
+random.shuffle(pos)
+n = 0
+for e in pos:
+    if n >= 500: break
+    tgt = e["target"]; para = e["para"]
+    q = _BRACKET.sub(" ", para)                                # drop [Para..]/[pin] refs
+    for t in name_tokens(meta[tgt].get("case_name")):          # strip the cited case's distinctive name tokens
+        q = re.sub(r"\b" + re.escape(t) + r"\b", " ", q, flags=re.I)
+    q = CITE_PAT.sub(" ", q)                                    # strip citations
+    q = re.sub(r"\b(?:v|vs|versus)\.?\b", " ", q, flags=re.I)   # drop "X v Y" cross-ref connectors
+    q = re.sub(r"[^A-Za-z0-9 .,'-]", " ", q)
+    q = re.sub(r"\s+", " ", q).strip(" .,-;:")
+    words = q.split()
+    if len(words) < 12: continue                               # too short after stripping
+    if CITE_PAT.search(q) or re.search(r"\b(scc|scr|air|insc)\b", q, re.I): continue   # residual citation -> leaky
+    if sum(1 for w in words if w.lower() in FUNC) < 3: continue                        # must read like prose
+    if sum(1 for w in words if w.isupper() and len(w) > 1) > len(words) * 0.3: continue # too many ALLCAPS names
+    qid += 1; n += 1
+    queries.append((qid, "citing_passage", q))
+    qrels.append((qid, tgt, 3))
+n_cite_passage = n
+
+# --- slice 2 + 3: known-item lookups (grade 3, single target) ---
+docs = [r for r in meta.values() if r.get("neutral_citation") and r.get("case_name")]
+random.shuffle(docs)
+n_ki_cite = n_ki_name = 0
+for r in docs:
+    d = r["doc_id"]
+    if n_ki_cite < 150:
+        qid += 1; n_ki_cite += 1
+        queries.append((qid, "known_item_cite", r["neutral_citation"]))
+        qrels.append((qid, d, 3))
+    elif n_ki_name < 150:
+        nm = re.sub(r"\s*&\s*(anr|ors)\.?", "", r["case_name"], flags=re.I).strip()
+        nm = re.sub(r"\s+", " ", nm)
+        qid += 1; n_ki_name += 1
+        queries.append((qid, "known_item_name", nm))
+        qrels.append((qid, d, 3))
+    if n_ki_cite >= 150 and n_ki_name >= 150: break
+
+# --- write frozen files ---
+with open(f"{OUT}/queries.tsv", "w") as f:
+    for q, intent, text in queries: f.write(f"{q}\t{intent}\t{text}\n")
+with open(f"{OUT}/qrels.tsv", "w") as f:
+    for q, d, g in qrels: f.write(f"{q}\t{d}\t{g}\n")
+with open(f"{OUT}/bad_law_docids.txt", "w") as f:
+    f.write("\n".join(bad_ids) + "\n")
+
+print(f"citing_passage : {n_cite_passage}")
+print(f"known_item_cite: {n_ki_cite}")
+print(f"known_item_name: {n_ki_name}")
+print(f"TOTAL queries  : {len(queries)}   qrels rows: {len(qrels)}   bad-law deny-list: {len(bad_ids)}")
+print("--- sample citing_passage queries (name/cite stripped) ---")
+for q, intent, text in [x for x in queries if x[1] == "citing_passage"][:4]:
+    print(f"  q{q}: {text[:120]}")

phase1/eval/classify_intent.py

@@ -0,0 +1,67 @@
+"""Round-0 intent classifier (the 'router' as a state-setter). Labels each query AUTHORITY
+(wants the leading/landmark case on a doctrine -> apply the authority prior) vs SPECIFIC (a
+particular case / fact-pattern / narrow holding -> no prior). DeepSeek, parallel, cached to JSON.
+
+Usage: THEMIS_QFILE=authority_queries.tsv OUT=intent_authority.json python classify_intent.py
+Reads .env for DEEPSEEK_API_KEY. Cached by qid so re-runs are free; delete the OUT file to refresh.
+"""
+import os, json, sys, time, concurrent.futures as cf
+import requests
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+def _load_env(p):
+    if os.path.exists(p):
+        for l in open(p):
+            l = l.strip()
+            if l and not l.startswith("#") and "=" in l:
+                k, v = l.split("=", 1); os.environ.setdefault(k.strip(), v.strip().strip('"').strip("'"))
+_load_env(os.path.join(HERE, "..", "scripts", ".env"))
+KEY = os.environ["DEEPSEEK_API_KEY"]
+HDR = {"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"}
+QFILE = os.environ.get("THEMIS_QFILE", "authority_queries.tsv")
+OUT = os.environ.get("OUT", "intent.json")
+
+SYS = ("You route a legal search query for an Indian Supreme Court case-law engine. "
+       "Decide what the user is after:\n"
+       "AUTHORITY = they want the leading / landmark / controlling case(s) on a legal PRINCIPLE, "
+       "doctrine, right, or test (e.g. 'is privacy a fundamental right', 'test for sedition', "
+       "'doctrine of basic structure').\n"
+       "SPECIFIC = they want a particular named case, a narrow fact-pattern match, a specific "
+       "statutory provision's application, or a procedural/factual lookup where the single most "
+       "authoritative landmark is NOT necessarily the right answer.\n"
+       "Reply with EXACTLY one word: AUTHORITY or SPECIFIC.")
+
+def classify(text):
+    for attempt in range(3):
+        try:
+            r = requests.post("https://api.deepseek.com/chat/completions", headers=HDR, timeout=40,
+                json={"model": "deepseek-chat", "temperature": 0, "max_tokens": 4,
+                      "messages": [{"role": "system", "content": SYS}, {"role": "user", "content": text}]})
+            if r.status_code == 200:
+                t = r.json()["choices"][0]["message"]["content"].strip().upper()
+                return "AUTHORITY" if "AUTHORITY" in t else "SPECIFIC"
+        except Exception:
+            time.sleep(2 * (attempt + 1))
+    return "SPECIFIC"  # fail-safe: no prior
+
+def main():
+    rows = []
+    for l in open(QFILE, encoding="utf-8"):
+        qid, intent, text = l.rstrip("\n").split("\t", 2); rows.append((qid, text))
+    cache = json.load(open(OUT)) if os.path.exists(OUT) else {}
+    todo = [(qid, text) for qid, text in rows if qid not in cache]
+    print(f"{len(rows)} queries, {len(todo)} to classify ({len(cache)} cached)", flush=True)
+    t0 = time.time()
+    with cf.ThreadPoolExecutor(max_workers=24) as ex:
+        futs = {ex.submit(classify, text): qid for qid, text in todo}
+        done = 0
+        for f in cf.as_completed(futs):
+            cache[futs[f]] = f.result(); done += 1
+            if done % 50 == 0:
+                json.dump(cache, open(OUT, "w")); print(f"  {done}/{len(todo)} {time.time()-t0:.0f}s", flush=True)
+    json.dump(cache, open(OUT, "w"))
+    n_auth = sum(1 for v in cache.values() if v == "AUTHORITY")
+    print(f"done {len(cache)} -> {OUT} | AUTHORITY={n_auth} ({100*n_auth/len(cache):.0f}%) SPECIFIC={len(cache)-n_auth}", flush=True)
+
+if __name__ == "__main__":
+    main()

phase1/eval/embed_chunks.py

@@ -0,0 +1,20 @@
+"""Re-embed the corpus chunks with BGE-small on the GPU to regenerate escr_vectors.npy locally
+(faster than transferring the 1.9GB float32 matrix over Thor's slow link). Embeds in FILE ORDER so
+the vector index matches escr_chunks.jsonl line order, exactly as serve.py expects. Documents are
+embedded PLAIN (no query instruction prefix — that's query-side only)."""
+import json, time, os
+import numpy as np
+from sentence_transformers import SentenceTransformer
+DATA = os.environ.get("THEMIS_DATA", ".")
+DEV  = os.environ.get("THEMIS_DEVICE", "cuda")
+texts = []
+with open(os.path.join(DATA, "escr_chunks.jsonl"), encoding="utf-8") as f:
+    for l in f:
+        texts.append(json.loads(l)["text"])
+print(f"{len(texts)} chunks; embedding on {DEV} ...", flush=True)
+st = SentenceTransformer("BAAI/bge-small-en-v1.5", device=DEV)
+t0 = time.time()
+M = st.encode(texts, batch_size=512, normalize_embeddings=True, convert_to_numpy=True,
+              show_progress_bar=True).astype(np.float32)
+np.save(os.path.join(DATA, "escr_vectors.npy"), M)
+print(f"done {M.shape} {M.dtype} in {time.time()-t0:.0f}s -> escr_vectors.npy", flush=True)

phase1/eval/fetch_feedback.py

@@ -0,0 +1,72 @@
+#!/usr/bin/env python3
+"""Pull lawyer feedback bundles from the private HF dataset and digest them.
+
+Feedback Mode (the Vercel UI toggle) posts judgment bundles to /api/feedback_bundle,
+which the Space pushes durably to the private dataset vg15o2/themis-feedback
+(one JSON per share under feedback/YYYY-MM-DD/). This script is the owner side:
+download everything, print a digest, dump a flat JSONL, and optionally emit
+qrels-candidate rows for the eval set.
+
+Grade map (placement -> graded relevance): top5=3, top10=2, after10=1, irrelevant=0.
+
+Usage:
+  python phase1/eval/fetch_feedback.py              # digest + feedback_dump.jsonl
+  python phase1/eval/fetch_feedback.py --to-qrels   # + feedback_qrels_candidates.tsv
+Needs the HF write token (read suffices) in ~/.git-credentials or HF_TOKEN env.
+"""
+import glob, json, os, socket, subprocess, sys
+
+_o = socket.getaddrinfo
+socket.getaddrinfo = lambda h, p, f=0, *a, **k: _o(h, p, socket.AF_INET, *a, **k)  # IPv6-first DNS hangs on this box
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+GRADE = {"top5": 3, "top10": 2, "after10": 1}
+
+def token():
+    t = os.environ.get("HF_TOKEN")
+    if t: return t
+    return subprocess.run(["bash", "-c",
+        "grep -m1 'huggingface.co' ~/.git-credentials | sed -E 's#https://[^:]*:([^@]+)@.*#\\1#'"],
+        capture_output=True, text=True).stdout.strip()
+
+def main():
+    from huggingface_hub import snapshot_download
+    local = snapshot_download("vg15o2/themis-feedback", repo_type="dataset", token=token())
+    files = sorted(glob.glob(os.path.join(local, "feedback", "**", "*.json"), recursive=True))
+    bundles = []
+    for f in files:                                   # daily arrays (current) or single objects (legacy)
+        data = json.load(open(f, encoding="utf-8"))
+        bundles.extend(data if isinstance(data, list) else [data])
+    print(f"[feedback] {len(bundles)} bundle(s) across {len(files)} file(s)\n")
+
+    dump = os.path.join(HERE, "feedback_dump.jsonl")
+    with open(dump, "w", encoding="utf-8") as f:
+        for b in bundles: f.write(json.dumps(b, ensure_ascii=False) + "\n")
+
+    for b in bundles:
+        judged = [r for r in b.get("results", []) if r.get("placement") or r.get("irrelevant") or (r.get("comment") or "").strip()]
+        print(f"— {b.get('server_ts', b.get('ts',''))[:16]}  {b.get('name','?')}  [{b.get('mode','auto')}]")
+        print(f"  Q: {b.get('q','')[:100]}")
+        for r in judged:
+            mark = "IRRELEVANT" if r.get("irrelevant") else (r.get("placement") or "")
+            c = (r.get("comment") or "").strip()
+            print(f"    #{r.get('rank_shown','?'):>3} {mark:<10} {r.get('case_name','')[:52]}" + (f'  "{c[:70]}"' if c else ""))
+        if (b.get("missing_case") or "").strip(): print(f"    MISSING: {b['missing_case'][:100]}")
+        if (b.get("additional") or "").strip():   print(f"    NOTE: {b['additional'][:140]}")
+        print()
+    print(f"[feedback] dump -> {dump}")
+
+    if "--to-qrels" in sys.argv:
+        out = os.path.join(HERE, "feedback_qrels_candidates.tsv")
+        with open(out, "w", encoding="utf-8") as f:
+            f.write("# qid(query text)\tdoc_id\tgrade\tsource\n")
+            for b in bundles:
+                for r in b.get("results", []):
+                    if r.get("irrelevant"): g = 0
+                    elif r.get("placement") in GRADE: g = GRADE[r["placement"]]
+                    else: continue
+                    f.write(f"{b.get('q','')}\t{r['doc_id']}\t{g}\t{b.get('name','?')}\n")
+        print(f"[feedback] qrels candidates -> {out} (review before merging into the eval set)")
+
+if __name__ == "__main__":
+    main()

phase1/eval/gold_foundational.json

@@ -0,0 +1,17 @@
+[
+ {"id":"fact-1","query":"husband and his family harassing wife for dowry, can the FIR under section 498A be quashed if the parties reach a settlement","control":false,"foundational":[["gian","singh","punjab"],["jitendra","raghuvanshi"]]},
+ {"id":"fact-2","query":"cheque issued on behalf of a company was dishonoured, is the director personally liable under section 138 of the Negotiable Instruments Act","control":false,"foundational":[["aneeta","hada"],["pharmaceuticals","neeta"]]},
+ {"id":"fact-3","query":"employee dismissed from service without a departmental inquiry, whether the termination violates principles of natural justice","control":false,"foundational":[["tulsiram","patel"],["maneka","gandhi"]]},
+ {"id":"fact-4","query":"accused seeking anticipatory bail in an economic offence involving diversion of investor money","control":false,"foundational":[["chidambaram"],["sushila","aggarwal"]]},
+ {"id":"fact-5","query":"government acquired private land and the owner claims the compensation awarded is far below the market value","control":false,"foundational":[]},
+ {"id":"fact-6","query":"person contracted a second marriage while the divorce petition from the first marriage was still pending, validity and bigamy","control":false,"foundational":[["sarla","mudgal"]]},
+ {"id":"issue-1","query":"whether a dying declaration alone, without corroboration, is sufficient to sustain a conviction","control":false,"foundational":[["khushal","rao"]]},
+ {"id":"issue-2","query":"the tests for grant of a temporary injunction: prima facie case, balance of convenience and irreparable injury","control":false,"foundational":[["dalpat","kumar"],["gujarat","bottling"]]},
+ {"id":"issue-3","query":"scope of judicial review of administrative action on the ground of arbitrariness under Article 14","control":false,"foundational":[["shrilekha","vidyarthi"],["royappa"]]},
+ {"id":"issue-4","query":"whether bail once granted can be cancelled merely on the basis of subsequent developments","control":false,"foundational":[["dolat","ram"]]},
+ {"id":"vague-1","query":"the supreme court judgment holding that privacy is a fundamental right, connected with the aadhaar matter","control":false,"foundational":[["puttaswamy"]]},
+ {"id":"vague-2","query":"constitution bench decision on reservation in promotion for scheduled caste and scheduled tribe employees","control":false,"foundational":[["indra","sawhney"],["jarnail","singh"]]},
+ {"id":"citation-1","query":"(2017) 10 SCC 1","control":true,"foundational":[["puttaswamy"]]},
+ {"id":"casename-1","query":"K.S. Puttaswamy v Union of India","control":true,"foundational":[["puttaswamy"]]},
+ {"id":"casename-2","query":"Vishaka v State of Rajasthan","control":true,"foundational":[["vishaka","rajasthan"]]}
+]