Spaces:

vg15o2
/

themis

Running

App Files Files Community

themis / phase1 /HANDOFF.md

vg15o2

Moonley backend (HF Space build)

9e07f5b 3 days ago

preview code

Raw

History Blame Contribute Delete

24.6 kB

Themis — Search-Relevance Handoff

Owner handoff for whoever drives Themis search quality next. Covers: what the product is, exactly how search works today (fast + deep), step by step, the known weaknesses, and — most importantly — the autoresearch methodology, the eval rig, the compute setup, and the experiment discipline so you can run the propose→measure→commit loop yourself from day one. Read §4–§7 before touching any code.

1. What Themis is & the goal

Grounded AI legal research over 37,898 reportable Indian Supreme Court judgments (the open AWS registry indian-supreme-court-judgments, ap-south-1). It retrieves on-point cases for a lawyer's query and produces a grounded summary (every asserted point backed by a verbatim quote from a retrieved case).

North star right now: make search quality good enough for a live pilot with practising lawyers (via Hitin, the lawyer co-founder). A blind benchmark vs CaseMine put us at nDCG@5 0.77 vs 0.64 — but it was a 3–3 split and partly self-graded (see §4), and the real, repeatedly-confirmed gap is foundational-authority recall on doctrinal queries (landmark cases like Khushal Rao, Indra Sawhney rank too low or don't surface).

2. How search works TODAY — step by step

2.0 Serving stack

phase1/scripts/serve.py (FastAPI) + frontend.html (vanilla JS). The corpus is chunked: 1,299,748 passages. Each chunk has a BGE-small-en-v1.5 embedding (384-d, float32 matrix M = escr_vectors.npy) and sits in a BM25 index. The serving LLM is DeepSeek (deepseek-chat) via `/.../.env DEEPSEEK_API_KEY. Artifacts: escr_chunks.jsonl(chunk text),escr_vectors.npy, escr_meta.jsonl(per-judgment metadata incl.issue/heldheadnotes),edges.jsonl(citation graph, 86,702 edges),good_law.jsonl(citator),escr_pdfmap.jsonl`.

Shared retrieval primitives

dense(q) — embed "Represent this sentence for searching relevant passages: " + q with BGE-small, cosine M @ qv, take top CAND=40 chunks.
bm25_top(q) — BM25 over tokenized chunks, top 40. ⚠ rank_bm25.get_scores scans 1.3M postings in pure Python = ~68 s/query — the reason the eval can't use the full serve path (see §6).
candidates(q) — RRF-fuse dense+BM25 (k=60), top ~48 chunk candidates.
rerank(q,cand,k) — cross-encoder ms-marco-MiniLM-L-6-v2 scores (q, chunk), dedups to best chunk per judgment, returns top-k docs (rr = cross-encoder score).
verify(q,results) — a DeepSeek "paralegal" labels each result relevant/partial/not, seeing only a ~280-char passage_snippet (NOT the holding). On JSON parse failure → everything defaults to partial.
Grounding gate (grounded_answer/verify_claims) — DeepSeek emits {claim, n, quote} triples; the gate drops any claim whose quote isn't a ≥4-word verbatim substring of cited case n's chunk; renders only survivors. (Checks provenance, not entailment — a real quote can be misread; that's a known residual.)
Good-law mask is "dark": only confirmed-overruled is flagged; unknown shows nothing.

FAST mode — `GET /api/search_stream` (SSE, stepwise)

F0 ROUTER identity_hits(q) — if q is a bare citation (regex) or "X v Y" name (fuzzy name_search, difflib + cite_indeg salience tiebreak), it's a lookup → metadata id_card, bypass the pipeline.
F1 RETRIEVE candidates(q) → ~48 hybrid chunks.
F2 RERANK cross-encoder → top 12 judgments.
F3 REVIEW verify → keep relevant+partial, drop not.
F4 THIN-GUARD if kept<4 → one DeepSeek query rewrite → re-retrieve/rerank/verify, add new.
F5 ORDER sort relevant-first, then by rr; top 8.
F6 GOOD-LAW dark mask. ⚠ In fast mode this is currently a no-op — the SSE label says "Checked which results are still good law" but NO fast-path code filters/demotes overruled cases (deep mode does). Fix pending.
F7 ANSWER grounded summary (claims + verbatim-quote gate).

DEEP mode — `GET /api/deep_search_stream`

D0 ROUTER same identity bypass.
D1 PLAN DeepSeek decomposes the issue into 1–3 sub-issues + names ≤5 leading authorities a lawyer expects (names only; each grounded via name_search — a hallucinated name simply fails to resolve).
D2 SEED retrieve(q,12) then verify → relevant/partial seed set.
D3 EXPAND (the only agentic step, one batch): (a) ground each named authority via name_search; (b) citation neighbours — from top-6 seed docs, pull what THEY cite (out_edges), take 6 most-common; (c) card_for_doc per added doc (reranks only its first 6 chunks — cis[:6], a real bug: a landmark's holding is often deeper → it gets mis-scored and dropped) → verify → keep relevant/partial AND not-overruled.
D4 AUTHORITY-CHECK report which named authorities landed (conflates "found but off-point" vs "not in corpus").
D5 RANK a reviewer-confirmed PLAN authority that's relevant gets a top slot; then relevant by rr; then partials; top 10. ⚠ authority bonus is binary + AND-gated on the snippet-derived relevant label.
D6 GOOD-LAW + ANSWER same as fast.

Other endpoints

/api/judgment?id= (full judgment view — metadata, issue/held headnote, citator, dark good-law, cleaned text, + /api/pdf?id= which pulls the official SCR PDF from the open registry and caches ≤20 locally, embedded inline), /api/search (non-stream fallback). The "ask this judgment" feature was removed.

3. Known weaknesses (panel-reviewed, grounded in serve.py) — prioritized

The foundational-recall gap is created in F1 (shallow 48-chunk pool), never recovered because ranking ignores authority (F5/D5 use raw rr; cite_indeg/bench/date sit on every card unused), and actively worsened by F3 deleting weakly-phrased landmarks on a 280-char snippet and D3's cis[:6] bug. Plus the F6 fast-mode good-law no-op (a trust defect: an overruled case can rank #1 under a false "checked" label; only ~40 of 43,175 docs are ever flaggable, partly_overruled in the D3 filter is dead code, doubted is wrongly omitted). Both models (BGE-small embedder, ms-marco reranker) are general web English, not legal/India-tuned.

Full per-step verdicts + the prioritized sequence are in the memory file index and the panel transcripts (§9).

4. The AUTORESEARCH approach (the methodology — read this)

Modeled on Karpathy's AutoResearch (Mar 2026): an agent runs experiments in a loop — read code → propose ONE change → run a short job → measure ONE mechanical metric → git commit if it improved / git revert if not → repeat. Three pillars: (1) a hard constraint, (2) one mechanical metric, (3) autonomous propose/score/commit. Plus his older "Recipe" discipline: become one with the data first, dumb baseline, change ONE thing at a time, never add unverified complexity.

How we adapt it (the rules — do not break these):

The metric is a FROZEN qrels file scored by pure arithmetic — NO LLM at score time. This makes it millisecond-cheap to recompute and non-self-gradeable by construction.
The judge must NEVER be the serving model family. Our live verify gate is DeepSeek, so DeepSeek is banned from labeling the eval (that's why the old 0.77 number is inadmissible — 20_score_benchmark.py graded DeepSeek-with-DeepSeek). Labels came from Claude (a different family); Codex is the cross-family second judge; the citation graph + known-item are model-free anchors.
Commit a change only if its metric delta clears the paired-bootstrap 95% CI AND no guardrail regresses.
Cheap vs expensive experiments: anything that only re-ranks (pool depth, authority α, reranker swap, the ordering rule) is a ~65 s offline re-score → run hundreds. Anything that re-embeds the 1.3M corpus (swapping the embedder) is expensive and gated behind a proven "landmarks enter the pool but rank low" signal.
Precision-first: a wrong overruled-case shown to a lawyer is the catastrophic error → the bad-law@10 guardrail and known-item success@1 can never regress for a commit to count.

5. The EVAL RIG (the measurement substrate) — `phase1/eval/`

Frozen set: 800 queries (queries.tsv + qrels.tsv + bad_law_docids.txt):

500 SILVER doctrinal / fact-pattern / vague — generated by Claude subagents (a Workflow, 25 agents) that read a judgment's held/issue headnote and wrote 2 natural, anti-leak queries it answers (NEVER naming the case/citation); gold = that source case (grade 3). gen_sample.json = the 250 sampled judgments. This is silver → to be audited by Hitin (double-label a stratified ~200 sample, compute Cohen's κ, promote to gold). The Claude+Codex dual-judge for graded multi-relevant labels (pooling top-k from fast/deep/BM25) is the next eval upgrade.
300 known-item (150 neutral-citation + 150 case-name) — gold = own doc; the success@1 control.
104 bad-law deny-list (overruled/doubted/per_incuriam doc_ids) — the model-free precision guardrail.

⚠ Critical caveat: the silver gold = a random source case, not a landmark → this set measures GENERAL retrieval, NOT foundational-authority recall. The authority prior (PageRank/cite_indeg) must be tested on a landmark set (extend eval/gold_foundational.json), where gold IS a landmark — on the silver set it catastrophically hurts (see §7). Building a bigger landmark/foundational gold set is a top eval to-do.

Metric suite (score_qrels.py, pure numpy, frozen, no network):

PRIMARY: nDCG@10 (graded, 2^g−1 gain) + nDCG@5.
GUARDRAILS: known-item success@1 (must stay ~1 once the router is in the harness) and bad-law@10 (lower is better; a commit that raises it is auto-rejected).
DIAGNOSTICS: recall@10, MAP@20, MRR, per-intent breakdown.
All with bootstrap-over-queries 95% CIs.

Harnesses:

lean_run.py — serial, dense+CE, drops BM25 (the 68s killer), env knobs. ~0.74 s/q.
batched_run.py — GPU-batched (encode all queries → one dense matmul → ONE batched cross-encoder pass) → full 800-query run in ~65 s (~9× the serial). This is the loop harness.
sweep.py — loads the corpus once and scores many (reranker × CAND × ALPHA) configs with paired bootstrap vs baseline. The autonomous experiment driver.
embed_chunks.py — re-embed the corpus on GPU (exists but too slow ~80 min — transfer original vectors instead).

6. COMPUTE — where & how to run

The Windows GPU box (the experiment machine)

ssh admin@100.81.98.43 (tailnet). RTX 5060 Ti 16 GB (Blackwell sm_120), 62 GB RAM, Python 3.11, default shell PowerShell. App dir C:\Users\admin\themis. Setup gotchas you WILL hit:

My key is in C:\ProgramData\ssh\administrators_authorized_keys with an icacls perms lock — required or sshd silently ignores it (admin accounts).
torch must be +cu128 for Blackwell. The venv's old pip backtracks to the CPU wheel — pip install -U pip first, then pip install torch --index-url https://download.pytorch.org/whl/cu128 (NOT --extra-index-url, which re-picks the CPU build). Current: torch 2.11.0+cu128, CUDA True.
PYTHONUTF8=1 is MANDATORY — Windows open() defaults to cp1252 → UnicodeDecodeError on the legal text.
Box is set to never-sleep + tailscale up --unattended so it stays reachable logged-out.

Run an experiment (the loop in practice):

cd $env:USERPROFILE\themis
$env:PYTHONUTF8="1"; $env:THEMIS_DATA="."; $env:THEMIS_EVAL="."; $env:THEMIS_DEVICE="cuda"
# one config:
$env:CAND="40"; $env:ALPHA="0"; $env:THEMIS_RERANKER="cross-encoder/ms-marco-MiniLM-L-6-v2"
.\venv\Scripts\python.exe batched_run.py        # -> run.tsv  (~65s)
.\venv\Scripts\python.exe score_qrels.py run.tsv
# or sweep many at once:
.\venv\Scripts\python.exe sweep.py

Getting artifacts onto the box (it was painful — documented so you don't repeat it)

Mac→box tailnet is DERP-relayed (~~0.8 MB/s, useless). Mac can't reach Thor from its network. Thor (the old Jetson, now 100.99.130.27 tailnet, flaky) shares a LAN with the box → the box pulls Thor:`~~/backup over LAN (192.168.1.76, ~3–5 MB/s). We added the box's own SSH key to Thor (from the Mac, retrying through Thor's flakiness). **Re-embedding on the 5060 Ti is too slow (~80 min, CPU-tokenization-bound) — transferring the original escr_vectors.npy` (~10 min) beats it and is exact.**

Other places it runs

Local serve (the live app, for demoing/QA): on the Mac, .venv (py3.12) + torch CPU, launched from the artifacts dir: THEMIS_LOG_DIR=… .venv/bin/uvicorn --app-dir phase1/scripts serve:app --host 127.0.0.1 --port 8000 (artifacts opened CWD-relative; scripts/.env holds the DeepSeek key; no passcode on localhost).
Pilot deploy: phase1/deploy/ has a ready Caddy + systemd + runbook for a Hetzner CPX41 (Caddy auto-TLS → uvicorn + a passcode gate, themis.apexflo.ai). Serving is CPU-only — no GPU needed to host. Not yet provisioned.

7. Results so far (the experiment log — keep appending to this)

Config	nDCG@10	recall@10	succ@1 (known)	bad-law@10	Δ vs baseline (paired bootstrap)	Verdict
ms-marco, CAND=40, α=0 (baseline)	0.517	0.595	0.31	0.037	—	baseline
ms-marco, CAND=100, α=0	0.532	0.611	0.34	0.036	+0.015 [+0.005, +0.026] ✓sig	COMMIT (deeper pool helps)
ms-marco, CAND=40, α=0.5	0.231	0.495	0.02	0.102	−0.286 [−0.309, −0.262]	REJECT (authority prior wrong on silver)
ms-marco, CAND=100, α=0.5	0.131	0.299	0.00	0.179	−0.386	REJECT

7b. The AUTHORITY slice — `authority_{queries,qrels,badlaw}` (150 landmark doctrinal queries)

Built to make the rig see the foundational-authority gap the silver set hides (silver gold is the random source-case, not the landmark; so it actively penalizes authority). Gold = the doctrine's landmark (grade 3) + strong-citation progeny (grade 2). Run with THEMIS_QFILE=authority_queries.tsv THEMIS_QRELS=authority_qrels.tsv THEMIS_QUERIES=authority_queries.tsv THEMIS_BADLAW=authority_badlaw.txt.

Config	nDCG@10	nDCG@5	MRR	recall@10	bad-law@10	Verdict
ms-marco, α=0 (baseline)	0.282 [.253,.312]	—	0.551	0.204	0.113	baseline — the gap, quantified (vs 0.517 silver)
bge-reranker-base, α=0	0.283 [.251,.313]	0.315	0.553	0.195	0.127	REJECT — Δ+0.001, a better topical reranker does NOT find foundational law
ms-marco, α=0.3	0.354 [.324,.385]	—	0.692	0.195	0.173	win on nDCG but bad-law regresses (boosts overruled ex-landmarks)
ms-marco, α=0.6	0.353	—	0.686	0.194	0.180	α plateaus past 0.3
ms-marco, α=0.3 + good-law filter	0.351 [.320,.384]	0.416	0.693	0.193	0.000	COMMIT — full nDCG gain held, bad-law → 0

Reading — three decisive results:

Reranker swap is dead. bge ≈ ms-marco (+0.001) on the exact slice it was meant to fix. Topical rerankers under-score old-language landmarks regardless of model. Don't ship the 1.1GB model / 10× slower CPU pass.
The authority prior is the lever: +0.072 nDCG@10 (+24% rel), MRR 0.551→0.692; recall@10 flat → it's a pure ranking fix (landmarks were always in-pool, just low) — exactly the diagnosis the pool-probe gave.
It ships gated by good-law: authority alone lifts bad-law (0.113→0.173); the denylist filter holds the gain (0.351) and drops bad-law to 0.000. Authority prior + good-law filter are one unit, never shipped apart.

The architecture consequence: the prior is −0.29 on general silver, +0.072 on doctrinal → it must be query-routed (ON for doctrinal/principle-seeking, OFF for known-item/fact lookup). That swing is the empirical mandate for a cheap LLM query-classifier at the front. Caveats: silver labels (Hitin audit pending); mild circularity (slice gold & prior both key off cite_indeg → magnitude may inflate, direction is sound); bad-law→0 is only as real as the denylist, so it depends on F6 good-law being real in production.

7c. STAGE 0 of the agentic build — routed authority prior + good-law spine (committed 2026-06-25)

First stage of AGENTIC_SEARCH_SPEC.md. Folds the §7b lever into one query-routed spine and measures it end-to-end. Round-0 intent classifier (classify_intent.py, DeepSeek: AUTHORITY vs SPECIFIC) gates the prior; good-law filter drops the 43 confirmed-bad docs (goodlaw_badlaw.txt, the real product signal — not the ad-hoc authority_badlaw.txt). Harness: spine_run.py (loads corpus once, CE once/slice, derives baseline + spine). Routing fires on 69% of authority queries, 6% of silver — aggressive-OFF on general, as designed.

Slice	paired ΔnDCG@10 vs baseline	Δsucc@1	bad-law@10
AUTHORITY (150)	+0.048 [+0.028, +0.070] SIG	+0.12 SIG	0.033 → 0.000
SILVER all-800	−0.013 [−0.021, −0.006] SIG	−0.020	0.013 → 0.000

The silver regression is confined to silver_doctrinal (−0.036 SIG); factpattern, vague, known-item are all flat/ns. It is an eval artifact: silver_doctrinal gold is the arbitrary source case, not the doctrine's landmark, so it penalizes us for correctly surfacing leading authority — the same query type scores +0.048 on the authority slice where the gold is the landmark. The real (correctly-specified) guardrails — known-item success@1, factpattern, vague, bad-law — all hold. Verdict: PASS; proceed to Stage 1.

Open caveats logged for Hitin's audit: (a) classifier precision — it over-fires AUTHORITY on a few narrow doctrinal queries ("frustration in a statutory tenancy") where a specific case may beat the landmark; correct gold is needed to tune this. (b) good-law coverage — only 43 docs flagged bad in 38k; the filter is perfect within what's labeled but labeling is thin (the F6 dependency). (c) silver_doctrinal needs landmark gold to be a valid instrument for authority features.

Files: classify_intent.py, spine_run.py, goodlaw_badlaw.txt, intent_{authority,silver}.json, {authority,silver}_{baseline,spine}.tsv.

7d. STAGE 1 — the agentic controller (two-turn parallel) — PASS (2026-06-26)

Tool-rich ReAct agent (founder-chosen over the binary router), shaped by the panel as two-turn parallel, not serial N-step ReAct: 1 plan LLM call (intent + expected authorities + statute refs + HyDE) → ALL retrieval tools fan out at once (vector + authority + name_lookup(plan authorities) + statute/cases_on_section

hyde + citation-graph) → merge → one batched CE rerank → authority-prior rank → good-law flag-don't-drop -for-authority. Files: scripts/tools.py (14-tool registry), scripts/agent.py (controller), eval/agentic_run.py.

Authority slice (150)	nDCG@10	MRR	succ@1	recall@10	bad-law@10
baseline	0.282	0.551	0.380	0.204	0.033
spine (Stage 0)	0.330	0.638	0.500	0.194	0.000
agent	0.389	0.788	0.687	0.181	0.160

agent vs spine paired ΔnDCG@10 = +0.0585 [+0.028,+0.089] SIG; Δrecall@10 ns (no real loss). The controlling authority is #1 in 69% of queries (succ@1 0.500→0.687).
Recall-recovery slice (8 landmarks OUTSIDE the dense pool): recall@10 = 0.75 vs spine ≈ 0 — the agent (name_lookup of LLM-named authorities + graph + hyde) solves what one-shot retrieval structurally cannot.
bad-law@10 0.160 is 88% artifact: 23/26 hits are high-authority FALSE-POSITIVE landmarks (Maneka-type) that flag-don't-drop keeps WITH A WARNING; only 3 are genuine dead law (≈ spine's leak). The headline rise is the good-law data's mislabels, not a precision regression. → Hitin's good-law audit is now the #1 data item.

Latency: the Mac eval is ~11s/query (CPU, multiple sequential CE passes) — production (GPU box, async parallel tools) hits the panel's 15s budget. name_lookup indexed (token postings) so it's no longer O(corpus). NEXT: product path (2nd LLM turn = judge + streamed grounded answer + SSE steps), wire as serve.py adaptive endpoint (keep fast/deep as fallback), per-tool ablation (cut tools that don't earn latency), good-law audit.

8. The stepwise plan (roadmap — panel-prioritized)

Step 0 — eval rig ✅ (this doc / §5).
Step 1 — cheap honesty/correctness (ship together, low risk): make F6 good-law real in fast mode (demote/ tag the ~40 confirmed-overruled), fix the D3 good-law token set (drop partly_overruled, add doubted), log the verify parse-failure no-op, sigmoid-normalize rr, share ONE F0/D0 router fn, emit D4 got/miss to the usage log.
Step 2 — pool depth (CAND↑ + diversity) — first signal positive (+0.015).
Step 3 — ranking core (now data-backed, §7b): ship the authority prior sigmoid(rr)+α·log1p(cite_indeg) at α=0.3, query-routed (ON for doctrinal/principle queries only — it's −0.29 on general, +0.072 on doctrinal) and gated by the good-law filter (the two are one unit). Needs (a) a cheap LLM query-classifier at the front, (b) F6 good-law real. headnote-fed verify (down-weight not drop high-authority), fix card_for_doc cis[:6]. NOTE: the bge-reranker swap is REJECTED (§7b) — do not pursue; the reranker is not the lever.
Step 4 — router guards (question-word before bypass; bare-name lookup) + A/B-and-maybe-cut the F4 rewrite.
Step 5 — answer: deep synthesis over top-8 + force-include confirmed authorities + a Qwen/cross-family entailment gate after the verbatim gate.
Stage 2 (post-pilot, measured): swap to a legal/Indic embedder (needs a re-embed of the corpus — expensive; note the bge-reranker is already ruled out by §7b, so this is an embedder bet, not a reranker one); the citation-resolution rebuild (it's a MISSING-KEYS problem — SCC resolves at 0% — not name-matching; tiered exact-lookup spine + trust firewall); the **statute layer (BNS↔IPC / BNSS↔CrPC / BSA↔Evidence)
- unified intent→plan→tools** agentic search (collapse fast/deep into one adaptive-effort pipeline); PageRank / Personalized PageRank for authority (gated on clean Tier-1/2 edges).

9. Rigor principles (the culture to keep)

Verify against the real code/data — the panels repeatedly caught wrong assumptions (e.g., "registry has SCC cites" was false; "re-scrape the registry" buys 0 keys; the "76 overruled" figure was wrong — it's 40). Read the file before you claim.
Never self-grade. Judge family ≠ serving family. Anchor in model-free truth (citation graph, known-item) where possible; reserve LLMs for the gaps and audit them.
One change at a time → paired-bootstrap CI → commit or revert. No unverified complexity.
Precision-first guardrails are non-negotiable (bad-law@10, known-item succ@1).
Reproducibility/backups: artifacts live on the Mac (phase1/data/thor_artifacts/, byte-verified), Thor ~/backup, and the GPU box; code on GitHub. The eval set + qrels are frozen and versioned.

10. Pointers

Code: phase1/scripts/serve.py, frontend.html; eval: phase1/eval/{build_qrels,score_qrels,lean_run,batched_run,sweep,embed_chunks}.py + queries.tsv/qrels.tsv/bad_law_docids.txt; deploy: phase1/deploy/.
Design records (in the user's auto-memory): themis-eval-rig, themis-citation-resolution, themis-agentic-search-design, themis-benchmark-v1, thor-gpu-embedding (the box saga + gotchas).
The panel reviews (per-step search critique, eval-metric design, citation-resolution) ran as multi-agent Workflows; their full transcripts are in the session's subagents/workflows/ dirs.

First thing to do when you pick this up: ssh the GPU box, run sweep.py, read the bge-reranker-base row, and either commit it (if it clears the CI and guardrails hold) or move to Step 1. The loop is live — turn the crank.