fieldnotes-rag / VALIDATION.md
iankamar's picture
Update walkthrough and record verified deployed relevance fixes
3add3e3 verified
|
Raw History Blame Contribute Delete
8.26 kB

Current validation — post-audit relevance fix

37 deterministic Node tests pass across all five *.test.mjs files. The real-model browser audit passed 10/10 development checks, with two labeled semantic fallbacks, in one observed local run of 56.5 seconds. Both natural ownership/approval questions that failed the portfolio audit now return the correct source quotations; the compound question preserves both Luis and Maya. Unsupported budget and production-launch questions are withheld. The rollback threshold remains 2 percent for 15 consecutive minutes.

This is regression evidence, not held-out accuracy. The two failed audit questions are now included in the ten public cases. Rule-based question constraints are conservative heuristics and can reject valid evidence; there is no general entailment verifier. The answer path considers eight passages while the historical retrieval-only benchmark still measures top-three retrieval. Models are now revision-pinned in worker.mjs. Earlier measurements below describe earlier implementations and must not be attributed to this release.

Public release f33fa8f782abb68f2d5a7bfd04cb0b8f0eb38717 was then smoke-tested on Hugging Face: the approval question returned Maya Chen with source [1], the compound ownership question returned Luis and Maya with source [3], and the budget question withheld an answer. All 11 uploaded files were fetched back and matched the local files byte for byte.

Historical validation log

Fieldnotes v2 validation — November 11, 2026

Executed before browser deployment

23 deterministic Node tests pass. Coverage includes BM25 retrieval, no-match behavior, overlap, source selection validation, untrusted-data placement, separate indexes, source mapping, semantic paraphrase recovery with controlled vectors, weak-match abstention, provenance, metric arithmetic/denominators, benchmark label integrity, export content, and embedding dimension errors. Syntax checks pass for the app and worker.

The original BM25 baseline on the new fixed synthetic corpus: Hit@1 20/24 (83.3%), Recall@3 21/24 (87.5%), MRR 0.854, unsupported rejection 0/6 (0%). These are retrieval metrics, not answer accuracy.

An actual live MiniLM run measured Hit@1 21/24, Recall@3 24/24, MRR 0.931, and unsupported rejection 4/6. The first five-question incident brief completed but revealed a wrong decision-owner selection and use of the obsolete policy. These are documented failures, not passing answer checks.

Fixes added from those observations: explicit current queries exclude provenance marked superseded; a relevance guard checks the LLM sentence choice against the leading passage and semantic margin, using a clearly labeled semantic fallback on disagreement. Three regression tests cover the selection guard and another covers superseded-source filtering.

The first automatic download verification stalled in the test browser. Exports now have a visible text preview and a save link so users can copy text when downloads are blocked. Final browser results are recorded below after verification.

Scope

30 agent-authored synthetic queries, 24 answerable and six unsupported, over 24 passages in nine fictional documents. Not independent, not held out from development, not a real enterprise benchmark. The corpus includes a superseded policy and adversarial exercise, but no claim of reliable conflict resolution or prompt-injection defense is made. Model repository revisions are not pinned.

The prior version's two successful live/local answer checks are historical and do not substitute for testing this upgrade. The account owner has not personally reviewed or run the implementation. AI assistance created and tested it.

Final browser observations

At commit 5126c58 the 30-query live evaluation reported: BM25 Hit@1 20/24, Recall@3 21/24, MRR 0.8541667, unsupported rejection 0/6; final hybrid + provenance rule Hit@1 22/24, Recall@3 24/24, MRR 0.9513889, unsupported rejection 4/6. Runtime reported 1,274 ms for that run, including the setup done in that run; this is one observation, not a latency benchmark.

evaluation-results.json preserves every observed ranking and recomputed matching metrics. It was transcribed from the visible live JSON export; per-query similarity details are omitted in that saved snapshot. Users can rerun the benchmark for the complete raw export.

The five-step brief ran with real model inference. Cause and impact source quotations matched the case. The owner and current-policy steps triggered explicit semantic fallbacks; the owner evidence identified Luis as incident commander and the policy evidence used 2 percent / 15 minutes. The prevention step returned Avery Stone with the November 14, 2026 due date. The completed brief contained five findings: three retained LLM selections and two explicit semantic fallbacks. No general answer accuracy percentage is claimed.

The incident-brief export preview contained all five questions, source excerpts, timestamps, model identifiers, answer methods, and measured timings (6,948 characters in the observed run). It was checked through the visible textarea for cause, impact, owner, policy, and timing content.

The evaluation export preview displayed the complete JSON and a save link. Automated file downloading is not verified in the embedded test browser; the copyable text preview is the verified fallback.

No exhaustive PDF/layout/mobile testing, independent adversarial study, or real-user productivity measurement was performed. No paid API or compute was enabled.

Post-publication local note: final commit a845a0e2f4742054506012cca2ec5f82f1388a88 was smoke-tested live on the decision-owner question. The labeled fallback preserved the full ownership passage, explicitly stating Luis owns rollback and Maya approves resuming. The live run reported 1.0 seconds retrieval / 9.2 seconds total. This is one observation, not a performance claim.

Five-minute finishing pass

28 deterministic tests pass, including five new checks proving that wrong facts, invalid citations, and irrelevant answers to unsupported questions fail the answer auditor. Added eight-case actual-model audit with visible pass/fail, fallback routes, sources, timing, and JSON export. Live run results are separate from the unit test count. No model training or paid services were added.

Live answer-audit result at release 46ce932: 7/8 checks passed, including all six factual checks and one of two unsupported-question refusals, with three explicitly labeled fallbacks. UI reported 51.9 seconds for the run. The budget question returned an irrelevant coordinator-count sentence and correctly failed the audit. The production-launch question was withheld. These are eight synthetic fact-string/citation checks, not a claim of 87.5 percent general answer accuracy. A summary is saved in answer-audit-results.json; the full per-answer JSON can be generated and exported in the app.

Link and editorial audit

33 deterministic tests pass, including five new citation-link checks (source destinations, invalid IDs, brief namespaces, preservation of text, and safe handling of markup). All documentation/model/CDN URLs and the navigation destinations were checked by HTTP HEAD. Local module imports, HTML asset references, static anchor IDs, and 26 UI targets were checked for missing destinations. Main-answer citation navigation was exercised in the browser and focused the matching source passage. Full brief citation navigation and live-file verification are recorded in the accompanying finishing check.

Removed slogan-style headings, repeated defensive copy, external font loading, and a misleading external-link arrow on the demo-loader button. Added direct Guide / Code / Test report links and formatted the source for readability.

Browser finishing check: main citations focused their matching passage. A decision-owner citation in the completed five-question brief opened its collapsed supporting passages and focused the correct ownership record. The Markdown save link downloaded an actual fieldnotes-incident-brief.md; its contents included all five findings and processing metadata. No browser download guarantee is made for every browser configuration.