Blame for docs/verification.md

# CROSSWIRE Verification Report

**Date:** 23 October 2026  
**Commit:** `codex/crosswire-design`  
**Platform:** Windows (Python 3.12.10 64-bit, Node.js v20+, Chromium)  
**Record type:** Full verification report. Current upgrade evidence includes safe calibration, model-mismatch diagnostics, independent protocol reliability, adaptive branching calibration, online change-point recovery with detection latency, and factored candidate spaces (357 passed tests).  
**Overall Status:** PASS (357/357 tests passing, clean wheel install verified, browser e2e verified, 997 total empirical runs across 13 benchmarks)

---

## 1. Automated Test Suite Summary

The automated test suite runs via `pytest` without external dependencies (except optional pinned `aalpy==1.6.2` for baseline testing).

```

============================= 357 passed =============================

```

### 1.1 Test Suite Breakdown by Subsystem

| Test File | Tests Passed | Subsystem & Invariant Checked |
|---|---:|---|
| `tests/test_advanced_features.py` | 6 | BALD mutual information profiles and minimax probe ranking (`information_profile`, `rank_probes`); contingent policy tree structural/entropy metrics (`policy_metrics`); adaptive policy Simplex recovery (`mode="recovery"`); certified soft return-to-home state stabilization (`synthesize_safe_return`). |
| `tests/test_adaptive.py` | 5 | Observation-dependent contingent policy synthesis (AND/OR tree search); independent policy certificate verification (branch completeness, horizon bounds, zero hazard); certificate tampering rejection; search cap handling; full execution trace replay and audit. |
| `tests/test_audit.py` | 6 | Executed prefix reconstruction; counterfactual trajectory analysis without side-effects; unexecuted suffixes withheld from hazard attribution; causal hazard averted classification. |
| `tests/test_baselines.py` | 3 | Pinned AALpy 1.6.2 $L^*$ integration; budget-limited SUL and equivalence oracle; comparison transition step counting; global RNG isolation. |

| `tests/test_branching_calibration.py` | 10 | Interactive within-query action selection (`policy="branching"`); evaluates conditional predictive entropy and model safety under surviving belief at every action dispatch; halts early upon contract readiness; tie-breaking in zero-entropy and delayed settings. |

| `tests/test_calibration_safety.py` | 8 | Pre-probe safety checks preventing hazardous queries during calibration under finite model assumptions. |

| `tests/test_cegp.py` | 3 | Decision-focused counterexample-guided active probing (CEGP); $O(W \cdot N)$ marginal frequency pair disagreement counting; dynamic synthesis of distinguishing witnesses beyond fixed probe pool. |

| `tests/test_certificates.py` | 11 | Independent finite public-evidence certificate generation and validation; clean re-enumeration of static/latency-1 adapter spaces; verification of surviving belief sets from public transcripts; deletion-minimal contradiction core extraction; verification of admitted plans and policy trees; unmodeled target scope boundaries. |

| `tests/test_change_points.py` | 7 | Explicit change-point fault schedules in `_Engine` and `RunConfig` (5 fault transformations: `sensor_drift`, `actuator_swap`, `delay_spike`, `stuck_output`, `semantic_swap`); detection latency measurement ($\Delta t = t_{\text{detect}} - t_{\text{fault}}$); causal hazard aversion under shielded abort and online replanned safe detours; negative control verification (`semantic_swap`). |

| `tests/test_cli.py` | 3 | CLI commands `run`, `study`, `verify`, `replay`, `catalog`, `serve`, and `check` with JSON evidence envelope verification. |

| `tests/test_core.py` | 14 | 144 bijective adapter permutations; prefix hazard safety; true-map retention; equivalence classes; empty-family non-vacuous handling; shortest lexicographic product counterexample BFS. |

| `tests/test_ec2.py` | 6 | Equivalence Class Edge Cutting (EC2) active probing; contract equivalence class partitioning; cross-class Jaccard distance weighting; zero intra-class score filtering; query cost normalization; temporal model compatibility. |

| `tests/test_escalation.py` | 7 | Information-theoretic Shannon entropy probe selection; deterministic tie-breaking and empty belief handling; autonomous model-class escalation ($\mathcal{H}_0 \to \mathcal{H}_1$) on delayed interfaces; structural contradiction certificate emission when nested classes are refuted; in-family non-spurious escalation; Simplex safe fallback recovery execution; configuration validation guards. |

| `tests/test_factored_space.py` | 6 | Factored permutation space decomposing candidate adapters into orthogonal factor spaces $\mathcal{P}_I \times \mathcal{P}_O$; lazy adapter proxy indexing; independent anchor pruning on $\mathcal{P}_O$; exact mathematical equivalence on 144 and 720 models; sub-second scaling to 17,280 candidate models with complete-survivor semantics. |

| `tests/test_fixtures.py` | 37 | Procedural vs. table target implementations; 17 fixture families; grounded anchor validation; hidden permutations; private raw machine extraction; deterministic seeded variation; routing control. |

| `tests/test_independent_instances.py` | 8 | Independent structural instance generation, protocol hashing, and instance seeds. |

| `tests/test_interfaces.py` | 7 | Subprocess lifecycle; exact-PID cleanup; line-delimited JSON protocol; malformed message recovery; timeout enforcement; attempted vs. successful call accounting. |

| `tests/test_model_diagnostics.py` | 8 | Sequential forecasts, first refutation witnesses, deterministic reset consistency checking, enabled-class diagnostics. |

| `tests/test_realistic_benchmarks.py` | 7 | Realistic robotics (`robotics_airlock`) and protocol conformance (`protocol_session`) benchmarks; exhaustive table/procedural parity; zero-hazard `model_safe` calibration without anchors; 4-step out-of-horizon synthesis; branching calibration and change-point shield latency. |

| `tests/test_receipts.py` | 3 | Canonical JSON serialization; SHA-256 payload and scientific hashing; envelope validation; tampering rejection; exact scientific replay. |

| `tests/test_reliability.py` | 8 | Statistical reliability metrics across independent protocol instances. |

| `tests/test_replan.py` | 5 | Receding-horizon dynamic replanning shield (`replan` mode); pre-step and post-step suffix risk detection; online synthesis of certified safe alternative detours (`REPLANNED_ONLINE_SAFE`); fallback to recovery/abort when no detour exists. |

| `tests/test_report.py` | 14 | Study reporting, stratified comparisons, SVG plot generation, and individual-run inspection. |

| `tests/test_risk_controlled.py` | 4 | Risk-controlled version-space admission (`risk_controlled` mode); empirical version-space hazard and goal calculation; zero-tolerance consensus equivalence; positive-tolerance majority admission; certificate verification and tampering rejection. |

| `tests/test_roadmap_integration.py` | 4 | End-to-end roadmap integration workflows. |

| `tests/test_safe_probes.py` | 6 | Pre-query safety guards across multiple probe policies. |

| `tests/test_server.py` | 6 | Async job queue; thread-safe scientific execution; cancellable import and replay jobs; static file path traversal rejection (`..` escaping blocked); origin protection; raw text receipt payload import. |

| `tests/test_shield.py` | 20 | Closed-loop dynamic runtime shield; mid-deployment anomaly detection when survivors collapse; pre-step preemptive hazard interception before target dispatch (any surviving interpretation); conservative Simplex safe fallback recovery requirements (consistent non-empty belief, remaining horizon, zero modeled hazards across all survivors); immediate abstention on contradiction. |

| `tests/test_study.py` | 23 | Bounded run/study configuration validation; freeze-before-audit digest; unified reset+step system-call cap; partial deployment prefix auditing; forced-fit residual diagnostics; exact-cell condition-stratified paired comparisons (matching execution mode, escalation, split, anchors, and horizon); paired admission contrasts; AALpy configuration guards. |

| `tests/test_temporal.py` | 10 | Temporal latency-aware adapter algebra ($|\mathcal{A}|=720$ with delay-1 and init-output); augmented transducer signatures tracking `(state, buffer)`; 3-tuple product Mealy state comparison; exact product equivalence merging. |

| **Total** | **357** | **Complete local coverage across 30 test modules verifying safe calibration, change-point fault recovery, factored candidate space scaling, adaptive policies, model escalation, evidence integrity, Simplex recovery, EC2 active probing, and risk-controlled admission.** |



---



## 2. Integration Review Findings Resolution



Following independent code review before benchmark execution, six specific actionable findings were identified and verified by test-driven regression:



1. **Deployment Prefix Auditing (P1):**

   - *Issue:* Auditor previously executed full words even if target subprocess failed midway, counting phantom hazards.
   - *Fix:* Reconstruct only the executed prefix; failed calls emit no engine step; unexecuted suffixes are never simulated as if they occurred.
   - *Regression Test:* `tests/test_study.py::test_failed_deployment_never_audits_an_unexecuted_suffix`.
2. **Unified System-Call Cap (P1):**
   - *Issue:* Resets were not charged against the active probe budget, violating the same-call-cap comparison.
   - *Fix:* `budget` is now enforced as a shared system-call cap ($N_{\text{calls}} = N_{\text{resets}} + N_{\text{steps}}$). Both learner loops and AALpy count resets.

   - *Regression Test:* `tests/test_study.py::test_calibration_budget_counts_resets_as_well_as_actions`.
3. **Deployment Interface Error Accounting (P2):**
   - *Issue:* Post-admission deployment failures were omitted from interface error summaries.
   - *Fix:* Count both acquisition and deployment interface errors explicitly in summaries without mutating frozen decisions.
   - *Regression Test:* `tests/test_study.py::test_failed_deployment_never_audits_an_unexecuted_suffix`.
4. **Forced-Fit Residual Diagnostics (P2):**
   - *Issue:* Forced-fit baseline discarded the selected adapter and residual loss under misspecification.
   - *Fix:* Added `fit_diagnostics` to baseline output detailing chosen adapter, raw mismatches, anchor mismatches, tied maps, and total residual.
   - *Regression Test:* `tests/test_study.py::test_forced_fit_exposes_residual_instead_of_implying_consistency`.
5. **AALpy Comparison Transition Work Accounting (P2):**
   - *Issue:* Product-automaton comparisons for AALpy hypotheses were not counting transition steps.
   - *Fix:* Accumulated `comparison_source_steps` and `comparison_hypothesis_steps` in baseline metadata.
   - *Regression Test:* `tests/test_baselines.py::test_completed_classical_model_reports_comparison_transition_work`.
6. **Budget-Stratified Paired Contrasts (P2):**
   - *Issue:* Paired comparisons pooled all budgets into a single parent mean.
   - *Fix:* Stratified paired contrasts strictly within identical call-cap levels, matching cell by cell inside each parent cluster.
   - *Regression Test:* `tests/test_study.py::test_paired_comparison_never_matches_different_budget_conditions`.
7. **Asynchronous Import and Replay (P2):**
   - *Issue:* Receipt import and replay ran synchronously on HTTP handler threads, blocking server worker queues.
   - *Fix:* Converted `/api/import` and `/api/runs/{id}/replay` to submit cancellable background jobs to the scientific worker.
   - *Regression Test:* `tests/test_server.py::test_http_run_export_import_and_replay_use_real_worker`.

---

## 3. Package Parity & Clean Wheel Installation

A clean packaging verification was executed using standard Python build tooling:
1. Wheel built with `pyproject.toml` via `python -m build --wheel --outdir $TEMP/crosswire-wheel-check/dist`.
2. Clean virtual environment initialized in `$TEMP/crosswire-wheel-check/env`.
3. Built wheel installed via `pip install`.
4. Verified package asset inclusion:
   ```

   ['app.js', 'index.html', 'styles.css']

   ```
5. Verified CLI operation from clean wheel environment:
   - `crosswire run --family latch --budget 16 --out latch.json` (exit 0)
   - `crosswire verify latch.json` (exit 0: "Scientific decisions, probes, costs and audits replay exactly")
   - `crosswire run --family latch --policy aalpy --budget 8` without optional dependency produces clear, expected `ModuleNotFoundError: No module named 'aalpy'`.

---

## 4. Browser Workbench End-to-End Automation

Browser automation was conducted using Playwright on headless Chromium against a live instance of the local server (`http://127.0.0.1:8790/`):

- **Script:** `scratch/e2e.js`
- **Output Artifacts:** `output/playwright/*.png`
- **Test Scenarios Exercised:**
  1. **Empty State:** Initial load confirms `connection: Local worker · 0.1.0`, skip link, accessible control labels, and empty state glyph. (`01-empty.png`)
  2. **Live Experiment Execution:** Submitted `latch` with seed 17, anchors enabled, budget 16. Polling tracked progress through worker completion. Displayed reference state graph, candidate mapping table, probe history, and private audit breakdown. (`02-latch-run.png`)
  3. **Receipt Export:** Clicked `#export-button`, received valid `crosswire-receipt-v1` JSON file (5,187 bytes).
  4. **Replay Verification:** Clicked `#replay-button`, tracked async worker job, verified receipt notice: `Replay verified: Scientific decisions, probes, costs and audits replay exactly`.
  5. **Receipt Import:** Ingested exported receipt file via `#receipt-file`, server validated digest and exact execution, rendered imported results.
  6. **Semantic Corruption Control:** Executed `semantic_corruption` fixture. UI accurately displayed decision admitted with contract violation flagged in red by independent audit. (`03-semantic-corruption.png`)
  7. **Study Dispatch & Cancellation:** Dispatched paired study via `#study-button`, triggered `#cancel-button`. Verified job cancelled cleanly without hanging child processes.
  8. **Study Execution & Plot:** Dispatched and completed paired study. Verified rendering of SVG useful-coverage plot and paired summary tables. (`04-study.png`)
  9. **Responsive Mobile Viewport:** Set viewport to $390 \times 900$. Verified 0 horizontal overflow pixels (`document.documentElement.scrollWidth === document.documentElement.clientWidth`). (`05-mobile.png`)
  10. **Error Log Audit:** Exactly zero console errors or uncaught page errors observed.

---

## 5. Scientific Receipt Provenance & Reproducibility

Ten systematic empirical benchmarks were executed and preserved as self-contained scientific receipts:

| Benchmark | Output Receipt | Protocol Hash | Scientific Hash | Total Runs |
|---|---|---|---|---:|
| **Development Smoke** | `reports/smoke-receipt.json` | `18f9b1ef3bce37f8…` | `f08e0e7316980cb7…` | 27 |
| **In-Family Benchmark** | `reports/pilot-in-family-receipt.json` | `9977cf45469275ff…` | `64c084a09fa36956…` | 350 |
| **Out-of-Family Stress** | `reports/pilot-stress-receipt.json` | `1dc8cc430b960163…` | `ad1c49f782e756cb…` | 216 |
| **Open-Loop Stress Baseline** | `reports/pilot-shield-openloop-receipt.json` | `a4fcb06f0e492da4…` | `5bdf9766d4ab4946…` | 36 |
| **Shielded Closed-Loop Benchmark** | `reports/pilot-shield-shielded-receipt.json` | `a88c531ea7fad346…` | `924a7b12de35f87c…` | 36 |
| **Temporal Adapter Synthesis** | `reports/pilot-temporal-receipt.json` | `674907c7f02dcdb7…` | `f0eee9109c032893…` | 4 |
| **CEGP Distinguishing Witness Benchmark** | `reports/pilot-cegp-longtrace-receipt.json` | `968ed5dfe5874e08…` | `e931c96b3d027d34…` | 4 |
| **Adaptive Contingent Policy Benchmark** | `reports/pilot-adaptive-receipt.json` | `6e07673c27941fc7…` | `cbc85a6cbc260e98…` | 48 |
| **Pilot Escalation, Entropy & Recovery** | `reports/pilot-escalation-receipt.json` | `74257f9947c445e3…` | `741aac4f5f86eb51…` | 36 |
| **EC2, Replanning Shield & Risk Control** | `reports/pilot-replanning-ec2-receipt.json` | `88c6426ff020be84…` | `e76ceb0c228977fd…` | 96 |
| **Model-Safe Probing Calibration** | `reports/pilot-calibration-safe-receipt.json` | `25e80327cf35821a…` | `6cd97b3fc725d461…` | 8 |
| **Unrestricted Calibration Baseline** | `reports/pilot-calibration-unrestricted-receipt.json` | `c6725eb7f3f6cbb6…` | `f0df1cc43058eacc…` | 8 |
| **Independent Protocols Reliability** | `reports/pilot-independent-protocols-receipt.json` | `9d66e5b7df0645b2…` | `039cc5257f627ff4…` | 128 |
| **Total Empirical Runs** | | | | **997** |

Receipts bind explicit source and protocol hashes. Historical receipts require their original implementation revision; the current source must reject mismatched revisions rather than silently relabeling them.