| # Mindweave experiment — 2026-07-23 |
|
|
| This report records the first external-data experiment for CFO-HRM. The result |
| is a shadow-mode learning artifact, not evidence that the model is ready for a |
| real accounting close. |
|
|
| ## Data and split |
|
|
| The experiment uses revision-pinned, CC BY-NC 4.0 Mindweave bank and accounting |
| ledger datasets. Exact revisions, file hashes, and attribution are recorded in |
| `THIRD_PARTY_DATA.md` and each run's `dataset_manifest.json`. |
|
|
| The adapter verified: |
|
|
| - 3,208 unique provenance-derived bank-to-cash-GL pairs; |
| - 24 source-less monthly bank fees, conservatively labeled unmatched; |
| - 3,232 bank rows in 145 independently shuffled fixed-size cases; and |
| - no unresolved rows in the pinned snapshot. |
|
|
| The chronological split prevents period leakage: |
|
|
| | Split | Periods | Cases | Bank rows | Matches | Fees | |
| | --- | --- | ---: | ---: | ---: | ---: | |
| | Train | 2024-01 through 2025-06 | 92 | 2,021 | 2,003 | 18 | |
| | Validation | 2025-07 through 2025-09 | 23 | 509 | 506 | 3 | |
| | Test | 2025-10 through 2025-12 | 30 | 702 | 699 | 3 | |
|
|
| Independent deterministic shuffling reduced same-position bank/GL targets from |
| 98.6% to 4.1%, removing a positional shortcut. |
|
|
| ## Training |
|
|
| The robust run starts from the synthetic-pretrained checkpoint at step 900 and |
| uses: |
|
|
| - 150 outer steps and four HRM deep-supervision segments; |
| - 75% Mindweave and 25% synthetic retention batches; |
| - 0.25 reference, 0.10 description, and 0.05 source-module dropout on external |
| training rows only; and |
| - external validation plus a fixed, disjoint synthetic retention evaluation. |
|
|
| Checkpoint selection prioritizes external match F1, external exception macro |
| recall, zero false auto-matches, synthetic retention metrics, and then external |
| calibration. It selected step 50. |
|
|
| ## Held-out results |
|
|
| All selective metrics use a 0.90 confidence threshold. |
|
|
| ### Raw test split |
|
|
| | System | Assignment accuracy | Match F1 | Exception accuracy | Bank-fee recall | ECE | Auto-match coverage | False auto-match rate | |
| | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | |
| | Synthetic-pretrained HRM | 99.43% | 99.43% | 7.41% | 100% | 14.76% | 52.99% | 0% | |
| | Deterministic evidence baseline | 100% | 100% | 99.57% | 0% | 0.83% | 99.57% | 0% | |
| | Robust fine-tuned HRM, step 50 | 100% | 100% | 100% | 100% | 7.95% | 83.90% | 0% | |
|
|
| The baseline's perfect matching result is important: this synthetic dataset is |
| largely solvable with exact amount, date, and provenance evidence. The HRM does |
| not establish an advantage on matching. Its useful measured improvement here is |
| exception classification, particularly recognizing bank fees. |
|
|
| ### Stress tests |
|
|
| | Test | Assignment accuracy | Match F1 | Bank-fee recall | Selective coverage | False auto-match rate | |
| | --- | ---: | ---: | ---: | ---: | ---: | |
| | Reference masked | 95.44% | 95.42% | 33.33% | 80.06% | 0% | |
| | Source module, reference, and description masked | 96.30% | 96.28% | 0% | 48.29% | 0% | |
|
|
| The model abstained safely at the chosen threshold in these tests, but the |
| accuracy and fee-recall degradation show that it still depends heavily on clean |
| synthetic identifiers and contextual fields. |
|
|
| On the run's fixed synthetic retention split, the selected checkpoint achieved |
| 95.71% match F1 and 34.22% exception macro recall. The corresponding starting |
| checkpoint achieved 95.41% and 37.86%, respectively: matching was retained, |
| while exception behavior still shifted across domains. |
|
|
| ## End-to-end controlled execution |
|
|
| Held-out test case 8 contains 19 bank rows, including one monthly fee. The |
| promoted checkpoint proposed 18 matches and one unmatched bank fee. Go controls |
| reported: |
|
|
| ```text |
| review_queue: exception_review |
| controls_passed: 18 |
| blocked_candidates: 0 |
| unmatched_bank_rows: 1 |
| requires_human_review: true |
| posting_authorized: false |
| audit verification: 1 hash-chained event verified |
| ``` |
|
|
| The review request, decision, and audit files were all written with mode |
| `0600`. No component contains an ERP posting or approval client. |
|
|
| ## Decision |
|
|
| Promote `runs/mindweave-robust-20260723/best.pt` only as the current local |
| shadow-mode checkpoint. Keep the deterministic baseline beside it and report |
| both. The next meaningful evaluation needs harder grouped transactions, |
| reference corruption, split/aggregate payments, GL-only items, and ultimately |
| approved human-reconciled real exports. |
|
|
