environment/training-source/source-snapshot/experiments/AUDIT.md

# Audit record for the Meganeura case study

Date: 2026-08-01 (America/Los_Angeles)

## Decision

The original `REPORT.md` is not admissible as paper evidence. It is a useful
engineering narrative, but several headline results were either evaluated on
training examples, measured with a biased timing protocol, or produced by an
incorrect DINOv3 graph. The case study can become paper material after the
fresh Quest correctness and measurement gates in `experiments/README.md` are
complete. Corrected three-seed held-out decoder evaluation and publication of
the host artifact are now complete.

The defensible case-study claim is deliberately narrow:

> A decoder graph is trained through Meganeura autodiff and Adam on the host,
> then its forward path is joined to a frozen DINOv3 encoder and compiled by
> the same Meganeura graph compiler/runtime for Vulkan inference in an Android
> XR application sharing Blade's graphics context and queue.

Training caches frozen encoder features and executes a batched decoder-only
graph. Deployment executes a batch-one joined encoder/decoder graph. They
share decoder construction, parameters, IR operations, compiler, and runtime;
they are not literally the same complete graph.

## Findings that invalidate historical numbers

| Historical evidence | Audit finding | Disposition |
|---|---|---|
| Decoder PSNR in `REPORT.md` | The trainer reconstructed the first six cached training images and averaged their PSNR. There was no held-out split, per-image record, or seed replication. | Replaced by an upstream-defined Imagenette validation split, all 3,925 images, per-image PSNR/SSIM/MAE, exact hashes, and three initializations. |
| Isolated device timings | The benchmark used one untimed warmup and reported the minimum retained sample as throughput. Clock ramp and process-to-process state were not controlled. | Rerun with at least five warmups, 20 retained samples, median/IQR/raw samples, and three fresh processes in a declared headset state. |
| DINOv3 encoder correctness | The app passed `[tokens, hidden]` data to Meganeura's NCHW-flat `mul_per_channel` as if it broadcast over the trailing dimension. Only the first token received the intended LayerScale update; later token residual branches were effectively suppressed. | Fixed by transposing to `[hidden, tokens]`, flattening for the NCHW operator, applying the hidden gain with `spatial=tokens`, reshaping, and transposing back. All old decoders and device timings are invalid. |
| Live XR “inference Hz” | Two asynchronous eye completions were combined into one count, and the raw-camera toggle still submitted hidden inference. | App records each eye independently and true raw-camera mode bypasses the model. |
| 240-pixel, f16, and operation-share conclusions | These were microbenchmark observations or extrapolations, not matched end-to-end experiments or a deployed-graph profile. | Exclude causal or “closed” claims unless the matched experiments are run. |

The invalid pre-fix decoder runs are preserved locally under
`C:\tmp\dinovision-runs`; they must not be uploaded or cited. The corrected
matrix uses a separate root so no artifact can be mistaken for a valid run.

## Independent encoder validation after the fix

The reference is `facebook/dinov3-vits16-pretrain-lvd1689m`, executed in
Torch 2.13.0+cpu and Transformers 5.14.1 from the same deterministic normalized
pixel tensor and exact checkpoint. Meganeura uses f32 scalar matmul on the RTX
3050 for this comparison. Thresholds were declared before the corrected run:
relative L2 at most 0.01, CLS cosine above 0.999, and every patch-token cosine
above 0.999.

| Depth | Relative L2 | CLS cosine | Worst patch cosine | Result |
|---:|---:|---:|---:|---|
| 1 layer | 0.000781 | 1.000000 | 0.999990 | pass |
| 3 layers (deployed) | 0.001403 | 1.000000 | 0.999995 | pass |
| 12 layers (upstream depth) | 0.002253 | 0.999997 | 0.999996 | pass |

At the first layer, embeddings, LayerNorm, Q/K/V projections, and RoPE agree
to the displayed six decimal places. Attention differs by 0.005852 relative
L2; after learned LayerScale the residual differs by 0.000566. Exact-shape
tests independently compare Meganeura attention and the corrected trailing
LayerScale broadcast against CPU implementations.

A deliberately small deterministic replay also ran the same 20-update,
16-image, seed-20260801 cell twice. The two serialized decoders are
byte-identical (SHA-256
`2959b06784576bb1bc495b4e0a0d2f4aeebb216d56f7fc078843df4b7ec12686`).
This validates the seeded code path and serialization, not full-run
repeatability across machines or drivers.

The audit also recommends hardening Meganeura's `mul_per_channel` builder.
Its documented flat NCHW input/gate contract is not currently asserted, and
the compiler derives the dispatch length from `shape[0]`. Validating rank,
element count, channel/spatial divisibility, and gate length at graph
construction would have rejected the original misuse before execution. This
is not needed to reinterpret the corrected results, but it is a worthwhile
paper-artifact change in Meganeura itself.

## Evidence status

- **Complete:** the public `mad-bot/dinovision` artifact binds all three
  decoders to the corrected source snapshots, DINO checkpoint hash,
  dataset-manifest hash, depth, resolution, and seeds, and retains the
  fail-closed validation manifest.
- **Complete:** fixed-frame host/Quest comparison passes for bit-identical
  patches, encoder output (relative L2 0.001059, cosine 0.99999944, minimum
  token cosine 0.999973), spatial decoder input (relative L2 0.001063), and
  reconstruction (relative L2 0.000450, cosine 0.99999994).
- Fresh isolated Quest timings and live-worn XR chunk sweep from the corrected
  graph, with raw samples and state snapshots.
- Clean, immutable DinoVision and Meganeura revisions. The current worktree is
  a development freeze candidate, not yet a citable revision.
- A validation-clean debug host context. Numerical tests pass, but the
  resolved Blade Vulkan setup currently emits two device-creation VUIDs; this
  is distinct from the corrected Meganeura tensor results and should be
  tracked rather than hidden.

Until those items are complete, the correct paper status is **promising but
not merge-ready**.