ORACLE-LIMITS.md

# The oracle's real limits

The verifier is the reason to use PyBytecode at all, so its limits belong in front of a user, not
in an appendix. Everything here is measured; sources are named per section.

---

## 1. The pre-flight 100% proves almost nothing. Read this before quoting it.

Every grading command prints `PRE-FLIGHT 600/600 = 100%` before it scores. **That number is
trivial by construction and is not evidence of soundness.**

Pre-flight grades each reference label against itself. The oracle asks whether
`compile(prediction)` and `compile(reference)` produce the same code object — so at pre-flight it
is comparing `compile(x)` with `compile(x)`. It would return 100% for *any* deterministic
function of the source, including a stub that hashes the input string and ignores the bytecode
entirely.

What pre-flight actually detects is a **broken harness**: a benchmark whose `.pyc` files do not
match their sources, a Python version mismatch (3.11 or 3.13 against a 3.12 benchmark), a corrupt
row. Those are real failure modes and worth catching, which is why it runs. But a passing
pre-flight says the instrument is plugged in, not that it measures anything.

**Soundness evidence comes from the mutation test and the blind-spot probes, not from
pre-flight**: corrupt a label and require the oracle to reject it. Measured
(`evidence/ORACLE-MUTATION.md`): **0 true survivors in 1,239 mutants**, and
**18/18 targeted blind-spot probes** behave as required — including the historical failure where
a `try:` body and a `try/else:` body were indistinguishable, docstring changes, docstring
removal, float-vs-int, bool-vs-int and `-0.0` vs `0.0`.

Even the mutation kill rate is weak evidence on its own: for a byte-identical oracle a kill is
close to tautological, since a mutant survives only if it compiles to a structurally identical
code object. The probes are the load-bearing test, because they ask the question that actually bit
us once — *is a behaviourally load-bearing field missing from the fingerprint?*

**Caveat on mutation supply, stated rather than hidden:** 188 of 600 wild rows (31%) produced no
effective mutant within 30 tries, and 323 void attempts were discarded. The wild kill rate is
measured on the 412 rows that did produce one.

## 2. The 0.33% wild false-reject floor

Against `.pyc` files built by someone else, the oracle refuses a small fraction of correct
answers. Measured on 600 wild install-time `.pyc` from installed site-packages
(`evidence/GATE-RESULT.md`):

| | certified | false reject | false accepts |
|---|---|---|---|
| L0 (old constant encoding) | 585/600 = 97.5% | 15 = 2.5% | 0 / 1,274 |
| **L1 (shipping)** | **598/600 = 99.67%** | **2 = 0.33%** | 0 / 1,274 |

13 of the 15 L0 failures were our own defect — `repr()` of a `set`/`frozenset`/`dict` follows the
compiling process's hash seed, which also made the L0 verdict **non-deterministic** (585 / 592 /
584 / 585 / 589 under `PYTHONHASHSEED` 0–4). L1 fixes it and returns 598 under all five seeds.

The remaining **0.33% is a real floor and is not fixable.** One distinct module
(`pandas/_testing/__init__.py`) compiles differently under CPython 3.12.3 than under 3.12.13 —
`co_code` 2,692 vs 2,696 bytes, and a differing `co_exceptiontable`. The source is correct; the
*compiler patch release* differs. No normalisation removes this without abandoning the
byte-identical guarantee.

**It degrades to a false REJECT, never a false accept.** You are told "unknown" about a correct
answer; you are never told "verified" about a wrong one. That is the safe direction, and it is the
direction the design chose deliberately.

## 3. Optimization level must match the producer's, or verification collapses

A `.pyc` built with `-O` or `-OO` is a different code object. Measured on 679 sources compiled by
a foreign interpreter at each level and graded at each level:

| producer ↓ / grader → | 0 | 1 | 2 |
|---|---|---|---|
| **0** | **100.0%** | 95.43% | 23.86% |
| **1** | 95.43% | **100.0%** | 24.15% |
| **2** | 23.86% | 24.15% | **100.0%** |

The diagonal is 679/679 at every level. Guessing wrong is **not** a graceful degradation — it
collapses to ~24%. There are only three levels and trying all three costs three compiles, so this
is *"needs normalisation to survive"*, not *"breaks"*. **The harness does not currently search the
three levels automatically; a user verifying a foreign `.pyc` must do it.**

**What trying all three costs you, and it is not nothing.** The certificate changes meaning from
*"byte-identical to the code object the original source compiles to"* to *"byte-identical to the
code object that was actually shipped"*. At `optimize>=1` docstrings are **absent from the `.pyc`**,
and at `>=2` asserts are gone too. So against an `-O` artifact, **docstring recovery cannot be
certified at all** — the information is not in the file. That bears directly on our headline
differentiator: 115 of 679 benchmark rows carry a real docstring, and none of that could be proven
against an `-O` `.pyc`. It is a limit of the artifact, not unsoundness in the oracle.

## 4. What the oracle deliberately ignores

Excluded from the fingerprint, with how often each would have caused a false reject on the 600
wild rows had it been included:

| Field | In fingerprint? | Rows differing / 600 |
|---|---|---|
| `co_filename` | excluded | **600 (100%)** — every wild `.pyc` carries its builder's absolute path |
| `co_linetable` | excluded | **122 (20.3%)** — varies across builds with no semantic content |
| `co_firstlineno` | excluded | 0 |

Docstrings, `co_consts`, `co_names` and `co_exceptiontable` are **included** — the first because
docstring fidelity is a claim we make, the last because omitting it once produced a false proof.

## 5. Unverified means unknown, not wrong

The oracle is **sound but incomplete**:

```
verified   = PROVABLY correct. Identical code object => identical behaviour. No false positives.
unverified = UNKNOWN. A correct decompilation that compiles differently — a `while` where the
             original had a `for`, a differently-ordered but equivalent boolean — does not verify.
```

Reported accuracy is therefore a **lower bound on correctness**, not an estimate of it. Treating
the unverified remainder as errors understates the model; treating it as correct is unsafe.

## 6. Not tested — unknown, not claimed

- **Cross-minor (3.13).** No 3.13 interpreter on the measurement box; nothing was downloaded.
  The benchmark and the model are 3.12 only.
- **PyInstaller / Nuitka containers.** `import PyInstaller` → `ModuleNotFoundError`. Not measured.
- **`.pyc` from non-CPython or patched builds.** Not measured. Given that a *patch release*
  already produces the 0.33% floor, a patched build is a live risk, not a theoretical one.
- **Obfuscated or deliberately adversarial bytecode.** Not measured. No malware was fetched.

## 7. Where these limits are stated to users

| Limit | Stated in |
|---|---|
| Pre-flight is trivial | this file; `EVAL.md`; `harness/README.md`; both benchmark data cards |
| 0.33% wild false-reject floor | this file; `EVAL.md`; `weights/MODEL-CARD.md` |
| `-O` mismatch collapse, docstrings unprovable | this file; `EVAL.md`; `weights/MODEL-CARD.md` |
| unverified ≠ wrong | this file; `EVAL.md`; `weights/MODEL-CARD.md`; `harness/README.md` |
| 3.13 / PyInstaller untested | this file; `EVAL.md`; `weights/MODEL-CARD.md` |