ORACLE-LIMITS.md

The oracle's real limits

The verifier is the reason to use PyBytecode at all, so its limits belong in front of a user, not in an appendix. Everything here is measured; sources are named per section.

---

1. The pre-flight 100% proves almost nothing. Read this before quoting it.

Every grading command prints PRE-FLIGHT 600/600 = 100% before it scores. That number is trivial by construction and is not evidence of soundness.

Pre-flight grades each reference label against itself. The oracle asks whether compile(prediction) and compile(reference) produce the same code object — so at pre-flight it is comparing compile(x) with compile(x). It would return 100% for any deterministic function of the source, including a stub that hashes the input string and ignores the bytecode entirely.

What pre-flight actually detects is a broken harness: a benchmark whose .pyc files do not match their sources, a Python version mismatch (3.11 or 3.13 against a 3.12 benchmark), a corrupt row. Those are real failure modes and worth catching, which is why it runs. But a passing pre-flight says the instrument is plugged in, not that it measures anything.

Soundness evidence comes from the mutation test and the blind-spot probes, not from pre-flight: corrupt a label and require the oracle to reject it. Measured (evidence/ORACLE-MUTATION.md): 0 true survivors in 1,239 mutants, and 18/18 targeted blind-spot probes behave as required — including the historical failure where a try: body and a try/else: body were indistinguishable, docstring changes, docstring removal, float-vs-int, bool-vs-int and -0.0 vs 0.0.

Even the mutation kill rate is weak evidence on its own: for a byte-identical oracle a kill is close to tautological, since a mutant survives only if it compiles to a structurally identical code object. The probes are the load-bearing test, because they ask the question that actually bit us once — is a behaviourally load-bearing field missing from the fingerprint?

Caveat on mutation supply, stated rather than hidden: 188 of 600 wild rows (31%) produced no effective mutant within 30 tries, and 323 void attempts were discarded. The wild kill rate is measured on the 412 rows that did produce one.

2. The 0.33% wild false-reject floor

Against .pyc files built by someone else, the oracle refuses a small fraction of correct answers. Measured on 600 wild install-time .pyc from installed site-packages (evidence/GATE-RESULT.md):

	certified	false reject	false accepts
L0 (old constant encoding)	585/600 = 97.5%	15 = 2.5%	0 / 1,274
L1 (shipping)	598/600 = 99.67%	2 = 0.33%	0 / 1,274

13 of the 15 L0 failures were our own defect — repr() of a set/frozenset/dict follows the compiling process's hash seed, which also made the L0 verdict non-deterministic (585 / 592 / 584 / 585 / 589 under PYTHONHASHSEED 0–4). L1 fixes it and returns 598 under all five seeds.

The remaining 0.33% is a real floor and is not fixable. One distinct module (pandas/_testing/__init__.py) compiles differently under CPython 3.12.3 than under 3.12.13 — co_code 2,692 vs 2,696 bytes, and a differing co_exceptiontable. The source is correct; the compiler patch release differs. No normalisation removes this without abandoning the byte-identical guarantee.

It degrades to a false REJECT, never a false accept. You are told "unknown" about a correct answer; you are never told "verified" about a wrong one. That is the safe direction, and it is the direction the design chose deliberately.

3. Optimization level must match the producer's, or verification collapses

A .pyc built with -O or -OO is a different code object. Measured on 679 sources compiled by a foreign interpreter at each level and graded at each level:

producer ↓ / grader →	0	1	2
0	100.0%	95.43%	23.86%
1	95.43%	100.0%	24.15%
2	23.86%	24.15%	100.0%

The diagonal is 679/679 at every level. Guessing wrong is not a graceful degradation — it collapses to ~24%. There are only three levels and trying all three costs three compiles, so this is "needs normalisation to survive", not "breaks". The harness does not currently search the three levels automatically; a user verifying a foreign .pyc must do it.

What trying all three costs you, and it is not nothing. The certificate changes meaning from "byte-identical to the code object the original source compiles to" to "byte-identical to the code object that was actually shipped". At optimize>=1 docstrings are absent from the .pyc, and at >=2 asserts are gone too. So against an -O artifact, docstring recovery cannot be certified at all — the information is not in the file. That bears directly on our headline differentiator: 115 of 679 benchmark rows carry a real docstring, and none of that could be proven against an -O .pyc. It is a limit of the artifact, not unsoundness in the oracle.

4. What the oracle deliberately ignores

Excluded from the fingerprint, with how often each would have caused a false reject on the 600 wild rows had it been included:

Field	In fingerprint?	Rows differing / 600
co_filename	excluded	600 (100%) — every wild .pyc carries its builder's absolute path
co_linetable	excluded	122 (20.3%) — varies across builds with no semantic content
co_firstlineno	excluded	0

Docstrings, co_consts, co_names and co_exceptiontable are included — the first because docstring fidelity is a claim we make, the last because omitting it once produced a false proof.

5. Unverified means unknown, not wrong

The oracle is sound but incomplete:

verified   = PROVABLY correct. Identical code object => identical behaviour. No false positives.
unverified = UNKNOWN. A correct decompilation that compiles differently — a `while` where the
             original had a `for`, a differently-ordered but equivalent boolean — does not verify.

Reported accuracy is therefore a lower bound on correctness, not an estimate of it. Treating the unverified remainder as errors understates the model; treating it as correct is unsafe.

6. Not tested — unknown, not claimed

• Cross-minor (3.13). No 3.13 interpreter on the measurement box; nothing was downloaded. The benchmark and the model are 3.12 only.
• PyInstaller / Nuitka containers. import PyInstaller → ModuleNotFoundError. Not measured.
• .pyc from non-CPython or patched builds. Not measured. Given that a patch release already produces the 0.33% floor, a patched build is a live risk, not a theoretical one.
• Obfuscated or deliberately adversarial bytecode. Not measured. No malware was fetched.

7. Where these limits are stated to users

Limit	Stated in
Pre-flight is trivial	this file; EVAL.md; harness/README.md; both benchmark data cards
0.33% wild false-reject floor	this file; EVAL.md; weights/MODEL-CARD.md
-O mismatch collapse, docstrings unprovable	this file; EVAL.md; weights/MODEL-CARD.md
unverified ≠ wrong	this file; EVAL.md; weights/MODEL-CARD.md; harness/README.md
3.13 / PyInstaller untested	this file; EVAL.md; weights/MODEL-CARD.md