EVAL.md

# PyBytecode v3 — full evaluation

The model card is deliberately short. This file is the long version: every number, the interval
around it, the set it was measured on, and the command that reproduces it. Nothing here is
softened relative to the card; the card is a subset.

Companion files: [`ORACLE-LIMITS.md`](ORACLE-LIMITS.md) for what the check can and cannot do,
[`DATA-CARD-training-corpus.md`](DATA-CARD-training-corpus.md) for training-data lineage.

---

## 1. What "certified" means

The oracle is **sound but incomplete**:

```
verified   = PROVABLY correct. Recompiling the prediction yields a code object byte-identical to
             the target's. Identical code object => identical behaviour. No false positives.
unverified = UNKNOWN, not wrong. A correct decompilation that compiles differently — a `while`
             where the original had a `for`, a differently-ordered but equivalent boolean —
             does not certify.
```

Three consequences:

- **Reported accuracy is a lower bound on correctness.** Everything counted here is proven; some
  of what is not counted is right too.
- **Best-of-N buys real accuracy, not a better guess.** Sample until something certifies, then
  stop. The verifier decides when to stop, so extra budget cannot mislead you.
- **The check is exact.** Comparison runs over the real code object, recursively, including
  docstrings and `co_exceptiontable` — not over a textual disassembly. That is not decoration: an
  earlier representation omitted the exception table's `end`, which made a `try:` body and a
  `try/else:` body indistinguishable, and it once certified a wrong prediction.

Strict L1 oracle throughout this file. Every figure carries a 95% interval and names its set.

---

## 2. Headline results — `csn-3.12-licensed`, n=600 across 117 repositories

| | certified | 95% CI |
|---|---|---|
| **PyBytecode v3, greedy** | **506 / 600 = 84.33%** | [80.48, 87.94] |
| **PyBytecode v3, verified best-of-32** | **562 / 600 = 93.67%** | [90.86, 96.08] |
| Untuned `Qwen2.5-Coder-1.5B-Instruct`, greedy | **4 / 600 = 0.67%** | [0.16, 1.35] |

Intervals are **repo-clustered** (bootstrap over repositories, 10,000 resamples), because rows
from one repository share an author and a house style and are not independent. Design effects
1.65 / 1.79 / 0.84.

The benchmark ships with the model, licence-resolved per row at the exact commit, so these numbers
are recomputable from files in this repository rather than taken on faith.

### The untuned base scores ~0, which is what makes the benchmark trustworthy

The same base model, given the same prompt, the same decoding budget and the same oracle, certifies
**4 of 600** here and **0 of 400** on our earlier benchmark. Under looser oracles it is clearer
still: the base produces **syntactically valid Python on 52% of rows** and reaches AST-identity on
**0%**. It paraphrases the disassembly into plausible-looking code with invented helpers rather
than inverting it.

A strong general code model cannot guess its way to a single point on this task. The benchmark is
not solvable by pattern-matching, and everything v3 scores was learned from the fine-tune. The
paired difference is not close to the noise floor: exact McNemar on 502 discordant pairs in our
favour and 0 against gives p < 1e-12.

### The benchmark rebuild reproduced the result

The published 600-row set was built from scratch: different rows (only 85 of 600 appear in the
earlier 400-row set), 117 repositories instead of 24, and a 1% per-repo cap replacing a 15%
concentration. It lands in the same place — certified@32 **93.67%** against **93.25%**, greedy
**84.33%** against **83.75%**. The difference of **+0.42pp** carries a 95% interval of
[−4.38, +5.22], so it is well inside noise, and the two size profiles are near-identical (median
59 vs 62 rep lines; 93.3% vs 94.0% of rows under 200), which is the composition factor that would
otherwise move the number. Standardised to the older set's size mix the new set reads 94.48%.
A result that survives a rebuild of the set it was measured on is worth more than the same result
quoted twice.

### Which numbers came from where

| Set | Published? | Why |
|---|---|---|
| `csn-3.12-licensed` (600) | **yes** | every row licence-resolved at its commit |
| `pybytecode-mbpp-3.12` (383) | **yes** | MBPP, CC-BY-4.0 |
| CSN-3.12 (400) | no | 45% of its rows are not redistributable |
| Held-out (279) | no | per-row attribution was not retained |

Our own results are measured on both the published set and the earlier pair. The figures in
*Comparison with other systems* exist only on the earlier pair, because that is where those runs
were done; they are labelled there because you cannot re-run them from what we shipped.

---

## 3. How much sampling budget to spend

The best-of-32 budget is **1 greedy decode + 31 sampled candidates at temperature 0.8**, with
verified early stop — sampling halts on the first candidate that certifies, so the mean cost is
far below 32. Across the 94 greedy failures it took a mean of 15.16 samples and recovered 56:

| budget | certified | |
|---|---|---|
| @1 (greedy alone) | 506 / 600 | 84.33% |
| @2 | 525 / 600 | 87.50% |
| @4 | 546 / 600 | 91.00% |
| @8 | 555 / 600 | 92.50% |
| @16 | 558 / 600 | 93.00% |
| @32 | 562 / 600 | 93.67% |

Most of the recovery arrives in the first few samples: @4 already captures two thirds of what @32
gets. Because early stop depends only on the index of the first passing sample, this curve is
exact rather than an estimate — the same cached generations give every point.

---

## 4. Accuracy by input size — the full curve

The size axis is **rep lines**: the number of lines in the disassembly handed to the model. One
line measures it:

```python
from harness.pybytecode_core.rep import disassemble_v2
rep_lines = disassemble_v2(code_object).count("\n")
```

### On the published 600-row benchmark

Recomputable from the files in this repo with `harness/size_curve.py`:

| rep lines | rows | v3 greedy | 95% CI | v3 best-of-32 | 95% CI |
|---|---|---|---|---|---|
| 0–49 | 224 | 95.98% | [93.01, 98.51] | 98.21% | [96.31, 99.58] |
| 50–99 | 224 | 89.73% | [85.17, 93.93] | 98.21% | [96.26, 99.57] |
| 100–199 | 112 | 65.18% | [55.36, 74.14] | 85.71% | [77.57, 92.98] |
| 200–299 | 27 | 51.85% | — | 77.78% | — |
| 300–399 | 5 | 60.00% | — | 100.00% | — |
| 400–599 | 5 | 0.00% | — | 0.00% | — |
| 600+ | 3 | 0.00% | — | 0.00% | — |

Intervals are repo-clustered, and omitted below 30 rows / 10 repositories rather than printed at a
width that would imply precision we do not have. The 300–399 bucket reading above the one below it
is n=5 noise, not a recovery. Median input in this benchmark is **59** rep lines; p90 is 167, max
1,622.

On this set the top two buckets stay at **0% even with 32 samples** — above ~400 rep lines,
sampling did not rescue a single one of the 8 rows. **96.64%** of greedy certifications and
**95.37%** of best-of-32 certifications come from units under 200 rep lines.

The four-row grouping on the model card is this same table pooled: under 100 rep lines is
416 / 448 = 92.86% greedy and 440 / 448 = 98.21% at 32; 200–399 is 17 / 32 = 53.12% and
26 / 32 = 81.25%; 400+ is 0 / 8 at both budgets.

### Pooled over both earlier benchmarks (n=679)

Pooled so the thin upper buckets carry as many rows as we can give them. This is also the only
table where PyLingual appears per bucket:

| rep lines | rows | v3 greedy | v3 best-of-32 | untuned base | PyLingual k=32 |
|---|---|---|---|---|---|
| 0–49 | 311 | 94.21% | 98.71% | 0.96% | 99.04% |
| 50–99 | 229 | 88.65% | 96.94% | 0.00% | 95.63% |
| 100–199 | 105 | 78.10% | 88.57% | 0.00% | 84.76% |
| 200–299 | 19 | 47.37% | 84.21% | 0.00% | 84.21% |
| 300–399 | 11 | 18.18% | 54.55% | 0.00% | 81.82% |
| 400–599 | 2 | 0.00% | 50.00% | 0.00% | 100.00% |
| 600+ | 2 | 0.00% | 0.00% | 0.00% | 50.00% |

- **The greedy knee is at ~200 rep lines**: 78.1% → 47.4% → 18.2% → 0% across four consecutive
  buckets.
- **Best-of-32 postpones the knee to ~300; it does not remove it.** Sampling 32 times buys roughly
  one bucket of headroom.
- Almost every point we score comes from small units. On the earlier CSN-3.12 set (n=400),
  **97.61%** of greedy certifications and **95.98%** of best-of-32 certifications are units under
  200 rep lines; on the published 600-row set it is **96.64%** of greedy certifications. The
  headline accuracy is a statement about small units.
- The top two buckets are n=2 each and carry nothing on their own. What carries the conclusion is
  the monotone decline through n=311/229/105/19/11 below them.

---

## 5. Comparison with other systems

[PyLingual](https://github.com/syssec-utd/pylingual) is another system that performs this task, by
symbolic reconstruction rather than generation. We ran it on the same sets, under the same oracle,
at a matched budget.

| CSN-3.12, n=400, 24 repos | certified | 95% CI (repo-clustered) |
|---|---|---|
| untuned base, greedy | 0 / 400 = 0.00% | — |
| v3 greedy | 335 / 400 = 83.75% | [77.84, 88.89] |
| v3 certified@32 | 373 / 400 = 93.25% | [88.82, 96.89] |
| PyLingual k=32 | 377 / 400 = 94.25% | [91.06, 96.80] |
| **union** | **393 / 400 = 98.25%** | [96.81, 99.51] |

| Held-out, n=279 | certified | 95% CI (Wilson) |
|---|---|---|
| untuned base, greedy | 3 / 279 = 1.08% | [0.37, 3.11] |
| v3 greedy | 254 / 279 = 91.04% | [87.11, 93.86] |
| v3 certified@32 | 272 / 279 = 97.49% | [94.91, 98.78] |
| PyLingual k=32 | 267 / 279 = 95.70% | [92.63, 97.52] |
| **union** | **275 / 279 = 98.57%** | [96.37, 99.44] |

**Neither difference is statistically significant** (exact paired McNemar, p = 0.6177 on CSN and
p = 0.2266 on held-out). We claim no accuracy advantage, and none is claimed against us. On cost
the comparison runs against us: ~1.5B parameters against their ~320M, and slower wall-clock.

Docstring recovery on the 115 held-out rows carrying a real docstring: **115 / 115 = 100%**
(PyLingual: 113/115 = 98.26%, under an oracle that does not score docstrings at all).

The held-out interval is **Wilson, not clustered**: every row of that set carries the same
placeholder repository value, so its clustering cannot be assessed. The CSN design effects of
2.34–2.69 show why that matters — on a set where one repository supplied 15% of rows, a naive
interval would have been far too narrow.

These figures were measured on our earlier benchmark pair, which is **not** published. PyLingual
has not been run on the 600-row licensed benchmark released alongside this model.

### The two approaches fail on different inputs

The more interesting measured result is not which number is larger, but that the two systems lose
on different inputs, along an axis that turns out to be **unit size**:

| rep lines | rows | PyBytecode greedy | PyBytecode @32 | PyLingual k=32 |
|---|---|---|---|---|
| 100–199 | 105 | 78.10% | 88.57% | 84.76% |
| 200–299 | 19 | 47.37% | 84.21% | 84.21% |
| 300–399 | 11 | **18.18%** | 54.55% | **81.82%** |
| 400–599 | 2 | **0.00%** | 50.00% | **100.0%** |

A generative model degrades as its input grows; a symbolic reconstruction does not. Two things
follow, both useful:

- **Large units are not intrinsically unsolvable.** A symbolic system handles them at 81.82% where
  we score 18.18%. What our curve measures is a capability limit of a 1.5B model, not a property
  of the task — which is a statement about where to spend the next round of scale.
- **The union exceeds either system**, 98.25% and 98.57% above. Because verification is sound, the
  union costs nothing to take: run either, keep the answer that certifies, fall through to the
  other. Neither system can certify a wrong answer, so combining them cannot mislead you.

---

## 6. Limits, in full

- **Large units — this is the measured limit.** Certification falls from 94% under 50
  representation lines to 18% at 300–399 and 0% above 400. The curve in §4 is the honest
  specification of what this model does. It was trained on functions; a whole module is a
  different regime in both size and shape, and we have not evaluated one.
- **Python 3.12 only.** A scope statement, not the explanation for the size behaviour above:
  trained and measured on 3.12, and the oracle refuses cross-minor input by design. Every number
  in this file is in-version, so nothing here is contaminated by a version mismatch.
- **It did not decompile real malware.** On the one wild PyInstaller-packed sample in scope, the
  entry-point module produced **nothing certifiable** — the prediction did not even compile. 3 of
  9 units in that sample certified, and those three were PyInstaller's own bootstrap plus a
  bundled stdlib module. Extraction and representation work (1,510/1,510 code objects recovered);
  end-to-end decompilation of real malware logic is **not demonstrated**.

  That entry point was both 3.13 *and* 491 representation lines, so **size and version are
  confounded in it and it is evidence for neither** on its own. It is reported because you should
  know the result, not because it diagnoses anything. The benign in-version curve in §4 is the
  instrument, and it already shows the model failing at that size on clean 3.12 input.
- **Annotated functions** are handicapped: the training corpus was compiled with PEP-563
  stringised annotations inherited, real `.pyc` files are not, and the benchmarks deliberately do
  not hand the model its training-time distribution back.

### The oracle's limits, where you will meet them

Full detail in [`ORACLE-LIMITS.md`](ORACLE-LIMITS.md).

- **0.33% false-reject floor on foreign `.pyc`.** On 600 wild install-time `.pyc`, 598 certify;
  2 do not, because CPython 3.12.3 and 3.12.13 generate different code for the same source. No
  normalisation fixes that without abandoning the byte-identical guarantee. It fails to a **false
  reject** — you are told "unknown" about a correct answer, never "verified" about a wrong one.
- **Optimization level must match the producer's.** A `.pyc` built with `-O` is a different code
  object. The wrong level does not degrade gracefully, it collapses to ~24%. Try all three; it
  costs three compiles. But note that at `optimize>=1` docstrings are absent from the `.pyc`, so
  **docstring recovery becomes unprovable** against such a file — a limit of the artifact, not of
  the oracle.
- **The 100% pre-flight the harness prints proves nothing about soundness.** It grades each
  reference against itself, i.e. compares `compile(x)` with `compile(x)`, so any deterministic
  function of the source scores 100% — including a stub that ignores the bytecode entirely. It
  detects a broken harness and nothing more. The soundness evidence is the mutation test (0 true
  survivors in 1,239 mutants) and 18 targeted blind-spot probes (18/18).
- **Formatting the AST does not capture can still change the bytecode, so byte-exact
  certification has a ceiling on some code.** CPython 3.12 inlines comprehensions (PEP 709) and
  emits a cleanup entry in the exception table for the inlined scope. How the comprehension is
  *line-wrapped* changes that entry. Two source texts with an identical AST, identical
  `ast.unparse` output and identical behaviour therefore compile to different code objects:

  ```python
  def f(xs): return sum([1 for a in range(len(xs)) for b in range(a, len(xs)) if xs[b] == a])

  def f(xs): return sum([1 for a in range(len(xs)) for b in range(a, len(   # same AST,
      xs)) if xs[b] == a])                                                 # different bytecode
  ```

  Measured on this interpreter: over 516 standard-library modules, rewriting each through
  `ast.unparse(ast.parse(src))` — a formatting-only change — preserves the compiled fingerprint
  for 416, **80.62%**. Roughly one wild module in five compiles differently after a rewrite that
  changes no code. (Byte-identical *text* round-trip is 0.6%, as expected, since `unparse`
  normalises formatting by design. 80.62% is the figure that matters, because certification
  compares code objects, not text.)

  On the published benchmarks, an AST-identical re-wrap changes the fingerprint for **26 of 600**
  rows of `csn-3.12-licensed` (4.33%) and **20 of 383** rows of the MBPP set (5.22%). 12.8% and
  19.1% of their rows contain a comprehension at all.

  **This is a property of byte-exact certification, not of this model, and it applies to every
  system scored this way — including PyLingual in §5.** A decompiler that recovers correct source
  but wraps it differently is recorded as failing on those inputs. It fails safe: the verdict is
  "not proven", never "verified" for a wrong answer.

  It costs the numbers in this file nothing measurable. Both benchmarks' reference text is
  `ast.unparse` output and the model reproduces that canonical single-line form, so of the 94
  greedy failures on `csn-3.12-licensed`, **0** have an AST identical to the reference. The
  ceiling limits what certification can *prove* about differently-formatted source; it does not
  inflate or deflate the reported scores.

  Two consequences worth stating plainly. First, both published benchmarks are **internally
  self-consistent**: the reference source and the `.pyc` are compiled from the same normalised
  string — verified 600/600 and 383/383 — which is what makes the oracle sound. Second, that
  normalised string is not the upstream text. For the MBPP set, **1 of 383 rows**
  (`src/00117.py`, MBPP `task_id` 757) compiles to different bytecode than the original MBPP
  solution, because that solution wraps a nested comprehension across lines. The row remains a
  valid decompilation task and its rewrite is declared in the dataset's `provenance.transform`
  and `NOTICES.md`; it is recorded here rather than quietly re-derived.
- **Untested:** cross-minor 3.13, Nuitka, non-CPython builds, obfuscated bytecode.

---

## 7. Reproduce every number

No model, no GPU, no network, no third-party decompiler — CPython 3.12 standard library only:

```bash
cd harness
python3 grade.py --bench ../benchmarks/csn-3.12-licensed/bench.jsonl --self-test-only --out /tmp/st.json
python3 analyze_scores.py --bench ../benchmarks/csn-3.12-licensed/bench.jsonl \
    --greedy ../generations/gen_v3_csn600.jsonl \
    --base   ../generations/gen_base_csn600.jsonl \
    --out /tmp/scores.json --rows-out /tmp/rows.jsonl
python3 size_curve.py --bench ../benchmarks/csn-3.12-licensed/bench.jsonl \
    --greedy ../generations/gen_v3_csn600.jsonl \
    --base   ../generations/gen_base_csn600.jsonl \
    --out /tmp/size_curve.json
```

Per-row verdicts for all 600 rows — repo, function, commit SHA, SPDX, tuned verdict, base verdict
— land in `rows.jsonl`. Every aggregate in §2 and §4 is recomputable from that file.

The figures in §5 are the exception: they were produced on the unpublished earlier benchmark pair
and against a GPL-3.0 third-party decompiler we do not vendor, so they cannot be re-run from what
ships here.

---

## 8. Training

| | |
|---|---|
| Base | [`Qwen/Qwen2.5-Coder-1.5B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct) (Apache-2.0) |
| Method | LoRA r=16, α=32, all attention + MLP projections, 1 epoch, lr 2e-4, merged into the base |
| Rows | 48,196 pairs of Python 3.12 disassembly → source |
| Corpus | `codeparrot/github-code-clean`, per-row filtered to 7 permissive licences; all GPL/LGPL/AGPL/MPL/EPL dropped before extraction |
| Decontamination | shard-disjoint, repo-disjoint, and identifier-blind fingerprint-disjoint |

**The training corpus is not distributed.** Per-row attribution was not retained during
extraction, so the corpus cannot be redistributed without dropping required MIT/BSD/Apache
notices. That is a property of the artifact, not of the licences — nothing in it is unlicensed or
reciprocally licensed. Full lineage, including what was dropped and why:
[`DATA-CARD-training-corpus.md`](DATA-CARD-training-corpus.md).