harness/README.md

# PyBytecode evaluation harness

Grades Python 3.12 decompilations under a **sound** oracle: recompile the prediction and require
the resulting code object to be byte-identical to the reference's, recursively, including
docstrings and `co_exceptiontable`. A pass is a proof, not a plausibility judgement.

Its limits are real and are documented in **[`../ORACLE-LIMITS.md`](../ORACLE-LIMITS.md)**:
a 0.33% false-reject floor on foreign `.pyc`, a hard dependency on matching the producer's
optimization level, and the fact that the 100% pre-flight this harness prints proves far less
than it looks like it does. Read that before quoting any number from here.

**Everything here runs from a fresh clone with the Python standard library alone.** No model, no
GPU, no network, no API key, no PyLingual, and no path outside the clone. Verified — see
*Fresh-clone verification* below.

---

## Requirements

| To do this | You need |
|---|---|
| Grade cached generations (every published number) | CPython **3.12.x**. Nothing else. |
| Generate new predictions | the above + any OpenAI-compatible server |
| Rebuild a benchmark from its source dataset | the above + `pip install -r requirements.txt` + authenticated `gh` |
| Compare against PyLingual | the above + the optional extra (`requirements-pylingual.txt`) |

CPython 3.12 is not a preference. The benchmark is 3.12 bytecode; on 3.11 or 3.13 the reference
`.pyc` files will not compare and pre-flight will fail loudly rather than score silently.

Configuration is environment-only — `PYBYTECODE_ENDPOINT`, `PYBYTECODE_MODEL`,
`PYBYTECODE_API_KEY` (see `config.py`). Nothing is hard-coded to a machine.

---

## Reproduce every published number, from cache

No model, no GPU, no network. The expensive part — the generations — is cached in this bundle, so
every number below is re-derivable for the cost of a few CPU-minutes.

```bash
cd harness
B=../benchmarks/csn-3.12-licensed/bench.jsonl
G=../generations

# 1. Prove the harness is sound before believing any score it prints.
python3 grade.py --bench $B --self-test-only --out ../results/selftest_csn.json

# 2. Full scoring: tuned greedy, untuned-base control, verified best-of-N,
#    with repo-clustered confidence intervals and per-row verdicts.
python3 analyze_scores.py --bench $B \
    --greedy  $G/gen_v3_csn600.jsonl \
    --samples $G/boN_v3_csn600.jsonl \
    --base    $G/gen_base_csn600.jsonl \
    --out ../results/scores_csn600.json --rows-out ../results/rows_csn600.jsonl
```

`../results/scores_csn600.json` holds the aggregates and intervals;
`../results/rows_csn600.jsonl` holds a per-row verdict for all 600 rows — repo, function, commit
SHA, SPDX, instruction count, whether tuned greedy certified, whether the untuned base certified,
and the index of the first passing sample. Every headline number is recomputable from that file
alone.

The MBPP set is graded the same way:

```bash
python3 grade.py --bench ../benchmarks/mbpp-ood/bench.jsonl \
                 --self-test-only --out ../results/selftest_mbpp.json
```

### Confidence intervals are repo-clustered

Rows from the same repository are not independent — shared author, house style, shared helpers —
so a plain binomial interval understates uncertainty. `analyze_scores.py` resamples
**repositories** with replacement (the cluster bootstrap, 10,000 draws) and reports the 2.5th and
97.5th percentiles, alongside the naive binomial interval and the design effect so the cost of
clustering is visible rather than assumed.

This is meaningful here only because the benchmark caps any repository at ~1% of rows. On the
superseded 400-row set, one repository supplied 15% and a clustered interval would have been the
only honest one to quote — and none was.

## Generate predictions (needs a model)

```bash
export PYBYTECODE_ENDPOINT=http://localhost:1234/v1
export PYBYTECODE_MODEL=pybytecode-v3-1.5b

python3 generate.py --bench ../benchmarks/csn-3.12-licensed/bench.jsonl --out gen.jsonl
python3 generate.py --bench ../benchmarks/csn-3.12-licensed/bench.jsonl \
                    --out boN.jsonl --temperature 0.8 --samples 32     # for best-of-N
python3 grade.py --bench ../benchmarks/csn-3.12-licensed/bench.jsonl \
                 --gen gen.jsonl --out ../results/mine.json
```

`generate.py` is resumable: an interrupted run is completed by re-running the same command.

---

## PyLingual — optional, user-installed, never vendored

PyLingual is `GPL-3.0-only`. It is **not** a dependency of this harness and is **not** included
in this repository in any form. We import exactly one symbol from it,
`pylingual.equivalence_check.compare_pyc`, and only at grading time, so that the head-to-head
comparison uses *their* definition of a perfect decompilation rather than our reimplementation of
it. Nobody can say we loosened their bar.

Without it: `dual_oracle.py` runs and reports `pylingual_available: false`, with their-oracle
columns `null` — explicitly absent, never silently zeroed. `grade.py --oracle theirs` exits with
an explanation. Everything else is unaffected.

Install instructions and the exact commit we measured: `requirements-pylingual.txt`.

---

## The self-tests, and why they can refuse

Two gates run before any score is printed:

- **Pre-flight** — grade every reference label against itself. A byte-perfect model must score
  100%. Anything less means the harness is broken, not that the model is bad.
  **It is trivial by construction and is NOT evidence of soundness**: it compares `compile(x)`
  with `compile(x)`, so any deterministic function of the source scores 100%, including a stub
  that ignores the bytecode. It detects a broken harness (mismatched `.pyc`, wrong Python minor,
  corrupt row) and nothing more. Soundness evidence is the mutation test and the 18 blind-spot
  probes — see `../ORACLE-LIMITS.md` §1.
- **Mutation test** — corrupt each label (swap `+`/`-`, flip a comparison, break a `return`) and
  confirm the oracle kills it. A grader that passes mutants is a stub and its scores are noise.

If either is below 100% the command **refuses to print a score**. That is intended behaviour.

Mutation candidates that do not change the program's AST are discarded rather than counted, so a
comment-only rewrite cannot be mistaken for a surviving mutant. This matters: the original harness
generated `return None` → `return None #None` and would have scored that as a survivor.
It reported 131/131 only because no row in its sample had a bare `return None`; the new 600-row
benchmark has three, and the un-filtered generator scored 97.48% and correctly refused to run.
The fix is in the mutation generator; the oracle was never loosened.

Measured on the current benchmarks:

| Benchmark | Pre-flight | Mutation kill rate |
|---|---|---|
| `csn-3.12-licensed` (n=600) | 600/600 = 100% | 116/116 = 100% |
| `mbpp-ood` (n=383) | 383/383 = 100% | 199/199 = 100% |

---

## Fresh-clone verification

Performed 2026-08-04. `git clone` into a scratch directory, no `PYBYTECODE_*` variables set, no
PyLingual installed (`ModuleNotFoundError`), no model running, working directory not the original
path:

```
1. self-test csn-3.12-licensed   PRE-FLIGHT 600/600 = 100.0%   MUTATION 116/116 = 100.0%   SOUND
2. dual_oracle on cached CSN     n=400  ours 335 = 83.75%   pylingual_available: false
3. bestofn_grade on cached CSN   greedy 335   certified@32 373 = 93.25%
```

Identical to the published values. Git LFS is required to materialise the cached generations —
see `../RELEASE-BLOCKERS.md`, which records the state of the LFS objects on the remote.

---

## Files

```
config.py            paths + endpoint; the only environment-aware module
common.py            oracles, fence stripping, pre-flight, mutation test
grade.py             score under one oracle (ours by default), self-tests first
dual_oracle.py       score under both oracles; degrades gracefully without PyLingual
bestofn_grade.py     verified best-of-N from cached samples
generate.py          the only script that needs a model
pybytecode_core/     verify.py + rep.py, copied verbatim from scripts/pybytecode/
```

`pybytecode_core/` is a byte-identical copy so the harness is self-contained. Verify it:

```
sha256  45ca921f86c73622d5b46295b9264e4a68bd697dce94e27a786c73f4416319b9  verify.py
sha256  1ba81c307c72e83d2c6a6eceb9936bf3f13645d267df3c7c676d421b9b2f43d9  rep.py
```