DATA-CARD-training-corpus.md

# Data card — PyBytecode v2 / v3 training corpus

The corpus behind `pybytecode-v2-1.5b` and `pybytecode-v3-1.5b`. Written 2026-08-04; no data card
existed before.

**This corpus is not distributed.** Section 5 states why, as a property of the artifact rather
than an apology. The model weights are unaffected — see `WEIGHTS-LICENSE-PROPOSAL.md`.

---

## 1. What it is

| | |
|---|---|
| Task | Python 3.12 bytecode disassembly → original source |
| Rows | **48,196** (identical row set in v2 and v3) |
| Row shape | `{"input": <disassembly>, "output": <source>}` |
| Source dataset | `codeparrot/github-code-clean` (the dataset itself is Apache-2.0) |
| Shards used for training | 0–5 |
| Unit | one top-level function plus its transitive helpers and imports |
| Python | 3.12, `optimize=0` |
| Files | `data/foundry/pybytecode-v2_train.jsonl`, `-v3_train.jsonl`, `-v3-sft_train.jsonl` |

v3 differs from v2 in the **input representation only**. v2's `rep.py` omitted the exception
table's `end`, so a bare `try:` body and a `try/else:` body that compile to the same instruction
stream were byte-identical in the model's input. v3 emits `EXC try=Ls..Le -> ...` and the end
label joins the label set, making the two distinguishable. 14,122 of 48,196 inputs changed;
no row was added, removed or relabelled (`data/foundry/pybytecode-artifacts/v3_build_report.json`).
Any v2→v3 delta is therefore attributable to the representation fix alone.

## 2. Licence filtering — what was dropped, and why

`scripts/pybytecode/extract_v2.py` filters **per row** on `github-code-clean`'s `license` column
and keeps seven permissive values:

```
mit   apache-2.0   bsd-2-clause   bsd-3-clause   isc   unlicense   cc0-1.0
```

Every GPL, LGPL, AGPL, MPL and EPL row is dropped before extraction, along with everything the
column does not positively identify. The intent was to avoid training on reciprocally-licensed
source; the filter is a hard gate, not a preference.

Rows are then dropped by a chain of quality gates, each counted rather than silently applied:
unparseable; not stable under `ast.unparse` round-trip (`canonicalise(canon) != canon`); does not
compile; disassembly or source over the size ceiling; duplicate of an already-kept unit
(SHA-1 of the canonical source).

## 3. Decontamination against the test sets

Three layers, because one is never enough (`scripts/pybytecode/build_final.py`):

1. **Shard-disjoint** — training from shards 0–5, held-out pool from shards 8–9. The held-out
   pool was never read during training extraction.
2. **Repo-disjoint** — any repo appearing in training is removed from the held-out pool outright.
   A repo can span shards, so layer 1 does not imply this.
3. **Fingerprint-disjoint** — an identifier-blind structural fingerprint (every `Name`/`arg`/
   attribute → placeholder, every literal → its type name). This catches the same algorithm under
   renamed variables, which exact match misses. It caught 15 items exact match missed.

The CSN benchmarks apply the same identifier-blind fingerprint against this corpus. On the
600-row licensed rebuild it removed 3 rows.

## 4. Labelling

Labels are the canonical source itself, so the supervision is exact by construction. The grading
tier attached to each unit was assigned **by running an oracle**, not by inspection:

- **tier A / behavioural** — differential execution. Kills 100% of injected semantic bugs; the
  gold tier.
- **tier B / stub** — stubbed execution. Kills 78.9%, so scores on it are an **upper bound**.
- **tier C / AST-exact** — undercounts by roughly 2.4×, so scores on it are a **lower bound**.

Only 5.43% of all real top-level Python functions survive to be behaviourally adjudicable. That
ceiling is why the byte-identical recompile oracle exists: it needs no runnable environment and
so has 100% coverage.

## 5. The corpus cannot be redistributed

**Per-row attribution was not retained.** `extract_v2.py` carried `repo` and `license` on every
row through extraction, and `build_final.py:118` writes only `{"input", "expected"}` when it
emits the final splits. The intermediate pool that still held the metadata was written to `/tmp`
and no longer exists.

The consequence is specific: the corpus is 48,196 excerpts of MIT-, BSD-, Apache-, ISC- and
public-domain-licensed source, and every one of those licences except the two public-domain
dedications requires the copyright notice to be reproduced with the copy. We cannot produce those
notices, because we no longer know which row came from which repository. Redistributing the file
would strip required notices from tens of thousands of copyright holders.

This is a limitation of the artifact, not of the licences: nothing about the corpus is unlicensed
or reciprocally licensed. It is not fixable by adding a licence file, and it is not repaired by
listing the source dataset — attribution under these licences is per-work, not per-collection.

**It is fixable by rebuilding.** Extraction is deterministic and the source dataset is public.
Carrying `repo`, `license` and file path through `build_final.py`'s writer — one line — produces
an equivalent corpus that *is* redistributable with a `NOTICES` file. That is the recommended
fix for a v4 and it does not require retraining anything to be useful.

## 6. Held-out evaluation sets built from the same corpus

`data/foundry/pybytecode-v3-ood-{behavioural,stub,ast,doc}_test.jsonl` are drawn from the
held-out shards of this same corpus and **inherit section 5 exactly** — they carry no attribution
and cannot be redistributed either.

They are named `-ood-` but they are **not out-of-distribution**: they are the same source and the
same distribution as training, held out three ways. Matching all 974 canonicalised
`google-research-datasets/mbpp` rows against them yields **0 matches** in all four files, while
the v1-era `pybytecode-ood_test.jsonl` matches at **400/400**. Only that v1-era file is MBPP.
Full evidence in `LICENSING-DETERMINATION.md` §4.

They remain sound *generalisation* tests, and the numbers measured on them stand. Only the label
is wrong, and it should be corrected wherever it appears.

**On the tier-A set specifically (n=279, the set behind the 91.04% and 97.49% figures):** it is
the complete behaviourally-adjudicable population of the held-out pool — the 400-row cap did not
truncate it, whereas the stub and ast tiers both hit that cap. The pool was therefore larger than
400 units, but its exact size is **not recoverable**: `build_final.py` printed its census to
stdout and no run log was kept. The framing "279 of 400 = 70% of the set" does not describe these
files (the four tiers are largely disjoint populations — the 279 behavioural rows share 9 function
names with the 400 stub rows and 8 with the 400 ast rows) and should not be used. Report the
denominator as 279 with the sentence above, or rebuild the pool to recover the true rate.

## 7. Known confound, stated rather than engineered away

`extract_v2.py` carries `from __future__ import annotations`, and its `compile()` call inherits
the flag, so the training bytecode has PEP-563 stringised annotations. **Real `.pyc` files are not
compiled that way.** The benchmarks deliberately compile with `dont_inherit=True`, giving the
model an input distribution it was not trained on for annotated functions. Any resulting handicap
is a real property of the model and is counted against it.

## 8. Provenance summary

| Field | Value |
|---|---|
| Base model | `Qwen/Qwen2.5-Coder-1.5B-Instruct` (Apache-2.0) |
| Rows | 48,196 |
| Method | LoRA r=16, α=32, all attention + MLP projections, 1 epoch, lr 2e-4 |
| Build scripts | `scripts/pybytecode/{extract_v2,build_final,build_v3,rep,gen,verify}.py` |
| Grader hashes | all six match `grader.sha256` in `data/models/models.jsonl` |
| Redistributable | **No** — §5 |