evidence/documents/DATA.md

# DATA: corpus provenance, filtering, and the safety review

What goes into Wisp, where it came from, what was removed, and what was checked.
This is the record the model card is built from, so it is written to be read by
someone deciding whether to trust the weights.

## Sources

| Source | Subset | Configured weight | Realized train tokens | Source terms |
|---|---|---:|---:|---|
| `bigcode/starcoderdata` | python | 0.24 | 1,200,566,506 | card label `other`; original repository terms apply |
| `bigcode/starcoderdata` | javascript | 0.13 | 650,722,190 | as above |
| `bigcode/starcoderdata` | typescript | 0.12 | 600,701,168 | as above |
| `bigcode/starcoderdata` | go | 0.09 | 451,081,764 | as above |
| `bigcode/starcoderdata` | rust | 0.09 | 451,511,705 | as above |
| `bigcode/starcoderdata` | java | 0.08 | 400,525,337 | as above |
| `bigcode/starcoderdata` | c | 0.05 | 250,363,883 | as above |
| `bigcode/starcoderdata` | shell | 0.03 | 150,445,965 | as above |
| `bigcode/starcoderdata` | sql | 0.02 | 100,765,073 | as above |
| `bigcode/starcoderdata` | markdown | 0.07 | 351,026,251 | as above |
| `HuggingFaceFW/fineweb-edu` | sample-10BT | 0.08 | 400,290,897 | ODC-By 1.0 and Common Crawl terms |

The final build log survived and is now preserved byte-for-byte as
`evidence/run1_corpus_build.log`. Its mtime matches the final shard index, and
the source totals sum exactly to the index's 5,008,000,739 train tokens. The
realized train split is 92.006972 percent StarCoderData, including 84.997663
percent implementation-language code and 7.009309 percent Markdown, plus
7.993028 percent FineWeb-Edu. The web share exists to keep natural-language
comments, docstrings, and identifiers from drifting, and was cut from 15 percent
to 8 to buy back budget for code when the niche was fixed as FIM-first.

This recovered log closes the aggregate train-token-count gap. It does not
recover accepted row identities, rejection counts, source metadata, per-source
validation overshoot, or row-level licensing obligations.

`starcoderdata` is gated. Access is granted per account by accepting the terms on
the dataset page. Gated access is not a permissive-license attestation. Its card
declares `license: other`; the original repository licenses and attribution
clauses remain applicable, together with the dataset's update and removal terms.
FineWeb-Edu is distributed under ODC-By 1.0 and remains subject to Common Crawl
terms.

All ten code subsets above were verified to stream after access. Run 1 retained
tokenized text only. It did not preserve StarCoderData row IDs, repository names
or paths, FineWeb IDs or URLs, a per-row license mapping, or an attribution index.
The resulting shards cannot support a local per-file licensing or attribution
audit. Apache 2.0 applies to the Wisp artifact; it does not override source-data
terms or the licenses of code the model may generate.

## The malware scan finding, resolved

Hugging Face flagged four files in `bigcode/starcoderdata` as unsafe. They were
checked individually rather than taken on the file name, because three of the four
names are ambiguous across subsets.

| Flagged file | Subset it is actually in | Used by Wisp |
|---|---|---|
| `train-00008-of-00055.parquet` | `git-commits-cleaned` | no |
| `train-00019-of-00055.parquet` | `git-commits-cleaned` | no |
| `train-00027-of-00055.parquet` | `git-commits-cleaned` | no |
| `train-00043-of-00059.parquet` | `github-issues-filtered-structured` | no |

**None of the four are in a subset this project reads.** The fourth needed
resolving: a file of that exact name exists in both
`github-issues-filtered-structured` and in `python`, which Wisp does use. Querying
the per-file scan status settles it. The `github-issues` copy is `unsafe`; the
`python` copy is `queued`, which means unscanned, not flagged.

Worth stating plainly, since "queued" is most of the repository: the majority of
files carry no scan verdict either way. Four are affirmatively flagged, none of
them ours, and the rest are largely unscanned rather than certified clean.

### Why this is a low risk here regardless

- **Parquet has no execution vector on read.** It is columnar data parsed by
  pyarrow. This is not the pickle situation, where deserialising *is* execution.
- **A flag on a GitHub-derived corpus is expected.** Commit messages and issue
  threads contain malware source, exploit proofs of concept, and EICAR test
  strings, because those things live on GitHub. The scanner is matching content,
  and it is right to.
- **Nothing in this pipeline executes corpus content.** The path is stream, then
  tokenize, then write uint16 shards. Where enabled, `ast.parse` and `json.loads`
  parse without evaluating.
- **The real exposure is to the model, not the machine.** Training on malware
  source teaches malware patterns. That argues for excluding such data on model
  safety grounds, and it is excluded here, since the affected subsets are not in
  the mixture.

## The quality gate

Run 1 applied the structural checks below to every source. Verifier-first was
borrowed from `agent-toolkit`'s `verify/verifiers.py` in stance rather than in
code: executing scraped code to decide whether to train on it would be the
security incident.

Rejected:

- Longest line over 1000 characters. Minified bundles, generated files, data on
  one line.
- Mean line length over 120 characters. Past anything hand written.
- Alphanumeric-plus-whitespace fraction under 0.55 in the first 20KB. Base64,
  hex dumps, embedded binary assets.
- Anything under 64 characters.

An earlier draft also claimed that run 1 rejected Python that failed `ast.parse`
and JSON that failed `json.loads`. That was wrong. The Hub loader requested the
column `path`, while StarCoderData exposes `max_stars_repo_path`. Its rows
therefore reached the structural checks without a file extension, and neither
extension parser activated during the run 1 Hub build.

The 92.7 percent retention figure came from a local forced-Python pilot, not from
the final streamed corpus. It remains evidence that the structural gate is not
vacuous, but it is not a final-corpus retention measurement.

After run 1 launched, the loader was repaired to map the pinned StarCoderData
schema, fail closed if its path field disappears, and remove the leading
`<reponame>` metadata line only for syntax parsing. The original text remains the
payload. Synthetic mutation checks prove invalid Python is rejected. This repair
applies to future source streaming only; it does not change the active process or
existing shards. E2 must not use the repaired stream to select a different
document population. Its no-FIM data is instead produced by the registered
deterministic normalization of run 1's token stream.

The rationale is capacity, not tidiness. At 108M parameters, tokens spent
modelling a webpack bundle are tokens not spent on code.

## The tokenizer, benchmarked rather than assumed

32,768 byte-level BPE, digits split individually, whitespace runs allowed to
merge, trained on 400,000 documents drawn round-robin across the eleven
configured source entries. The tokenizer sampler did not apply the configured
token weights. Each language entry therefore contributed about one eleventh of
the document sample, while Python contributed 24 percent of the later shard
budget. This is a reproducibility limitation and a future tokenizer experiment,
not a reason to alter the tokenizer under an active model.

Measured on 366KB of fresh starcoderdata across Python, JavaScript, Go, and Rust,
none of it in the tokenizer's training sample:

| tokenizer | vocab | chars/token |
|---|---|---|
| wisp code32k | 32,768 | **3.52** |
| bigcode/starcoder2-15b | 49,152 | 3.54 |
| Qwen/Qwen2.5-Coder | 151,643 | 4.06 |

Compression matters because it sets how much code 5B tokens actually buys. A
tokenizer 15 percent worse means 15 percent less code for the same compute.

The 32K tokenizer therefore matches StarCoder2's compression while carrying a
third less vocabulary. Qwen's 15.6 percent advantage is not practical at this
scale: a 151,643-token embedding at d_model 768 alone costs 116M parameters,
larger than this entire model.

That comparison did not answer the more relevant challenge, 16K versus 32K.
`benchmark_tokenizer_vocab.py` now does. Greedy BPE learns merges sequentially,
so truncating the frozen tokenizer at the 16K merge boundary gives the exact
same-training-sample 16K merge prefix. Both tokenizers were measured on all 343
documents in the clean, source-weighted publication holdout:

| vocab | tokens | chars/token | tied embedding | model if only vocab changes | training head MACs/token |
|---|---:|---:|---:|---:|---:|
| 16,384 | 1,117,129 | 3.2123 | 12.58M | 95.57M | 37.75M |
| 32,768 | 1,069,055 | 3.3568 | 25.17M | 108.16M | 75.50M |

The 16K tokenizer needs 4.50 percent more tokens for the same text, or sees 4.30
percent fewer characters at a fixed token budget. In return it saves 12,582,912
parameters, 11.63 percent of the whole 32K model, and halves the width of each of
the three output projections evaluated per training position.

So 32K was not forced, and the old wording claiming that was wrong. Run 1 remains
32K because its tokenizer, corpus, and active weights are already frozen. The
evidence makes 16K a serious future-successor candidate, but compression and
projection arithmetic do not establish end-to-end speed or model quality. Those
require a post-run Metal throughput arm and a matched training-quality arm.
Registered run 2 remains a 32K no-FIM ablation.
`config/tokenizer_vocab_receipt.json` records the exact hashes and measurements.

## Fill in the middle, and what the window geometry does to it

The configured chunk-level rate is not the rate a sampled training window sees,
and the gap is large enough to matter.

After tokenization, each source document is split into chunks of at most 1024
tokens. FIM is selected independently for each chunk at prepare time. Training
then samples random 2051-token windows out of the concatenated stream. A window
can see a complete frame, a fragment such as a suffix with no prefix sentinel,
or no frame at all. Measured while comparing the discarded whole-document build
with the final chunked build, sampling 4,000 windows per shard:

| build | complete frame | partial | none |
|---|---|---|---|
| whole-document FIM | 43.2% / 62.1% / 69.9% | 9 to 19% | 20 to 38% |
| 1024-token chunks | **90.1% / 89.8% / 91.2%** | 5 to 7% | 3 to 4% |

The headline capability was being diluted by roughly a fifth before the fix.

**The arithmetic, and its side effect.** With chunks of 1024 and windows of 2051,
a window spans about two chunks, and a complete frame appears if *either* chunk was
transformed. At `fim_rate` 0.7 that is `1 - 0.3^2 = 91%`, which matches the
measurement. The same arithmetic says pure left-to-right windows fall to about
9%, well under the 30% that would follow from treating the configured value as a
window-level rate.

That is a real change to the mixture and it was not the goal, it is a consequence.
Kept deliberately, for two reasons. Every token is still predicted
autoregressively, so next-token capability is not lost; what shrinks is exposure to
the *unframed* format, not to left-to-right prediction itself. And the deployment
target is a cursor inside an existing file, which is the framed case.

If a future run wants the window-level rate to match a 70 percent target,
solve `1 - (1 - r)^2 = 0.7` for `r`, giving `fim_rate` about 0.45. The research
design is insulated from this either way: the primary endpoint compares FIM against
FIM-with-shuffled-suffix, both of which are framed, so format exposure cancels.
The FIM over L2R secondary number is the one this affects.

## Fill in the middle

Applied at prepare time, not train time, independently to 70 percent of tokenized
chunks of at most 1024 tokens, split evenly between PSM and SPM orderings. It was
not applied once per source document. The rate was raised from 0.5 when the niche
was fixed, since cursor completion is the deployment target. `fim_rate` now lives
in the config and the CLI flag overrides it.

## Validation split in the final corpus

The first corpus build filled one shared validation counter from Python and was
discarded. The final corpus was rebuilt after `prepare_data.py` changed to
allocate validation tokens per source in the same proportions as training.
Run 1 therefore uses a source-stratified validation shard with 20,006,112
tokens. The discarded build is not used by the trainer.

## Scheduled tokens are not unique corpus coverage

Run 1 draws random 2051-token windows with replacement. Its 19,073 steps,
8-example micro batch, and 16-way accumulation schedule 2,441,344 windows and
4,999,872,512 prediction positions. The registered step-300 sampler reset
repeats the first 38,400 window coordinates, leaving 2,402,944 distinct
scheduled draws at most.

Under the sampler's uniform-start distribution, the interval-coverage
approximation `1 - exp(-4,921,229,312 / 5,008,000,739)` is 62.57 percent.
Roughly 37.43 percent of corpus positions are therefore expected not to appear
in a prediction window, while other positions repeat. This is expected
with-replacement sampling, not shard corruption. A future run should compare a
checkpointed shuffled block order without replacement so nearly the full corpus
is exposed before repetition. E2 keeps run 1's sampler semantics for a matched
comparison.

## Corpus-build attestation

Run 1's existing index did not include shard hashes, so a full read-only
attestation now binds all 52 files, 10,056,013,702 bytes, the source index, the
tokenizer, and the normalization program in
`config/run1_shard_integrity_receipt.json`. The scan found 27 detectable FIM
frames split by literal EOS tokens inside their payloads. It reassembles those
visible groups and restores 33 internal EOS tokens. The resulting normalized
totals are 4,992,043,184 train tokens and 19,949,502 validation tokens.

This is deterministic token-stream normalization, not proof of exact original
examples. An EOS inside unframed content, an EOS after the final FIM marker, and
raw content that already has the visible grammar of a valid FIM frame are
observationally indistinguishable from preprocessing structure after
tokenization. The receipt records `exact_original_units_proven: false`, the
visible grouping manifest, and domain-separated source, recovered, and
destination hashes.

The run 2 builder never reopens a Hub source. It requires the attested run 1
files, a missing or empty real output directory, and the exact registered
derivation strategy. It rejects symbolic links, unsafe or duplicate paths,
mutation during reads, malformed FIM grammar, cross-shard recovery, residual FIM
sentinels, and any count or hash mismatch. It writes one destination shard and
one fixed-width unit-boundary sidecar per source shard. Only after independently
rereading every output does it atomically publish a schema-3 index with
per-shard hashes, sidecar hashes, and split-level normalization evidence.

Both normalized splits fall just below the old requested 5B and 20M corpus
floors because three generated sentinels are removed from every visible FIM
frame. The fixed run 2 schedule still samples 4,999,872,512 token positions.
This changes neither registered E2 endpoint nor the model geometry, but it
supersedes the earlier claim that exact raw examples could be recovered.

This was added before any run2 corpus existed. The amendment is recorded in
`config/eval_format_ablation_receipt.json`; it changes build provenance and
failure behavior, not either registered endpoint. Run1's checkpoint did not
record source-file hashes or the sampler state. Its step-300 process recovery
reset the legacy shard sampler and therefore repeats 78,643,200 scheduled token
positions, 1.57 percent of the training budget. That observed reset is now
registered in both E2 arms as `sampler_reset_steps: [300]`. New checkpoints
store the exact sampler RNG state, and legacy checkpoints reconstruct the
registered position. This matches reset timing, not raw examples across
different arm shards. The model and optimizer are unchanged, but exact run1
source-state equivalence cannot be proven.

Run 1 also did not pin Hub revisions at build time or save an ordered raw-source
manifest. Revisions captured from the local cache afterward are useful
repository-level evidence, but they cannot reconstruct the accepted rows,
repository paths, rejection counts, validation overshoot by source, or row-level
obligations. Aggregate realized train-token totals survive in the preserved
final build log.
`config/training_data_receipt.json` binds these limitations, the audited source
cards and schemas, the prospective parser repair, and the deterministic
normalization limitation.

## Held out data

The publication evaluation corpus is not the old local-file fallback. It is built
from `config/eval_repos.json`, which pins 29 repository-language strata at exact
commits. Repositories are non-forks with allowed licenses, GitHub creation dates
after 2024-01-01, and no reachable root commit before that cutoff. The selected
files carry source URLs, exact content hashes, and first-history dates.

The source-weighted build produced 350 candidates across all ten training
languages. `check_holdout_overlap.py` then scanned up to 32 exact 64-token anchors
per document against all 5,008,000,739 final training tokens. Seven documents had
at least one exact hit and were excluded wholesale, leaving 343 clean documents.
The registered 200-example evaluation draw covers all ten languages and 24
repositories, with every shuffled suffix matched to the same language and a
different repository.

`config/eval_holdout_receipt.json` records the candidate and clean SHA-256 values,
tokenizer and training-index hashes, scan parameters, language counts, and all
seven exclusions. A clean anchor scan is not proof that no shorter fragment
overlaps training; it is an auditable conservative check at the recorded
64-token granularity.