evidence/documents/DATA.md

DATA: corpus provenance, filtering, and the safety review

What goes into Wisp, where it came from, what was removed, and what was checked. This is the record the model card is built from, so it is written to be read by someone deciding whether to trust the weights.

Sources

Source	Subset	Configured weight	Realized train tokens	Source terms
bigcode/starcoderdata	python	0.24	1,200,566,506	card label other; original repository terms apply
bigcode/starcoderdata	javascript	0.13	650,722,190	as above
bigcode/starcoderdata	typescript	0.12	600,701,168	as above
bigcode/starcoderdata	go	0.09	451,081,764	as above
bigcode/starcoderdata	rust	0.09	451,511,705	as above
bigcode/starcoderdata	java	0.08	400,525,337	as above
bigcode/starcoderdata	c	0.05	250,363,883	as above
bigcode/starcoderdata	shell	0.03	150,445,965	as above
bigcode/starcoderdata	sql	0.02	100,765,073	as above
bigcode/starcoderdata	markdown	0.07	351,026,251	as above
HuggingFaceFW/fineweb-edu	sample-10BT	0.08	400,290,897	ODC-By 1.0 and Common Crawl terms

The final build log survived and is now preserved byte-for-byte as evidence/run1_corpus_build.log. Its mtime matches the final shard index, and the source totals sum exactly to the index's 5,008,000,739 train tokens. The realized train split is 92.006972 percent StarCoderData, including 84.997663 percent implementation-language code and 7.009309 percent Markdown, plus 7.993028 percent FineWeb-Edu. The web share exists to keep natural-language comments, docstrings, and identifiers from drifting, and was cut from 15 percent to 8 to buy back budget for code when the niche was fixed as FIM-first.

This recovered log closes the aggregate train-token-count gap. It does not recover accepted row identities, rejection counts, source metadata, per-source validation overshoot, or row-level licensing obligations.

starcoderdata is gated. Access is granted per account by accepting the terms on the dataset page. Gated access is not a permissive-license attestation. Its card declares license: other; the original repository licenses and attribution clauses remain applicable, together with the dataset's update and removal terms. FineWeb-Edu is distributed under ODC-By 1.0 and remains subject to Common Crawl terms.

All ten code subsets above were verified to stream after access. Run 1 retained tokenized text only. It did not preserve StarCoderData row IDs, repository names or paths, FineWeb IDs or URLs, a per-row license mapping, or an attribution index. The resulting shards cannot support a local per-file licensing or attribution audit. Apache 2.0 applies to the Wisp artifact; it does not override source-data terms or the licenses of code the model may generate.

The malware scan finding, resolved

Hugging Face flagged four files in bigcode/starcoderdata as unsafe. They were checked individually rather than taken on the file name, because three of the four names are ambiguous across subsets.

Flagged file	Subset it is actually in	Used by Wisp
train-00008-of-00055.parquet	git-commits-cleaned	no
train-00019-of-00055.parquet	git-commits-cleaned	no
train-00027-of-00055.parquet	git-commits-cleaned	no
train-00043-of-00059.parquet	github-issues-filtered-structured	no

None of the four are in a subset this project reads. The fourth needed resolving: a file of that exact name exists in both github-issues-filtered-structured and in python, which Wisp does use. Querying the per-file scan status settles it. The github-issues copy is unsafe; the python copy is queued, which means unscanned, not flagged.

Worth stating plainly, since "queued" is most of the repository: the majority of files carry no scan verdict either way. Four are affirmatively flagged, none of them ours, and the rest are largely unscanned rather than certified clean.

Why this is a low risk here regardless

• Parquet has no execution vector on read. It is columnar data parsed by pyarrow. This is not the pickle situation, where deserialising is execution.
• A flag on a GitHub-derived corpus is expected. Commit messages and issue threads contain malware source, exploit proofs of concept, and EICAR test strings, because those things live on GitHub. The scanner is matching content, and it is right to.
• Nothing in this pipeline executes corpus content. The path is stream, then tokenize, then write uint16 shards. Where enabled, ast.parse and json.loads parse without evaluating.
• The real exposure is to the model, not the machine. Training on malware source teaches malware patterns. That argues for excluding such data on model safety grounds, and it is excluded here, since the affected subsets are not in the mixture.

The quality gate

Run 1 applied the structural checks below to every source. Verifier-first was borrowed from agent-toolkit's verify/verifiers.py in stance rather than in code: executing scraped code to decide whether to train on it would be the security incident.

Rejected:

• Longest line over 1000 characters. Minified bundles, generated files, data on one line.
• Mean line length over 120 characters. Past anything hand written.
• Alphanumeric-plus-whitespace fraction under 0.55 in the first 20KB. Base64, hex dumps, embedded binary assets.
• Anything under 64 characters.

An earlier draft also claimed that run 1 rejected Python that failed ast.parse and JSON that failed json.loads. That was wrong. The Hub loader requested the column path, while StarCoderData exposes max_stars_repo_path. Its rows therefore reached the structural checks without a file extension, and neither extension parser activated during the run 1 Hub build.

The 92.7 percent retention figure came from a local forced-Python pilot, not from the final streamed corpus. It remains evidence that the structural gate is not vacuous, but it is not a final-corpus retention measurement.

After run 1 launched, the loader was repaired to map the pinned StarCoderData schema, fail closed if its path field disappears, and remove the leading <reponame> metadata line only for syntax parsing. The original text remains the payload. Synthetic mutation checks prove invalid Python is rejected. This repair applies to future source streaming only; it does not change the active process or existing shards. E2 must not use the repaired stream to select a different document population. Its no-FIM data is instead produced by the registered deterministic normalization of run 1's token stream.

The rationale is capacity, not tidiness. At 108M parameters, tokens spent modelling a webpack bundle are tokens not spent on code.

The tokenizer, benchmarked rather than assumed

32,768 byte-level BPE, digits split individually, whitespace runs allowed to merge, trained on 400,000 documents drawn round-robin across the eleven configured source entries. The tokenizer sampler did not apply the configured token weights. Each language entry therefore contributed about one eleventh of the document sample, while Python contributed 24 percent of the later shard budget. This is a reproducibility limitation and a future tokenizer experiment, not a reason to alter the tokenizer under an active model.

Measured on 366KB of fresh starcoderdata across Python, JavaScript, Go, and Rust, none of it in the tokenizer's training sample:

tokenizer	vocab	chars/token
wisp code32k	32,768	3.52
bigcode/starcoder2-15b	49,152	3.54
Qwen/Qwen2.5-Coder	151,643	4.06

Compression matters because it sets how much code 5B tokens actually buys. A tokenizer 15 percent worse means 15 percent less code for the same compute.

The 32K tokenizer therefore matches StarCoder2's compression while carrying a third less vocabulary. Qwen's 15.6 percent advantage is not practical at this scale: a 151,643-token embedding at d_model 768 alone costs 116M parameters, larger than this entire model.

That comparison did not answer the more relevant challenge, 16K versus 32K. benchmark_tokenizer_vocab.py now does. Greedy BPE learns merges sequentially, so truncating the frozen tokenizer at the 16K merge boundary gives the exact same-training-sample 16K merge prefix. Both tokenizers were measured on all 343 documents in the clean, source-weighted publication holdout:

vocab	tokens	chars/token	tied embedding	model if only vocab changes	training head MACs/token
16,384	1,117,129	3.2123	12.58M	95.57M	37.75M
32,768	1,069,055	3.3568	25.17M	108.16M	75.50M

The 16K tokenizer needs 4.50 percent more tokens for the same text, or sees 4.30 percent fewer characters at a fixed token budget. In return it saves 12,582,912 parameters, 11.63 percent of the whole 32K model, and halves the width of each of the three output projections evaluated per training position.

So 32K was not forced, and the old wording claiming that was wrong. Run 1 remains 32K because its tokenizer, corpus, and active weights are already frozen. The evidence makes 16K a serious future-successor candidate, but compression and projection arithmetic do not establish end-to-end speed or model quality. Those require a post-run Metal throughput arm and a matched training-quality arm. Registered run 2 remains a 32K no-FIM ablation. config/tokenizer_vocab_receipt.json records the exact hashes and measurements.

Fill in the middle, and what the window geometry does to it

The configured chunk-level rate is not the rate a sampled training window sees, and the gap is large enough to matter.

After tokenization, each source document is split into chunks of at most 1024 tokens. FIM is selected independently for each chunk at prepare time. Training then samples random 2051-token windows out of the concatenated stream. A window can see a complete frame, a fragment such as a suffix with no prefix sentinel, or no frame at all. Measured while comparing the discarded whole-document build with the final chunked build, sampling 4,000 windows per shard:

build	complete frame	partial	none
whole-document FIM	43.2% / 62.1% / 69.9%	9 to 19%	20 to 38%
1024-token chunks	90.1% / 89.8% / 91.2%	5 to 7%	3 to 4%

The headline capability was being diluted by roughly a fifth before the fix.

The arithmetic, and its side effect. With chunks of 1024 and windows of 2051, a window spans about two chunks, and a complete frame appears if either chunk was transformed. At fim_rate 0.7 that is 1 - 0.3^2 = 91%, which matches the measurement. The same arithmetic says pure left-to-right windows fall to about 9%, well under the 30% that would follow from treating the configured value as a window-level rate.

That is a real change to the mixture and it was not the goal, it is a consequence. Kept deliberately, for two reasons. Every token is still predicted autoregressively, so next-token capability is not lost; what shrinks is exposure to the unframed format, not to left-to-right prediction itself. And the deployment target is a cursor inside an existing file, which is the framed case.

If a future run wants the window-level rate to match a 70 percent target, solve 1 - (1 - r)^2 = 0.7 for r, giving fim_rate about 0.45. The research design is insulated from this either way: the primary endpoint compares FIM against FIM-with-shuffled-suffix, both of which are framed, so format exposure cancels. The FIM over L2R secondary number is the one this affects.

Fill in the middle

Applied at prepare time, not train time, independently to 70 percent of tokenized chunks of at most 1024 tokens, split evenly between PSM and SPM orderings. It was not applied once per source document. The rate was raised from 0.5 when the niche was fixed, since cursor completion is the deployment target. fim_rate now lives in the config and the CLI flag overrides it.

Validation split in the final corpus

The first corpus build filled one shared validation counter from Python and was discarded. The final corpus was rebuilt after prepare_data.py changed to allocate validation tokens per source in the same proportions as training. Run 1 therefore uses a source-stratified validation shard with 20,006,112 tokens. The discarded build is not used by the trainer.

Scheduled tokens are not unique corpus coverage

Run 1 draws random 2051-token windows with replacement. Its 19,073 steps, 8-example micro batch, and 16-way accumulation schedule 2,441,344 windows and 4,999,872,512 prediction positions. The registered step-300 sampler reset repeats the first 38,400 window coordinates, leaving 2,402,944 distinct scheduled draws at most.

Under the sampler's uniform-start distribution, the interval-coverage approximation 1 - exp(-4,921,229,312 / 5,008,000,739) is 62.57 percent. Roughly 37.43 percent of corpus positions are therefore expected not to appear in a prediction window, while other positions repeat. This is expected with-replacement sampling, not shard corruption. A future run should compare a checkpointed shuffled block order without replacement so nearly the full corpus is exposed before repetition. E2 keeps run 1's sampler semantics for a matched comparison.

Corpus-build attestation

Run 1's existing index did not include shard hashes, so a full read-only attestation now binds all 52 files, 10,056,013,702 bytes, the source index, the tokenizer, and the normalization program in config/run1_shard_integrity_receipt.json. The scan found 27 detectable FIM frames split by literal EOS tokens inside their payloads. It reassembles those visible groups and restores 33 internal EOS tokens. The resulting normalized totals are 4,992,043,184 train tokens and 19,949,502 validation tokens.

This is deterministic token-stream normalization, not proof of exact original examples. An EOS inside unframed content, an EOS after the final FIM marker, and raw content that already has the visible grammar of a valid FIM frame are observationally indistinguishable from preprocessing structure after tokenization. The receipt records exact_original_units_proven: false, the visible grouping manifest, and domain-separated source, recovered, and destination hashes.

The run 2 builder never reopens a Hub source. It requires the attested run 1 files, a missing or empty real output directory, and the exact registered derivation strategy. It rejects symbolic links, unsafe or duplicate paths, mutation during reads, malformed FIM grammar, cross-shard recovery, residual FIM sentinels, and any count or hash mismatch. It writes one destination shard and one fixed-width unit-boundary sidecar per source shard. Only after independently rereading every output does it atomically publish a schema-3 index with per-shard hashes, sidecar hashes, and split-level normalization evidence.

Both normalized splits fall just below the old requested 5B and 20M corpus floors because three generated sentinels are removed from every visible FIM frame. The fixed run 2 schedule still samples 4,999,872,512 token positions. This changes neither registered E2 endpoint nor the model geometry, but it supersedes the earlier claim that exact raw examples could be recovered.

This was added before any run2 corpus existed. The amendment is recorded in config/eval_format_ablation_receipt.json; it changes build provenance and failure behavior, not either registered endpoint. Run1's checkpoint did not record source-file hashes or the sampler state. Its step-300 process recovery reset the legacy shard sampler and therefore repeats 78,643,200 scheduled token positions, 1.57 percent of the training budget. That observed reset is now registered in both E2 arms as sampler_reset_steps: [300]. New checkpoints store the exact sampler RNG state, and legacy checkpoints reconstruct the registered position. This matches reset timing, not raw examples across different arm shards. The model and optimizer are unchanged, but exact run1 source-state equivalence cannot be proven.

Run 1 also did not pin Hub revisions at build time or save an ordered raw-source manifest. Revisions captured from the local cache afterward are useful repository-level evidence, but they cannot reconstruct the accepted rows, repository paths, rejection counts, validation overshoot by source, or row-level obligations. Aggregate realized train-token totals survive in the preserved final build log. config/training_data_receipt.json binds these limitations, the audited source cards and schemas, the prospective parser repair, and the deterministic normalization limitation.

Held out data

The publication evaluation corpus is not the old local-file fallback. It is built from config/eval_repos.json, which pins 29 repository-language strata at exact commits. Repositories are non-forks with allowed licenses, GitHub creation dates after 2024-01-01, and no reachable root commit before that cutoff. The selected files carry source URLs, exact content hashes, and first-history dates.

The source-weighted build produced 350 candidates across all ten training languages. check_holdout_overlap.py then scanned up to 32 exact 64-token anchors per document against all 5,008,000,739 final training tokens. Seven documents had at least one exact hit and were excluded wholesale, leaving 343 clean documents. The registered 200-example evaluation draw covers all ten languages and 24 repositories, with every shuffled suffix matched to the same language and a different repository.

config/eval_holdout_receipt.json records the candidate and clean SHA-256 values, tokenizer and training-index hashes, scan parameters, language counts, and all seven exclusions. A clean anchor scan is not proof that no shorter fragment overlaps training; it is an auditable conservative check at the recorded 64-token granularity.