guide/02-data.md

02 — Data

The mix is the part you cannot re-run cheaply after training starts. Build it once, publish it reproducibly, and prove it is clean before a single GPU second is spent on it.

1. Design for abilities, not for items

_{Sources: Phase 1 of the constitution · D-009}

Decide what the finished model must do — commonsense and physical inference, science QA, breadth of factual knowledge, factuality, arithmetic, multi-step reasoning — then reason from what kinds of text build those abilities to proportions. Write a rationale per source, citing the corpus property you relied on.

The line that keeps this honest: proportions justified by corpus properties are legitimate; proportions justified by looking at evaluation questions are contamination. If a source's justification reads like a benchmark name, that's the tell.

• Prefer few large, well-attributed sources over many small scraped ones. Every source is a maintenance cost and a contamination surface.
• Check what a source's maintainers already exclude. Some curated math sets decontaminate against test splits only — which silently leaves validation in your train data if you don't look.
• Question every modality you are told to include. This project's constitution said "general English plus math and code", and code was dropped (D-009): the corpus exposed whole-repository rows behind a nested struct, honest per-file token accounting was a subproject, and at 100M parameters with GSM8K being word problems, those tokens bought more elsewhere. An instruction in a plan is not a measurement.

2. Size the mix with headroom, then let dedup eat it

Target N tokens in the final mix; assemble 1.1–1.25× that, because dedup, exclusion masks and the held-out split all subtract and you cannot top up after publication. Reserve validation (~2 %) before sharding, with the same tokenizer, and keep it out of every training shard.

3. Tokenise once, pack, shard for exact positional resumption

Emit raw uint16/uint32 token arrays per shard plus a manifest.

Decision	What this project chose	Why, and the alternative
Document boundaries	Contiguous packing, no separator token, no block-diagonal mask (D-008)	The seam cost was measured at ~0.16 % of windows before it was accepted. A separator plus attention mask is the textbook answer and costs throughput; the wrong move is to pick either without measuring the seam rate.
Sample length	Fixed multiple of seq_len; tail dropped	Record the dropped count — it belongs in the reproducibility proof.
Shard size	Small enough that a job reads a few at a time	Select shards by step range and stream. A training job must never materialise the whole mix on an instance disk.

4. Exact positional resumption is a data-structure problem, not a state problem

sample k must be a pure function of (dataset fingerprint, shuffle_seed, k).

Layout	Verdict
Shuffle the list of samples with a recorded seed; persist samples_consumed	Works — recompute everything else
Shard-major, offset-major traversal; cursor = (shard, offset)	Works
Anything relying on Python iteration order, an open file handle, unsaved DataLoader worker state, or the set of files on disk at the time	Fails — and fails quietly, which is worse

Persist in the checkpoint, all of them: seq_len, samples_consumed, shuffle_seed, shuffle_perm_sha, dataset_files_sha, step, tokens_consumed. The hashes let a resume detect that it is resuming into a different dataset than it was trained on — raise, never silently re-shuffle.

Prove it three ways: recorded index/hash, token-offset check, and loss continuity across the break within noise. Then resume twice in sequence — one interruption is a fluke. Position must never drift backwards or re-read consumed data.

5. The contamination audit is mechanical, and its scope is narrow

Audit against validation/dev material only. test splits stay untouched until the benchmark phase. This is a hard boundary, not a conservatism: reading test items — even to count overlap against them — is the one thing that can quietly invalidate the entire result set. Compute overlap without ever printing an item.

Method: normalise (lowercase, strip punctuation/whitespace), count exact long-n-gram matches (13-grams are the usual standard) between mix documents and each task's non-test reference split, report per-source with an overlap_total. Drop or clean any source that fails; re-run to zero.

Three traps, all of which produce a passing audit that means nothing:

1. Zero denominator. If extraction pulled 0 reference strings, overlap_total 0 is true and worthless. Assert a minimum row count per task, and print denominators beside numerators.
2. Wrong split. Verify per task which split the harness actually scores. In current lm-eval several standard academic tasks are scored on validation, and some datasets expose no test at all. Auditing test because that's the intuitive name measures the wrong bytes and breaks the rule above.
3. The audit's own artefacts leak benchmark text. Exception strings, sample matches and debug dumps get serialised into audit.json and published with the mix, quoting real cells. Publish counts, and scrub every string that could carry an item — this happened here before it was caught.

Keep the re-runnable script and the row-count-bearing report in the repo. "We decontaminated" is a claim; a script another party can run against the published mix is evidence.

6. Publish so a stranger can rebuild it

• manifest.json: per-source token counts, train/val totals, shard counts and sizes, tokenizer identity (hub repo, revision, vocab size, tokenizer file hashes), the packing rule, the seed.
• The exclusion masks and dedup keys, not just the result — otherwise the build is unreviewable.
• A content fingerprint over the ordered file set (hash of concatenated per-file sha256s) plus the per-file sha256s. Training asserts it matches; that is what makes dataset_files_sha above meaningful.
• Self-consistency: totals equal the sum of parts, shard counts equal the file listing, no empty shards. Write the checker; don't eyeball a table. A manifest whose numbers don't add up is worse than none, because it reads like a guarantee.

Then verify the published artefact: re-download the manifest anonymously, re-hash shards, and feed the downloaded bytes to the same reader the trainer uses. The dataset is done when an instance with an empty cache can start from the repo id alone and read the right tokens.