xuan/data/README.md

# FineWeb-Edu 10B / GPT-2 tokenized `.bin` shards (LoopResidual)

Pre-tokenized pretraining data for the LoopResidual looped-Transformer experiments.
Drop-in for `LoopResidual/train.py` (`--train_files`, `--val_files`).

## Contents

| path | shards | tokens | bytes |
|---|---|---|---|
| `fineweb_edu_train/train_0000{00..20}.bin` | 21 | **10,400,062,869** (10.400B) | ~20 GB |
| `fineweb_edu_val/val_000000.bin` | 1 | **100,002,688** (0.100B) | 191 MB |

Training budget this is sized for: exactly **10B tokens**
(19,073 steps x 524,288 tokens/step = 9,999,745,024). The train pool carries a ~4%
margin on purpose — the loader cycles files with `itertools.cycle`, so a pool of
exactly 10B would start re-reading already-trained data in the final steps, and each
shard also discards a tail fragment smaller than one batch span (<16,384 tokens).
The margin guarantees all 10B training tokens are distinct.

## Provenance

- Source: [`HuggingFaceFW/fineweb-edu`](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu),
  subset **`sample/100BT`**, 24 parquet shards (51.65 GB) downloaded 2026-07-30.
- Taken **as-is, with no score re-bucketing**. That subset is already the score>=3
  educational filter; measured `int_score` distribution over 40k sampled docs:
  **3 -> 85.8%, 4 -> 14.1%, 5 -> 0.09%**. (Data quality is a nuisance variable held
  fixed here, not the thing under study, so it is not re-partitioned.)
- Tokenizer: **tiktoken `gpt2`** (vocab 50257), one EOT (50256) appended per document.
  Measured EOT rate 0.089-0.105% per shard, i.e. ~1000 tokens/doc.
- 10,087,135 train documents / 97,741 val documents.

## Validation split is shard-disjoint

Validation is **2 whole held-out parquet shards** (`002_00002.parquet`,
`002_00003.parquet`) that never enter training — *not* a prefix of each training
file. Verified at document level: **0 of 16,025,000 train doc IDs appear among the
1,453,000 val doc IDs**.

This matters because the obvious alternative (the tokenizer's built-in
`--val_tokens`, which takes the first N tokens of every file and trains on the
remainder of the *same* files) leaves train and val sharing shards, so
near-duplicate documents can straddle the split. FineWeb's MinHash dedup is
per-dump, so cross-dump near-duplicates are not fully excluded by any split — but
whole-shard holdout is the standard defensible choice.

Residual caveat: exact duplicates are excluded; near-duplicates with different IDs
are not provably absent.

## Binary format

`nanoGPT`-style: a 256 x `int32` header followed by a `uint16` token payload.

| header index | value |
|---|---|
| 0 | magic `20240520` |
| 1 | version `1` |
| 2 | number of tokens |

```python
import numpy as np

def load_shard(path):
    with open(path, "rb") as f:
        header = np.frombuffer(f.read(256 * 4), dtype=np.int32)
    assert header[0] == 20240520 and header[1] == 1
    return np.memmap(path, dtype=np.uint16, mode="r",
                     offset=256 * 4, shape=(int(header[2]),))
```

## Download

```bash
huggingface-cli download YoungXuan/MS-backup --repo-type dataset \
    --include "xuan/data/fineweb_edu_*/*" --local-dir ./bins
```

## Reproduce

```bash
# in LoopResidual/
HF_CACHE_ROOT=<cache> python data/download_fineweb.py \
    --repo_id HuggingFaceFW/fineweb-edu --subset "sample/100BT" \
    --output_dir <parquet_dir> --max_files 24 --batch_files 24 --continuous
bash data/prepare_fineweb_edu.sh     # shard-disjoint split + tokenize + verify
```

Integrity checks (`data/verify_bins.py`): header magic/version, declared token count
vs actual file size (catches truncated writes), all ids < 50257 (no uint16 wrap),
non-zero EOT rate, and a content fingerprint per shard to catch a collapsed
train/val split. All 22 shards passed.