| # Reproducing the Dropout Decay Experiments |
|
|
| This repository is intended to be runnable without checking out nanochat. The |
| implementation is derived from nanochat and retains its MIT attribution, but |
| runtime commands use this package, local cached data, and a local MPS-capable |
| Python environment. |
|
|
| ## Requirements |
|
|
| - macOS with Apple Silicon MPS available. |
| - Python 3.10-3.12 recommended for PyTorch MPS wheels. |
| - A project-local virtual environment at `.venv`. |
| - MPS-capable PyTorch. CPU and CUDA runs are intentionally refused by the |
| runner. |
|
|
| Create the environment: |
|
|
| ```bash |
| python3.11 -m venv .venv |
| .venv/bin/python -m pip install --upgrade pip |
| .venv/bin/python -m pip install -e . |
| ``` |
|
|
| Verify MPS: |
|
|
| ```bash |
| .venv/bin/python - <<'PY' |
| import torch |
| print(torch.__version__) |
| print(torch.backends.mps.is_built(), torch.backends.mps.is_available()) |
| PY |
| ``` |
|
|
| Both booleans must be `True`. |
|
|
| ## Data |
|
|
| The runner supports two modes: |
|
|
| - `--use-cached-data --cache-dir .cache/dropout_decay` |
| - `--corpus` / `--corpus-glob` to build a cache from raw text or parquet. |
|
|
| The repo includes the curated source corpora needed by the current regimes: |
|
|
| ```text |
| data/openwebtext10k/base_data_climbmix/shard_*.parquet |
| data/openwebtext10k/openwebtext10k.txt |
| data/tinystories/train-00000-of-00004.parquet |
| data/wikitext103_raw/train-00001-of-00002.parquet |
| ``` |
|
|
| The experiments in the current report used these repo-local caches: |
|
|
| ```text |
| .cache/dropout_decay/tokenizer-v4096.json |
| .cache/dropout_decay/tokens-v4096-uint16.npy |
| .cache/dropout_decay_tinystories/tokenizer-v4096.json |
| .cache/dropout_decay_tinystories/tokens-v4096-uint16.npy |
| .cache/dropout_decay_wikitext103/tokenizer-v4096.json |
| .cache/dropout_decay_wikitext103/tokens-v4096-uint16.npy |
| ``` |
|
|
| These binary artifacts are intended to be published with the repository through |
| the configured large-file storage rules. The local OpenWebText-style cached |
| split used by the completed OpenWebText10K runs contains: |
|
|
| ```text |
| train tokens: 5,000,970 |
| validation tokens: 500,000 |
| vocab size: 4,096 |
| ``` |
|
|
| The public WikiText-103 holdout can be rebuilt from source: |
|
|
| ```bash |
| .venv/bin/python scripts/prepare_wikitext103.py |
| ``` |
|
|
| The pinned parquet source is verified as: |
|
|
| ```text |
| bytes: 156,700,942 |
| sha256: 75aa65dee9de2a7c10ba1808efd2408c3f4eb008104c3ccac47f8ed19300ebdd |
| ``` |
|
|
| The public TinyStories holdout can also be rebuilt from source: |
|
|
| ```bash |
| .venv/bin/python scripts/prepare_tinystories.py |
| ``` |
|
|
| The pinned parquet source is verified as: |
|
|
| ```text |
| bytes: 248,731,111 |
| sha256: 77cf780cebe52b6e83e3a2ac84bc56d8059363113e41d17a023f1d8b2ed0fc0b |
| ``` |
|
|
| ## Smoke Test |
|
|
| This verifies cached-data loading without running a Torch experiment: |
|
|
| ```bash |
| PYTHONPATH=src .venv/bin/python - <<'PY' |
| from pathlib import Path |
| from dropout_decay.data import load_cached_splits |
| |
| tok, splits = load_cached_splits( |
| cache_dir=Path(".cache/dropout_decay"), |
| vocab_size=4096, |
| max_required_train_tokens=4_000_000, |
| val_tokens=500_000, |
| allow_short_corpus=False, |
| ) |
| print(tok.vocab_size) |
| print(len(splits.train), len(splits.val)) |
| PY |
| ``` |
|
|
| Expected: |
|
|
| ```text |
| 4096 |
| 5000970 500000 |
| ``` |
|
|
| ## Headline Formula |
|
|
| The tested formula is: |
|
|
| ```text |
| p = clamp(0.02, 0.65, |
| 0.154 * log10(params / unique_tokens) |
| + 0.249 * log10(cumulative_sampled_tokens / unique_tokens) |
| - 0.210) |
| ``` |
|
|
| For the standard protocol: |
|
|
| - stream prefixes: `250000 500000 1000000 2000000 4000000` |
| - stage steps: `1000` |
| - batch size: `16` |
| - block size: `128` |
| - cumulative sampled tokens after stage `i`: `i * 1000 * 16 * 128` |
|
|
| ## Reproduce Model-Size Validation |
|
|
| Example L12 command: |
|
|
| ```bash |
| PYTHONPATH=src .venv/bin/python scripts/run_experiments.py \ |
| --mode locked_stream \ |
| --use-cached-data \ |
| --cache-dir .cache/dropout_decay \ |
| --output-dir runs/reproduce_l12_formula \ |
| --models L12_H8_D320=12x8x320 \ |
| --seeds 1 2 3 \ |
| --stream-token-caps 250000 500000 1000000 2000000 4000000 \ |
| --dropout-rates 0.09 0.14 0.18 0.20 0.26 0.30 \ |
| --anchor-decays pressure_formula_l12:250000=0.300,500000=0.260,1000000=0.180,2000000=0.090,4000000=0.020 \ |
| --stage-steps 1000 \ |
| --batch-size 16 \ |
| --block-size 128 \ |
| --eval-batches 64 \ |
| --train-eval-batches 32 \ |
| --trace-eval-batches 8 \ |
| --log-every 500 \ |
| --vocab-size 4096 \ |
| --val-tokens 500000 \ |
| --lr 0.0003 \ |
| --weight-decay 0.1 \ |
| --grad-clip 1.0 |
| ``` |
|
|
| Completed reference result: |
|
|
| ```text |
| pressure formula final validation: 4.4812 +/- 0.0062 |
| best static final validation: 4.5183 |
| ``` |
|
|
| ## Reproduce Architecture-Shape Holdout |
|
|
| Deep/narrow holdout: |
|
|
| ```bash |
| PYTHONPATH=src .venv/bin/python scripts/run_experiments.py \ |
| --mode locked_stream \ |
| --use-cached-data \ |
| --cache-dir .cache/dropout_decay \ |
| --output-dir runs/reproduce_arch_deep_narrow \ |
| --models deep_narrow_L18_H8_D256=18x8x256 \ |
| --seeds 1 2 3 \ |
| --stream-token-caps 250000 500000 1000000 2000000 4000000 \ |
| --dropout-rates 0.02 0.08 0.14 0.18 0.20 0.26 0.30 \ |
| --anchor-decays formula_deep_narrow_l18_h8:250000=0.297,500000=0.250,1000000=0.173,2000000=0.083,4000000=0.020 \ |
| --stage-steps 1000 \ |
| --batch-size 16 \ |
| --block-size 128 \ |
| --eval-batches 64 \ |
| --train-eval-batches 32 \ |
| --trace-eval-batches 8 \ |
| --log-every 500 \ |
| --vocab-size 4096 \ |
| --val-tokens 500000 \ |
| --lr 0.0003 \ |
| --weight-decay 0.1 \ |
| --grad-clip 1.0 |
| ``` |
|
|
| Completed reference result: |
|
|
| ```text |
| formula final validation: 4.5286 +/- 0.0118 |
| best static final validation: 4.5564 +/- 0.0127 |
| ``` |
|
|
| ## Reproduce Width-Heavy Holdout |
|
|
| The width-heavy architecture holdout is the paired complement to the deep/narrow |
| holdout above: |
|
|
| ```bash |
| PYTHONPATH=src .venv/bin/python scripts/run_experiments.py \ |
| --mode locked_stream \ |
| --use-cached-data \ |
| --cache-dir .cache/dropout_decay \ |
| --output-dir runs/architecture_shape_holdout_wide_h8 \ |
| --models wide_L8_H8_D384=8x8x384 \ |
| --seeds 1 2 3 \ |
| --stream-token-caps 250000 500000 1000000 2000000 4000000 \ |
| --dropout-rates 0.02 0.08 0.14 0.18 0.20 0.26 0.30 \ |
| --anchor-decays formula_wide_l8_h8:250000=0.301,500000=0.254,1000000=0.177,2000000=0.087,4000000=0.020 \ |
| --stage-steps 1000 \ |
| --batch-size 16 \ |
| --block-size 128 \ |
| --eval-batches 64 \ |
| --train-eval-batches 32 \ |
| --trace-eval-batches 8 \ |
| --log-every 500 \ |
| --vocab-size 4096 \ |
| --val-tokens 500000 \ |
| --lr 0.0003 \ |
| --weight-decay 0.1 \ |
| --grad-clip 1.0 |
| ``` |
|
|
| Completed reference result: |
|
|
| ```text |
| formula final validation: 4.4658 +/- 0.0065 |
| best static final validation: 4.4946 +/- 0.0087 |
| best mean trajectory: static 0.18, 4.9064 vs formula 4.9073 |
| ``` |
|
|
| ## Reproduce WikiText-103 Corpus Holdout |
|
|
| Prepare the public corpus first: |
|
|
| ```bash |
| .venv/bin/python scripts/prepare_wikitext103.py |
| ``` |
|
|
| Then run the frozen L12 formula against a broad static grid: |
|
|
| ```bash |
| PYTHONPATH=src .venv/bin/python scripts/run_experiments.py \ |
| --mode locked_stream \ |
| --corpus data/wikitext103_raw/train-00001-of-00002.parquet \ |
| --cache-dir .cache/dropout_decay_wikitext103 \ |
| --output-dir runs/corpus_holdout_wikitext103_l12 \ |
| --models L12_H8_D320=12x8x320 \ |
| --seeds 1 2 3 \ |
| --stream-token-caps 250000 500000 1000000 2000000 4000000 \ |
| --dropout-rates 0.00 0.02 0.06 0.08 0.10 0.14 0.18 0.20 0.26 0.30 \ |
| --anchor-decays formula_l12_wikitext103:250000=0.300,500000=0.260,1000000=0.180,2000000=0.090,4000000=0.020 \ |
| --stage-steps 1000 \ |
| --batch-size 16 \ |
| --block-size 128 \ |
| --eval-batches 64 \ |
| --train-eval-batches 32 \ |
| --trace-eval-batches 8 \ |
| --log-every 500 \ |
| --vocab-size 4096 \ |
| --val-tokens 500000 \ |
| --lr 0.0003 \ |
| --weight-decay 0.1 \ |
| --grad-clip 1.0 |
| ``` |
|
|
| Completed reference result: |
|
|
| ```text |
| formula final validation: 4.0836 +/- 0.0258 |
| best static final validation: 4.1081 +/- 0.0258 |
| best mean trajectory: formula, 4.5728 vs static 0.18, 4.5759 |
| ``` |
|
|
| ## Reproduce TinyStories Corpus Holdout |
|
|
| Prepare the public corpus first: |
|
|
| ```bash |
| .venv/bin/python scripts/prepare_tinystories.py |
| ``` |
|
|
| Then run the frozen L12 formula against a broad static grid: |
|
|
| ```bash |
| PYTHONPATH=src .venv/bin/python scripts/run_experiments.py \ |
| --mode locked_stream \ |
| --corpus data/tinystories/train-00000-of-00004.parquet \ |
| --cache-dir .cache/dropout_decay_tinystories \ |
| --output-dir runs/corpus_holdout_tinystories_l12 \ |
| --models L12_H8_D320=12x8x320 \ |
| --seeds 1 2 3 \ |
| --stream-token-caps 250000 500000 1000000 2000000 4000000 \ |
| --dropout-rates 0.00 0.02 0.06 0.08 0.10 0.14 0.18 0.20 0.26 0.30 0.40 0.50 \ |
| --anchor-decays formula_l12_tinystories:250000=0.300,500000=0.260,1000000=0.180,2000000=0.090,4000000=0.020 \ |
| --stage-steps 1000 \ |
| --batch-size 16 \ |
| --block-size 128 \ |
| --eval-batches 64 \ |
| --train-eval-batches 32 \ |
| --trace-eval-batches 8 \ |
| --log-every 500 \ |
| --vocab-size 4096 \ |
| --val-tokens 500000 \ |
| --lr 0.0003 \ |
| --weight-decay 0.1 \ |
| --grad-clip 1.0 |
| ``` |
|
|
| Completed reference result: |
|
|
| ```text |
| frozen formula final validation: 2.7016 +/- 0.0212 |
| best static final validation: 2.7009 +/- 0.0216 |
| paired frozen-formula wins: 1/3 |
| ``` |
|
|
| The TinyStories holdout exposed a useful boundary condition: the frozen |
| pressure formula starts too high for this easier corpus. The focused low-decay |
| follow-up can be reproduced with: |
|
|
| ```bash |
| PYTHONPATH=src .venv/bin/python scripts/run_experiments.py \ |
| --mode locked_stream \ |
| --use-cached-data \ |
| --cache-dir .cache/dropout_decay_tinystories \ |
| --output-dir runs/corpus_holdout_tinystories_l12_lowdecay \ |
| --models L12_H8_D320=12x8x320 \ |
| --seeds 1 2 3 \ |
| --stream-token-caps 250000 500000 1000000 2000000 4000000 \ |
| --dropout-rates \ |
| --anchor-decays \ |
| tinystories_low_decay_014_002:250000=0.140,500000=0.140,1000000=0.100,2000000=0.060,4000000=0.020 \ |
| tinystories_low_decay_014_006:250000=0.140,500000=0.140,1000000=0.100,2000000=0.080,4000000=0.060 \ |
| tinystories_monotone_oracle:250000=0.140,500000=0.140,1000000=0.140,2000000=0.080,4000000=0.080 \ |
| tinystories_low_decay_010_006:250000=0.100,500000=0.100,1000000=0.080,2000000=0.080,4000000=0.060 \ |
| --stage-steps 1000 \ |
| --batch-size 16 \ |
| --block-size 128 \ |
| --eval-batches 64 \ |
| --train-eval-batches 32 \ |
| --trace-eval-batches 8 \ |
| --log-every 500 \ |
| --vocab-size 4096 \ |
| --val-tokens 500000 \ |
| --lr 0.0003 \ |
| --weight-decay 0.1 \ |
| --grad-clip 1.0 |
| ``` |
|
|
| Completed reference result: |
|
|
| ```text |
| best low-decay path: 0.14 -> 0.14 -> 0.10 -> 0.06 -> 0.02 |
| best low-decay final validation: 2.6859 +/- 0.0286 |
| best static final validation: 2.7009 +/- 0.0216 |
| paired low-decay wins: 3/3 |
| ``` |
|
|
| ## Notes for Publication |
|
|
| - Do not claim the formula is universal. |
| - The supported claim is final-validation improvement under this |
| expanding-prefix protocol. |
| - PDFs are generated artifacts and are ignored by Git. |
| - The exact cached token file should be published through an appropriate binary |
| artifact mechanism once dataset provenance is finalized. |
|
|
