| # TinyStories demo slice (~700 KB train + 70 KB valid) |
|
|
| This is a small slice of the **TinyStories** dataset by Eldan & Li (2023): |
| - Source: https://huggingface.co/datasets/roneneldan/TinyStories |
| - Original file: `TinyStoriesV2-GPT4-valid.txt` (~22 MB) |
| - This slice: the first ~700 KB of stories from that file, packed as raw uint8 bytes |
| - License: the upstream dataset is CC-BY-4.0; this redistribution preserves that license |
|
|
| ## Files |
|
|
| | File | Size | Purpose | |
| |---|---|---| |
| | `train.bin` | ~700 KB | training shard (uint8 byte sequence) | |
| | `valid.bin` | ~70 KB | held-out validation shard | |
|
|
| ## Why this slice and not the full thing |
|
|
| The full TinyStories train file is ~2 GB. We didn't want every kit user to download |
| 2 GB just to do their first smoke-training run. 700 KB is enough to: |
|
|
| - Run 50–500 training steps in a few minutes on CPU and see loss fall |
| - Verify your install end-to-end |
| - Get a feel for how the trainer behaves before committing to a real run |
|
|
| For a real ~10M-param training run you want millions of bytes minimum; download |
| the full dataset from the source URL above and point `--data-dir` at it. |
|
|
| ## Format |
|
|
| Files are flat sequences of `uint8` bytes — no headers, no separators between |
| stories beyond the natural `<|endoftext|>` strings inside the text. The trainer |
| memmaps these and samples random windows of `seq_len` bytes. Each byte IS a |
| token (vocabulary size = 256). |
|
|
