| --- |
| license: mit |
| library_name: pytorch |
| language: |
| - en |
| tags: |
| - nanochat |
| - language-model |
| - pre-1930 |
| - historical |
| - vintage |
| - research |
| --- |
| |
| # BART Experiments |
|
|
| The complete training archive behind [BART](https://huggingface.co/jbduran/bart) — 39 runs, |
| their checkpoints, tokenizers, and evaluations, including every dead end. |
|
|
| 📝 [Read the write-up](https://www.unboundedlab.com/blog/bart) · 🌐 [Unbounded Labs](https://unboundedlab.com) |
|
|
| If you want the model itself, use [bart](https://huggingface.co/jbduran/bart) or |
| [bart-sft](https://huggingface.co/jbduran/bart-sft). This repository is for reproducing or |
| inspecting how they were reached. |
|
|
| ## Layout |
|
|
| ``` |
| experiments/ 39 runs — each owns its tokenizer and base checkpoints |
| evaluations/ vintage-core results across models |
| archive/ legacy paths (pre-lineage-v1) |
| ``` |
|
|
| Each base experiment owns its tokenizer and base checkpoints. SFT runs nest under their exact base |
| parent (`experiments/<base>/sft/<sft-run>/`), and post-training runs nest under their exact SFT |
| parent. |
|
|
| ## Reading experiment names |
|
|
| | Fragment | Meaning | |
| |---|---| |
| | `d12`, `d24`, `d32` | model depth | |
| | `r11`–`r30` | target parameter-to-data ratio | |
| | `ctx4096`, `ctx8192` | max sequence length | |
| | `sssl` | window pattern | |
| | `fulltok`, `randtok` | tokenizer variant | |
|
|
| The prefix tells you the corpus generation: |
|
|
| | Prefix | Corpus | |
| |---|---| |
| | `think-` | [bart-dataset-v1](https://huggingface.co/datasets/jbduran/bart-dataset-v1) | |
| | `thinkcleaned-` | [v2](https://huggingface.co/datasets/jbduran/bart-dataset-v2) | |
| | `clean1930s-` | [v3](https://huggingface.co/datasets/jbduran/bart-dataset-v3) | |
| | `Think.Unbounded-` | v3 plus [midtraining](https://huggingface.co/datasets/zachnorton03/bart-midtrain) | |
|
|
| ## The runs that matter |
|
|
| `Think.Unbounded-d32` is the first d32 run, trained on midtrain mixtures whose ratios were computed |
| by **document count**. Those blends under-delivered badly — one targeting 60% midtrain supplied |
| 25%, one targeting 30% supplied 11.5%. |
|
|
| `Think.Unbounded-d32-v2mix-cont` is the repair: it branches from that run at step 5500, carries the |
| optimizer state, and continues on corrected **token-based** mixtures. It is the model shipped as |
| [bart](https://huggingface.co/jbduran/bart). The two configs side by side are the clearest record |
| of that bug and its fix. |
|
|
| Its `sft/` directory holds six fine-tuning variants. The one shipped as |
| [bart-sft](https://huggingface.co/jbduran/bart-sft) is |
| `pre1930-curriculum-c3-robust-v2`; the others — `karpathy-modern-sft-v1`, |
| `nanochat-default-datamatch-v1`, `c3-robust`, and two `c3-robust-v3` configs — are kept here. |
|
|
| `clean1930s-d24-r12-ctx4096-sssl-fulltok-v1` is the d24 / 1.38B run; its tokenizer is the one every |
| midtrain mixture was built against. |
|
|
| > **Note.** Each `config.json` records repo and dataset names as they were at training time |
| > (`jbduran/think.nano`, `think-dataset-clean-1930s`, `think-midtrain`). Repo renames still resolve |
| > via Hub redirects, but the midtrain mixture subfolders they reference (`mixed/v2/ratio_21/data`) |
| > moved to `mixtures/v2-by-tokens/ratio_21/data` when that dataset was restructured. Configs are |
| > kept as historical records rather than rewritten. |
|
|
| ## Related |
|
|
| [bart](https://huggingface.co/jbduran/bart) · |
| [bart-sft](https://huggingface.co/jbduran/bart-sft) · |
| [bart-dataset-v3](https://huggingface.co/datasets/jbduran/bart-dataset-v3) · |
| [bart-midtrain](https://huggingface.co/datasets/zachnorton03/bart-midtrain) · |
| [bart-dataset-scripts](https://github.com/OwenVoorhees/bart-dataset-scripts) · |
| [bart-midtrain-scripts](https://github.com/OwenVoorhees/bart-midtrain-scripts) |
|
|
| ## License |
|
|
| MIT |
|
|
| --- |
|
|
| Built by [Unbounded Labs](https://unboundedlab.com). |
|
|
