| --- |
| license: mit |
| library_name: pytorch |
| language: |
| - en |
| tags: |
| - nanochat |
| - language-model |
| - pre-1930 |
| - historical |
| - vintage |
| --- |
| |
| # BART |
|
|
| A 2.8B-parameter language model whose knowledge ends in **1930**. BART is trained from scratch on |
| pre-1930 public-domain books — no modern text anywhere in the corpus — to study what a model learns |
| when its entire world is historical. |
|
|
| 📝 [Read the write-up](https://www.unboundedlab.com/blog/bart) · 🌐 [Unbounded Labs](https://unboundedlab.com) |
|
|
| This repository holds **the final base model only**. For the other 38 training runs see |
| [bart-experiments](https://huggingface.co/jbduran/bart-experiments); for the instruction-tuned |
| version see [bart-sft](https://huggingface.co/jbduran/bart-sft). |
|
|
| ## Specifications |
|
|
| | | | |
| |---|---| |
| | Parameters | 2,818,575,450 (~2.8B, including embeddings) | |
| | Depth | 32 | |
| | Context | 4,096 (SSSL window pattern) | |
| | Vocab | 32,768 (`rustbpe`) | |
| | Training tokens | 20.13B | |
| | Precision | fp8 (tensorwise) | |
| | Final step | 9,600 | |
| | Run | `Think.Unbounded-d32-v2mix-cont` | |
|
|
| > **On the parameter count.** `scaling_params` in `config.json` reads 1,677,724,672 — that figure is the scaling-law budget and excludes the value-embedding matrices. The model carries a full `32768 x 2048` value embedding on every other layer (16 of 32), which is another 1,073,741,824 parameters. Counting everything, the model is 2,818,575,450 parameters. |
| |
| ## Contents |
| |
| ``` |
| model_009600.pt final weights (8.99 GB) |
| meta_009600.json checkpoint metadata |
| config.json full training configuration |
| run.json lineage record |
| summary.json run summary |
| tokenizer/ rustbpe tokenizer, vocab 32,768 |
| evals/ val_bpb, samples, vintage-core results |
| ``` |
| |
| The optimizer shard and the eight earlier checkpoints (steps 6000–9500) live in |
| [bart-experiments](https://huggingface.co/jbduran/bart-experiments) under |
| `experiments/Think.Unbounded-d32-v2mix-cont/`. |
| |
| ## Training |
| |
| Pretrained on [bart-dataset-v3](https://huggingface.co/datasets/jbduran/bart-dataset-v3), then |
| midtrained on [bart-midtrain](https://huggingface.co/datasets/zachnorton03/bart-midtrain) with a |
| three-stage schedule that shifts the mixture from 0% → 21% → 45% midtrain while the learning rate |
| decays (nanochat WSD, decay from ~35%). |
| |
| This run branches from `Think.Unbounded-d32` at step 5500, carrying optimizer state, and continues |
| on corrected token-based mixtures. The parent had computed its mixture ratios by document count, |
| which under-delivered midtrain content badly — this continuation is the repair. |
| |
| | Stage | Dataset | |
| |---|---| |
| | Pretraining | [v3](https://huggingface.co/datasets/jbduran/bart-dataset-v3) ← [v2](https://huggingface.co/datasets/jbduran/bart-dataset-v2) ← [v1](https://huggingface.co/datasets/jbduran/bart-dataset-v1) | |
| | Midtraining | [bart-midtrain](https://huggingface.co/datasets/zachnorton03/bart-midtrain) | |
| |
| Pipelines: [bart-dataset-scripts](https://github.com/OwenVoorhees/bart-dataset-scripts) · |
| [bart-midtrain-scripts](https://github.com/OwenVoorhees/bart-midtrain-scripts) |
| |
| ## What it's for |
| |
| BART exists to answer what a model trained *only* on historical text actually knows and how it |
| writes. Because nothing after 1930 appears in training, it can be probed for period-authentic prose |
| and for historical reasoning without contamination from modern sources — and measured on whether |
| modern concepts leak in at all. The companion benchmarks |
| ([vintage-core](https://huggingface.co/datasets/jbduran/vintage-core), |
| [vintage-gsm8k](https://huggingface.co/datasets/jbduran/vintage-gsm8k), |
| [history-event-reconstruction](https://huggingface.co/datasets/jbduran/history-event-reconstruction)) |
| are period-adapted so the model isn't penalized for lacking knowledge it was never given. |
| |
| This is a base model shipped as a raw nanochat checkpoint — not instruction-tuned and not a chat |
| model. Use [bart-sft](https://huggingface.co/jbduran/bart-sft) for that. |
| |
| > **Note.** `config.json` records repo and dataset names as they were at training time |
| > (`jbduran/think.nano`, `think-dataset-clean-1930s`, `think-midtrain`). Those still resolve via |
| > Hub redirects, but the midtrain mixture subfolders it points at have since moved. The config is |
| > kept as a historical record of the run rather than rewritten. |
| |
| ## License |
| |
| MIT |
| |
| --- |
| |
| Built by [Unbounded Labs](https://unboundedlab.com). |
| |
