README.md

---
license: mit
library_name: pytorch
language:
- en
tags:
- nanochat
- language-model
- pre-1930
- historical
- vintage
---

# BART

A 2.8B-parameter language model whose knowledge ends in **1930**. BART is trained from scratch on
pre-1930 public-domain books — no modern text anywhere in the corpus — to study what a model learns
when its entire world is historical.

📝 [Read the write-up](https://www.unboundedlab.com/blog/bart) · 🌐 [Unbounded Labs](https://unboundedlab.com)

This repository holds **the final base model only**. For the other 38 training runs see
[bart-experiments](https://huggingface.co/jbduran/bart-experiments); for the instruction-tuned
version see [bart-sft](https://huggingface.co/jbduran/bart-sft).

## Specifications

| | |
|---|---|
| Parameters | 2,818,575,450 (~2.8B, including embeddings) |
| Depth | 32 |
| Context | 4,096 (SSSL window pattern) |
| Vocab | 32,768 (`rustbpe`) |
| Training tokens | 20.13B |
| Precision | fp8 (tensorwise) |
| Final step | 9,600 |
| Run | `Think.Unbounded-d32-v2mix-cont` |

> **On the parameter count.** `scaling_params` in `config.json` reads 1,677,724,672 — that figure is the scaling-law budget and excludes the value-embedding matrices. The model carries a full `32768 x 2048` value embedding on every other layer (16 of 32), which is another 1,073,741,824 parameters. Counting everything, the model is 2,818,575,450 parameters.

## Contents

```
model_009600.pt     final weights (8.99 GB)
meta_009600.json    checkpoint metadata
config.json         full training configuration
run.json            lineage record
summary.json        run summary
tokenizer/          rustbpe tokenizer, vocab 32,768
evals/              val_bpb, samples, vintage-core results
```

The optimizer shard and the eight earlier checkpoints (steps 6000–9500) live in
[bart-experiments](https://huggingface.co/jbduran/bart-experiments) under
`experiments/Think.Unbounded-d32-v2mix-cont/`.

## Training

Pretrained on [bart-dataset-v3](https://huggingface.co/datasets/jbduran/bart-dataset-v3), then
midtrained on [bart-midtrain](https://huggingface.co/datasets/zachnorton03/bart-midtrain) with a
three-stage schedule that shifts the mixture from 0% → 21% → 45% midtrain while the learning rate
decays (nanochat WSD, decay from ~35%).

This run branches from `Think.Unbounded-d32` at step 5500, carrying optimizer state, and continues
on corrected token-based mixtures. The parent had computed its mixture ratios by document count,
which under-delivered midtrain content badly — this continuation is the repair.

| Stage | Dataset |
|---|---|
| Pretraining | [v3](https://huggingface.co/datasets/jbduran/bart-dataset-v3) ← [v2](https://huggingface.co/datasets/jbduran/bart-dataset-v2) ← [v1](https://huggingface.co/datasets/jbduran/bart-dataset-v1) |
| Midtraining | [bart-midtrain](https://huggingface.co/datasets/zachnorton03/bart-midtrain) |

Pipelines: [bart-dataset-scripts](https://github.com/OwenVoorhees/bart-dataset-scripts) ·
[bart-midtrain-scripts](https://github.com/OwenVoorhees/bart-midtrain-scripts)

## What it's for

BART exists to answer what a model trained *only* on historical text actually knows and how it
writes. Because nothing after 1930 appears in training, it can be probed for period-authentic prose
and for historical reasoning without contamination from modern sources — and measured on whether
modern concepts leak in at all. The companion benchmarks
([vintage-core](https://huggingface.co/datasets/jbduran/vintage-core),
[vintage-gsm8k](https://huggingface.co/datasets/jbduran/vintage-gsm8k),
[history-event-reconstruction](https://huggingface.co/datasets/jbduran/history-event-reconstruction))
are period-adapted so the model isn't penalized for lacking knowledge it was never given.

This is a base model shipped as a raw nanochat checkpoint — not instruction-tuned and not a chat
model. Use [bart-sft](https://huggingface.co/jbduran/bart-sft) for that.

> **Note.** `config.json` records repo and dataset names as they were at training time
> (`jbduran/think.nano`, `think-dataset-clean-1930s`, `think-midtrain`). Those still resolve via
> Hub redirects, but the midtrain mixture subfolders it points at have since moved. The config is
> kept as a historical record of the run rather than rewritten.

## License

MIT

---

Built by [Unbounded Labs](https://unboundedlab.com).