---
license: mit
library_name: pytorch
language:
- en
tags:
- nanochat
- language-model
- pre-1930
- historical
- vintage
- sft
- conversational
---

# BART SFT

The instruction-tuned [BART](https://huggingface.co/jbduran/bart): a 2.8B-parameter model whose
knowledge ends in **1930**, fine-tuned to hold a conversation in period-appropriate voice.

📝 [Read the write-up](https://www.unboundedlab.com/blog/bart) · 🌐 [Unbounded Labs](https://unboundedlab.com)

Run `Think.Unbounded-d32-v2mix-cont-pre1930-curriculum-c3-robust-v2`, branched from
[BART](https://huggingface.co/jbduran/bart) at base step 9,600.

## Specifications

| | |
|---|---|
| Base model | [`bart`](https://huggingface.co/jbduran/bart) @ step 9,600 |
| Parameters | 2,818,575,450 (~2.8B, including embeddings) |
| Final step | 42 |
| Validation bpb | 0.7037 |
| Recipe | `curriculum` (C3Rv2, staged) |

## Contents

```
checkpoints/model_000042.pt    final weights (8.99 GB)
checkpoints/optim_000042_rank0.pt
checkpoints/meta_000042.json
checkpoints/eval_metrics.json
config.json                    full SFT configuration
run.json                       lineage record
evals/chatcore.json
```

## The curriculum

Training runs in three staged passes over the task routes, with a 30% noise rate throughout, then a
robustness pass.

| Stage | Routes | Authentic data |
|---|---|---|
| 1 | `knowledge_qa` | single-turn |
| 2 | `reasoning_qa`, `stem_reasoning`, `how_to_qa`, `opinion_qa`, `composition_qa`, `verse_qa` | calibration QA |
| 3 | `multiturn_qa`, `narrative_grounded`, `narrative_fiction` | multi-turn |

The robustness pass (1 epoch) adds `conversation_qa`, `unparseable_qa`, `typo_qa`, `era_qa`, and
`conversation_multiturn` — teaching the model to cope with malformed input, typos, and questions
about its own era.

### Training data

| Dataset | Role |
|---|---|
| [synthetic-pre1930-sft](https://huggingface.co/datasets/zachnorton03/synthetic-pre1930-sft) | the task routes (~416K rows, eleven routes) |
| [authentic-pre1930-sft-conversational](https://huggingface.co/datasets/zachnorton03/authentic-pre1930-sft-conversational) | authentic single- and multi-turn rows from 27 public-domain texts |
| [vintage-sft-robustness](https://huggingface.co/datasets/zachnorton03/vintage-sft-robustness) | the robustness pass (7,338 rows, five categories) |

## Evaluation

ChatCORE (karpathy suite) at step 42:

| Task | Score |
|---|---:|
| ARC-Easy | 0.2458 |
| ARC-Challenge | 0.2304 |
| MMLU | 0.2314 |
| GSM8K | 0.0000 |
| HumanEval | 0.0000 |
| **ChatCORE** | **-0.0113** |

These are *modern* benchmarks, and the multiple-choice scores sit at chance. That is the expected
result for a 2.8B model that has never seen a word written after 1930 — GSM8K and HumanEval in
particular test knowledge and formats that simply are not in the training data. The period-adapted
suites ([vintage-core](https://huggingface.co/datasets/jbduran/vintage-core),
[vintage-gsm8k](https://huggingface.co/datasets/jbduran/vintage-gsm8k),
[history-event-reconstruction](https://huggingface.co/datasets/jbduran/history-event-reconstruction))
exist precisely because these numbers don't measure what the model was built to do.

## Related

[bart](https://huggingface.co/jbduran/bart) (base) ·
[bart-experiments](https://huggingface.co/jbduran/bart-experiments) (all 39 runs, including five
other SFT variants) ·
[bart-dataset-v3](https://huggingface.co/datasets/jbduran/bart-dataset-v3) ·
[bart-midtrain](https://huggingface.co/datasets/zachnorton03/bart-midtrain)

## License

MIT

---

Built by [Unbounded Labs](https://unboundedlab.com).