Mr. Chatterbox (group B): a Victorian chatbot trained from scratch

A 352-million-parameter transformer trained from scratch on British Library books published 1837–1898, then fine-tuned to hold a conversation in a Victorian voice. It was built by students in DSCI 340 (group "chatterbox-b") as a class replication of Mr. Chatterbox, or, The Modern Prometheus by Trip (Estragon Newsletter, 30 March 2026), using Andrej Karpathy's nanochat.

This is a teaching project, not a product. The model is often wrong, invents facts, mostly does not follow instructions, and can repeat views from its period that are offensive today.

Try it: the Space emory-dsci340/mr-chatterbox-b serves the 10B-2 checkpoint from this repository.

What is in this repository

These are nanochat checkpoints, not transformers models. AutoModel.from_pretrained will not load them.

Path What it is
chatsft_checkpoints/s10_10B2/model_000013.pt, meta_000013.json 10B-2, the deployed model: the two-stage conversational fine-tune
chatsft_checkpoints/s11_11A/model_000010.pt, meta_000010.json 11A: 10B-2 after a small instruction-tuning run. Kept for the record; it scored worse (see Evaluation)
tokenizer/tokenizer.json, tokenizer/token_bytes.pt Byte-level BPE tokenizer trained on the corpus, vocabulary 49,152
MANIFEST.json, SHA256SUMS.txt Sizes and SHA256 of every file, the model configuration and the special-token IDs

Each model file is 1,409,347,167 bytes of fp32 weights. The folder is laid out like nanochat's base directory, so pointing NANOCHAT_BASE_DIR at a download of it is enough for nanochat's own loader.

Loading

The models need nanochat at commit 92d63d4 with the course patch (tied embedding and output head, no value-embedding tables, HuggingFace tokenizer). Unpatched nanochat refuses to load them. The patched files and a working loader are in the Space: see chatterbox.py and nanochat/ there. In short:

import os, torch
os.environ["NANOCHAT_BASE_DIR"] = "/path/to/a/download/of/this/repository"
from nanochat.checkpoint_manager import build_model   # the patched copy
from nanochat.engine import Engine

model, tokenizer, meta = build_model(
    "/path/to/a/download/of/this/repository/chatsft_checkpoints/s10_10B2", 13,
    torch.device("cpu"), phase="eval")
assert model.lm_head.weight is model.transformer.wte.weight   # the embeddings must be tied
engine = Engine(model, tokenizer)

Prompts are rendered with tokenizer.render_for_completion, with no system message (the models were trained without one). All evaluation used temperature 0.8, top-k 50, at most 256 new tokens, no top-p and no repetition penalty, stopping at <|assistant_end|> or <|bos|>.

Model details

Architecture nanochat GPT: 24 layers, width 1,024, 16 heads (head dimension 64), context 2,048 tokens, full attention in every layer (window pattern "L"), rotary positions, tied input embedding and output head
Parameters 352,321,610
Tokenizer Byte-level BPE, 49,152 tokens, trained on the corpus; nine special tokens at IDs 0–8 (`<
Language English, as printed 1837–1898 (with OCR errors)
Knowledge Nothing after 1898 by design
Precision Trained in bf16 on one NVIDIA L40S; the Space runs it on a CPU in fp32

Training

Every stage ran on a shared university AWS cluster. Counts are from the group's step logs.

1. Pretraining corpus

The British Library Books dataset, read from its hosted parquet files at revision b085f99d…, filtered to:

  • publication year 1837–1901 (the hosted files end in 1898, and most of the 1890s is missing);
  • first listed language English;
  • mean page OCR confidence of the book at least 0.65, then pages below 0.50 dropped.

Result: 17,222 books (17,086 for training, 136 held out for validation), 9.17 billion characters, 2.08 billion training tokens. The text keeps its OCR errors, running page headers and page numbers. This is smaller than the original author's corpus (28,035 books, 2.93 billion tokens): it is English-only and most of the 1890s was unavailable.

2. Base model

Trained from scratch for 6,496 steps of 524,288 tokens (about 3.41 billion tokens, 1.64 passes over the corpus) in 19.0 hours on one L40S. Final validation loss 0.919 bits per byte (the original author reports 0.973 on a different corpus and tokenizer, so the two are not directly comparable). Books were cut into passages of 1,000–8,000 characters first, because nanochat's loader otherwise keeps only the start of each long document.

3. Conversational fine-tuning data

Source How it was made In the final mix
Rewritten dialogue pairs 25,102 adjacent speaker turns extracted from 3,550 of the training books; 16,125 kept after a sentence-similarity filter and hand review; the first turn of each rewritten into modern English with GPT-4o mini (Azure OpenAI); 15,713 completed, 412 blocked by the content filter and left out. The reply is the original Victorian text. 10,044
Extended corpus conversations A rewritten pair plus 2–3 generated exchanges (GPT-4.1 mini) 4,671
Fully synthetic conversations Generated from a topic, a persona and an opening style (GPT-4.1 mini) 4,598
Corpus-grounded conversations Generated about one 200–400-word passage from a training book (GPT-4.1 mini) 4,203
Behavioural conversations Generated greetings, goodbyes, identity questions, abuse, gibberish, simple and modern facts (GPT-4.1 mini) 1,739
Catechisms Nineteenth-century question-and-answer school books (church history, general school questions, science, geography, household topics), supplied by the instructor 1,690
Play scenes Two-speaker stretches from 13 plays by Oscar Wilde and Bernard Shaw (Project Gutenberg texts), converted without an API 825
Total after cleaning, exact and near de-duplication and a per-cluster cap 27,770 conversations, 4.89 million tokens

The generated conversations put a modern user opposite an unnamed, educated Victorian of 1880–1899 who knows nothing later. Generated-assistant stock openers ("Indeed,", "Certainly,") were stripped in cleaning; mid-reply flattery was only partly removed. 200 conversations were held out and never trained on.

4. Fine-tuning runs

All runs use nanochat's chat_sft.py with small recorded edits, on one L40S.

Run Starts from Data Schedule In this repository
10A (a deliberate negative result) base 15,280 single-turn rewritten pairs 2 epochs, 133 steps no
10B-1 base the 27,770-conversation mix 2 epochs, 598 steps of 16,384 tokens no
10B-2 10B-1 the 1,739 behavioural conversations again 1 epoch, 13 steps of 8,192 tokens yes (deployed)
11A 10B-2 270 instruction pairs: 18 hand-written, 252 printed catechism answers given hand-written instructions 1 epoch, 10 steps of 2,048 tokens yes

Known quirks of 10B-2's run: the loader left 122 of the 1,739 conversations (the shortest) untrained, and its last step ran at a learning rate of zero.

Evaluation

All scoring was done by hand, blind, by two students working separately, on replies generated with the settings above and seeds 20261108 and 20261109. A reply is "usable" if it responds to the message, stops properly and is appropriate.

Step 10 battery (40 prompts Γ— 2 seeds: greetings, goodbyes, identity, factual, Victorian topics, modern questions, nonsense or abuse, and two-turn probes):

Usable base 10A 10B-1 10B-2 11A
Step 10 scorers (two students) 0% / 0% 32.5% / 46.3% 67.5% / 72.5% 75% / 75% β€”
Step 11 scorers (two other students) β€” β€” β€” 78.8% / 70.0% 58.8% / 48.8%

For 10B-2, the Step 11 scorers also found: in character 67.5% / 60.0%; Victorian register 90.0% / 87.5%; no knowledge after 1898 in 95.0% of replies; coherent second turn in 7 of 12 two-turn probes.

Instruction battery (21 prompts Γ— 2 seeds, strict all-or-nothing criteria): instruction followed in 2 of 42 conversations for 10B-2 and 1 of 42 for 11A, by both scorers.

Why 10B-2 is the deployed model. The instruction-tuning run (11A) did not improve instruction following and made ordinary conversation worse (usable down about 20 points for both scorers).

Deployment checks (Step 12). The Space's code, with the frozen seeds, gave replies identical to a laptop run on all 92 battery turns and 22 extra probe turns. Against the cluster's GPU replies for the same prompts (different random samples, so no reply is identical), 89 of 92 turns ended the same way and median length was 43.5 tokens against 36; two students judged 13 of 13 compared replies to be the same character. In ten probes with an explicit rule (one sentence, a list of four, a polite rewrite, a four-line poem, five words or fewer) the rule was followed 0 times.

Caveats.

  • Small samples: 40 prompts and two seeds, with six outcomes per prompt type. Differences are descriptive, not statistically tested.
  • The two Step 10 scorers disagreed widely on Victorian register (90.0% against 43.1%); the two Step 11 scorers agreed much better.
  • One battery greeting ("Hello!") appears word for word in the training data of 10B-1 and 10B-2.
  • Scores from different pairs of scorers are not comparable with each other.

Known failure modes

  • Instructions are mostly ignored. Requests for a set length, a list, a rewrite or a poem are usually answered as ordinary conversation.
  • Long conversations drift. After several turns it stops tracking what was said earlier.
  • Replies that do not end. A few percent of replies run to the 256-token limit (3 of 92 in our CPU test), sometimes repeating a phrase.
  • Invented facts. It writes confidently in period style whether or not the content is true.
  • OCR artefacts. Stray page numbers, running headers, broken words and odd spacing learned from the scanned books can appear.
  • Period views. The training books and plays carry Victorian attitudes on race, religion, class, gender and empire, including slurs; the model can reproduce them. Its medical and child-care advice is that of the period.
  • Odd input. Strings of repeated characters and nonsense produce broken replies.
  • Flattery. Some generated training conversations praised the user; traces remain.

Intended use and limits

Intended for teaching and for curiosity: seeing what a small language model trained only on nineteenth-century text does. Do not use it for facts, advice (medical, legal, financial or otherwise), or any decision. It is not a historical source: it imitates period prose, it does not quote it. It should not be presented as a person.

Energy and compute

Recorded by every step with the course's energy tracker (CPU and memory are estimated with Cloud Carbon Footprint coefficients; GPU energy is measured from nvidia-smi power logs) and a data-centre overhead (PUE) of 1.135.

Steps 2–11
Energy of the jobs themselves about 8.07 kWh: 7.55 kWh GPU (measured) + 0.52 kWh CPU and memory (estimated)
GPU time about 20.7 hours on one NVIDIA L40S (19.9 recorded + about 0.75 estimated for one job that left no record); 19 of them are the pretraining run
CPU-partition time about 8.6 vCPU-hours
Idle and boot time of cluster nodes (estimated, separate) about 0.5 kWh for the steps that estimated it; two steps did not
API use (Azure OpenAI) about 3.3 million tokens for rewriting ($0.71) and about 18.5 million for generation ($13.22)
COβ‚‚e about 3.1 kg for the job energy, at 0.379 kg COβ‚‚e per kWh (the Cloud Carbon Footprint factor for AWS us-east-1 used in the step logs)

Not included: the cluster's always-on head node and storage, the students' laptops, the energy of the API calls, and this Space. Over 97% of the energy is the single pretraining run.

Licences and credits

  • Original project and write-up: Trip, Mr. Chatterbox, or, The Modern Prometheus, Estragon Newsletter, 30 March 2026.
  • Training framework: nanochat by Andrej Karpathy, MIT licence, commit 92d63d4, with a course patch.
  • Pretraining text: the British Library's digitised nineteenth-century books, via TheBritishLibrary/blbooks (see that dataset's card for its terms).
  • Plays: Oscar Wilde and Bernard Shaw, from Project Gutenberg editions.
  • Catechisms: nineteenth-century question books, compiled by the course instructor.
  • Generated and rewritten data: made with Azure OpenAI models (GPT-4o mini, GPT-4.1 mini).
  • Built by the students of DSCI 340, group chatterbox-b, autumn 2026, with scripts drafted using Claude Code. Every evaluation score was given by students, by hand.

The licence for these weights has not been set yet.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Dataset used to train emory-dsci340/mr-chatterbox-b

Space using emory-dsci340/mr-chatterbox-b 1