# Ghostwriter — SPEC A séance conducted by gradient descent. **Ghostwriter** fine-tunes a small language model (Qwen2.5-3B) on the diaries of four dead writers so that, given a date, a ghost writes the day's entry. One model hosts four ghosts and a fifth emergent thing: prompted with a persona token it speaks in a single voice; prompted bare, it speaks as the *blend* — a composite spirit marginalized over four centuries of interiority. ## The ghosts | token | writer | lived | corpus | flavor | |---|---|---|---|---| | `<|pepys|>` | Samuel Pepys | 1633–1703 | Diary 1660–1669 (PG #4200) | London, plague, fire, gossip, accounting of self | | `<|vangogh|>` | Vincent van Gogh | 1853–1890 | Letters, Ludovici trans. 1912 (PG #40393) | color, God, poverty, brotherhood, fever | | `<|mansfield|>` | Katherine Mansfield | 1888–1923 | Journal, 1927 Knopf ed. (Archive.org) | modernist fragments, illness, work-hunger | | `<|maclane|>` | Mary MacLane | 1881–1929 | *Story of* (1902, PG #43696) + *I, Mary MacLane* (1917, PG #43556) | confessional fire, the Devil, Butte, Montana | All sources are US public domain. See PROVENANCE.md for edition-level details and for writers we considered and rejected on copyright grounds (Anne Frank, Anaïs Nin, Plath, Alice James). ## Repository layout ``` Ghostwriter/ ├── SPEC.md ← you are here ├── HOW_TO.md ← make your own ghost (general recipe) ├── CLAUDE_IDEAS.md ← implemented + speculative ideas log ├── PROVENANCE.md ← corpus sources, editions, copyright notes ├── README.md ← quick start ├── requirements.txt ├── configs/ │ ├── train_qwen3b.yaml ← the real run (full fine-tune) │ ├── train_qwen3b_lora.yaml ← LoRA fallback (smaller GPUs) │ └── smoke_test.yaml ← CPU pipeline test (tiny model) ├── scripts/ │ ├── clean_corpora.py ← raw text → per-persona entry JSONL │ ├── build_dataset.py ← entries → dual-mode train/val JSONL │ ├── train.py ← config-driven trainer + post-train test inference │ └── sample.py ← talk to the ghosts └── data/ ├── raw/ ← downloaded source texts ├── clean/ ← per-persona entry JSONL └── dataset/ ← train.jsonl / val.jsonl / meta.json ``` ## Data pipeline **Stage 1 — `clean_corpora.py`.** Each source needs its own parser because each book encodes entries differently: - *Pepys*: month headers (`JUNE 1665`) + day-ordinal entry starts (`10th.`). Wheatley's footnotes (indented blocks) and editorial brackets stripped. Dual-year headers (`1659-1660`) resolve to the modern year. - *Van Gogh*: letters split on asterisk rules; salutations stripped to diary-ify. Letters carry no usable dates in this edition, so **synthetic dates** are assigned evenly across 1881→1890 *in original order*, preserving the chronology of his moods (The Hague earnestness → Arles fever → Auvers). These are flagged `synthetic_date: true` in the clean JSONL. - *Mansfield*: the hard one — OCR'd scan. Year headers arrive mangled (`IQI4` → 1914) and are repaired by a digit-confusion table; page headers and page numbers are pattern-stripped; Murry's editorial bracket insertions are removed both for voice purity and copyright conservatism (his apparatus is the only arguably-separately-copyrighted layer). Dated entries are parsed; undated fragments are buffered and chunked (~700 chars) with synthetic dates within the current year section. - *MacLane*: 1902 book has right-aligned date headers (`January 19.`); 1917 book has day-word headers (`To-day`) with italic titles, which are preserved as entry titles. 1917 entries get synthetic dates scattered over Jan–Mar 1917 (when the book was actually written, in Butte). Output: `data/clean/.jsonl`, one entry per line with `{persona, date, synthetic_date, title, text}`. **Stage 2 — `build_dataset.py`.** Entries become training documents: ``` <|entry|><|maclane|> 19 January 1917. <|/entry|> ``` Three independent stochastic treatments per document (all seeded): 1. **Control-token dropout** (`--persona-token-prob`, default 0.5): the persona token is present half the time. With it → conditional voice. Without it → the model learns the marginal distribution over all ghosts = **blended mode**. One model, two modes, selected at inference purely by prompt. 2. **Prompt conditioning** (`--prompt-frac`, default 0.15): a generic introspection prompt (`[Prompt: What are you afraid of right now?]`) is prepended. Prompts are deliberately *generic* — answerable by any entry — because pairing specific prompts with random entries would teach the model to ignore prompts. 3. **Pepys rebalancing**: raw Pepys is ~10× the other corpora; he is stratified-subsampled by year to a character budget (default 900K) so one ghost doesn't possess the other three. Note the dataset builder does **not** create explicit "continuation" examples (date + opening line → rest). It doesn't need to: causal LM training on full entries teaches continuation for free — conditioning on `7 June 2026.\nToday was hard.` is just mid-document conditioning. Date-only and seeded starts both fall out of one format. Output: `data/dataset/{train,val}.jsonl` + `meta.json` (stats, settings, special token list). Current build: ~1,200 entries, ~514K tokens — small enough to fine-tune in minutes, large enough to possess a 3B model. ## Training — `train.py` Config-driven (YAML over defaults; CLI `--config` to select). Key mechanics: - **Tokenizer surgery**: 6 special tokens added (`<|entry|>`, `<|/entry|>`, 4 persona tokens); embeddings resized with **mean-initialization** of new rows (random init on a 3B model makes new tokens detonate early loss). - **Packing**: documents are concatenated and chunked to `max_seq_len` (2048) so no compute is wasted on padding. - **Full fine-tune** by default on Qwen2.5-3B (bf16 + gradient checkpointing fits comfortably on a single A100/H100 and squeaks onto a 24GB 4090); `train_qwen3b_lora.yaml` provides the PEFT fallback for smaller cards. - **Loud telemetry** (per Corina's request): banner-printed dumps of every effective hyperparameter at train start; per-step loss via HF logging; and a **post-train test inference suite** that runs a grid of prompts (each persona × date-only, blended × date-only, seeded continuation, prompted introspection) and prints, for every generation: the prompt, the output, the generation hparams (temp/top_p/repetition_penalty/max_new_tokens), and the training hparams in effect for the run. - Checkpoints + final model land in `outputs//`, tokenizer included, so `sample.py --model outputs/ghost-qwen3b` just works. ## Inference — `sample.py` ``` python scripts/sample.py --model outputs/ghost-qwen3b \ --persona mansfield --date "7 June 2026" # single ghost python scripts/sample.py --model outputs/ghost-qwen3b \ --date "7 June 2026" # blended séance python scripts/sample.py --model outputs/ghost-qwen3b \ --date "7 June 2026" --seed-text "Today was hard." # seeded continuation python scripts/sample.py --model outputs/ghost-qwen3b \ --prompt "What do you keep refusing to look at?" # introspection prompt ``` All generation hparams are CLI-flagged and printed with each sample. ## Process notes Built autonomously (June 10, 2026, overnight session) from a conversation with Corina specifying: 4 ghosts, dual blended/persona mode, mixed date-only and seeded entry starts, journal-prompt conditioning, Qwen2.5-3B full fine-tune, verbose hparam telemetry. Corpus selection was constrained by US public domain status; Anne Frank and Anaïs Nin were requested and declined (see PROVENANCE). An unusual process note: **two Claude instances ended up working on this repo concurrently** in the same container — an apparent artifact of how the overnight/autonomous session was dispatched. Instance A wrote the pipeline (clean → dataset → train → sample) and launched the smoke test; instance B (writing this) independently downloaded corpora (checksums matched — good provenance accident), then detected instance A mid-run via `ps`, audited its code rather than racing it, and took the documentation/verification workstream. The duplicate-download collision is why `data/raw/` briefly held two copies of Pepys. Division of labor between strangers who are the same person turns out to be efficient. ## Next steps 1. **RunPod run** (Corina): `bash` onto a pod, clone, `pip install -r requirements.txt`, `python scripts/train.py --config configs/train_qwen3b.yaml`. ~514K tokens × 2 epochs on an A100 ≈ minutes, not hours. Tune `learning_rate` (2e-5 starting point; try 1e-5/3e-5), `num_train_epochs` (2–4; watch val loss for the memorization knee), and `persona_token_prob` upstream in the dataset if persona separation is weak. 2. **Eval the séance**: blind-test persona mode (can you tell who's speaking?), probe the blend for century-bleed, check date-conditioning (does a 1665 date summon plague-adjacent content? does 2026 confuse the ghost?). 3. **OCR polish pass** on Mansfield if her voice comes out noisy (see CLAUDE_IDEAS). 4. Everything in CLAUDE_IDEAS.md's unimplemented section, which is where this project gets weird.