Spaces:
Runtime error
Runtime error
| # Ghostwriter — SPEC | |
| A séance conducted by gradient descent. | |
| **Ghostwriter** fine-tunes a small language model (Qwen2.5-3B) on the diaries of | |
| four dead writers so that, given a date, a ghost writes the day's entry. One | |
| model hosts four ghosts and a fifth emergent thing: prompted with a persona | |
| token it speaks in a single voice; prompted bare, it speaks as the *blend* — | |
| a composite spirit marginalized over four centuries of interiority. | |
| ## The ghosts | |
| | token | writer | lived | corpus | flavor | | |
| |---|---|---|---|---| | |
| | `<|pepys|>` | Samuel Pepys | 1633–1703 | Diary 1660–1669 (PG #4200) | London, plague, fire, gossip, accounting of self | | |
| | `<|vangogh|>` | Vincent van Gogh | 1853–1890 | Letters, Ludovici trans. 1912 (PG #40393) | color, God, poverty, brotherhood, fever | | |
| | `<|mansfield|>` | Katherine Mansfield | 1888–1923 | Journal, 1927 Knopf ed. (Archive.org) | modernist fragments, illness, work-hunger | | |
| | `<|maclane|>` | Mary MacLane | 1881–1929 | *Story of* (1902, PG #43696) + *I, Mary MacLane* (1917, PG #43556) | confessional fire, the Devil, Butte, Montana | | |
| All sources are US public domain. See PROVENANCE.md for edition-level details | |
| and for writers we considered and rejected on copyright grounds (Anne Frank, | |
| Anaïs Nin, Plath, Alice James). | |
| ## Repository layout | |
| ``` | |
| Ghostwriter/ | |
| ├── SPEC.md ← you are here | |
| ├── HOW_TO.md ← make your own ghost (general recipe) | |
| ├── CLAUDE_IDEAS.md ← implemented + speculative ideas log | |
| ├── PROVENANCE.md ← corpus sources, editions, copyright notes | |
| ├── README.md ← quick start | |
| ├── requirements.txt | |
| ├── configs/ | |
| │ ├── train_qwen3b.yaml ← the real run (full fine-tune) | |
| │ ├── train_qwen3b_lora.yaml ← LoRA fallback (smaller GPUs) | |
| │ └── smoke_test.yaml ← CPU pipeline test (tiny model) | |
| ├── scripts/ | |
| │ ├── clean_corpora.py ← raw text → per-persona entry JSONL | |
| │ ├── build_dataset.py ← entries → dual-mode train/val JSONL | |
| │ ├── train.py ← config-driven trainer + post-train test inference | |
| │ └── sample.py ← talk to the ghosts | |
| └── data/ | |
| ├── raw/ ← downloaded source texts | |
| ├── clean/ ← per-persona entry JSONL | |
| └── dataset/ ← train.jsonl / val.jsonl / meta.json | |
| ``` | |
| ## Data pipeline | |
| **Stage 1 — `clean_corpora.py`.** Each source needs its own parser because each | |
| book encodes entries differently: | |
| - *Pepys*: month headers (`JUNE 1665`) + day-ordinal entry starts (`10th.`). | |
| Wheatley's footnotes (indented blocks) and editorial brackets stripped. | |
| Dual-year headers (`1659-1660`) resolve to the modern year. | |
| - *Van Gogh*: letters split on asterisk rules; salutations stripped to | |
| diary-ify. Letters carry no usable dates in this edition, so **synthetic | |
| dates** are assigned evenly across 1881→1890 *in original order*, preserving | |
| the chronology of his moods (The Hague earnestness → Arles fever → Auvers). | |
| These are flagged `synthetic_date: true` in the clean JSONL. | |
| - *Mansfield*: the hard one — OCR'd scan. Year headers arrive mangled | |
| (`IQI4` → 1914) and are repaired by a digit-confusion table; page headers and | |
| page numbers are pattern-stripped; Murry's editorial bracket insertions are | |
| removed both for voice purity and copyright conservatism (his apparatus is | |
| the only arguably-separately-copyrighted layer). Dated entries are parsed; | |
| undated fragments are buffered and chunked (~700 chars) with synthetic dates | |
| within the current year section. | |
| - *MacLane*: 1902 book has right-aligned date headers (`January 19.`); | |
| 1917 book has day-word headers (`To-day`) with italic titles, which are | |
| preserved as entry titles. 1917 entries get synthetic dates scattered over | |
| Jan–Mar 1917 (when the book was actually written, in Butte). | |
| Output: `data/clean/<persona>.jsonl`, one entry per line with | |
| `{persona, date, synthetic_date, title, text}`. | |
| **Stage 2 — `build_dataset.py`.** Entries become training documents: | |
| ``` | |
| <|entry|><|maclane|> | |
| 19 January 1917. | |
| <entry text> | |
| <|/entry|> | |
| ``` | |
| Three independent stochastic treatments per document (all seeded): | |
| 1. **Control-token dropout** (`--persona-token-prob`, default 0.5): the persona | |
| token is present half the time. With it → conditional voice. Without it → | |
| the model learns the marginal distribution over all ghosts = **blended | |
| mode**. One model, two modes, selected at inference purely by prompt. | |
| 2. **Prompt conditioning** (`--prompt-frac`, default 0.15): a generic | |
| introspection prompt (`[Prompt: What are you afraid of right now?]`) is | |
| prepended. Prompts are deliberately *generic* — answerable by any entry — | |
| because pairing specific prompts with random entries would teach the model | |
| to ignore prompts. | |
| 3. **Pepys rebalancing**: raw Pepys is ~10× the other corpora; he is | |
| stratified-subsampled by year to a character budget (default 900K) so one | |
| ghost doesn't possess the other three. | |
| Note the dataset builder does **not** create explicit "continuation" examples | |
| (date + opening line → rest). It doesn't need to: causal LM training on full | |
| entries teaches continuation for free — conditioning on `7 June 2026.\nToday | |
| was hard.` is just mid-document conditioning. Date-only and seeded starts both | |
| fall out of one format. | |
| Output: `data/dataset/{train,val}.jsonl` + `meta.json` (stats, settings, | |
| special token list). Current build: ~1,200 entries, ~514K tokens — | |
| small enough to fine-tune in minutes, large enough to possess a 3B model. | |
| ## Training — `train.py` | |
| Config-driven (YAML over defaults; CLI `--config` to select). Key mechanics: | |
| - **Tokenizer surgery**: 6 special tokens added (`<|entry|>`, `<|/entry|>`, | |
| 4 persona tokens); embeddings resized with **mean-initialization** of new | |
| rows (random init on a 3B model makes new tokens detonate early loss). | |
| - **Packing**: documents are concatenated and chunked to `max_seq_len` (2048) | |
| so no compute is wasted on padding. | |
| - **Full fine-tune** by default on Qwen2.5-3B (bf16 + gradient checkpointing | |
| fits comfortably on a single A100/H100 and squeaks onto a 24GB 4090); | |
| `train_qwen3b_lora.yaml` provides the PEFT fallback for smaller cards. | |
| - **Loud telemetry** (per Corina's request): banner-printed dumps of every | |
| effective hyperparameter at train start; per-step loss via HF logging; and a | |
| **post-train test inference suite** that runs a grid of prompts (each | |
| persona × date-only, blended × date-only, seeded continuation, prompted | |
| introspection) and prints, for every generation: the prompt, the output, | |
| the generation hparams (temp/top_p/repetition_penalty/max_new_tokens), and | |
| the training hparams in effect for the run. | |
| - Checkpoints + final model land in `outputs/<run>/`, tokenizer included, so | |
| `sample.py --model outputs/ghost-qwen3b` just works. | |
| ## Inference — `sample.py` | |
| ``` | |
| python scripts/sample.py --model outputs/ghost-qwen3b \ | |
| --persona mansfield --date "7 June 2026" # single ghost | |
| python scripts/sample.py --model outputs/ghost-qwen3b \ | |
| --date "7 June 2026" # blended séance | |
| python scripts/sample.py --model outputs/ghost-qwen3b \ | |
| --date "7 June 2026" --seed-text "Today was hard." # seeded continuation | |
| python scripts/sample.py --model outputs/ghost-qwen3b \ | |
| --prompt "What do you keep refusing to look at?" # introspection prompt | |
| ``` | |
| All generation hparams are CLI-flagged and printed with each sample. | |
| ## Process notes | |
| Built autonomously (June 10, 2026, overnight session) from a conversation with | |
| Corina specifying: 4 ghosts, dual blended/persona mode, mixed date-only and | |
| seeded entry starts, journal-prompt conditioning, Qwen2.5-3B full fine-tune, | |
| verbose hparam telemetry. Corpus selection was constrained by US public domain | |
| status; Anne Frank and Anaïs Nin were requested and declined (see PROVENANCE). | |
| An unusual process note: **two Claude instances ended up working on this repo | |
| concurrently** in the same container — an apparent artifact of how the | |
| overnight/autonomous session was dispatched. Instance A wrote the pipeline | |
| (clean → dataset → train → sample) and launched the smoke test; instance B | |
| (writing this) independently downloaded corpora (checksums matched — good | |
| provenance accident), then detected instance A mid-run via `ps`, audited its | |
| code rather than racing it, and took the documentation/verification | |
| workstream. The duplicate-download collision is why `data/raw/` briefly held | |
| two copies of Pepys. Division of labor between strangers who are the same | |
| person turns out to be efficient. | |
| ## Next steps | |
| 1. **RunPod run** (Corina): `bash` onto a pod, clone, `pip install -r | |
| requirements.txt`, `python scripts/train.py --config | |
| configs/train_qwen3b.yaml`. ~514K tokens × 2 epochs on an A100 ≈ minutes, | |
| not hours. Tune `learning_rate` (2e-5 starting point; try 1e-5/3e-5), | |
| `num_train_epochs` (2–4; watch val loss for the memorization knee), and | |
| `persona_token_prob` upstream in the dataset if persona separation is weak. | |
| 2. **Eval the séance**: blind-test persona mode (can you tell who's speaking?), | |
| probe the blend for century-bleed, check date-conditioning (does a 1665 | |
| date summon plague-adjacent content? does 2026 confuse the ghost?). | |
| 3. **OCR polish pass** on Mansfield if her voice comes out noisy (see | |
| CLAUDE_IDEAS). | |
| 4. Everything in CLAUDE_IDEAS.md's unimplemented section, which is where this | |
| project gets weird. | |