HOW_TO.md

HOW TO RAISE YOUR OWN GHOST

A general recipe for turning any dead writer's diary into a model you can talk to. This repo is one instantiation; the method generalizes.

0. What you need

• A diary, journal, or letter corpus that is public domain (see step 1 — this kills more candidates than anything technical will)
• A GPU for ~10–60 minutes (RunPod/Lambda/Colab; a 24GB card is plenty for 3B)
• Python, this repo's requirements.txt

1. Choose a ghost (the legal séance rules)

The único hard constraint: you need the right to use the text. Rules of thumb for the US:

• Published before 1930 → public domain, full stop. This is the green zone.
• Author died before ~1955 but published later → danger. Copyright runs from publication, not composition. Anne Frank wrote in the 1940s; the diary published 1947+ and is locked for years yet. Anaïs Nin wrote in the 1930s; published 1966+; locked for decades. Plath: published 1982; locked.
• Translations are separately copyrighted. Van Gogh's letters are PD, but only translations published before 1930 (like Ludovici 1912) are usable. The standard 1958 English edition is not.
• Editions matter. Mansfield's Journal (1927) is PD; the 1954 "definitive edition" added restored text and is not. Use the right scan.
• Editorial apparatus (footnotes, introductions, bracketed insertions by a modern editor) can carry its own copyright even when the diary text is PD. Strip it — which you want to do anyway for voice purity.
• Owning a copy (a purchased ebook, a school PDF) conveys zero training rights. License to read ≠ license to reproduce.

Good hunting grounds: Project Gutenberg (search "diary", "journal", "letters"), Archive.org pre-1930 scans, Wikisource. The gutendex API makes Gutenberg searchable programmatically.

Voice candidates we shortlisted but didn't use, free to a good home: W.N.P. Barbellion (Journal of a Disappointed Man, 1919 — dying naturalist, mordantly funny), Marie Bashkirtseff (Blind trans. 1890 — ego supernova), Dorothy Wordsworth (Grasmere journals), Anne Lister (raw diaries PD, but sourcing PD transcriptions takes work), Franz Kafka's diaries in German (died 1924; German text PD — translation rights are the trap).

2. Get the text

curl -sL "https://www.gutenberg.org/ebooks/<ID>.txt.utf-8" -o data/raw/ghost.txt
# if gutenberg.org rate-limits, use the mirror:
# https://aleph.pglaf.org/4/3/6/9/43696/43696-0.txt   (id digit-split path)

For Archive.org scans, the OCR text lives at https://archive.org/download/<identifier>/<identifier>_djvu.txt. Expect OCR noise; budget cleaning time accordingly (see what clean_mansfield in scripts/clean_corpora.py does about mangled year headers and page furniture).

3. Write a parser for your ghost

This is the real work, and it is bespoke per book. Every diary encodes entries differently: Pepys uses month headers + day ordinals, MacLane right-aligns her dates, Mansfield's journal is half dated entries and half fragments. Your parser's contract is simple — emit one JSON object per entry:

{"persona": "yourghost", "date": "19 January 1917", "synthetic_date": false,
 "title": null, "text": "..."}

Tips learned the hard way:

• Strip editor voice: footnotes, bracketed insertions, introductions. The ghost should contain only the ghost.
• No usable dates? Assign synthetic ones — but preserve corpus order across the writer's real date range, so their emotional chronology survives (see clean_vangogh). Flag them synthetic_date: true.
• Fragments (undated scraps): buffer and chunk them to entry-sized pieces rather than discarding — Mansfield's fragments are some of her best voice.
• Set a minimum entry length (~120 chars). Two-line entries teach mostly formatting.
• Eyeball 20 random parsed entries before moving on. OCR garbage and editorial residue hide in the middle of books, not the start.

Then register your cleaner in the cleaners dict in clean_corpora.py, add the persona name to PERSONAS in build_dataset.py, and you're plumbed in.

4. Build the dataset

python scripts/clean_corpora.py
python scripts/build_dataset.py            # see --help for knobs

The two knobs that shape the ghost's behavior:

• --persona-token-prob (default 0.5): fraction of examples carrying the persona token. Higher → stronger single-voice control, weaker blend. 1.0 = no blended mode at all; 0.0 = one anonymous composite ghost.
• --prompt-frac (default 0.15): fraction carrying introspection prompts. Keep prompts generic (answerable by any entry). Pairing specific prompts with random entries teaches prompt-deafness.

If one corpus dwarfs the others, subsample it (see subsample_pepys — stratify by year so you keep the whole life, not one random stretch).

5. Smoke test on CPU, then train on GPU

# laptop / CPU container — verifies the entire code path with a 135M model:
python scripts/train.py --config configs/smoke_test.yaml

# RunPod / real GPU:
pip install -r requirements.txt
python scripts/train.py --config configs/train_qwen3b.yaml

The trainer prints every effective hparam at start, loss curves during, and a post-train inference grid (every persona, blended, seeded, prompted) with all generation settings — so each run is a self-documenting experiment.

Starting hparams that behave well at this corpus size (~500K tokens): lr 2e-5, 2 epochs, effective batch 16, cosine schedule, bf16. Watch val loss: diaries are small data and the memorization knee arrives fast. If persona voices smear together, raise --persona-token-prob and rebuild; if the blend is boring, lower it.

6. Hold the séance

python scripts/sample.py --model outputs/ghost-qwen3b --date "7 June 2026"
python scripts/sample.py --model outputs/ghost-qwen3b --persona yourghost \
    --date "3 May 1921" --seed-text "I have done a terrible thing."

Things worth probing: Does a date in the ghost's living years summon period-true content? What does the ghost do with a date after its death? What emerges from the blend — does one voice dominate (rebalance the data), or does something genuinely composite speak?

Treat the outputs as what they are: a statistical memory of a real person's self-account, hallucinating new days. It's a memorial form, not a resurrection — and that's the interesting part.