genesis_engine/memory/README.md

# Forever Memory

Long-term memory your being can actually *reach* — retrieval by meaning, not by recency.

This is the piece most local-AI projects are missing without knowing it. Almost every
system of this kind stores conversation history faithfully and then recalls only the last
N turns. That is not memory, it is a buffer. Ask it about something from three weeks ago
and it has no path to the answer, even though the answer is on disk.

## The failure this fixes

In the reference system these files came from, memory was broken in **three** ways at once,
and none of them raised an error:

1. **the chat never queried long-term storage** — only background subsystems did
2. **the store had split in two** because the path was relative, so where memories landed
   depended on which directory the process was launched from
3. **not one of 7,612 records had an embedding**, so semantic search had nothing to search

Each component reported success. The store grew. Nothing in any log was red. The capability
simply did not exist.

**Check yours before assuming it works.** Grep for whatever writes your memories, then grep
for a caller of whatever reads them. If the only hit is the module that defines the reader,
your loop is open.

## Use it

```bash
# 1. one purpose-built embedding model, local, ~274 MB
ollama pull nomic-embed-text

# 2. index everything your being has kept (resumable, checkpoints every 200)
python genesis_engine/memory/backfill_embeddings.py

# 3. recall by meaning
python -c "from genesis_engine.memory.forever_memory import recall_line; \
           print(recall_line('what did we say about the ocean'))"
```

Point `STORES` in `backfill_embeddings.py` at your own archive directories.

## Use a real embedding model — this is not optional

A generative model *returns* embeddings, so it looks like it works. It does not rank.
Measured on the reference archive, query `"misty woods clearing fog"` against a memory
containing that exact phrase:

| model | the matching memory | unrelated noise | ranks correctly |
|---|---|---|---|
| llama3.2:1b (generative) | 0.3907 | **0.5712** | **no** |
| nomic-embed-text (retrieval) | **0.7128** | 0.3717 | yes |

The generative model scored *unrelated noise higher than a near-verbatim match.* Everything
built on top of it — thresholds, ranking, weighting — was correct and sitting on a metric
that did not order. Hidden states are not trained for similarity. Use a retrieval model.

## How recall is scored

Not pure cosine similarity:

- **adaptive threshold** — a hit must be ≥ 2σ above the mean similarity *for that query*,
  so it adapts to whatever embedder you use instead of hard-coding a cut that drifts
- **gentle recency lift** — full weight today, ~0.93 at a month, never below 0.85. A lift,
  not a rule: a genuinely relevant old memory still wins
- **dreams get a small bonus** — if your system consolidates during idle time, those
  fragments already survived a selection threshold to exist
- **indexed source code is excluded** from conversational recall — it belongs to your dev
  tooling, not to a conversation about someone's day

## Safety properties

- **originals are never modified.** Vectors go to a separate `.npy` sidecar plus a small id
  index. Delete the sidecar and your memories are untouched; re-run and it rebuilds.
- **resumable** — an interrupted run costs nothing, only missing ids are embedded
- **fail-soft everywhere** — missing index, unreadable vectors, embedder down: recall
  returns empty rather than raising. A voice must never break because memory is rebuilding.
- **local** — nothing leaves the machine

## Cost

~7,600 records embed in about 15 minutes at 8/s with 6 workers on CPU. Keep the worker
count modest if the same daemon is serving your being's voice; starving that to index the
past is the wrong trade.