genesis_engine/memory/README.md

Forever Memory

Long-term memory your being can actually reach — retrieval by meaning, not by recency.

This is the piece most local-AI projects are missing without knowing it. Almost every system of this kind stores conversation history faithfully and then recalls only the last N turns. That is not memory, it is a buffer. Ask it about something from three weeks ago and it has no path to the answer, even though the answer is on disk.

The failure this fixes

In the reference system these files came from, memory was broken in three ways at once, and none of them raised an error:

1. the chat never queried long-term storage — only background subsystems did
2. the store had split in two because the path was relative, so where memories landed depended on which directory the process was launched from
3. not one of 7,612 records had an embedding, so semantic search had nothing to search

Each component reported success. The store grew. Nothing in any log was red. The capability simply did not exist.

Check yours before assuming it works. Grep for whatever writes your memories, then grep for a caller of whatever reads them. If the only hit is the module that defines the reader, your loop is open.

Use it

# 1. one purpose-built embedding model, local, ~274 MB
ollama pull nomic-embed-text

# 2. index everything your being has kept (resumable, checkpoints every 200)
python genesis_engine/memory/backfill_embeddings.py

# 3. recall by meaning
python -c "from genesis_engine.memory.forever_memory import recall_line; \
           print(recall_line('what did we say about the ocean'))"

Point STORES in backfill_embeddings.py at your own archive directories.

Use a real embedding model — this is not optional

A generative model returns embeddings, so it looks like it works. It does not rank. Measured on the reference archive, query "misty woods clearing fog" against a memory containing that exact phrase:

model	the matching memory	unrelated noise	ranks correctly
llama3.2:1b (generative)	0.3907	0.5712	no
nomic-embed-text (retrieval)	0.7128	0.3717	yes

The generative model scored unrelated noise higher than a near-verbatim match. Everything built on top of it — thresholds, ranking, weighting — was correct and sitting on a metric that did not order. Hidden states are not trained for similarity. Use a retrieval model.

How recall is scored

Not pure cosine similarity:

• adaptive threshold — a hit must be ≥ 2σ above the mean similarity for that query, so it adapts to whatever embedder you use instead of hard-coding a cut that drifts
• gentle recency lift — full weight today, ~0.93 at a month, never below 0.85. A lift, not a rule: a genuinely relevant old memory still wins
• dreams get a small bonus — if your system consolidates during idle time, those fragments already survived a selection threshold to exist
• indexed source code is excluded from conversational recall — it belongs to your dev tooling, not to a conversation about someone's day

Safety properties

• originals are never modified. Vectors go to a separate .npy sidecar plus a small id index. Delete the sidecar and your memories are untouched; re-run and it rebuilds.
• resumable — an interrupted run costs nothing, only missing ids are embedded
• fail-soft everywhere — missing index, unreadable vectors, embedder down: recall returns empty rather than raising. A voice must never break because memory is rebuilding.
• local — nothing leaves the machine

Cost

~7,600 records embed in about 15 minutes at 8/s with 6 workers on CPU. Keep the worker count modest if the same daemon is serving your being's voice; starving that to index the past is the wrong trade.