code_release/docs/EVALUATION.md

Running the evaluation

All commands run from code/.

Environment

Python 3.11 on Linux x86-64 (the pins are manylinux wheels).

pip install --index-url https://download.pytorch.org/whl/cpu torch==2.13.0
pip install -r requirements/eval.txt                 # every method except memos
pip install --no-deps mem0ai==0.1.87                 # the Mem0 engine
python -m nltk.downloader punkt punkt_tab            # bm25_rag tokenization
bash vendor/fetch_engines.sh

memos runs in a second environment layered over the first, so that its pins (requirements/memos.txt) do not replace those of the other methods:

python -m venv .venv_memos                     # with the evaluation interpreter
python -c "import sysconfig; print(sysconfig.get_paths()['purelib'])" \
    > .venv_memos/lib/python3.11/site-packages/eval_base.pth
.venv_memos/bin/pip install --no-deps -r requirements/memos.txt

Run memos and its services with .venv_memos/bin/python.

Models

memebench/configs/backbone.yaml names every model; the evaluation refuses to start while a <...> value that the selected methods need is left. embedding.batch_size is the number of texts per embedding request. Method parameters, including the variables shared by several methods at the top of methods.yaml, are set in memebench/configs/methods.yaml; a method whose parameters still contain a <...> value is refused.

role	fields	used for
answer	model, base_url, api_key_env, reasoning, tokenizer	the runner's answering call (every method except hipporag, remem, letta and mirix)
judge	model, base_url, api_key_env, reasoning	grading each answer against the gold
embedding	model, base_url, api_key_env, dim, batch_size, tokenizer	every method that retrieves by vector similarity
memory	model, base_url, api_key_env, reasoning (all optional)	every LLM call inside a method, including the answer of hipporag, remem, letta and mirix

base_url is the endpoint of the role; null uses the client's default endpoint (OPENAI_BASE_URL when set). api_key_env names the environment variable that holds the key for that endpoint (OPENAI_API_KEY when null); each key is sent only to its endpoint, and roles that share an endpoint must name the same variable. memory.model: null uses the answer model at the answer base_url. reasoning: true marks a reasoning model: the methods' internal calls to it set max_completion_tokens instead of max_tokens, and the judge sends reasoning_effort only to a reasoning model. memory.reasoning: null takes the answer model's setting and must be set when the memory model differs from the answer model. answer.tokenizer (a tiktoken encoding name) counts the context budgets and chunk sizes; embedding.tokenizer counts embedding inputs. Methods whose store needs the embedding dimension take embedding.dim; letta and lightmem embed one text per request.

Running

python -m memebench.run_eval --list-patients
python -m memebench.run_eval --method bm25_rag --patient <patient>
python -m memebench.run_eval --method hipporag --patient <patient>
.venv_memos/bin/python -m memebench.run_eval --method memos --patient <patient> \
    --methods-config memebench/configs/methods.memos.yaml

• --method: one of long_context, bm25_rag, dense_rag, hipporag, remem, amem, letta, mirix, lightmem, mem0, memos; a comma list; or all (long_context, bm25_rag and dense_rag).
• --patient: P01 ... P23, a comma list, or all. Each (method, patient) pair is an independent run and can run as its own process.
• --methods-config: parameter file (default memebench/configs/methods.yaml); memos takes its parameters from methods.memos.yaml. --backbone-config: model file (default memebench/configs/backbone.yaml).
• --resume continues the newest run (not for memos); with --qids 0-8,40 it answers the selected questions again and merges them into that run. A run whose stored answers (graded or not) do not match the current questions file (a question index that no longer exists or another question id) is not resumed; start a fresh run. A run is resumed only with the settings it started with (run_settings.json).
• --rejudge re-grades every stored answer of the newest completed run with the configured judge, without re-answering; the previous files are kept as answers.before_rejudge_<stamp>.jsonl and report.before_rejudge_<stamp>.json.
• --top-k N sets the number of items every retrieving method retrieves for an answer (long_context retrieves nothing).

memos services

.venv_memos/bin/python services/memos_services.py download     # Java 21, Neo4j 5.26.12, Qdrant 1.16.3
.venv_memos/bin/python services/memos_services.py model        # the MemOS reranker at a pinned revision
.venv_memos/bin/python services/memos_services.py run          # supervises Neo4j, Qdrant and the reranker on loopback

Run them with the memos interpreter, or install requirements/services.txt into a separate Python 3.11 environment. --work-root and --local-state-root set where the services and their state live (--help); the Neo4j password is generated on first use into <work-root>/config/neo4j_password.txt.

Environment variables

variable	default	effect
OPENAI_API_KEY		API key of every endpoint whose api_key_env in backbone.yaml is not set otherwise
OPENAI_BASE_URL	the client's default	endpoint of every client without an explicit base_url
MEMEBENCH_MEMOS_NEO4J_URI, MEMEBENCH_MEMOS_QDRANT_URL, MEMEBENCH_MEMOS_RERANKER_URL	the loopback services	memos service addresses, used when the value in methods.memos.yaml is null
MEMEBENCH_MEMOS_NEO4J_PASSWORD		Neo4j password, read before neo4j_password_file

Timeouts and retry counts are the constants at the top of memebench/llm_client.py.

Outputs

outputs/<method>_<answer_model>/<patient>/<timestamp>/:

file	content
answers.jsonl	one row per question: qid, answer, verdict, pass, retrieved visits, memory_preview (the MEMORY block), error fields
answers_partial.jsonl	answered questions not yet graded; exists while a run is in progress
report.json	totals, accuracy, by_axis, by_answer_type, by_grounding, method parameters, models, memorize_wrapper (true when memory is built by an LLM), memorize_failures and consolidate_failures (visits and checkpoints whose memory construction failed; a run with any is reported as not clean and the command exits with status 1)
run_settings.json	the settings a resume must match: method, patient, method parameters, models and answering settings
memory_build.jsonl	one row per visit: memorize outcome and timing
memory_trace.jsonl	the method's memory events
memory_snapshots/	for methods that export their memory: the store at each checkpoint (ckpt_vNNNN) and at the end (final)
store/	the method's working store
usage.jsonl	token usage per call, tagged memorize / query / judge

latest points at the newest completed run of a patient. hipporag builds its graph at the first question after new visits, so its memory_snapshots/ckpt_vNNNN holds the store as of the previous checkpoint.