# Running the evaluation

All commands run from `code/`.

## Environment

Python 3.11 on Linux x86-64 (the pins are manylinux wheels).

```bash
pip install --index-url https://download.pytorch.org/whl/cpu torch==2.13.0
pip install -r requirements/eval.txt                 # every method except memos
pip install --no-deps mem0ai==0.1.87                 # the Mem0 engine
python -m nltk.downloader punkt punkt_tab            # bm25_rag tokenization
bash vendor/fetch_engines.sh
```

memos runs in a second environment layered over the first, so that its pins (`requirements/memos.txt`) do not
replace those of the other methods:

```bash
python -m venv .venv_memos                     # with the evaluation interpreter
python -c "import sysconfig; print(sysconfig.get_paths()['purelib'])" \
    > .venv_memos/lib/python3.11/site-packages/eval_base.pth
.venv_memos/bin/pip install --no-deps -r requirements/memos.txt
```

Run memos and its services with `.venv_memos/bin/python`.

## Models

`memebench/configs/backbone.yaml` names every model; the evaluation refuses to start while a `<...>` value that the
selected methods need is left. `embedding.batch_size` is the number of texts per embedding request. Method
parameters, including the variables shared by several methods at the top of `methods.yaml`, are set in
`memebench/configs/methods.yaml`; a method whose parameters still contain a `<...>` value is refused.

| role | fields | used for |
|---|---|---|
| answer | `model`, `base_url`, `api_key_env`, `reasoning`, `tokenizer` | the runner's answering call (every method except hipporag, remem, letta and mirix) |
| judge | `model`, `base_url`, `api_key_env`, `reasoning` | grading each answer against the gold |
| embedding | `model`, `base_url`, `api_key_env`, `dim`, `batch_size`, `tokenizer` | every method that retrieves by vector similarity |
| memory | `model`, `base_url`, `api_key_env`, `reasoning` (all optional) | every LLM call inside a method, including the answer of hipporag, remem, letta and mirix |

`base_url` is the endpoint of the role; null uses the client's default endpoint (`OPENAI_BASE_URL` when set).
`api_key_env` names the environment variable that holds the key for that endpoint (`OPENAI_API_KEY` when null); each
key is sent only to its endpoint, and roles that share an endpoint must name the same variable. `memory.model: null`
uses the answer model at the answer `base_url`. `reasoning: true` marks a reasoning model: the methods' internal
calls to it set `max_completion_tokens` instead of `max_tokens`, and the judge sends `reasoning_effort` only to a
reasoning model. `memory.reasoning: null` takes the answer model's setting and must be set when the memory model
differs from the answer model. `answer.tokenizer` (a tiktoken encoding name) counts the context budgets and chunk
sizes; `embedding.tokenizer` counts embedding inputs. Methods whose store needs the embedding dimension take
`embedding.dim`; letta and lightmem embed one text per request.

## Running

```bash
python -m memebench.run_eval --list-patients
python -m memebench.run_eval --method bm25_rag --patient <patient>
python -m memebench.run_eval --method hipporag --patient <patient>
.venv_memos/bin/python -m memebench.run_eval --method memos --patient <patient> \
    --methods-config memebench/configs/methods.memos.yaml
```

- `--method`: one of long_context, bm25_rag, dense_rag, hipporag, remem, amem, letta, mirix, lightmem, mem0,
  memos; a comma list; or `all` (long_context, bm25_rag and dense_rag).
- `--patient`: P01 ... P23, a comma list, or `all`. Each (method, patient) pair is an independent run and can run
  as its own process.
- `--methods-config`: parameter file (default `memebench/configs/methods.yaml`); memos takes its parameters from
  `methods.memos.yaml`. `--backbone-config`: model file (default `memebench/configs/backbone.yaml`).
- `--resume` continues the newest run (not for memos); with `--qids 0-8,40` it answers the selected questions again
  and merges them into that run. A run whose stored answers (graded or not) do not match the current questions file (a
  question index that no longer exists or another question id) is not resumed; start a fresh run. A run is resumed
  only with the settings it started with (`run_settings.json`).
- `--rejudge` re-grades every stored answer of the newest completed run with the configured judge, without
  re-answering; the previous files are kept as `answers.before_rejudge_<stamp>.jsonl` and
  `report.before_rejudge_<stamp>.json`.
- `--top-k N` sets the number of items every retrieving method retrieves for an answer (long_context retrieves
  nothing).

## memos services

```bash
.venv_memos/bin/python services/memos_services.py download     # Java 21, Neo4j 5.26.12, Qdrant 1.16.3
.venv_memos/bin/python services/memos_services.py model        # the MemOS reranker at a pinned revision
.venv_memos/bin/python services/memos_services.py run          # supervises Neo4j, Qdrant and the reranker on loopback
```

Run them with the memos interpreter, or install `requirements/services.txt` into a separate Python 3.11 environment.
`--work-root` and `--local-state-root` set where the services and their state live (`--help`); the Neo4j password
is generated on first use into `<work-root>/config/neo4j_password.txt`.

## Environment variables

| variable | default | effect |
|---|---|---|
| `OPENAI_API_KEY` | | API key of every endpoint whose `api_key_env` in `backbone.yaml` is not set otherwise |
| `OPENAI_BASE_URL` | the client's default | endpoint of every client without an explicit `base_url` |
| `MEMEBENCH_MEMOS_NEO4J_URI`, `MEMEBENCH_MEMOS_QDRANT_URL`, `MEMEBENCH_MEMOS_RERANKER_URL` | the loopback services | memos service addresses, used when the value in `methods.memos.yaml` is null |
| `MEMEBENCH_MEMOS_NEO4J_PASSWORD` | | Neo4j password, read before `neo4j_password_file` |

Timeouts and retry counts are the constants at the top of `memebench/llm_client.py`.

## Outputs

`outputs/<method>_<answer_model>/<patient>/<timestamp>/`:

| file | content |
|---|---|
| `answers.jsonl` | one row per question: qid, answer, verdict, pass, retrieved visits, `memory_preview` (the MEMORY block), error fields |
| `answers_partial.jsonl` | answered questions not yet graded; exists while a run is in progress |
| `report.json` | totals, accuracy, `by_axis`, `by_answer_type`, `by_grounding`, method parameters, models, `memorize_wrapper` (true when memory is built by an LLM), `memorize_failures` and `consolidate_failures` (visits and checkpoints whose memory construction failed; a run with any is reported as not clean and the command exits with status 1) |
| `run_settings.json` | the settings a resume must match: method, patient, method parameters, models and answering settings |
| `memory_build.jsonl` | one row per visit: memorize outcome and timing |
| `memory_trace.jsonl` | the method's memory events |
| `memory_snapshots/` | for methods that export their memory: the store at each checkpoint (`ckpt_vNNNN`) and at the end (`final`) |
| `store/` | the method's working store |
| `usage.jsonl` | token usage per call, tagged memorize / query / judge |

`latest` points at the newest completed run of a patient. hipporag builds its graph at the first question after new
visits, so its `memory_snapshots/ckpt_vNNNN` holds the store as of the previous checkpoint.