| # Running the evaluation |
|
|
| All commands run from `code/`. |
|
|
| ## Environment |
|
|
| Python 3.11 on Linux x86-64 (the pins are manylinux wheels). |
|
|
| ```bash |
| pip install --index-url https://download.pytorch.org/whl/cpu torch==2.13.0 |
| pip install -r requirements/eval.txt # every method except memos |
| pip install --no-deps mem0ai==0.1.87 # the Mem0 engine |
| python -m nltk.downloader punkt punkt_tab # bm25_rag tokenization |
| bash vendor/fetch_engines.sh |
| ``` |
|
|
| memos runs in a second environment layered over the first, so that its pins (`requirements/memos.txt`) do not |
| replace those of the other methods: |
|
|
| ```bash |
| python -m venv .venv_memos # with the evaluation interpreter |
| python -c "import sysconfig; print(sysconfig.get_paths()['purelib'])" \ |
| > .venv_memos/lib/python3.11/site-packages/eval_base.pth |
| .venv_memos/bin/pip install --no-deps -r requirements/memos.txt |
| ``` |
|
|
| Run memos and its services with `.venv_memos/bin/python`. |
|
|
| ## Models |
|
|
| `memebench/configs/backbone.yaml` names every model; the evaluation refuses to start while a `<...>` value that the |
| selected methods need is left. `embedding.batch_size` is the number of texts per embedding request. Method |
| parameters, including the variables shared by several methods at the top of `methods.yaml`, are set in |
| `memebench/configs/methods.yaml`; a method whose parameters still contain a `<...>` value is refused. |
|
|
| | role | fields | used for | |
| |---|---|---| |
| | answer | `model`, `base_url`, `api_key_env`, `reasoning`, `tokenizer` | the runner's answering call (every method except hipporag, remem, letta and mirix) | |
| | judge | `model`, `base_url`, `api_key_env`, `reasoning` | grading each answer against the gold | |
| | embedding | `model`, `base_url`, `api_key_env`, `dim`, `batch_size`, `tokenizer` | every method that retrieves by vector similarity | |
| | memory | `model`, `base_url`, `api_key_env`, `reasoning` (all optional) | every LLM call inside a method, including the answer of hipporag, remem, letta and mirix | |
|
|
| `base_url` is the endpoint of the role; null uses the client's default endpoint (`OPENAI_BASE_URL` when set). |
| `api_key_env` names the environment variable that holds the key for that endpoint (`OPENAI_API_KEY` when null); each |
| key is sent only to its endpoint, and roles that share an endpoint must name the same variable. `memory.model: null` |
| uses the answer model at the answer `base_url`. `reasoning: true` marks a reasoning model: the methods' internal |
| calls to it set `max_completion_tokens` instead of `max_tokens`, and the judge sends `reasoning_effort` only to a |
| reasoning model. `memory.reasoning: null` takes the answer model's setting and must be set when the memory model |
| differs from the answer model. `answer.tokenizer` (a tiktoken encoding name) counts the context budgets and chunk |
| sizes; `embedding.tokenizer` counts embedding inputs. Methods whose store needs the embedding dimension take |
| `embedding.dim`; letta and lightmem embed one text per request. |
|
|
| ## Running |
|
|
| ```bash |
| python -m memebench.run_eval --list-patients |
| python -m memebench.run_eval --method bm25_rag --patient <patient> |
| python -m memebench.run_eval --method hipporag --patient <patient> |
| .venv_memos/bin/python -m memebench.run_eval --method memos --patient <patient> \ |
| --methods-config memebench/configs/methods.memos.yaml |
| ``` |
|
|
| - `--method`: one of long_context, bm25_rag, dense_rag, hipporag, remem, amem, letta, mirix, lightmem, mem0, |
| memos; a comma list; or `all` (long_context, bm25_rag and dense_rag). |
| - `--patient`: P01 ... P23, a comma list, or `all`. Each (method, patient) pair is an independent run and can run |
| as its own process. |
| - `--methods-config`: parameter file (default `memebench/configs/methods.yaml`); memos takes its parameters from |
| `methods.memos.yaml`. `--backbone-config`: model file (default `memebench/configs/backbone.yaml`). |
| - `--resume` continues the newest run (not for memos); with `--qids 0-8,40` it answers the selected questions again |
| and merges them into that run. A run whose stored answers (graded or not) do not match the current questions file (a |
| question index that no longer exists or another question id) is not resumed; start a fresh run. A run is resumed |
| only with the settings it started with (`run_settings.json`). |
| - `--rejudge` re-grades every stored answer of the newest completed run with the configured judge, without |
| re-answering; the previous files are kept as `answers.before_rejudge_<stamp>.jsonl` and |
| `report.before_rejudge_<stamp>.json`. |
| - `--top-k N` sets the number of items every retrieving method retrieves for an answer (long_context retrieves |
| nothing). |
| |
| ## memos services |
| |
| ```bash |
| .venv_memos/bin/python services/memos_services.py download # Java 21, Neo4j 5.26.12, Qdrant 1.16.3 |
| .venv_memos/bin/python services/memos_services.py model # the MemOS reranker at a pinned revision |
| .venv_memos/bin/python services/memos_services.py run # supervises Neo4j, Qdrant and the reranker on loopback |
| ``` |
| |
| Run them with the memos interpreter, or install `requirements/services.txt` into a separate Python 3.11 environment. |
| `--work-root` and `--local-state-root` set where the services and their state live (`--help`); the Neo4j password |
| is generated on first use into `<work-root>/config/neo4j_password.txt`. |
|
|
| ## Environment variables |
|
|
| | variable | default | effect | |
| |---|---|---| |
| | `OPENAI_API_KEY` | | API key of every endpoint whose `api_key_env` in `backbone.yaml` is not set otherwise | |
| | `OPENAI_BASE_URL` | the client's default | endpoint of every client without an explicit `base_url` | |
| | `MEMEBENCH_MEMOS_NEO4J_URI`, `MEMEBENCH_MEMOS_QDRANT_URL`, `MEMEBENCH_MEMOS_RERANKER_URL` | the loopback services | memos service addresses, used when the value in `methods.memos.yaml` is null | |
| | `MEMEBENCH_MEMOS_NEO4J_PASSWORD` | | Neo4j password, read before `neo4j_password_file` | |
|
|
| Timeouts and retry counts are the constants at the top of `memebench/llm_client.py`. |
|
|
| ## Outputs |
|
|
| `outputs/<method>_<answer_model>/<patient>/<timestamp>/`: |
|
|
| | file | content | |
| |---|---| |
| | `answers.jsonl` | one row per question: qid, answer, verdict, pass, retrieved visits, `memory_preview` (the MEMORY block), error fields | |
| | `answers_partial.jsonl` | answered questions not yet graded; exists while a run is in progress | |
| | `report.json` | totals, accuracy, `by_axis`, `by_answer_type`, `by_grounding`, method parameters, models, `memorize_wrapper` (true when memory is built by an LLM), `memorize_failures` and `consolidate_failures` (visits and checkpoints whose memory construction failed; a run with any is reported as not clean and the command exits with status 1) | |
| | `run_settings.json` | the settings a resume must match: method, patient, method parameters, models and answering settings | |
| | `memory_build.jsonl` | one row per visit: memorize outcome and timing | |
| | `memory_trace.jsonl` | the method's memory events | |
| | `memory_snapshots/` | for methods that export their memory: the store at each checkpoint (`ckpt_vNNNN`) and at the end (`final`) | |
| | `store/` | the method's working store | |
| | `usage.jsonl` | token usage per call, tagged memorize / query / judge | |
|
|
| `latest` points at the newest completed run of a patient. hipporag builds its graph at the first question after new |
| visits, so its `memory_snapshots/ckpt_vNNNN` holds the store as of the previous checkpoint. |
|
|
