| # MEME-BENCH: code |
|
|
| Code for MEME-BENCH, a benchmark of long-term memory over longitudinal patient histories: the evaluation runner and |
| judge, the adapters of the long-context baseline and the eleven evaluated memory systems, the pipeline that |
| constructed the benchmark, and its prompts. The benchmark data (23 patients, 15,049 visit dialogues, 1,646 questions) |
| is released separately. |
|
|
| ## Contents |
|
|
| ``` |
| code/ |
| memebench/ evaluation runner, judge, method adapters, configs -> docs/EVALUATION.md |
| vendor/ licenses, patches, and the download script of the engines |
| services/ backing services for MemOS |
| construction/ the pipeline that built the benchmark (reference) -> construction/build_patient.sh |
| requirements/ pinned environments |
| prompts/ the prompts of the construction pipeline, the evaluation and the methods -> prompts/README.md |
| docs/ EVALUATION.md |
| ``` |
|
|
| ## Data |
|
|
| The evaluation reads the patient folders of the data release (`P01` ... `P23`, each with `dialogues.jsonl` and |
| `questions.jsonl`). Place them under `data/` next to `code/`: the default patient list, |
| `code/memebench/configs/patients.yaml`, reads `../data/P*`. `--patients-config` takes another such file, listing |
| other locations. |
|
|
| ## Quick start |
|
|
| ```bash |
| cd code |
| # set up the environment as in docs/EVALUATION.md (Environment) |
| export OPENAI_API_KEY=... |
| # set every <...> value in memebench/configs/backbone.yaml and max_context_tokens at the top of |
| # memebench/configs/methods.yaml |
| python -m memebench.run_eval --method bm25_rag --patient <patient> |
| ``` |
|
|
| Results are written to `code/outputs/<method>_<answer_model>/<patient>/`; see |
| [docs/EVALUATION.md](docs/EVALUATION.md). The construction pipeline (`code/construction/`) documents how the |
| benchmark was built; two of its inputs are not distributed, so it cannot be run end to end. |
|
|
| ## License |
|
|
| Apache-2.0 ([LICENSE](LICENSE)); the third-party engines keep their own licenses |
| ([THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md)). |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{memebench2026, |
| title = {MEME-BENCH: Long-Term Memory over Longitudinal Patient Histories}, |
| author = {Anonymous}, |
| note = {Under review}, |
| year = {2026} |
| } |
| ``` |
|
|
