code_release/README.md

MEME-BENCH: code

Code for MEME-BENCH, a benchmark of long-term memory over longitudinal patient histories: the evaluation runner and judge, the adapters of the long-context baseline and the eleven evaluated memory systems, the pipeline that constructed the benchmark, and its prompts. The benchmark data (23 patients, 15,049 visit dialogues, 1,646 questions) is released separately.

Contents

code/
  memebench/      evaluation runner, judge, method adapters, configs              -> docs/EVALUATION.md
  vendor/         licenses, patches, and the download script of the engines
  services/       backing services for MemOS
  construction/   the pipeline that built the benchmark (reference)              -> construction/build_patient.sh
  requirements/   pinned environments
prompts/     the prompts of the construction pipeline, the evaluation and the methods -> prompts/README.md
docs/        EVALUATION.md

Data

The evaluation reads the patient folders of the data release (P01 ... P23, each with dialogues.jsonl and questions.jsonl). Place them under data/ next to code/: the default patient list, code/memebench/configs/patients.yaml, reads ../data/P*. --patients-config takes another such file, listing other locations.

Quick start

cd code
# set up the environment as in docs/EVALUATION.md (Environment)
export OPENAI_API_KEY=...
# set every <...> value in memebench/configs/backbone.yaml and max_context_tokens at the top of
# memebench/configs/methods.yaml
python -m memebench.run_eval --method bm25_rag --patient <patient>

Results are written to code/outputs/<method>_<answer_model>/<patient>/; see docs/EVALUATION.md. The construction pipeline (code/construction/) documents how the benchmark was built; two of its inputs are not distributed, so it cannot be run end to end.

License

Apache-2.0 (LICENSE); the third-party engines keep their own licenses (THIRD_PARTY_NOTICES.md).

Citation

@misc{memebench2026,
  title  = {MEME-BENCH: Long-Term Memory over Longitudinal Patient Histories},
  author = {Anonymous},
  note   = {Under review},
  year   = {2026}
}