# MEME-BENCH: code

Code for MEME-BENCH, a benchmark of long-term memory over longitudinal patient histories: the evaluation runner and
judge, the adapters of the long-context baseline and the eleven evaluated memory systems, the pipeline that
constructed the benchmark, and its prompts. The benchmark data (23 patients, 15,049 visit dialogues, 1,646 questions)
is released separately.

## Contents

```
code/
  memebench/      evaluation runner, judge, method adapters, configs              -> docs/EVALUATION.md
  vendor/         licenses, patches, and the download script of the engines
  services/       backing services for MemOS
  construction/   the pipeline that built the benchmark (reference)              -> construction/build_patient.sh
  requirements/   pinned environments
prompts/     the prompts of the construction pipeline, the evaluation and the methods -> prompts/README.md
docs/        EVALUATION.md
```

## Data

The evaluation reads the patient folders of the data release (`P01` ... `P23`, each with `dialogues.jsonl` and
`questions.jsonl`). Place them under `data/` next to `code/`: the default patient list,
`code/memebench/configs/patients.yaml`, reads `../data/P*`. `--patients-config` takes another such file, listing
other locations.

## Quick start

```bash
cd code
# set up the environment as in docs/EVALUATION.md (Environment)
export OPENAI_API_KEY=...
# set every <...> value in memebench/configs/backbone.yaml and max_context_tokens at the top of
# memebench/configs/methods.yaml
python -m memebench.run_eval --method bm25_rag --patient <patient>
```

Results are written to `code/outputs/<method>_<answer_model>/<patient>/`; see
[docs/EVALUATION.md](docs/EVALUATION.md). The construction pipeline (`code/construction/`) documents how the
benchmark was built; two of its inputs are not distributed, so it cannot be run end to end.

## License

Apache-2.0 ([LICENSE](LICENSE)); the third-party engines keep their own licenses
([THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md)).

## Citation

```bibtex
@misc{memebench2026,
  title  = {MEME-BENCH: Long-Term Memory over Longitudinal Patient Histories},
  author = {Anonymous},
  note   = {Under review},
  year   = {2026}
}
```