Evaluating Agent Memory Honestly: Benchmarks, Failure Modes, and an Open Interchange Format
Long-horizon agents fail in a way that benchmark leaderboards mostly don't capture: not because the model can't reason, but because what the system remembers is stale, contradictory, or silently wrong. This post is about taking agent memory seriously as an evaluation problem. We'll cover why memory is an active process rather than a retrieval index, what the current benchmarks (LoCoMo, LongMemEval, MemoryAgentBench) actually measure and where they fall short, the temporal-correctness bugs we shipped and fixed in our own open-source system, and why we've adopted an open interchange format so results — and memories themselves — are reproducible across tools.
We build Memanto, an MIT-licensed companion memory agent, and Moorcheh, the information-theoretic retrieval engine it uses under the hood. Both come up below as worked examples, but the methodology arguments stand on their own, and every harness discussed here is public and pluggable — the point is that you can run all of this against whatever memory system you prefer.
Memory is a process, not a store
The dominant abstraction in agent frameworks treats memory as passive storage: add(text) on the way in, search(query) on the way out, usually backed by an embedding index. This abstraction is convenient and quietly wrong, because it leaves nobody responsible for the epistemic state of the store.
Consider what actually has to happen for an agent's knowledge to remain true over weeks of operation. New observations must be assessed for significance (most turns contain nothing worth keeping). Incoming facts must be reconciled against existing ones — and when they conflict, resolution has to favor recency while preserving history, because "the user preferred Python until May 14, TypeScript after" is different information than "the user prefers TypeScript." Validity is temporal: facts have valid_from and often valid_until, and queries like as-of ("what did we believe on June 1?") and changed-since ("what's new since the last deploy?") need to be answerable. And forgetting is a feature: unmaintained stores converge on retrieving stale contradictions with high confidence.
None of this is retrieval. It's maintenance, and it's cognitive work. Our design conclusion — the central bet of Memanto — is that this work should be owned by a dedicated agent running alongside your task agents, rather than smeared across the task agent's prompt or delegated to a schema. The task agent stays on task; the companion watches the interaction stream, extracts, consolidates, resolves, and expires. Whether or not you use our implementation, we'd argue the architectural claim generalizes: if no process in your system is accountable for memory truth, your memory is an append-only log wearing a knowledge base costume.
What the benchmarks measure — and what they miss
Three datasets dominate agent-memory evaluation right now.
LoCoMo tests question-answering over very long multi-session conversations — the classic "did the system retain and retrieve the right fact from session 3 of 30" setting. LongMemEval broadens the ability coverage: information extraction, multi-session reasoning, knowledge updates, temporal reasoning, and abstention (knowing that you don't know). MemoryAgentBench pushes toward agentic settings, evaluating memory as part of an interactive loop rather than a static QA pass.
Our published results — 89.8% on LongMemEval and 87.1% on LoCoMo, methodology in our paper (arXiv:2604.22085) — are competitive, and we're proud of them. But the more useful contribution, we think, is the critique you can only make after living inside these benchmarks for a while:
Knowledge updates and temporal reasoning remain the discriminating categories. Most systems, ours included, score high on single-fact retrieval; the spread between systems concentrates almost entirely in questions where a fact changed and the system must prefer the current version, or reason about when something was true. If you're comparing memory systems, look at per-category breakdowns, not the headline aggregate — the aggregate is mostly measuring the easy categories.
Abstention is under-weighted everywhere. A memory system that confidently answers from a stale or wrong memory is worse than one that says "I don't have that," but aggregate accuracy treats a lucky stale hit as a win.
Static QA benchmarks can't see maintenance failures. A system can score well on LoCoMo while its consolidation pipeline slowly corrupts state in ways that only manifest after weeks of writes. Which brings us to the failure catalog.
A field guide to temporal-correctness bugs (from our own changelog)
In Memanto v0.2.8 we fixed a batch of temporal bugs, and we're documenting them here because we believe every team building memory hits these — usually silently. Treat this as a test checklist:
| Bug | Symptom | Why it's dangerous |
|---|---|---|
| Tag filters silently ignored on temporal queries | recall --as-of --tag billing returns plausible but unfiltered results |
No error, no empty set — just wrong scope. Invisible in demos. |
Date-only as-of resolves to midnight |
"as of June 1" excludes everything that happened on June 1 | Off-by-one-day errors in audit and debugging workflows |
| Delete-and-recreate produces duplicates | Two versions of one memory retrieved; ranking decides which "wins" | Non-deterministic answers depending on retrieval order |
| Naive vs. timezone-aware timestamp mixing | Session expiry fires early/late across timezones | State corruption that correlates with user geography |
If your evaluation harness doesn't include cases for these four, your temporal-reasoning score is measuring your happy path. Ours didn't, at first. Now it does.
Reproducibility, or: numbers in a README are advertisements
The memory space has a benchmark credibility problem — nearly every vendor cites LoCoMo or LongMemEval numbers, few publish runnable harnesses, and evaluation choices (retrieval depth, judge prompts, chunking) can swing results by many points. Our position: if the harness isn't public, the number is an ad. That standard applies to us too.
So the harnesses live in the repo under examples/benchmarks/, in two forms. The first is a live head-to-head — Memanto vs. Mem0 — with a dashboard and identical conditions on both sides. The second is a multi-adapter retrieval suite with pluggable adapters for Mem0, Letta, Supermemory, Cognee, Zep/Graphiti, and a plain vector baseline, run against LoCoMo and MemoryAgentBench. The adapter interface is deliberately thin, so adding a system you care about is an afternoon, not a research project:
class MemoryAdapter(Protocol):
def ingest(self, session: list[Turn]) -> None: ...
def query(self, question: str, as_of: datetime | None = None) -> Answer: ...
We'd genuinely welcome adversarial PRs — if you believe our harness disadvantages a particular system's idiomatic usage, the fix belongs in the repo, not in a reply thread. A benchmark the community has kicked holes in and patched is worth ten benchmarks a vendor curated.
A note on the retrieval layer, since HF readers will ask: Memanto's retrieval backend is Moorcheh, which approaches search from information theory (signal-processing foundations rather than pure embedding cosine similarity; details in arXiv:2601.11557). The relevant claim for this post is narrower than the full paper: deterministic recall matters for evaluation, because a retrieval layer that returns different candidates across runs makes memory benchmarks noisy in ways that get misattributed to the memory logic. If you're benchmarking your own stack, fix your retrieval seed and verify run-to-run determinism before trusting any ablation.
Open weights taught us this lesson already; open memory is next
The Hugging Face community spent years establishing that models you can't download, inspect, and reproduce are claims, not artifacts. We think agent memory is at the same fork. Today, most production memory lives in proprietary stores — closed graphs and hosted indexes with no meaningful export — which means the knowledge your agents accumulate is exactly as portable as a closed model checkpoint: not at all.
In June, Google published the Open Knowledge Format (OKF): agent knowledge as a directory of markdown files with YAML frontmatter, one file per memory, tool-specific extensions namespaced (ours travel under x_memanto). It's a minimal spec, and that's its virtue — it standardizes the interchange surface without dictating a memory taxonomy.
v0.2.8 makes it a native Memanto format: memory export --okf, continuous sync --okf, and migrate --okf for imports, with lossless round trips as the hard requirement. For this audience, the interesting consequences are scientific as much as practical. A memory state that's a folder of markdown is a shareable experimental artifact — you can publish the exact memory bundle behind a benchmark run the way you'd publish weights, and others can reproduce your evaluation from identical state. It's diffable, so consolidation behavior can be inspected as version history rather than inferred from outputs. And it's system-independent, so cross-tool comparisons can start from literally the same memories. We'd love to see memory bundles show up alongside datasets on the Hub; the format finally makes that a sensible thing to do.
Where we'd point the community
Three open problems we think this community is unusually well-positioned to attack. First, maintenance-aware benchmarks: evaluations that run for simulated weeks with interleaved writes, contradictions, and deletions, scoring the trajectory of state quality rather than end-state QA. Second, abstention-weighted scoring for memory tasks, so confident staleness is penalized properly. Third, standardized memory artifacts — OKF bundles as first-class evaluation inputs, so "we tested on the same memories" becomes as normal as "we tested on the same split."
Everything referenced here is open: Memanto is MIT (pip install memanto), the harnesses are in examples/benchmarks/, the papers are arXiv:2604.22085 (Memanto) and arXiv:2601.11557 (Moorcheh), docs at docs.memanto.ai. Fork the harness, break our numbers, send the PR. That's the whole point of doing this in the open.
