Feather DB on LongMemEval: embedded retrieval beats full-context GPT-4o for $2.40
We benchmarked Feather DB — our embedded vector database — against LongMemEval, the ICLR 2025 standard for long-term agent memory.
TL;DR: 0.693 on the S variant. GPT-4o full-context baseline: 0.640. Full run with Gemini-2.5-Flash: $2.40.
What is LongMemEval?
LongMemEval (Wu et al., ICLR 2025) tests long-term memory in chat assistants. 500 questions paired with conversation histories up to 115,000 tokens. Five axes: information-extraction, multi-session reasoning, temporal reasoning, knowledge-update, and abstention.
The benchmark is designed to be unforgiving. There are no hints about which session contains the answer. Sloppy retrieval fails badly on temporal and multi-session axes.
Our approach
Feather DB is an embedded vector database — it runs inside your Python process as a single .feather file. No managed service. No server to provision.
For LongMemEval we store each conversation turn as a dense vector (Azure text-embedding-3-small) alongside a BM25 inverted index. At retrieval time:
- Run hybrid BM25 + dense search, fused via Reciprocal Rank Fusion (k=60)
- Return top 10 chunks
- Apply adaptive temporal decay (half_life=14d, time_weight=0.4)
Retrieval latency: p50 = 0.19ms. The entire cost budget lives in the LLM round-trip.
Results
| Configuration | Score | Cost |
|---|---|---|
| Feather + GPT-4o | 0.693 | ~$8 |
| Feather + Gemini-2.5-Flash | 0.657 | $2.40 |
| Full-context GPT-4o (paper baseline) | 0.640 | — |
Per-axis with GPT-4o:
| Axis | Score | Notes |
|---|---|---|
| Information-extraction | 0.942 | Strong — hybrid search retrieves single facts well |
| Knowledge-update | 0.714 | Flat across model classes — structural gap |
| Multi-session | 0.606 | Improves with context-graph traversal |
| Temporal | 0.477 | Weakest — needs timeline indexing (Phase 9) |
What the temporal score means
Questions like "what did I say before I changed jobs?" require event-sequence reasoning. Semantic similarity retrieval doesn't surface enough temporal signal. We know exactly what's needed: LLM-based atomic fact extraction at ingest time, building an explicit timestamped proposition index. That's Phase 9.
Reproduce
git clone https://github.com/feather-store/feather
pip install feather-db
python bench/run_longmemeval.py \
--embedder azure-3-small \
--answerer gemini-2.5-flash \
--decay half_life=14 time_weight=0.4
Raw audit JSONs: bench/results/. Every number above links to an auditable file.
Try it
pip install feather-db
MIT licensed. Docs and cloud waitlist at getfeather.store.