docs/EXPERIMENTS.md

# 实验记录

## 实验环境

- **数据**：VisTrajQA VAB smoke（1 sample, vab_minecraft, 30 步, 5 sessions, 45 QA）
- **转换模式**：text-only（image caption 内联到 user turn 文本）
- **Judge / Answer 模型**：从 `.env` 读取 `OPENAI_MODEL`
- **日期**：2026-04-15

---

## 总表

### Memory 维度

| Baseline | Recall ↑ | Update Recall ↑ | Correctness ↑ | Hallucination ↓ | Irrelevant ↓ | Update Score ↑ |
|----------|----------|-----------------|---------------|-----------------|--------------|----------------|
| Dummy | 95.0% | 90.0% | 50.0% | 5.0% | 45.0% | 71.4% |
| FUMemory | 95.0% | 90.0% | 80.0% | 10.0% | 10.0% | 69.0% |
| STMemory | 95.0% | 90.0% | 83.3% | 6.7% | 10.0% | 71.4% |
| LTMemory | 95.0% | 90.0% | 83.3% | 6.7% | 10.0% | 69.0% |
| GAMemory | 92.1% | 90.0% | 62.5% | 22.5% | 15.0% | 71.4% |
| MGMemory | 95.0% | 90.0% | 83.3% | 6.7% | 10.0% | 71.4% |
| RFMemory | 95.0% | 90.0% | 90.0% | 6.7% | 3.3% | 71.4% |
| MMMemory | 95.0% | 90.0% | 83.3% | 6.7% | 10.0% | 71.4% |
| MMFUMemory | 95.0% | 90.0% | 83.3% | 6.7% | 10.0% | 71.4% |
| NGMemory | 95.0% | 90.0% | 83.3% | 6.7% | 10.0% | 71.4% |
| AUGUSTUSMemory | 95.0% | 90.0% | 83.3% | 6.7% | 10.0% | 71.4% |
| UniversalRAGMemory | 95.0% | 90.0% | 80.0% | 10.0% | 10.0% | 71.4% |
| Mem0 | 47.9% | 56.7% | 57.3% | 18.0% | 24.7% | 81.0% |
| Mem0-Graph | 48.2% | 56.7% | 37.9% | 14.6% | 38.1% | 71.4% |
| SimpleMem | 95.0% | 90.0% | 48.3% | 5.0% | 46.7% | 71.4% |
| Omni-SimpleMem | — | — | — | — | — | — |
| MemVerse | 95.0% | 90.0% | 65.0% | 23.3% | 11.7% | 71.4% |
| Zep | — | — | — | — | — | — |
| A-Mem | 95.0% | 90.0% | 56.7% | 5.0% | 38.3% | 71.4% |
| MemoryOS | 47.1% | 41.0% | 33.8% | 3.5% | 62.8% | 50.0% |

### QA 维度

| Baseline | QA Correct ↑ | QA Hallucination ↓ | QA Omission ↓ | Evidence Coverage ↑ |
|----------|--------------|---------------------|---------------|---------------------|
| Dummy | 44.4% | 24.4% | 31.1% | 47.6% |
| FUMemory | 57.8% | 40.0% | 2.2% | 65.0% |
| STMemory | 31.1% | 33.3% | 35.6% | 32.0% |
| LTMemory | 40.0% | 26.7% | 33.3% | 51.5% |
| GAMemory | 37.8% | 20.0% | 42.2% | 51.5% |
| MGMemory | 60.0% | 35.6% | 4.4% | 68.9% |
| RFMemory | 60.0% | 37.8% | 2.2% | 63.1% |
| MMMemory | 57.8% | 40.0% | 2.2% | 66.0% |
| MMFUMemory | 60.0% | 37.8% | 2.2% | 67.0% |
| NGMemory | 62.2% | 24.4% | 13.3% | 71.8% |
| AUGUSTUSMemory | 64.4% | 22.2% | 13.3% | 68.9% |
| UniversalRAGMemory | 40.0% | 28.9% | 31.1% | 49.5% |
| Mem0 | 26.7% | 28.9% | 44.4% | 31.1% |
| Mem0-Graph | 31.1% | 20.0% | 48.9% | 32.0% |
| SimpleMem | 57.8% | 22.2% | 20.0% | 35.9% |
| Omni-SimpleMem | — | — | — | — |
| MemVerse | 40.0% | 28.9% | 31.1% | 40.8% |
| Zep | — | — | — | — |
| A-Mem | 46.7% | 20.0% | 33.3% | 50.5% |
| MemoryOS | 28.9% | 24.4% | 46.7% | 37.9% |

### QA 分类型正确率

| Baseline | FR | DU | MB | TR | KR | VFR | VS | VU | CMR |
|----------|----|----|----|----|----|----|----|----|-----|
| Dummy | 5/5 | 0/5 | 4/5 | 2/5 | 2/5 | 4/5 | 0/5 | 1/5 | 2/5 |
| FUMemory | 4/5 | 3/5 | 5/5 | 4/5 | 4/5 | 4/5 | 0/5 | 2/5 | 0/5 |
| STMemory | 0/5 | 0/5 | 5/5 | 1/5 | 3/5 | 2/5 | 0/5 | 0/5 | 3/5 |
| LTMemory | 2/5 | 1/5 | 5/5 | 1/5 | 3/5 | 4/5 | 0/5 | 0/5 | 2/5 |
| GAMemory | 4/5 | 0/5 | 5/5 | 0/5 | 2/5 | 4/5 | 0/5 | 0/5 | 2/5 |
| MGMemory | 4/5 | 3/5 | 5/5 | 4/5 | 4/5 | 4/5 | 0/5 | 1/5 | 2/5 |
| RFMemory | 4/5 | 3/5 | 5/5 | 4/5 | 4/5 | 4/5 | 0/5 | 1/5 | 2/5 |
| MMMemory | 4/5 | 3/5 | 5/5 | 4/5 | 4/5 | 4/5 | 0/5 | 2/5 | 0/5 |
| MMFUMemory | 4/5 | 3/5 | 5/5 | 4/5 | 4/5 | 4/5 | 0/5 | 2/5 | 1/5 |
| NGMemory | 5/5 | 3/5 | 5/5 | 3/5 | 3/5 | 4/5 | 0/5 | 2/5 | 3/5 |
| AUGUSTUSMemory | 5/5 | 2/5 | 5/5 | 4/5 | 3/5 | 4/5 | 1/5 | 2/5 | 3/5 |
| UniversalRAGMemory | 2/5 | 1/5 | 5/5 | 1/5 | 3/5 | 4/5 | 0/5 | 0/5 | 2/5 |
| Mem0 | 2/5 | 1/5 | 5/5 | 0/5 | 0/5 | 2/5 | 0/5 | 0/5 | 2/5 |
| Mem0-Graph | 2/5 | 0/5 | 5/5 | 0/5 | 1/5 | 3/5 | 0/5 | 0/5 | 3/5 |
| SimpleMem | 5/5 | 3/5 | 4/5 | 3/5 | 2/5 | 2/5 | 0/5 | 3/5 | 4/5 |
| Omni-SimpleMem | — | — | — | — | — | — | — | — | — |
| MemVerse | 5/5 | 0/5 | 5/5 | 0/5 | 2/5 | 3/5 | 0/5 | 0/5 | 3/5 |
| Zep | — | — | — | — | — | — | — | — | — |
| A-Mem | 3/5 | 3/5 | 5/5 | 0/5 | 2/5 | 4/5 | 0/5 | 0/5 | 4/5 |
| MemoryOS | 4/5 | 1/5 | 5/5 | 0/5 | 0/5 | 1/5 | 0/5 | 0/5 | 2/5 |