File size: 4,338 Bytes
85b19cf
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
# 实验记录

## 实验环境

- **数据**:VisTrajQA VAB smoke(1 sample, vab_minecraft, 30 步, 5 sessions, 45 QA)
- **转换模式**:text-only(image caption 内联到 user turn 文本)
- **Judge / Answer 模型**:从 `.env` 读取 `OPENAI_MODEL`
- **日期**:2026-04-15

---

## 总表

### Memory 维度

| Baseline | Recall ↑ | Update Recall ↑ | Correctness ↑ | Hallucination ↓ | Irrelevant ↓ | Update Score ↑ |
|----------|----------|-----------------|---------------|-----------------|--------------|----------------|
| Dummy | 95.0% | 90.0% | 50.0% | 5.0% | 45.0% | 71.4% |
| FUMemory | 95.0% | 90.0% | 80.0% | 10.0% | 10.0% | 69.0% |
| STMemory | 95.0% | 90.0% | 83.3% | 6.7% | 10.0% | 71.4% |
| LTMemory | 95.0% | 90.0% | 83.3% | 6.7% | 10.0% | 69.0% |
| GAMemory | 92.1% | 90.0% | 62.5% | 22.5% | 15.0% | 71.4% |
| MGMemory | 95.0% | 90.0% | 83.3% | 6.7% | 10.0% | 71.4% |
| RFMemory | 95.0% | 90.0% | 90.0% | 6.7% | 3.3% | 71.4% |
| MMMemory | 95.0% | 90.0% | 83.3% | 6.7% | 10.0% | 71.4% |
| MMFUMemory | 95.0% | 90.0% | 83.3% | 6.7% | 10.0% | 71.4% |
| NGMemory | 95.0% | 90.0% | 83.3% | 6.7% | 10.0% | 71.4% |
| AUGUSTUSMemory | 95.0% | 90.0% | 83.3% | 6.7% | 10.0% | 71.4% |
| UniversalRAGMemory | 95.0% | 90.0% | 80.0% | 10.0% | 10.0% | 71.4% |
| Mem0 | 47.9% | 56.7% | 57.3% | 18.0% | 24.7% | 81.0% |
| Mem0-Graph | 48.2% | 56.7% | 37.9% | 14.6% | 38.1% | 71.4% |
| SimpleMem | 95.0% | 90.0% | 48.3% | 5.0% | 46.7% | 71.4% |
| Omni-SimpleMem | — | — | — | — | — | — |
| MemVerse | 95.0% | 90.0% | 65.0% | 23.3% | 11.7% | 71.4% |
| Zep | — | — | — | — | — | — |
| A-Mem | 95.0% | 90.0% | 56.7% | 5.0% | 38.3% | 71.4% |
| MemoryOS | 47.1% | 41.0% | 33.8% | 3.5% | 62.8% | 50.0% |

### QA 维度

| Baseline | QA Correct ↑ | QA Hallucination ↓ | QA Omission ↓ | Evidence Coverage ↑ |
|----------|--------------|---------------------|---------------|---------------------|
| Dummy | 44.4% | 24.4% | 31.1% | 47.6% |
| FUMemory | 57.8% | 40.0% | 2.2% | 65.0% |
| STMemory | 31.1% | 33.3% | 35.6% | 32.0% |
| LTMemory | 40.0% | 26.7% | 33.3% | 51.5% |
| GAMemory | 37.8% | 20.0% | 42.2% | 51.5% |
| MGMemory | 60.0% | 35.6% | 4.4% | 68.9% |
| RFMemory | 60.0% | 37.8% | 2.2% | 63.1% |
| MMMemory | 57.8% | 40.0% | 2.2% | 66.0% |
| MMFUMemory | 60.0% | 37.8% | 2.2% | 67.0% |
| NGMemory | 62.2% | 24.4% | 13.3% | 71.8% |
| AUGUSTUSMemory | 64.4% | 22.2% | 13.3% | 68.9% |
| UniversalRAGMemory | 40.0% | 28.9% | 31.1% | 49.5% |
| Mem0 | 26.7% | 28.9% | 44.4% | 31.1% |
| Mem0-Graph | 31.1% | 20.0% | 48.9% | 32.0% |
| SimpleMem | 57.8% | 22.2% | 20.0% | 35.9% |
| Omni-SimpleMem | — | — | — | — |
| MemVerse | 40.0% | 28.9% | 31.1% | 40.8% |
| Zep | — | — | — | — |
| A-Mem | 46.7% | 20.0% | 33.3% | 50.5% |
| MemoryOS | 28.9% | 24.4% | 46.7% | 37.9% |

### QA 分类型正确率

| Baseline | FR | DU | MB | TR | KR | VFR | VS | VU | CMR |
|----------|----|----|----|----|----|----|----|----|-----|
| Dummy | 5/5 | 0/5 | 4/5 | 2/5 | 2/5 | 4/5 | 0/5 | 1/5 | 2/5 |
| FUMemory | 4/5 | 3/5 | 5/5 | 4/5 | 4/5 | 4/5 | 0/5 | 2/5 | 0/5 |
| STMemory | 0/5 | 0/5 | 5/5 | 1/5 | 3/5 | 2/5 | 0/5 | 0/5 | 3/5 |
| LTMemory | 2/5 | 1/5 | 5/5 | 1/5 | 3/5 | 4/5 | 0/5 | 0/5 | 2/5 |
| GAMemory | 4/5 | 0/5 | 5/5 | 0/5 | 2/5 | 4/5 | 0/5 | 0/5 | 2/5 |
| MGMemory | 4/5 | 3/5 | 5/5 | 4/5 | 4/5 | 4/5 | 0/5 | 1/5 | 2/5 |
| RFMemory | 4/5 | 3/5 | 5/5 | 4/5 | 4/5 | 4/5 | 0/5 | 1/5 | 2/5 |
| MMMemory | 4/5 | 3/5 | 5/5 | 4/5 | 4/5 | 4/5 | 0/5 | 2/5 | 0/5 |
| MMFUMemory | 4/5 | 3/5 | 5/5 | 4/5 | 4/5 | 4/5 | 0/5 | 2/5 | 1/5 |
| NGMemory | 5/5 | 3/5 | 5/5 | 3/5 | 3/5 | 4/5 | 0/5 | 2/5 | 3/5 |
| AUGUSTUSMemory | 5/5 | 2/5 | 5/5 | 4/5 | 3/5 | 4/5 | 1/5 | 2/5 | 3/5 |
| UniversalRAGMemory | 2/5 | 1/5 | 5/5 | 1/5 | 3/5 | 4/5 | 0/5 | 0/5 | 2/5 |
| Mem0 | 2/5 | 1/5 | 5/5 | 0/5 | 0/5 | 2/5 | 0/5 | 0/5 | 2/5 |
| Mem0-Graph | 2/5 | 0/5 | 5/5 | 0/5 | 1/5 | 3/5 | 0/5 | 0/5 | 3/5 |
| SimpleMem | 5/5 | 3/5 | 4/5 | 3/5 | 2/5 | 2/5 | 0/5 | 3/5 | 4/5 |
| Omni-SimpleMem | — | — | — | — | — | — | — | — | — |
| MemVerse | 5/5 | 0/5 | 5/5 | 0/5 | 2/5 | 3/5 | 0/5 | 0/5 | 3/5 |
| Zep | — | — | — | — | — | — | — | — | — |
| A-Mem | 3/5 | 3/5 | 5/5 | 0/5 | 2/5 | 4/5 | 0/5 | 0/5 | 4/5 |
| MemoryOS | 4/5 | 1/5 | 5/5 | 0/5 | 0/5 | 1/5 | 0/5 | 0/5 | 2/5 |