Yash030 commited on
Commit
dc238fd
·
1 Parent(s): bbea853

feat: dedup observations, fix dashboard buttons, add per-obs delete

Browse files

- Add SHA-256 content fingerprint dedup to folder_observe() — identical
text from the same (folder, agent) pair returns existing obs ID without
writing a duplicate. Per-pair threading.Lock prevents TOCTOU races.

- Add dedup_folder_observations() to clean up existing duplicates across
one or all (folder, agent) pairs, rebuilding the dedup index after.

- Add POST /agentmemory/folder/dedup REST endpoint (auth-gated).

- Add memory_dedup MCP tool (schema + dispatch).

- Viewer: fix Back/Delete-folder buttons to use data-action delegation
instead of fragile inline onclick with escaped string args.

- Viewer: add per-observation Delete button on each card in folder detail.

- Viewer: add Select-all checkbox + bulk Delete-selected with single
POST /forget call; cards removed from DOM without reload.

- Viewer: add one-click Dedup button in folder detail toolbar.

- Viewer: fix Tools tab TypeError — raw fetch() replaces apiFetchRaw()
so .text() is called on the Response object, not a parsed JS object.

.kiro/specs/dedup-and-dashboard-fixes/design.md ADDED
@@ -0,0 +1,150 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Design: Deduplication & Dashboard Fixes
2
+
3
+ ## 1. Root Cause Analysis
4
+
5
+ ### Duplicate observations
6
+ `folder_observe()` in `src/functions.py` calls `generate_id("fobs")` unconditionally and writes the result. There is no idempotency check. Agents that call `agent_observe` on every hook invocation with the same text (e.g. a status line logged on every file edit) produce hundreds of identical records.
7
+
8
+ ### Broken viewer buttons
9
+ - The folder detail view is rendered via `innerHTML` with inline `onclick="loadFolders()"` and `onclick="deleteFolderMemory(...)"`. These work in isolation but are brittle and inconsistent with the rest of the app's `data-action` delegation pattern.
10
+ - `deleteFolderMemory` calls `apiFetchRaw('/agentmemory/forget', {...})` — the path resolution in `apiFetchRaw` prepends `REST` correctly, but the `Content-Type` header is not set in the options object passed (only in `headers` sub-key), so Flask receives an empty body and `forget` gets `{}`.
11
+ - The Tools tab "Run" button calls `apiFetchRaw(...)` and then `await result.text()`. Because `apiFetchRaw` already calls `res.json()`, `result` is a plain JS object, not a `Response`, so `.text()` is not a function → `TypeError`.
12
+
13
+ ---
14
+
15
+ ## 2. Deduplication Design
16
+
17
+ ### 2a. Fingerprint scope
18
+ A new KV scope `mem:obs_dedup:{safe_path}:{agent_id}` stores:
19
+ - key = SHA-256 hex of `normalized_text` (lowercased, whitespace-collapsed, first 4000 chars)
20
+ - value = `{"obsId": "<first_obs_id>", "timestamp": "<ts>"}`
21
+
22
+ This is a O(1) lookup before any write.
23
+
24
+ ### 2b. Concurrency
25
+ A `threading.Lock` per `(folder_path, agent_id)` pair is held for the duration of the dedup-check + write window. A module-level `dict[str, Lock]` keyed by `f"{folder_path}:{agent_id}"` with a global meta-lock for dict access is sufficient for the single-process Flask/Werkzeug model.
26
+
27
+ ### 2c. `folder_observe` changes (functions.py)
28
+ ```
29
+ fingerprint = sha256(safe_text[:4000].strip().lower())
30
+ existing = kv.get(KV.obs_dedup(folder_path, agent_id), fingerprint)
31
+ if existing:
32
+ return {"observationId": existing["obsId"], "deduplicated": True}
33
+ # ... proceed with normal write ...
34
+ kv.set(KV.obs_dedup(folder_path, agent_id), fingerprint, {"obsId": obs_id, "timestamp": timestamp})
35
+ ```
36
+
37
+ ### 2d. New KV scope in `KV` class
38
+ ```python
39
+ @staticmethod
40
+ def obs_dedup(folder_path: str, agent_id: str) -> str:
41
+ safe_path = folder_path.replace("\\", "/").strip("/")
42
+ safe_agent = agent_id.strip()
43
+ return f"mem:obs_dedup:{safe_path}:{safe_agent}"
44
+ ```
45
+
46
+ Mirror in `src/storage/scopes.py`.
47
+
48
+ ### 2e. Cleanup endpoint `dedup_folder_observations()`
49
+ New function in `functions.py`:
50
+ 1. Load all obs for a (folder, agent) pair.
51
+ 2. Build fingerprint → list of obs sorted by timestamp asc.
52
+ 3. Keep the first, delete the rest via the existing `forget` path (which handles BM25/vector cleanup).
53
+ 4. Rebuild the dedup index from scratch for that pair.
54
+ 5. Return counts.
55
+
56
+ New route in `src/routes/observations.py`: `POST /agentmemory/folder/dedup`.
57
+
58
+ ---
59
+
60
+ ## 3. Viewer Dashboard Fixes
61
+
62
+ ### 3a. Observation cards — per-row delete button
63
+ In `loadFolderDetail`, each observation div gets:
64
+ ```html
65
+ <button class="btn btn-danger" style="..."
66
+ data-action="delete-obs"
67
+ data-obs-id="<id>"
68
+ data-folder-path="<fp>"
69
+ data-agent-id="<aid>">Delete</button>
70
+ ```
71
+ The global `document.addEventListener('click', ...)` handler gains a new branch:
72
+ ```js
73
+ if (action === 'delete-obs') {
74
+ var obsId = target.getAttribute('data-obs-id');
75
+ var fp = target.getAttribute('data-folder-path');
76
+ var aid = target.getAttribute('data-agent-id');
77
+ confirmDeleteObs(obsId, fp, aid);
78
+ }
79
+ ```
80
+ `confirmDeleteObs` shows a modal, then calls `apiPost('forget', {folderPath: fp, agentId: aid, observationIds: [obsId]})` and removes the card from the DOM.
81
+
82
+ ### 3b. Checkbox bulk delete
83
+ The folder detail toolbar gets a "Select all" checkbox (`id="obs-select-all"`) and a "Delete selected" button (`data-action="delete-obs-selected"`, disabled by default).
84
+
85
+ Each observation card wrapper gets `data-obs-id`, a checkbox `class="obs-checkbox"`.
86
+
87
+ `change` events on checkboxes update a `selectedObsIds` Set and toggle the button's disabled state.
88
+
89
+ ### 3c. Back button — data-action
90
+ Replace inline `onclick="loadFolders()"` with:
91
+ ```html
92
+ <button class="btn" data-action="back-to-folders">← Back</button>
93
+ ```
94
+ Handler in the global click dispatcher:
95
+ ```js
96
+ if (action === 'back-to-folders') { loadFolders(); return; }
97
+ ```
98
+
99
+ ### 3d. Delete folder button — data-action + data attributes
100
+ Replace inline `onclick="deleteFolderMemory(...)"` with:
101
+ ```html
102
+ <button class="btn btn-danger"
103
+ data-action="delete-folder"
104
+ data-folder-path="<fp>"
105
+ data-agent-id="<aid>">Delete folder memory</button>
106
+ ```
107
+ Handler:
108
+ ```js
109
+ if (action === 'delete-folder') {
110
+ var fp = target.getAttribute('data-folder-path');
111
+ var aid = target.getAttribute('data-agent-id');
112
+ confirmDeleteFolder(fp, aid);
113
+ }
114
+ ```
115
+ `confirmDeleteFolder` shows the modal, calls `apiPost('forget', {folderPath: fp, agentId: aid})`, then `loadFolders()`.
116
+
117
+ ### 3e. Tools tab run button — fix double-parse
118
+ Replace the `apiFetchRaw` call in the tool runner with a raw `fetch`:
119
+ ```js
120
+ var token = getViewerToken();
121
+ var headers = {'Content-Type': 'application/json'};
122
+ if (token) headers['Authorization'] = 'Bearer ' + token;
123
+ var rawRes = await fetch(REST + '/agentmemory/mcp/tools', {
124
+ method: 'POST',
125
+ headers: headers,
126
+ body: JSON.stringify({ name: name, arguments: args })
127
+ });
128
+ var text = await rawRes.text();
129
+ var ok = rawRes.ok;
130
+ ```
131
+
132
+ ---
133
+
134
+ ## 4. Files Changed
135
+
136
+ | File | Change |
137
+ |------|--------|
138
+ | `src/functions.py` | Add `KV.obs_dedup`, dedup lock dict, dedup check in `folder_observe`, new `dedup_folder_observations` function |
139
+ | `src/storage/scopes.py` | Mirror `KV.obs_dedup` |
140
+ | `src/routes/observations.py` | Add `POST /agentmemory/folder/dedup` route |
141
+ | `src/routes/mcp.py` | Add `memory_dedup` tool schema + dispatch |
142
+ | `src/viewer/index.html` | Fix all broken buttons, add per-obs delete, add bulk delete, fix tools tab |
143
+
144
+ ---
145
+
146
+ ## 5. Security Notes
147
+ - The dedup fingerprint is computed over already-stripped (private-data-removed) text, so secrets are not persisted in fingerprint keys.
148
+ - The cleanup endpoint is auth-gated identical to all other write endpoints.
149
+ - `data-` attribute values are HTML-escaped via the existing `esc()` function before injection into innerHTML — no XSS risk.
150
+ - The per-pair lock prevents TOCTOU races under concurrent agent writes.
.kiro/specs/dedup-and-dashboard-fixes/requirements.md ADDED
@@ -0,0 +1,56 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Requirements: Deduplication & Dashboard Fixes
2
+
3
+ ## Overview
4
+ Agents are producing duplicate observations in the folder memory store, causing memory bloat and potential hallucination. The viewer dashboard has broken navigation/delete buttons and lacks per-observation delete. This spec addresses the root cause and the UI gaps.
5
+
6
+ ---
7
+
8
+ ## Requirements
9
+
10
+ ### REQ-01: Observation Deduplication in `folder_observe`
11
+ - The system MUST compute a SHA-256 content fingerprint over `(folder_path, agent_id, text_normalized)` for every incoming observation.
12
+ - If an observation with the same fingerprint already exists in `KV.folder_obs(folder_path, agent_id)`, the system MUST return the existing observation ID without writing a duplicate.
13
+ - The fingerprint key stored in KV MUST use the scope `mem:obs_dedup:{folder_path}:{agent_id}` with key = fingerprint hex, value = existing obs_id.
14
+ - The dedup check MUST happen after normalization and private-data stripping but before writing to KV.
15
+ - Duplicate detection MUST be atomic: check + write must be protected against concurrent writes by a per-(folder, agent) lock.
16
+ - The response for a deduplicated observation MUST include `{"observationId": "<existing_id>", "deduplicated": true}`.
17
+
18
+ ### REQ-02: One-Shot Deduplication Cleanup Endpoint
19
+ - A new REST endpoint `POST /agentmemory/folder/dedup` MUST be added.
20
+ - It MUST accept `{"folderPath": "...", "agentId": "..."}` in the body.
21
+ - It MUST scan all observations for that pair, group by normalized text fingerprint, keep the earliest observation per group, delete all later duplicates using the existing `forget` path, and rebuild the dedup index.
22
+ - If `folderPath` and `agentId` are both omitted, it MUST run across ALL folder pairs.
23
+ - It MUST return `{"deduplicated": <count_removed>, "pairs_processed": <n>, "kept": <count_kept>}`.
24
+ - Auth check MUST be applied.
25
+
26
+ ### REQ-03: Per-Observation Delete in Folder Detail View
27
+ - The folder detail view in the viewer MUST show a "Delete" button on each observation card.
28
+ - Clicking delete on an observation MUST call `POST /agentmemory/forget` with `{"folderPath": "...", "agentId": "...", "observationIds": ["<id>"]}`.
29
+ - After deletion the observation card MUST be removed from the DOM without a full page reload.
30
+ - A confirmation modal MUST be shown before deletion (reuse the existing modal pattern).
31
+
32
+ ### REQ-04: Bulk Delete with Checkbox Selection
33
+ - Each observation card in the folder detail view MUST have a checkbox.
34
+ - A "Select all" checkbox MUST appear in the toolbar above the list.
35
+ - A "Delete selected (N)" button MUST appear in the toolbar, enabled only when ≥1 checkbox is checked.
36
+ - Clicking "Delete selected" MUST call `POST /agentmemory/forget` with all selected observation IDs in a single request.
37
+ - After bulk deletion all deleted cards MUST be removed from the DOM.
38
+
39
+ ### REQ-05: Fix Back Button in Folder Detail
40
+ - The "← Back" button MUST navigate back to the folder list without a full page reload.
41
+ - It MUST use the existing `data-action` event delegation pattern rather than inline `onclick`.
42
+
43
+ ### REQ-06: Fix Delete Folder Button
44
+ - The "Delete folder memory" button MUST use `data-action` event delegation.
45
+ - It MUST NOT use inline `onclick` with string-escaped arguments (brittle with special characters in paths).
46
+ - The folder path and agent ID MUST be stored in `data-` attributes on the button element.
47
+ - After successful deletion it MUST navigate back to the folder list.
48
+
49
+ ### REQ-07: Fix Tools Tab `apiFetchRaw` Double-Parse Bug
50
+ - The Tools tab "Run" button calls `apiFetchRaw()` then `.text()` on the result, but `apiFetchRaw` already calls `.json()` — this throws a TypeError.
51
+ - The fix MUST use the raw `fetch()` response directly in the Tools tab runner so `.text()` can be called on the `Response` object, not on a parsed JS object.
52
+
53
+ ### REQ-08: Add `POST /agentmemory/folder/dedup` to MCP Tools
54
+ - The dedup endpoint MUST be exposed as an MCP tool named `memory_dedup`.
55
+ - Schema: `{"folderPath": string (optional), "agentId": string (optional)}`.
56
+ - It MUST appear in `GET /agentmemory/mcp/tools` and be dispatchable via `POST /agentmemory/mcp/tools`.
.kiro/specs/dedup-and-dashboard-fixes/tasks.md ADDED
@@ -0,0 +1,95 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Tasks: Deduplication & Dashboard Fixes
2
+
3
+ ## Task 1: Add dedup KV scope and per-pair locking to functions.py
4
+ - [x] Add `KV.obs_dedup(folder_path, agent_id)` static method to the `KV` class in `src/functions.py`
5
+ - [x] Add module-level `_dedup_locks: Dict[str, threading.Lock] = {}` and `_dedup_locks_meta = threading.Lock()` in `src/functions.py`
6
+ - [x] Add helper `_get_dedup_lock(folder_path: str, agent_id: str) -> threading.Lock` in `src/functions.py`
7
+ - [x] Mirror `obs_dedup` scope in `src/storage/scopes.py`
8
+
9
+ ## Task 2: Add dedup check to `folder_observe` in functions.py
10
+ - [x] After `safe_text` is computed (post strip_private_data), compute fingerprint: `hashlib.sha256(safe_text[:4000].strip().lower().encode()).hexdigest()`
11
+ - [x] Acquire `_get_dedup_lock(folder_path, agent_id)` before the dedup check
12
+ - [x] Look up `kv.get(KV.obs_dedup(folder_path, agent_id), fingerprint)`
13
+ - [x] If found, release lock and return `{"observationId": existing["obsId"], "deduplicated": True}`
14
+ - [x] After the obs is written to KV (step 6 in the existing pipeline), write the dedup index entry: `kv.set(KV.obs_dedup(folder_path, agent_id), fingerprint, {"obsId": obs_id, "timestamp": timestamp})`
15
+ - [x] Release the lock after the dedup index write
16
+ - [x] Ensure the lock is always released (use try/finally)
17
+
18
+ ## Task 3: Add `dedup_folder_observations` function to functions.py
19
+ - [x] Define `def dedup_folder_observations(kv: StateKV, folder_path_raw: Optional[str], agent_id_raw: Optional[str]) -> Dict[str, Any]`
20
+ - [x] If both are None, iterate all pairs from `kv.list(KV.folders)`; otherwise validate and normalize the single pair
21
+ - [x] For each pair: load all obs, group by `sha256(obs["text"][:4000].strip().lower())`, keep earliest by timestamp, delete duplicates via `forget(kv, {"folderPath": fp, "agentId": aid, "observationIds": [ids_to_delete]})`
22
+ - [x] After deletion, rebuild the dedup index for the pair from scratch
23
+ - [x] Return `{"deduplicated": total_removed, "pairs_processed": n, "kept": total_kept}`
24
+
25
+ ## Task 4: Add `POST /agentmemory/folder/dedup` REST endpoint
26
+ - [x] Add route in `src/routes/observations.py`:
27
+ ```python
28
+ @observations_bp.route("/agentmemory/folder/dedup", methods=["POST"])
29
+ def api_folder_dedup():
30
+ ```
31
+ - [x] Apply `_check_auth()`
32
+ - [x] Parse `folderPath` and `agentId` from request body (both optional)
33
+ - [x] Call `functions.dedup_folder_observations(_get_kv(), folder_path, agent_id)`
34
+ - [x] Return result as JSON 200
35
+
36
+ ## Task 5: Add `memory_dedup` MCP tool
37
+ - [x] Add tool schema to `GET /agentmemory/mcp/tools` in `src/routes/mcp.py`:
38
+ ```python
39
+ {
40
+ "name": "memory_dedup",
41
+ "description": "Remove duplicate observations from a (folderPath, agentId) pair or all pairs.",
42
+ "inputSchema": {
43
+ "type": "object",
44
+ "properties": {
45
+ "folderPath": {"type": "string", "description": "Folder path to deduplicate (optional — omit for all)"},
46
+ "agentId": {"type": "string", "description": "Agent ID to deduplicate (optional — omit for all)"},
47
+ }
48
+ }
49
+ }
50
+ ```
51
+ - [x] Add dispatch branch in `POST /agentmemory/mcp/tools`:
52
+ ```python
53
+ elif name == "memory_dedup":
54
+ res = functions.dedup_folder_observations(kv, args.get("folderPath"), args.get("agentId"))
55
+ text_out = json.dumps(res, indent=2)
56
+ ```
57
+
58
+ ## Task 6: Fix viewer — Back button and Delete folder button (data-action pattern)
59
+ - [x] In `loadFolderDetail`, replace `onclick="loadFolders()"` on the Back button with `data-action="back-to-folders"`
60
+ - [x] Replace inline `onclick="deleteFolderMemory(...)"` on the Delete folder button with `data-action="delete-folder"` plus `data-folder-path` and `data-agent-id` attributes (values must go through `esc()`)
61
+ - [x] Add handler for `back-to-folders` in the global `document.addEventListener('click', ...)` dispatcher
62
+ - [x] Add handler for `delete-folder` in the dispatcher: reads data attributes, calls `confirmDeleteFolder(fp, aid)`
63
+ - [x] Implement `confirmDeleteFolder(fp, aid)`: shows modal with confirmation message, on confirm calls `apiPost('forget', {folderPath: fp, agentId: aid})` then `loadFolders()`
64
+ - [x] Remove the now-dead `deleteFolderMemory` function
65
+
66
+ ## Task 7: Add per-observation delete button to folder detail view
67
+ - [x] In `loadFolderDetail`, add to each observation card:
68
+ - A "Delete" button: `data-action="delete-obs"`, `data-obs-id`, `data-folder-path`, `data-agent-id` (all through `esc()`)
69
+ - [x] Add handler for `delete-obs` in the dispatcher: calls `confirmDeleteObs(obsId, fp, aid)`
70
+ - [x] Implement `confirmDeleteObs(obsId, fp, aid)`: shows modal, on confirm calls `apiPost('forget', {folderPath: fp, agentId: aid, observationIds: [obsId]})`, then removes the card element from the DOM
71
+
72
+ ## Task 8: Add bulk checkbox delete to folder detail view
73
+ - [x] Add a toolbar row above the observations list in `loadFolderDetail` with:
74
+ - "Select all" checkbox: `id="obs-select-all"`
75
+ - "Delete selected" button: `data-action="delete-obs-selected"`, disabled initially
76
+ - [x] Add a checkbox to each observation card wrapper: `class="obs-checkbox"`, `data-obs-id`
77
+ - [x] Wire `change` on `#obs-select-all` to toggle all checkboxes
78
+ - [x] Wire `change` on each `.obs-checkbox` to update a `selectedObsIds` Set and enable/disable the "Delete selected" button
79
+ - [x] Add handler for `delete-obs-selected` in the dispatcher: reads `selectedObsIds`, calls `apiPost('forget', {folderPath, agentId, observationIds: [...selectedObsIds]})`, removes all deleted cards from DOM
80
+
81
+ ## Task 9: Fix Tools tab double-parse bug
82
+ - [x] In `renderToolsTab`, replace the `apiFetchRaw` call inside the "Run" button handler with a direct `fetch()` call:
83
+ ```js
84
+ var token = getViewerToken();
85
+ var headers = {'Content-Type': 'application/json'};
86
+ if (token) headers['Authorization'] = 'Bearer ' + token;
87
+ var rawRes = await fetch(REST + '/agentmemory/mcp/tools', {
88
+ method: 'POST',
89
+ headers: headers,
90
+ body: JSON.stringify({ name: name, arguments: args })
91
+ });
92
+ var text = await rawRes.text();
93
+ var ok = rawRes.ok;
94
+ ```
95
+ - [x] Update history recording to use `ok` and `text` from the above
src/functions.py CHANGED
@@ -26,6 +26,8 @@ _embedding_provider = None
26
  _hybrid_search = HybridSearch(_bm25_index, _vector_index, None, None)
27
  _index_persistence = None
28
  _stream_broadcaster = None # Callable: (payload) -> None
 
 
29
 
30
  # KV scope registry — folder-based memory model
31
  class KV:
@@ -57,6 +59,16 @@ class KV:
57
  safe_agent = agent_id.strip()
58
  return f"mem:foldermeta:{safe_path}:{safe_agent}"
59
 
 
 
 
 
 
 
 
 
 
 
60
  # ---- Global / shared scopes (kept) ----
61
 
62
  # Long-term memories — unchanged from previous implementation.
@@ -225,6 +237,18 @@ def validate_agent_id(agent_id: str) -> str:
225
  return sanitized
226
 
227
 
 
 
 
 
 
 
 
 
 
 
 
 
228
  def auto_complete_old_active_sessions(kv: StateKV, current_session_id: str, project: Optional[str] = None, agent_id: Optional[str] = None) -> int:
229
  sessions = kv.list(KV.sessions)
230
  count = 0
@@ -1004,67 +1028,80 @@ def folder_observe(kv: StateKV, payload: Dict[str, Any]) -> Dict[str, Any]:
1004
  safe_text = strip_private_data(text_raw)
1005
  safe_text = safe_text[:4000]
1006
 
1007
- # 10. Enforce MAX_OBS_PER_FOLDER cap before writing (REQ-015)
1008
- max_obs = int(os.getenv("MAX_OBS_PER_FOLDER", "2000"))
1009
- if max_obs > 0:
1010
- existing_obs = kv.list(KV.folder_obs(folder_path, agent_id))
1011
- if len(existing_obs) >= max_obs:
1012
- raise ValueError(f"Folder observation limit reached ({max_obs})")
1013
-
1014
- # 4. Generate obs_id (REQ-010)
1015
- obs_id = generate_id("fobs")
1016
-
1017
- # 5. Determine optional fields
1018
- obs_type = payload.get("type")
1019
- if not obs_type:
1020
- # Use infer_type with no tool_name; pass "other" as hook_type
1021
- obs_type = infer_type(None, "other")
1022
-
1023
- title = payload.get("title")
1024
- if not title:
1025
- title = safe_text[:80]
1026
-
1027
- concepts = payload.get("concepts") or []
1028
- if not isinstance(concepts, list):
1029
- concepts = []
1030
-
1031
- files = payload.get("files")
1032
- if not isinstance(files, list):
1033
- # Try to extract file refs from the payload itself
1034
- files = extract_files(payload)
1035
-
1036
- raw_importance = payload.get("importance")
1037
- if raw_importance is None:
1038
- importance = 5
1039
- else:
1040
- try:
1041
- importance = max(1, min(10, int(raw_importance)))
1042
- except (TypeError, ValueError):
 
 
1043
  importance = 5
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1044
 
1045
- # 5. Build FolderObservation dict (REQ-003)
1046
- obs: Dict[str, Any] = {
1047
- "id": obs_id,
1048
- "folderPath": folder_path,
1049
- "agentId": agent_id,
1050
- "timestamp": timestamp,
1051
- "text": safe_text,
1052
- "type": obs_type,
1053
- "title": title,
1054
- "concepts": concepts,
1055
- "files": files,
1056
- "importance": importance,
1057
- }
1058
 
1059
- # 6. Write observation to KV (REQ-001)
1060
- obs_scope = KV.folder_obs(folder_path, agent_id)
1061
- kv.set(obs_scope, obs_id, obs)
 
 
1062
 
1063
- # Write coordinate lookup mapping
1064
- kv.set(KV.obs_lookup, obs_id, {
1065
- "folderPath": folder_path,
1066
- "agentId": agent_id,
1067
- })
1068
 
1069
  # 7. Upsert folder metadata (REQ-005)
1070
  meta_scope = KV.folder_meta(folder_path, agent_id)
@@ -1114,6 +1151,92 @@ def folder_observe(kv: StateKV, payload: Dict[str, Any]) -> Dict[str, Any]:
1114
  return {"observationId": obs_id}
1115
 
1116
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1117
  # =====================================================================
1118
  # Folder-Based Search (folder_search)
1119
  # =====================================================================
 
26
  _hybrid_search = HybridSearch(_bm25_index, _vector_index, None, None)
27
  _index_persistence = None
28
  _stream_broadcaster = None # Callable: (payload) -> None
29
+ _dedup_locks: Dict[str, threading.Lock] = {} # per-(folder, agent) write locks
30
+ _dedup_locks_meta = threading.Lock() # protects _dedup_locks dict itself
31
 
32
  # KV scope registry — folder-based memory model
33
  class KV:
 
59
  safe_agent = agent_id.strip()
60
  return f"mem:foldermeta:{safe_path}:{safe_agent}"
61
 
62
+ @staticmethod
63
+ def obs_dedup(folder_path: str, agent_id: str) -> str:
64
+ """Deduplication index scope for (folder, agent) pairs.
65
+ Key = SHA-256 fingerprint hex of normalized text.
66
+ Value = {"obsId": str, "timestamp": str}
67
+ """
68
+ safe_path = folder_path.replace("\\", "/").strip("/")
69
+ safe_agent = agent_id.strip()
70
+ return f"mem:obs_dedup:{safe_path}:{safe_agent}"
71
+
72
  # ---- Global / shared scopes (kept) ----
73
 
74
  # Long-term memories — unchanged from previous implementation.
 
237
  return sanitized
238
 
239
 
240
+ def _get_dedup_lock(folder_path: str, agent_id: str) -> threading.Lock:
241
+ """Return a per-(folder_path, agent_id) Lock, creating it if necessary.
242
+
243
+ Uses _dedup_locks_meta to protect concurrent creation of new lock entries.
244
+ """
245
+ key = f"{folder_path}:{agent_id}"
246
+ with _dedup_locks_meta:
247
+ if key not in _dedup_locks:
248
+ _dedup_locks[key] = threading.Lock()
249
+ return _dedup_locks[key]
250
+
251
+
252
  def auto_complete_old_active_sessions(kv: StateKV, current_session_id: str, project: Optional[str] = None, agent_id: Optional[str] = None) -> int:
253
  sessions = kv.list(KV.sessions)
254
  count = 0
 
1028
  safe_text = strip_private_data(text_raw)
1029
  safe_text = safe_text[:4000]
1030
 
1031
+ # 3a. Deduplication check compute fingerprint over normalized text (REQ-DEDUP)
1032
+ _dedup_fp = hashlib.sha256(safe_text[:4000].strip().lower().encode("utf-8")).hexdigest()
1033
+ _dedup_lock = _get_dedup_lock(folder_path, agent_id)
1034
+ _dedup_lock.acquire()
1035
+ try:
1036
+ _existing_dedup = kv.get(KV.obs_dedup(folder_path, agent_id), _dedup_fp)
1037
+ if _existing_dedup and isinstance(_existing_dedup, dict) and _existing_dedup.get("obsId"):
1038
+ return {"observationId": _existing_dedup["obsId"], "deduplicated": True}
1039
+
1040
+ # 10. Enforce MAX_OBS_PER_FOLDER cap before writing (REQ-015)
1041
+ max_obs = int(os.getenv("MAX_OBS_PER_FOLDER", "2000"))
1042
+ if max_obs > 0:
1043
+ existing_obs = kv.list(KV.folder_obs(folder_path, agent_id))
1044
+ if len(existing_obs) >= max_obs:
1045
+ raise ValueError(f"Folder observation limit reached ({max_obs})")
1046
+
1047
+ # 4. Generate obs_id (REQ-010)
1048
+ obs_id = generate_id("fobs")
1049
+
1050
+ # 5. Determine optional fields
1051
+ obs_type = payload.get("type")
1052
+ if not obs_type:
1053
+ obs_type = infer_type(None, "other")
1054
+
1055
+ title = payload.get("title")
1056
+ if not title:
1057
+ title = safe_text[:80]
1058
+
1059
+ concepts = payload.get("concepts") or []
1060
+ if not isinstance(concepts, list):
1061
+ concepts = []
1062
+
1063
+ files = payload.get("files")
1064
+ if not isinstance(files, list):
1065
+ files = extract_files(payload)
1066
+
1067
+ raw_importance = payload.get("importance")
1068
+ if raw_importance is None:
1069
  importance = 5
1070
+ else:
1071
+ try:
1072
+ importance = max(1, min(10, int(raw_importance)))
1073
+ except (TypeError, ValueError):
1074
+ importance = 5
1075
+
1076
+ # 5. Build FolderObservation dict (REQ-003)
1077
+ obs: Dict[str, Any] = {
1078
+ "id": obs_id,
1079
+ "folderPath": folder_path,
1080
+ "agentId": agent_id,
1081
+ "timestamp": timestamp,
1082
+ "text": safe_text,
1083
+ "type": obs_type,
1084
+ "title": title,
1085
+ "concepts": concepts,
1086
+ "files": files,
1087
+ "importance": importance,
1088
+ }
1089
 
1090
+ # 6. Write observation to KV (REQ-001)
1091
+ obs_scope = KV.folder_obs(folder_path, agent_id)
1092
+ kv.set(obs_scope, obs_id, obs)
 
 
 
 
 
 
 
 
 
 
1093
 
1094
+ # Write coordinate lookup mapping
1095
+ kv.set(KV.obs_lookup, obs_id, {
1096
+ "folderPath": folder_path,
1097
+ "agentId": agent_id,
1098
+ })
1099
 
1100
+ # Write dedup index entry — inside lock so check+write is atomic (REQ-DEDUP)
1101
+ kv.set(KV.obs_dedup(folder_path, agent_id), _dedup_fp, {"obsId": obs_id, "timestamp": timestamp})
1102
+
1103
+ finally:
1104
+ _dedup_lock.release()
1105
 
1106
  # 7. Upsert folder metadata (REQ-005)
1107
  meta_scope = KV.folder_meta(folder_path, agent_id)
 
1151
  return {"observationId": obs_id}
1152
 
1153
 
1154
+ # =====================================================================
1155
+ # Folder Deduplication (dedup_folder_observations)
1156
+ # =====================================================================
1157
+
1158
+ def dedup_folder_observations(
1159
+ kv: StateKV,
1160
+ folder_path_raw: Optional[str],
1161
+ agent_id_raw: Optional[str],
1162
+ ) -> Dict[str, Any]:
1163
+ """Remove duplicate observations from one or all (folder, agent) pairs.
1164
+
1165
+ For each pair, groups observations by SHA-256 fingerprint of their normalized
1166
+ text, keeps the earliest observation per group, and deletes the rest.
1167
+ Also rebuilds the dedup index for each processed pair.
1168
+
1169
+ Args:
1170
+ folder_path_raw: folder path to deduplicate; None = all pairs.
1171
+ agent_id_raw: agent ID to deduplicate; None = all pairs.
1172
+
1173
+ Returns:
1174
+ {"deduplicated": <count>, "pairs_processed": <n>, "kept": <count>}
1175
+ """
1176
+ # Determine which pairs to process
1177
+ if folder_path_raw and agent_id_raw:
1178
+ try:
1179
+ fp = normalize_folder_path(folder_path_raw)
1180
+ aid = validate_agent_id(agent_id_raw)
1181
+ except ValueError as exc:
1182
+ return {"success": False, "error": str(exc)}
1183
+ pairs = [{"folderPath": fp, "agentId": aid}]
1184
+ else:
1185
+ pairs = [
1186
+ {"folderPath": e.get("folderPath", ""), "agentId": e.get("agentId", "")}
1187
+ for e in kv.list(KV.folders)
1188
+ if e.get("folderPath") and e.get("agentId")
1189
+ ]
1190
+
1191
+ total_removed = 0
1192
+ total_kept = 0
1193
+
1194
+ for pair in pairs:
1195
+ fp = pair["folderPath"]
1196
+ aid = pair["agentId"]
1197
+ all_obs = kv.list(KV.folder_obs(fp, aid))
1198
+
1199
+ # Group by fingerprint, keeping earliest by timestamp
1200
+ fingerprint_map: Dict[str, Dict[str, Any]] = {}
1201
+ duplicates: List[str] = []
1202
+
1203
+ for obs in all_obs:
1204
+ text = obs.get("text") or ""
1205
+ fp_hash = hashlib.sha256(text[:4000].strip().lower().encode("utf-8")).hexdigest()
1206
+ if fp_hash not in fingerprint_map:
1207
+ fingerprint_map[fp_hash] = obs
1208
+ else:
1209
+ # Keep the one with the earlier timestamp
1210
+ existing_ts = fingerprint_map[fp_hash].get("timestamp", "")
1211
+ this_ts = obs.get("timestamp", "")
1212
+ if this_ts < existing_ts:
1213
+ # This one is older — demote the previously-kept one
1214
+ duplicates.append(fingerprint_map[fp_hash]["id"])
1215
+ fingerprint_map[fp_hash] = obs
1216
+ else:
1217
+ duplicates.append(obs["id"])
1218
+
1219
+ if duplicates:
1220
+ forget(kv, {"folderPath": fp, "agentId": aid, "observationIds": duplicates})
1221
+ total_removed += len(duplicates)
1222
+
1223
+ total_kept += len(fingerprint_map)
1224
+
1225
+ # Rebuild the dedup index for this pair from scratch
1226
+ dedup_scope = KV.obs_dedup(fp, aid)
1227
+ # Clear existing dedup entries by re-writing from surviving observations
1228
+ for fp_hash, obs in fingerprint_map.items():
1229
+ kv.set(dedup_scope, fp_hash, {"obsId": obs["id"], "timestamp": obs.get("timestamp", "")})
1230
+
1231
+ print(f"[dedup] Processed {len(pairs)} pair(s): removed {total_removed}, kept {total_kept}")
1232
+ return {
1233
+ "success": True,
1234
+ "deduplicated": total_removed,
1235
+ "pairs_processed": len(pairs),
1236
+ "kept": total_kept,
1237
+ }
1238
+
1239
+
1240
  # =====================================================================
1241
  # Folder-Based Search (folder_search)
1242
  # =====================================================================
src/routes/mcp.py CHANGED
@@ -217,6 +217,17 @@ def mcp_tools_list():
217
  },
218
  },
219
  },
 
 
 
 
 
 
 
 
 
 
 
220
  ]
221
  return jsonify({"tools": tools}), 200
222
 
@@ -424,6 +435,14 @@ def mcp_tools_call():
424
  )
425
  text_out = json.dumps(res, indent=2)
426
 
 
 
 
 
 
 
 
 
427
  else:
428
  return jsonify({"error": f"unknown tool: {name}"}), 400
429
 
 
217
  },
218
  },
219
  },
220
+ {
221
+ "name": "memory_dedup",
222
+ "description": "Remove duplicate observations from a (folderPath, agentId) pair or all pairs. Keeps the earliest observation per unique text fingerprint.",
223
+ "inputSchema": {
224
+ "type": "object",
225
+ "properties": {
226
+ "folderPath": {"type": "string", "description": "Folder path to deduplicate (optional — omit for all pairs)"},
227
+ "agentId": {"type": "string", "description": "Agent ID to deduplicate (optional — omit for all pairs)"},
228
+ },
229
+ },
230
+ },
231
  ]
232
  return jsonify({"tools": tools}), 200
233
 
 
435
  )
436
  text_out = json.dumps(res, indent=2)
437
 
438
+ elif name == "memory_dedup":
439
+ res = functions.dedup_folder_observations(
440
+ kv,
441
+ args.get("folderPath") or None,
442
+ args.get("agentId") or None,
443
+ )
444
+ text_out = json.dumps(res, indent=2)
445
+
446
  else:
447
  return jsonify({"error": f"unknown tool: {name}"}), 400
448
 
src/routes/observations.py CHANGED
@@ -238,6 +238,34 @@ def api_session_end():
238
  return jsonify({"success": True, "message": "Session model is now folder-based."}), 200
239
 
240
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
241
  # ---------------------------------------------------------------------------
242
  # GET /agentmemory/observations (legacy compat shim)
243
  # ---------------------------------------------------------------------------
 
238
  return jsonify({"success": True, "message": "Session model is now folder-based."}), 200
239
 
240
 
241
+ # ---------------------------------------------------------------------------
242
+ # GET /agentmemory/observations (legacy compat shim)
243
+ # ---------------------------------------------------------------------------
244
+
245
+ @observations_bp.route("/agentmemory/folder/dedup", methods=["POST"])
246
+ def api_folder_dedup():
247
+ """POST /agentmemory/folder/dedup — remove duplicate observations.
248
+
249
+ Body (both optional):
250
+ folderPath: str — deduplicate only this folder pair
251
+ agentId: str — deduplicate only this agent
252
+
253
+ If both are omitted all folder pairs are processed.
254
+ Returns: {"success": bool, "deduplicated": int, "pairs_processed": int, "kept": int}
255
+ """
256
+ auth_err = _check_auth()
257
+ if auth_err:
258
+ return auth_err
259
+ try:
260
+ body = request.get_json(force=True) or {}
261
+ folder_path = body.get("folderPath") or None
262
+ agent_id = body.get("agentId") or None
263
+ res = functions.dedup_folder_observations(_get_kv(), folder_path, agent_id)
264
+ return jsonify(res), 200
265
+ except Exception as e:
266
+ return jsonify({"error": str(e)}), 400
267
+
268
+
269
  # ---------------------------------------------------------------------------
270
  # GET /agentmemory/observations (legacy compat shim)
271
  # ---------------------------------------------------------------------------
src/storage/scopes.py CHANGED
@@ -31,6 +31,16 @@ class KV:
31
  safe_agent = agent_id.strip()
32
  return f"mem:foldermeta:{safe_path}:{safe_agent}"
33
 
 
 
 
 
 
 
 
 
 
 
34
  # ---- Global / shared scopes (kept) ----
35
 
36
  # Long-term memories — unchanged from previous implementation.
 
31
  safe_agent = agent_id.strip()
32
  return f"mem:foldermeta:{safe_path}:{safe_agent}"
33
 
34
+ @staticmethod
35
+ def obs_dedup(folder_path: str, agent_id: str) -> str:
36
+ """Deduplication index scope for (folder, agent) pairs.
37
+ Key = SHA-256 fingerprint hex of normalized text.
38
+ Value = {"obsId": str, "timestamp": str}
39
+ """
40
+ safe_path = folder_path.replace("\\", "/").strip("/")
41
+ safe_agent = agent_id.strip()
42
+ return f"mem:obs_dedup:{safe_path}:{safe_agent}"
43
+
44
  # ---- Global / shared scopes (kept) ----
45
 
46
  # Long-term memories — unchanged from previous implementation.
src/viewer/index.html CHANGED
@@ -1732,59 +1732,193 @@
1732
  });
1733
  }
1734
 
 
 
 
 
 
1735
  async function loadFolderDetail(folderPath, agentId) {
1736
  var el = document.getElementById('view-folders');
1737
  if (!el) return;
 
 
 
1738
  el.innerHTML = '<div style="padding:24px;"><div class="skeleton skeleton-card"></div><div class="skeleton skeleton-card"></div></div>';
1739
  try {
1740
- var params = 'folderPath=' + encodeURIComponent(folderPath) + '&agentId=' + encodeURIComponent(agentId);
1741
- var result = await apiFetchRaw('/agentmemory/folder/observations?' + params);
1742
  var obs = (result && result.observations) || [];
1743
 
1744
- var html = '<div style="padding:20px 24px;">';
1745
- html += '<div style="display:flex;align-items:center;gap:12px;margin-bottom:20px;">';
1746
- html += '<button onclick="loadFolders()" style="background:none;border:1px solid var(--border-light);padding:6px 14px;cursor:pointer;font-family:var(--font-ui);font-size:12px;border-radius:3px;">&larr; Back</button>';
 
 
1747
  html += '<span style="font-family:var(--font-mono);font-size:13px;color:var(--ink);">' + esc(folderPath) + '</span>';
1748
  html += '<span style="background:var(--bg-inset);padding:2px 8px;border-radius:3px;font-size:11px;font-weight:600;">' + esc(agentId) + '</span>';
1749
- html += '<span style="margin-left:auto;font-size:12px;color:var(--ink-muted);">' + obs.length + ' observations</span>';
 
 
1750
  html += '</div>';
1751
 
1752
  if (obs.length === 0) {
1753
- html += '<div class="empty-state"><p>No observations found.</p></div>';
1754
  } else {
 
 
 
 
 
 
 
 
 
 
1755
  obs.forEach(function(o) {
1756
  var ts = o.timestamp ? new Date(o.timestamp).toLocaleString() : '';
1757
  var typeColor = {'file_edit':'var(--blue)','command_run':'var(--orange)','search':'var(--green)','error':'var(--red)','conversation':'var(--purple)'}[o.type] || 'var(--ink-muted)';
1758
- html += '<div style="border:1px solid var(--border-light);border-radius:4px;padding:14px 16px;margin-bottom:10px;background:var(--bg);">';
1759
- html += '<div style="display:flex;align-items:center;gap:8px;margin-bottom:6px;">';
 
 
 
 
 
 
1760
  html += '<span style="font-size:10px;font-weight:700;text-transform:uppercase;letter-spacing:0.08em;color:' + typeColor + ';background:var(--bg-subtle);padding:2px 7px;border-radius:3px;">' + esc(o.type || 'other') + '</span>';
1761
  html += '<span style="font-size:11px;color:var(--ink-muted);font-family:var(--font-mono);">' + esc(ts) + '</span>';
 
 
1762
  html += '</div>';
1763
  if (o.title) html += '<div style="font-weight:600;font-size:14px;color:var(--ink);margin-bottom:4px;">' + esc(o.title) + '</div>';
1764
- if (o.text) html += '<div style="font-size:13px;color:var(--ink-secondary);line-height:1.5;white-space:pre-wrap;">' + esc(o.text.length > 300 ? o.text.slice(0,300)+'\u2026' : o.text) + '</div>';
 
1765
  html += '</div>';
1766
  });
 
1767
  }
1768
 
1769
- html += '<div style="margin-top:20px;padding-top:16px;border-top:1px solid var(--border-light);">';
1770
- html += '<button onclick="deleteFolderMemory(\'' + esc(folderPath).replace(/'/g,"\\'") + '\',\'' + esc(agentId).replace(/'/g,"\\'") + '\')" style="background:var(--accent);color:white;border:none;padding:8px 18px;cursor:pointer;font-family:var(--font-ui);font-size:13px;border-radius:3px;">Delete folder memory</button>';
1771
- html += '</div></div>';
1772
  el.innerHTML = html;
 
 
 
 
1773
  } catch(err) {
1774
- el.innerHTML = '<div style="padding:20px;"><button onclick="loadFolders()" style="background:none;border:1px solid var(--border-light);padding:6px 14px;cursor:pointer;">&larr; Back</button><div style="margin-top:16px;color:var(--accent);">Error: ' + esc(String(err)) + '</div></div>';
 
 
1775
  }
1776
  }
1777
 
1778
- async function deleteFolderMemory(folderPath, agentId) {
1779
- if (!confirm('Delete all observations for ' + folderPath + ' / ' + agentId + '?')) return;
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1780
  try {
1781
- await apiFetchRaw('/agentmemory/forget', { method: 'POST', body: JSON.stringify({ folderPath: folderPath, agentId: agentId }) });
1782
  loadFolders();
1783
  } catch(err) {
1784
  alert('Delete failed: ' + String(err));
1785
  }
1786
  }
1787
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1788
  async function apiFetchRaw(path, opts) {
1789
  opts = opts || {};
1790
  var headers = Object.assign({ 'Content-Type': 'application/json' }, opts.headers || {});
@@ -3416,6 +3550,56 @@
3416
  if (memoryId) confirmDeleteMemory(memoryId);
3417
  return;
3418
  }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3419
  if (action === 'save-viewer-token') {
3420
  var tokenInput = document.getElementById('viewer-auth-token');
3421
  var token = tokenInput ? tokenInput.value.trim() : '';
@@ -3701,12 +3885,18 @@
3701
  btn.textContent = 'Running...';
3702
  btn.disabled = true;
3703
  try {
3704
- var result = await apiFetchRaw('/agentmemory/mcp/tools', {
 
 
 
 
 
3705
  method: 'POST',
 
3706
  body: JSON.stringify({ name: name, arguments: args })
3707
  });
3708
- var text = await result.text();
3709
- var ok = result.ok;
3710
  respEl.textContent = text;
3711
  respEl.className = ok ? '' : 'error';
3712
  respEl.style.display = '';
 
1732
  });
1733
  }
1734
 
1735
+ // Per-folder-detail state for bulk selection
1736
+ var _folderDetailFp = '';
1737
+ var _folderDetailAid = '';
1738
+ var _selectedObsIds = new Set();
1739
+
1740
  async function loadFolderDetail(folderPath, agentId) {
1741
  var el = document.getElementById('view-folders');
1742
  if (!el) return;
1743
+ _folderDetailFp = folderPath;
1744
+ _folderDetailAid = agentId;
1745
+ _selectedObsIds = new Set();
1746
  el.innerHTML = '<div style="padding:24px;"><div class="skeleton skeleton-card"></div><div class="skeleton skeleton-card"></div></div>';
1747
  try {
1748
+ var qp = 'folderPath=' + encodeURIComponent(folderPath) + '&agentId=' + encodeURIComponent(agentId);
1749
+ var result = await apiFetchRaw('/agentmemory/folder/observations?' + qp);
1750
  var obs = (result && result.observations) || [];
1751
 
1752
+ var html = '<div id="folder-detail-root" style="padding:20px 24px;">';
1753
+
1754
+ // ── Header row ──────────────────────────────────────────────────
1755
+ html += '<div style="display:flex;align-items:center;gap:12px;margin-bottom:16px;flex-wrap:wrap;">';
1756
+ html += '<button class="btn" data-action="back-to-folders">&larr; Back</button>';
1757
  html += '<span style="font-family:var(--font-mono);font-size:13px;color:var(--ink);">' + esc(folderPath) + '</span>';
1758
  html += '<span style="background:var(--bg-inset);padding:2px 8px;border-radius:3px;font-size:11px;font-weight:600;">' + esc(agentId) + '</span>';
1759
+ html += '<span style="font-size:12px;color:var(--ink-muted);">' + obs.length + ' observation' + (obs.length !== 1 ? 's' : '') + '</span>';
1760
+ html += '<span style="flex:1;"></span>';
1761
+ html += '<button class="btn btn-danger" data-action="delete-folder" data-folder-path="' + esc(folderPath) + '" data-agent-id="' + esc(agentId) + '">Delete folder</button>';
1762
  html += '</div>';
1763
 
1764
  if (obs.length === 0) {
1765
+ html += '<div class="empty-state"><div class="empty-icon">&#128203;</div><p>No observations found.</p></div>';
1766
  } else {
1767
+ // ── Bulk-delete toolbar ─────────────────────────────────────────
1768
+ html += '<div id="obs-bulk-toolbar" style="display:flex;align-items:center;gap:10px;margin-bottom:14px;padding:10px 14px;background:var(--bg-alt);border:1px solid var(--border-light);">';
1769
+ html += '<label style="display:flex;align-items:center;gap:6px;cursor:pointer;font-family:var(--font-ui);font-size:12px;">';
1770
+ html += '<input type="checkbox" id="obs-select-all" style="accent-color:var(--ink);"> Select all</label>';
1771
+ html += '<button class="btn btn-danger" id="obs-delete-selected-btn" data-action="delete-obs-selected" disabled style="font-size:11px;padding:4px 12px;">Delete selected (0)</button>';
1772
+ html += '<button class="btn" data-action="run-dedup" data-folder-path="' + esc(folderPath) + '" data-agent-id="' + esc(agentId) + '" style="font-size:11px;padding:4px 12px;" title="Remove duplicate observations automatically">&#x2728; Dedup</button>';
1773
+ html += '</div>';
1774
+
1775
+ // ── Observation cards ───────────────────────────────────────────
1776
+ html += '<div id="obs-list">';
1777
  obs.forEach(function(o) {
1778
  var ts = o.timestamp ? new Date(o.timestamp).toLocaleString() : '';
1779
  var typeColor = {'file_edit':'var(--blue)','command_run':'var(--orange)','search':'var(--green)','error':'var(--red)','conversation':'var(--purple)'}[o.type] || 'var(--ink-muted)';
1780
+ html += '<div class="obs-detail-card" data-obs-id="' + esc(o.id) + '" style="border:1px solid var(--border-light);border-radius:4px;padding:12px 16px;margin-bottom:8px;background:var(--bg);display:flex;gap:10px;align-items:flex-start;">';
1781
+ // checkbox
1782
+ html += '<label style="padding-top:2px;cursor:pointer;flex-shrink:0;">';
1783
+ html += '<input type="checkbox" class="obs-checkbox" data-obs-id="' + esc(o.id) + '" style="accent-color:var(--ink);">';
1784
+ html += '</label>';
1785
+ // content
1786
+ html += '<div style="flex:1;min-width:0;">';
1787
+ html += '<div style="display:flex;align-items:center;gap:8px;margin-bottom:6px;flex-wrap:wrap;">';
1788
  html += '<span style="font-size:10px;font-weight:700;text-transform:uppercase;letter-spacing:0.08em;color:' + typeColor + ';background:var(--bg-subtle);padding:2px 7px;border-radius:3px;">' + esc(o.type || 'other') + '</span>';
1789
  html += '<span style="font-size:11px;color:var(--ink-muted);font-family:var(--font-mono);">' + esc(ts) + '</span>';
1790
+ html += '<span style="flex:1;"></span>';
1791
+ html += '<button class="btn btn-danger" data-action="delete-obs" data-obs-id="' + esc(o.id) + '" data-folder-path="' + esc(folderPath) + '" data-agent-id="' + esc(agentId) + '" style="font-size:9px;padding:2px 8px;">Delete</button>';
1792
  html += '</div>';
1793
  if (o.title) html += '<div style="font-weight:600;font-size:14px;color:var(--ink);margin-bottom:4px;">' + esc(o.title) + '</div>';
1794
+ if (o.text) html += '<div style="font-size:13px;color:var(--ink-secondary);line-height:1.5;white-space:pre-wrap;">' + esc(o.text.length > 300 ? o.text.slice(0, 300) + '\u2026' : o.text) + '</div>';
1795
+ html += '</div>';
1796
  html += '</div>';
1797
  });
1798
+ html += '</div>'; // #obs-list
1799
  }
1800
 
1801
+ html += '</div>'; // #folder-detail-root
 
 
1802
  el.innerHTML = html;
1803
+
1804
+ // Wire up checkbox logic after innerHTML is set
1805
+ _wireFolderDetailCheckboxes();
1806
+
1807
  } catch(err) {
1808
+ el.innerHTML = '<div style="padding:20px;">' +
1809
+ '<button class="btn" data-action="back-to-folders">&larr; Back</button>' +
1810
+ '<div style="margin-top:16px;color:var(--accent);">Error loading observations: ' + esc(String(err)) + '</div></div>';
1811
  }
1812
  }
1813
 
1814
+ function _wireFolderDetailCheckboxes() {
1815
+ var selectAll = document.getElementById('obs-select-all');
1816
+ var deleteBtn = document.getElementById('obs-delete-selected-btn');
1817
+ if (!selectAll || !deleteBtn) return;
1818
+
1819
+ function updateDeleteBtn() {
1820
+ deleteBtn.textContent = 'Delete selected (' + _selectedObsIds.size + ')';
1821
+ deleteBtn.disabled = _selectedObsIds.size === 0;
1822
+ }
1823
+
1824
+ selectAll.addEventListener('change', function() {
1825
+ document.querySelectorAll('.obs-checkbox').forEach(function(cb) {
1826
+ cb.checked = selectAll.checked;
1827
+ var oid = cb.getAttribute('data-obs-id');
1828
+ if (selectAll.checked) { _selectedObsIds.add(oid); }
1829
+ else { _selectedObsIds.delete(oid); }
1830
+ });
1831
+ updateDeleteBtn();
1832
+ });
1833
+
1834
+ document.querySelectorAll('.obs-checkbox').forEach(function(cb) {
1835
+ cb.addEventListener('change', function() {
1836
+ var oid = cb.getAttribute('data-obs-id');
1837
+ if (cb.checked) { _selectedObsIds.add(oid); }
1838
+ else { _selectedObsIds.delete(oid); }
1839
+ // Update select-all indeterminate state
1840
+ var total = document.querySelectorAll('.obs-checkbox').length;
1841
+ selectAll.indeterminate = _selectedObsIds.size > 0 && _selectedObsIds.size < total;
1842
+ selectAll.checked = _selectedObsIds.size === total;
1843
+ updateDeleteBtn();
1844
+ });
1845
+ });
1846
+ }
1847
+
1848
+ function confirmDeleteFolder(fp, aid) {
1849
+ var modal = document.getElementById('modal');
1850
+ var overlay = document.getElementById('modal-overlay');
1851
+ modal.innerHTML = '<h3>Delete Folder Memory</h3>' +
1852
+ '<p>Delete <strong>all observations</strong> for<br><code style="font-size:11px;">' + esc(fp) + '</code> / <code style="font-size:11px;">' + esc(aid) + '</code>?<br><br>This cannot be undone.</p>' +
1853
+ '<div class="modal-actions">' +
1854
+ '<button class="btn" data-action="close-modal">Cancel</button>' +
1855
+ '<button class="btn btn-danger" data-action="confirm-delete-folder" data-folder-path="' + esc(fp) + '" data-agent-id="' + esc(aid) + '">Delete all</button>' +
1856
+ '</div>';
1857
+ overlay.classList.add('open');
1858
+ }
1859
+
1860
+ async function execDeleteFolder(fp, aid) {
1861
+ closeModal();
1862
  try {
1863
+ await apiPost('forget', { folderPath: fp, agentId: aid });
1864
  loadFolders();
1865
  } catch(err) {
1866
  alert('Delete failed: ' + String(err));
1867
  }
1868
  }
1869
 
1870
+ function confirmDeleteObs(obsId, fp, aid) {
1871
+ var modal = document.getElementById('modal');
1872
+ var overlay = document.getElementById('modal-overlay');
1873
+ modal.innerHTML = '<h3>Delete Observation</h3>' +
1874
+ '<p>Delete this observation? This cannot be undone.</p>' +
1875
+ '<div class="modal-actions">' +
1876
+ '<button class="btn" data-action="close-modal">Cancel</button>' +
1877
+ '<button class="btn btn-danger" data-action="confirm-delete-obs" data-obs-id="' + esc(obsId) + '" data-folder-path="' + esc(fp) + '" data-agent-id="' + esc(aid) + '">Delete</button>' +
1878
+ '</div>';
1879
+ overlay.classList.add('open');
1880
+ }
1881
+
1882
+ async function execDeleteObs(obsId, fp, aid) {
1883
+ closeModal();
1884
+ try {
1885
+ await apiPost('forget', { folderPath: fp, agentId: aid, observationIds: [obsId] });
1886
+ // Remove card from DOM without reload
1887
+ var card = document.querySelector('.obs-detail-card[data-obs-id="' + obsId + '"]');
1888
+ if (card) card.remove();
1889
+ _selectedObsIds.delete(obsId);
1890
+ // Update count in header
1891
+ var remaining = document.querySelectorAll('.obs-detail-card').length;
1892
+ var countEl = document.querySelector('#folder-detail-root span[style*="ink-muted"]');
1893
+ if (countEl) countEl.textContent = remaining + ' observation' + (remaining !== 1 ? 's' : '');
1894
+ } catch(err) {
1895
+ alert('Delete failed: ' + String(err));
1896
+ }
1897
+ }
1898
+
1899
+ async function execDeleteObsSelected() {
1900
+ var ids = Array.from(_selectedObsIds);
1901
+ if (!ids.length) return;
1902
+ closeModal();
1903
+ try {
1904
+ await apiPost('forget', { folderPath: _folderDetailFp, agentId: _folderDetailAid, observationIds: ids });
1905
+ ids.forEach(function(oid) {
1906
+ var card = document.querySelector('.obs-detail-card[data-obs-id="' + oid + '"]');
1907
+ if (card) card.remove();
1908
+ _selectedObsIds.delete(oid);
1909
+ });
1910
+ var remaining = document.querySelectorAll('.obs-detail-card').length;
1911
+ var countEl = document.querySelector('#folder-detail-root span[style*="ink-muted"]');
1912
+ if (countEl) countEl.textContent = remaining + ' observation' + (remaining !== 1 ? 's' : '');
1913
+ var deleteBtn = document.getElementById('obs-delete-selected-btn');
1914
+ if (deleteBtn) { deleteBtn.textContent = 'Delete selected (0)'; deleteBtn.disabled = true; }
1915
+ var selectAll = document.getElementById('obs-select-all');
1916
+ if (selectAll) { selectAll.checked = false; selectAll.indeterminate = false; }
1917
+ } catch(err) {
1918
+ alert('Bulk delete failed: ' + String(err));
1919
+ }
1920
+ }
1921
+
1922
  async function apiFetchRaw(path, opts) {
1923
  opts = opts || {};
1924
  var headers = Object.assign({ 'Content-Type': 'application/json' }, opts.headers || {});
 
3550
  if (memoryId) confirmDeleteMemory(memoryId);
3551
  return;
3552
  }
3553
+ if (action === 'back-to-folders') {
3554
+ loadFolders();
3555
+ return;
3556
+ }
3557
+ if (action === 'delete-folder') {
3558
+ var fp = target.getAttribute('data-folder-path');
3559
+ var aid = target.getAttribute('data-agent-id');
3560
+ if (fp && aid) confirmDeleteFolder(fp, aid);
3561
+ return;
3562
+ }
3563
+ if (action === 'confirm-delete-folder') {
3564
+ var fp = target.getAttribute('data-folder-path');
3565
+ var aid = target.getAttribute('data-agent-id');
3566
+ if (fp && aid) execDeleteFolder(fp, aid);
3567
+ return;
3568
+ }
3569
+ if (action === 'delete-obs') {
3570
+ var obsId = target.getAttribute('data-obs-id');
3571
+ var fp = target.getAttribute('data-folder-path');
3572
+ var aid = target.getAttribute('data-agent-id');
3573
+ if (obsId && fp && aid) confirmDeleteObs(obsId, fp, aid);
3574
+ return;
3575
+ }
3576
+ if (action === 'confirm-delete-obs') {
3577
+ var obsId = target.getAttribute('data-obs-id');
3578
+ var fp = target.getAttribute('data-folder-path');
3579
+ var aid = target.getAttribute('data-agent-id');
3580
+ if (obsId && fp && aid) execDeleteObs(obsId, fp, aid);
3581
+ return;
3582
+ }
3583
+ if (action === 'delete-obs-selected') {
3584
+ if (_selectedObsIds.size > 0) execDeleteObsSelected();
3585
+ return;
3586
+ }
3587
+ if (action === 'run-dedup') {
3588
+ var fp = target.getAttribute('data-folder-path');
3589
+ var aid = target.getAttribute('data-agent-id');
3590
+ if (!fp || !aid) return;
3591
+ target.textContent = '⏳ Running…';
3592
+ target.disabled = true;
3593
+ apiPost('folder/dedup', { folderPath: fp, agentId: aid }).then(function(res) {
3594
+ var removed = res && res.deduplicated != null ? res.deduplicated : '?';
3595
+ target.textContent = '✓ Removed ' + removed + ' dupes';
3596
+ setTimeout(function() { loadFolderDetail(fp, aid); }, 1200);
3597
+ }).catch(function() {
3598
+ target.textContent = '✗ Failed';
3599
+ target.disabled = false;
3600
+ });
3601
+ return;
3602
+ }
3603
  if (action === 'save-viewer-token') {
3604
  var tokenInput = document.getElementById('viewer-auth-token');
3605
  var token = tokenInput ? tokenInput.value.trim() : '';
 
3885
  btn.textContent = 'Running...';
3886
  btn.disabled = true;
3887
  try {
3888
+ // Use raw fetch so we can call .text() on the Response — apiFetchRaw
3889
+ // already parses JSON and returns a plain object, not a Response.
3890
+ var _toolToken = getViewerToken();
3891
+ var _toolHeaders = { 'Content-Type': 'application/json' };
3892
+ if (_toolToken) _toolHeaders['Authorization'] = 'Bearer ' + _toolToken;
3893
+ var rawRes = await fetch(REST + '/agentmemory/mcp/tools', {
3894
  method: 'POST',
3895
+ headers: _toolHeaders,
3896
  body: JSON.stringify({ name: name, arguments: args })
3897
  });
3898
+ var text = await rawRes.text();
3899
+ var ok = rawRes.ok;
3900
  respEl.textContent = text;
3901
  respEl.className = ok ? '' : 'error';
3902
  respEl.style.display = '';