| # F3 — opencoti-server (Go companion daemon) |
|
|
| > Parent: [../MASTER_PLAN.md](../MASTER_PLAN.md) |
| > Status: **planning** |
| > Owner: TBD |
| > Reference: `/shared/dev/claude-hooks` (the inspiration; mirror its |
| > goals, then go further) |
|
|
| ## Problem |
|
|
| `claude-hooks` is a Python daemon that wraps Claude Code with a hook |
| API and adds persistent memory, multi-session coordination, tool |
| glue, and a small zoo of integrations. It works very well *for Claude |
| Code*. opencoti has the same shape of needs — and a few extra |
| because opencoti has its own tier engine, its own local llamafile, |
| and its own session model. |
|
|
| We want a Go daemon that is to opencoti what claude-hooks is to |
| Claude Code, **but deeper** because we control opencoti's source. |
| Instead of bolting onto a hook API from outside, opencoti-server can |
| be a first-class peer of opencoti's runtime. |
|
|
| ## Goals |
|
|
| - **G1.** Single Go binary, `opencoti-server`, that exposes a small, |
| well-versioned API to opencoti. |
| - **G2.** Persistent memory (vector + KG) shared across opencoti |
| sessions on a host. |
| - **G3.** Multi-session coordination: opencoti CLI instances on the |
| same host see each other, share context where the user wants it. |
| - **G4.** Tool glue: a place to register external tools (search, |
| filesystem extensions, MCP-like) once and have them appear in all |
| opencoti sessions. |
| - **G5.** Event sink for tier-engine routing decisions (so we can |
| log, audit, and later learn from them). |
| - **G6.** Hook fan-out compatible with the existing claude-hooks |
| ecosystem where it makes sense — but not bound by it. |
|
|
| ## Non-goals (for now) |
|
|
| - Replacing claude-hooks. opencoti-server is opencoti-specific. |
| - Becoming a generic "MCP host". MCP integration is a feature, not |
| the architecture. |
| - Multi-host federation. Single-host first. |
|
|
| ## Design sketch |
|
|
| ### Where it lives |
|
|
| - `opencoti/server/` — top-level non-bun source root. Go module. |
| Putting it outside `packages/` keeps the bun workspace clean. |
| - `packages/opencoti-server-client/` — TypeScript client used by |
| opencoti to talk to the server. |
| - **Surgical hooks** in opencode session lifecycle (session start, |
| session end, turn complete, tool invocation, model call result): |
| each hook is a one-line "if a client is available, notify it". |
| Listed in `docs/protocols/UPSTREAM_SYNC.md`. |
|
|
| ### Transport |
|
|
| Unix domain socket on Linux/macOS, named pipe on Windows. Loopback |
| TCP as a fallback. HTTP/JSON wire format for simplicity (no gRPC |
| build dependency for the client). |
|
|
| **Daemon-launch policy (applies to opencoti-server and every |
| opencoti daemon)**: |
|
|
| - Listen port MUST come from opencoti's reserved |
| **47000-48000 range** — never 38000-39000 or 18790-18811 |
| (claude-hooks territory). Tentative reservation table to |
| keep adjacent ports clear of accidental collision: |
| - **47092** — `@opencoti/embedder` (F4) |
| - **47190** — opencoti-server daemon (proposed default; revisit |
| when F3 M1 ships) |
| - **47191** — opencoti-server dashboard (proposed default) |
| - First-run setup MUST ask the user explicitly where the |
| daemon should bind: `127.0.0.1` (default), all interfaces, |
| or a specific IP. The default port from the reservation |
| table is proposed; the user may override either field within |
| the 47000-48000 range. Setup writes the validated choice to |
| the user's config. |
| |
| ### Persistence |
|
|
| Two backends, picked per-deployment: |
|
|
| - **pgvector** — same as solidPC's `claude-hooks` setup. Best when a |
| Postgres is already on the host. |
| - **sqlite-vec** — single-file, no external dependency. Default for |
| fresh installs. |
|
|
| The backend choice is config; the memory API is identical. |
|
|
| ### Surface (illustrative) |
|
|
| ``` |
| POST /v1/sessions # register a session |
| POST /v1/sessions/:id/turns # record a turn |
| POST /v1/memory # store a memory (M2) |
| POST /v1/memory/search # vector recall (M2 — POST because the embedding is multi-KB) |
| POST /v1/memory/collections # collections CRUD (M2) |
| PUT /v1/memory/collections/:name/acl # per-session ACL (M2) |
| GET /v1/memory/count # count (M2) |
| POST /v1/memory/kg/entities # KG ops |
| POST /v1/memory/kg/relations |
| POST /v1/tier-events # log a tier-engine decision |
| GET /v1/tools # list registered tools |
| POST /v1/tools/:name/invoke # invoke (server-side glue) |
| GET /v1/healthz # M1; M2 extended with store_ok/store_path/embedding_dim/store_error |
| ``` |
|
|
| The server is **embedder-agnostic** at M2: clients pass vectors, |
| not text. The text→vector convenience (e.g., |
| `GET /v1/memory?q=<text>` proxying to the embedder daemon) is a |
| deferred follow-up — adding it requires a runtime dep on |
| `@opencoti/embedder` (F4 M5) and is most naturally landed once |
| the TS client (M3) demonstrates the convenience is wanted at |
| the HTTP layer vs done client-side. |
|
|
| API is **versioned** (`/v1`) and the wire schema is owned by |
| `packages/opencoti-server-client/` (so opencoti can rev independently |
| of the server binary, within compatible versions). |
|
|
| ### Mirroring claude-hooks goals |
|
|
| Same goals, mapped: |
|
|
| | claude-hooks | opencoti-server | |
| | --- | --- | |
| | `claude-hooks` recall hooks | implicit recall on `/v1/memory` query, surfaced by opencoti's prompt builder | |
| | `mcp__pgvector__*` MCP tools | `/v1/memory` (and a thin MCP shim if external MCP clients want it) | |
| | Stop hook auto-ingest | server records turn-complete events, applies same heuristics | |
| | `claude-hooks` companion tools (Episodic etc.) | first-class tools registered via `/v1/tools` | |
|
|
| ### Deeper integration than claude-hooks |
|
|
| Because we own opencoti, we can: |
|
|
| - Receive **tier-engine** decisions directly (provider chosen, |
| escalation reasons, cost) without scraping logs. |
| - Receive **diff and tool-output** events as structured payloads, not |
| parsed from a transcript. |
| - Push back: server can suggest a memory recall payload that |
| opencoti **injects directly** into the prompt builder, instead of |
| appending to a transcript. |
|
|
| ## Milestones |
|
|
| ### M1 — Server skeleton + healthz *(2026-05-23 — shipped)* |
|
|
| - Go module under `opencoti/server/` (module |
| `github.com/mann1x/opencoti/server`, Go 1.22). Single dep: |
| `golang.org/x/sys` for the Windows `svc` packages. |
| - `opencoti-server serve` binds a UDS by default on Unix |
| (`$XDG_RUNTIME_DIR/opencoti/server.sock` or |
| `~/.opencoti/server.sock` fallback) or `tcp://127.0.0.1:47190` |
| on Windows. `--addr unix:///path` and `--addr tcp://host:port` |
| override. |
| - One HTTP endpoint: `GET /v1/healthz` returns 200 + JSON |
| `{status, version, started_at, uptime_seconds}`. All other |
| paths return a 404 envelope with code + message so the wire |
| format is consistent for M2+. `POST /v1/healthz` returns 405 |
| with the same envelope. |
| - Graceful shutdown on SIGTERM/SIGINT (Unix) or SCM |
| Stop/Shutdown (Windows). UDS file unlinked on Unix shutdown. |
| - **Windows service support shipped in M1 (per user directive |
| 2026-05-23):** |
| - `install` subcommand wraps SCM (`golang.org/x/sys/windows/svc/mgr`): |
| `--name`, `--display-name`, `--description`, |
| `--start-type=auto|manual|disabled`, `--addr`. Registers |
| the Event Log source under the same name. |
| - `uninstall` removes the SCM entry AND the Event Log source. |
| - `start` / `stop` wrap `mgr.Service.Start` / |
| `Service.Control(svc.Stop)`. `stop` waits up to 10s for the |
| state to become Stopped. |
| - When `svc.IsWindowsService()` returns true, `serve` enters |
| `svc.Run` with a handler that translates SCM Stop / |
| Shutdown into a graceful `http.Server.Shutdown`. Log |
| records flow to the Windows Application Event Log via |
| `eventlog.Open(name)`. |
| - Single source tree, build-tagged platform splits |
| (`*_windows.go` / `*_nonwindows.go`). Same binary surface |
| on every platform — `install` / `uninstall` / `start` / |
| `stop` on Unix print "Windows-only command" and exit 2. |
| - 7 cross-platform tests (parser + listener + healthz/404/405 |
| + UDS roundtrip + UDS unlink-on-cleanup + double-shutdown |
| safety) + 3 Windows-tagged tests for the `BuildMgrConfig` |
| install-config builder. All green on Linux; Windows |
| cross-compile clean. |
| - **Hook footprint: ZERO.** M1 is purely additive — no |
| `opencoti-hook:` markers, no UPSTREAM_SYNC.md registry rows. |
| The autostart hook lands in M3. |
| - **Deferred from M1:** Windows named-pipe TRANSPORT (use TCP |
| on Windows in M1; named-pipe support is a follow-up); |
| systemd/launchd unit files on Unix; CI workflows (the repo |
| has no CI yet — separate concern); the first-run setup |
| wizard step ("install as Windows service?" — comes when the |
| setup-flow picks up F3 concerns). |
| |
| ### M2 — Memory backend (sqlite-vec) + recall API *(2026-05-23 — shipped)* |
|
|
| - Go SQLite stack: `github.com/mattn/go-sqlite3` (cgo) + |
| `github.com/asg017/sqlite-vec-go-bindings/cgo`, sqlite-vec |
| statically linked via `sqlite_vec.Auto()` so the daemon ships |
| as a single binary (no `vec0.so` to bundle). CGO becomes a |
| hard build requirement at M2; cross-compile to Windows from |
| Linux needs MinGW-w64 (`apt install gcc-mingw-w64-x86-64`). |
| - `internal/store/store.go` defines a `Store` interface |
| mirroring the TS `MemoryStore` in |
| `packages/opencoti-memory/src/types.ts` 1:1. M7's pgvector |
| backend will implement the same interface, so the HTTP layer |
| doesn't change. |
| - `internal/store/sqlitevec/` is the M2 backend. DDL mirrors |
| `packages/opencoti-memory/src/schema.ts` byte-functionally: |
| four tables (`meta`, `collections`, `memories`, `session_acl`) |
| plus the `memory_vecs` vec0 virtual table; `SCHEMA_VERSION=1`; |
| `embedding_dim` parameter-substituted into the vec0 DDL at |
| first creation and stored in `meta` for validation on later |
| opens. Content idempotency: SHA256-hex with |
| `UNIQUE(collection_name, content_hash)`. Pure `ResolveAccessMode` |
| mirrors the TS resolver (global=r, session-owner=rw, |
| others=none; explicit `session_acl` rows override). Recall |
| is vec0 MATCH + overscan |
| (`k * min(8, max(2, len(candidates)))`) + join-back + |
| ACL filter, truncated to k. |
| - **DB schema is wire-compatible with the in-process |
| `@opencoti/memory` package** — a DB created by either side |
| opens cleanly under the other. This is the load-bearing |
| invariant for M3's path-B refactor. |
| - HTTP surface (eight new routes, all under `/v1/memory/*`): |
| - `POST /v1/memory/collections` — create (201 / 409 |
| collection_exists / 400 invalid_collection_name). |
| - `GET /v1/memory/collections?session_id=...` — list, |
| optionally filtered through the ACL resolver. |
| - `DELETE /v1/memory/collections/{name}` — cascade-deletes |
| memories + vec0 rows + session_acl entries. |
| - `PUT /v1/memory/collections/{name}/acl` — set per-session |
| ACL (`r`/`w`/`rw`/`none`). |
| - `GET /v1/memory/collections/{name}/acl?session_id=...` — |
| read the resolved access mode. |
| - `POST /v1/memory` — store. 422 dim_mismatch on length |
| mismatch; 403 write_denied on ACL deny. |
| - `POST /v1/memory/search` — vector recall (POST because |
| a 1024-element Float32 array doesn't fit a query string). |
| - `GET /v1/memory/count?collection=...` — count, optionally |
| scoped to one collection. |
| - All error responses use M1's `{error: {code, message}}` |
| envelope. Embeddings on the wire are JSON `[float32, ...]` |
| arrays (verbose but trivial for the M3 TS client; base64 raw |
| bytes is a follow-up if wire size measurably matters). |
| - `/v1/healthz` extended with `store_ok`, `store_path`, |
| `embedding_dim`, and `store_error` (omitempty). When the |
| store fails to open, the daemon still serves `/v1/healthz` |
| with `store_ok=false` and `/v1/memory/*` returns 503 |
| `store_unavailable` — partial degradation beats refuse-to-start |
| so the daemon stays observable when something's wrong with the DB. |
| - `serve` gets `--db-path` (default |
| `$HOME/.opencoti/memory/state.db` — matches the TS |
| `defaultDbPath()` exactly so both implementations point at the |
| same file by default) and `--embedding-dim` (default 1024, |
| matches `DEFAULT_EMBEDDING_DIM`). `install` bakes both into |
| the SCM service-start arguments. |
| - 30 tests cover the surface: 18 unit tests on the store |
| conformance (collection lifecycle, ACL resolver, idempotent |
| store, recall ordering, count, close+reopen, dim-mismatch |
| paths) + 12 HTTP integration tests via `httptest.NewServer` |
| (every endpoint, happy + 4xx + 5xx envelopes). |
| - `testdata/crosslang/` ships a Go↔Bun schema-compat probe: |
| Go writes a DB → optionally invokes a Bun script that opens |
| the same DB via `@opencoti/memory` and confirms readback. |
| Skips cleanly when bun or the workspace package isn't on |
| PATH. `make test-cross-lang` is always safe to run. |
| - **Hook footprint: ZERO** (unchanged from M1). M2 is purely |
| additive — the API exists but no surgical hooks into opencode |
| yet. |
| - **Deferred from M2:** text→vector convenience |
| (`GET /v1/memory?q=<text>`); streaming recall (SSE); |
| pagination on list-collections; authentication for TCP |
| transport (UDS owner-only on Unix is the M2 security |
| boundary); pgvector backend (M7); migration tools (schemas |
| are identical — the DB just opens); server-side embedder |
| cache; bulk-store / batch endpoints; base64 raw-byte |
| embedding wire format. |
| |
| ### M3 — TS client + opencoti hook to start the server *(2026-05-23 — shipped)* |
|
|
| - `@opencoti/opencoti-server-client` published in-workspace under |
| `packages/opencoti-server-client/`. Single class |
| `OpencotiServerClient` implements `MemoryStore` over HTTP — all |
| eight `/v1/memory/*` routes from M2 plus a `healthz()` probe. |
| Wire format is M2's verbatim: JSON `[float32, ...]` for embeddings; |
| M1/M2's `{error: {code, message}}` envelope decoded into a |
| sentinel `OpencotiServerError({status, code, message})` so callers |
| can match on `code === "store_unavailable"`, `dim_mismatch`, etc. |
| Per-request `AbortController` + 5 s default timeout. `close()` is |
| a documented no-op (no per-instance handle to close). |
| - **Transport** uses Bun's native `fetch` — `tcp://host:port` is |
| rewritten to `http://...` in `toFetchBase`; `unix:///path` passes |
| through verbatim (Bun supports UDS fetch natively, no custom |
| adapter needed). `defaultAddress(platform)` mirrors the Go |
| `DefaultAddress()` exactly: `$XDG_RUNTIME_DIR/opencoti/server.sock` |
| on Linux (`$HOME/.opencoti/server.sock` fallback), and |
| `tcp://127.0.0.1:47190` on Windows. |
| - **Autostart** lives in `src/autostart.ts`: |
| `startAutostart({address, binaryPath?, dbPath?, embeddingDim?, ...})` |
| probes `/v1/healthz` first (200 → `already_running`), locates the |
| binary via `Bun.which` if none was passed (→ `no_binary` when |
| missing), spawns with `stdio: "ignore"` (mirroring the |
| `@opencoti/embedder` manager pattern), and polls `/v1/healthz` |
| every 200 ms until the deadline (default 5 s; → |
| `ready_timeout` if never recovers). Returns a discriminated |
| `AutostartOutcome` so consumers can log five distinct states |
| (`already_running | spawned | no_binary | spawn_failed | ready_timeout`) |
| without exception handling. Test seams: `spawnImpl`, `fetchImpl`, |
| `whichImpl`, `sleepImpl`. |
| - **Surgical hook footprint: 3 markers + 1 package.json dep.** All |
| three TS markers land in `packages/opencode/src/config/config.ts` |
| (the same file that already carries `opencoti-default-plugins`): |
| the import next to it, the `opencoti.server.*` schema struct |
| (`autostart: boolean`, `address: string`, `binary_path: string`, |
| `db_path: string`, `embedding_dim: PositiveInt`), and the |
| fire-and-forget `void maybeStartOpencotiServer(...)` call site |
| right after the existing `applyDefaultPlugins(...)` call. The |
| hook's full substance lives in |
| `@opencoti/opencoti-server-client/autostart-hook` so the |
| opencode-side surface stays minimal — three lines plus the |
| package.json dep (no anchor comment in JSON, registered in the |
| table). |
| - **Plugin swap.** Both `@opencoti/memory-plugin` and |
| `@opencoti/tui-memory` accept a new optional |
| `server_address?: string`. The default `StoreFactory`: |
| if `serverAddress` is set, instantiate `OpencotiServerClient`, |
| call `healthz()`, and return the client when |
| `storeOk === true`; otherwise silently fall through to the |
| existing in-process `SqliteVecStore.open(...)`. The |
| `__setStoreFactory` test seam is unchanged — existing tests |
| continue to inject stubs that ignore `serverAddress`. |
| - **Health envelope mismatch detection.** Client validates that |
| `embedding_dim` from the server matches the configured dim; |
| on mismatch, `healthz()` returns `storeOk: false` with a |
| `dim mismatch` error so the plugin falls back to the |
| in-process store rather than corrupting the wire format. |
| This is **client-side**; the server's own dim check still |
| fires on `POST /v1/memory`. |
| - **Tests.** 56 new tests in the client package: 33 mocked-fetch |
| client conformance (every endpoint, happy + 4xx + 5xx envelopes, |
| dedup, dim mismatch, store_unavailable propagation), 11 pure |
| tests for `defaultAddress` / `toFetchBase` / `joinURL` / |
| `probeReady`, 6 mocked-spawn `startAutostart` cases (already |
| running, no binary, spawned-then-ready, ready timeout, |
| spawn failed, --addr passthrough). Plus 2 new cases in the |
| memory-plugin tests and 3 new in the tui-memory tests for the |
| `server_address` swap path. All 23 workspace packages |
| typecheck clean; 82 tests pass across the three affected |
| packages. |
| - **Live verification.** Manual end-to-end smoke against the |
| F3 M2 binary (TCP loopback): `healthz` returns the expected |
| shape (`store_ok: true`, `embedding_dim: 64`), `createCollection` |
| + `setSessionAcl(rw)` + `store(...)` (with `deduplicated: |
| false` then `true` on idempotency) + `recall(...)` (distance |
| 0 on exact match) + `count(...)` (returns 1) + `deleteCollection` |
| all return the expected shapes. UDS path tested at the |
| discover / autostart layer; full UDS smoke deferred to |
| M4 when session-event hooks bring it under day-to-day use. |
| - **Path-B loop closed.** With M3 in, the F4 M5 |
| in-process-then-daemon migration path described in |
| [memory_embedder.md](memory_embedder.md) is mechanically |
| complete: setting `opencoti.server.address` (or `autostart: |
| true`) in `opencode.jsonc` is the only change a user makes |
| to switch from the in-process store to the Go daemon. No |
| code paths change in `@opencoti/memory` itself. |
| - **Deferred from M3:** detached / setsid daemon lifecycle |
| (M3's spawn dies with opencode — cross-session sharing |
| requires running the daemon externally, e.g. the M1 Windows |
| service installer or a future systemd unit); the first-run |
| "install as a service?" wizard step (a follow-up once the |
| setup-flow picks up F3 concerns); a text-`q=` convenience |
| endpoint (`GET /v1/memory?q=<text>`) — every M3 caller has |
| the embedder in-process and can pre-embed; retry / backoff |
| on transient HTTP errors (single attempt, the plugin's |
| silent fallback handles the failure case); bulk-store / |
| batch endpoints; TLS / shared-secret authentication for TCP |
| transport (UDS owner-only remains the M3 security boundary). |
|
|
| ### M4 — Session + turn events flowing *(2026-05-23 — shipped)* |
|
|
| **Design pivot vs the original spec.** M4 was originally specced |
| as *surgical hooks at session start/end/turn complete*. Phase-1 |
| exploration confirmed opencode's `@opencode-ai/plugin` API |
| already exposes those events fully typed via `Hooks.event` |
| (session.created/updated/deleted/error/idle plus |
| session.status). Following the precedent set by M5-D2 |
| (`@opencoti/memory-plugin`), M4 ships as a plugin with **zero |
| opencoti-hook footprint** — strictly richer than a surgical hook |
| for this use case (typed payloads, zero upstream-source touch, |
| no UPSTREAM_SYNC.md row to maintain on every sync). Hook count |
| stays at 9 source markers + 4 JSON deps (unchanged from F3 M3). |
| |
| - **Per-feature schema versioning.** `meta.schema_version` stays |
| at 1 (memory tables — kept stable so M2/M3 TypeScript |
| `@opencoti/memory` clients still open M4 DBs cleanly). New |
| `meta.sessions_schema_version = 1` is written by M4+ daemons |
| and reported on `/v1/healthz`. M3 clients without the key |
| treat the daemon as pre-M4; M4 plugins disable forwarding |
| silently when `sessionsSchemaVersion < 1`. |
| - **Three new tables** in the existing `state.db` (no new |
| `--flag`, no migration tool): |
| - `sessions` — soft-delete via `deleted_at`. Indices on |
| `parent_id` (fork-tree queries) and `deleted_at` |
| (cheap `WHERE deleted_at IS NULL`). Upsert is idempotent |
| on `id`; `INSERT ... ON CONFLICT(id) DO UPDATE SET ...` |
| overwrites all columns from the latest snapshot. |
| - `session_events` — append-only audit log. `FOREIGN KEY ... |
| ON DELETE CASCADE` would normally wipe history on |
| `DELETE FROM sessions`; we use soft-delete instead so the |
| audit trail outlives the session row. |
| - `session_messages` — UPSERT on opencode `MessageID`. The |
| plugin re-sends the full latest snapshot on every turn |
| boundary; SQLite UPSERT handles dedup. |
| - **SessionStore interface** (`internal/store/store.go`): 8 |
| methods + sentinel errors (`ErrSessionNotFound`, |
| `ErrInvalidSessionID`, `ErrSessionPayloadTooLarge`). The |
| same `*sqlitevec.Store` implements both `Store` (memory) and |
| `SessionStore`; server.go type-asserts at each entry point. |
| M7 pgvector will implement the same interface. |
| - **HTTP surface** under `/v1/sessions/*`: |
| - `POST /v1/sessions` — upsert (201 on insert, 200 on update, |
| 400 invalid_session_id, 413 payload_too_large). |
| - `GET /v1/sessions[?parent_id=&include_deleted=&limit=&offset=]` |
| — list, default newest-first by `updated_at`, omitting |
| soft-deleted rows. |
| - `GET /v1/sessions/{id}` / `DELETE /v1/sessions/{id}` — |
| fetch / soft-delete. |
| - `POST /v1/sessions/{id}/events` / |
| `GET /v1/sessions/{id}/events[?type=&limit=&offset=]` — |
| append + list audit entries. |
| - `POST /v1/sessions/{id}/messages` / |
| `GET /v1/sessions/{id}/messages[?limit=&offset=&since=]` — |
| bulk-upsert (idempotent on message id) + list. |
| - `GET /v1/sessions/{id}/turns` — coarse server-side |
| projection grouping messages by user-message boundaries. |
| Authoritative turn-boundary logic stays in opencode's |
| `compaction.ts`; this view is for UI / debugging. |
| - **healthz extension**: `sessions_schema_version` field |
| added; M3 clients without the field treat the daemon as |
| pre-M4. |
| - **Payload size guards**: 8 MiB hard cap on |
| `session_events.payload` and the bulk-upsert messages body |
| (413 `payload_too_large`); the plugin chunks at a softer |
| 1 MiB default before posting. |
| - **TS client extension** (`@opencoti/opencoti-server-client`): |
| 8 new `OpencotiServerClient` methods (`upsertSession`, |
| `listSessions`, `getSession`, `deleteSession`, |
| `appendSessionEvent`, `listSessionEvents`, |
| `upsertSessionMessages`, `listSessionMessages`, |
| `listSessionTurns`). Wire format snake_case → camelCase via |
| private mappers (same pattern as M2's `mapHit` / |
| `mapCollectionInfo`). `healthz()` return type + |
| `ProbeResult` gain `sessionsSchemaVersion?: number`. |
| - **New plugin `@opencoti/session-events-plugin`**: |
| subscribes to `session.created/updated/deleted/error/idle` |
| and `session.status` (idle transition) via opencode's |
| typed `Hooks.event`; does **not** subscribe to |
| `message.updated` (fires thousands of times per turn — |
| text deltas, tool calls, reasoning chunks — and forwarding |
| every delta would melt the daemon). On idle, the plugin |
| fetches the session's latest messages via |
| `input.client.session.messages.list` and bulk-upserts in |
| byte-bounded chunks. Maintains a per-session merge cache so |
| opencode's partial `session.updated` payloads are reconciled |
| into a full snapshot before reaching the daemon. |
| Best-effort, never-throws: a forwarding plugin must not |
| propagate daemon failures into the opencode session. |
| - **session.error before sessionID exists** lands on a |
| synthetic `__pre_session_errors__` session (lazily upserted |
| on first such error) so the audit log captures pre-creation |
| failures without losing context. |
| - **Auto-wire**: one-line addition to |
| `@opencoti/tiers/default-plugins` `OPENCOTI_DEFAULT_PLUGINS`. |
| Any user with opencoti config in `opencode.jsonc` gets the |
| plugin automatically; without an `opencoti.server.address` |
| the plugin's healthz probe fails, the internal client stays |
| `undefined`, and every event handler early-returns — same |
| no-op-when-unconfigured ergonomic the memory-plugin uses. |
| - **Tests**: 26 new Go tests (16 store unit + 10 HTTP |
| integration) — total Go test count climbs from 30 → 63. |
| 14 new TS client tests (33 → 47). 16 new plugin tests in |
| `@opencoti/session-events-plugin`. All 24 workspace |
| packages typecheck clean. |
| - **Live verification**: end-to-end smoke against a real M4 |
| daemon (TCP loopback) confirmed healthz reports |
| `sessions_schema_version: 1`, full upsert → update → get → |
| list → events → messages → turns → soft-delete cycle |
| works, soft-deleted session's events + messages remain |
| queryable through `include_deleted=true`. |
| - **Deferred from M4**: |
| - **`message.updated` real-time forwarding** — coalescing on |
| idle is the right granularity; per-delta forwarding would |
| mean O(text-deltas) HTTP roundtrips per turn. |
| - **Per-tool-invocation events** as first-class rows — |
| captured today inside the message's `parts` JSON; can be |
| promoted to a `session_tool_calls` table later if needed. |
| - **WebSocket / SSE push from daemon → client** — |
| request-response only in M4. |
| - **TLS / authentication for `/v1/sessions/*` TCP transport** |
| — UDS owner-only on Unix remains the M4 security boundary. |
| |
| ### M5 — Tier-engine event sink (shipped 2026-05-23) |
|
|
| - **Daemon side: `/v1/tier-events`.** |
| - `POST /v1/tier-events` appends a row (201; 400 invalid_tier_event; |
| 413 payload_too_large at the same 8 MiB cap as `/v1/sessions/*`). |
| Body: `{event_type, ts?, session_id?, payload}` where `payload` |
| is opaque JSON — the daemon does not introspect it. |
| - `GET /v1/tier-events?session_id=&event_type=&since=&limit=&offset=` |
| returns the audit log newest-first. |
| - One new `tier_events` table: `(id PK AUTO, session_id TEXT, |
| event_type TEXT, payload TEXT, ts INTEGER)` + two indices |
| `(session_id, ts)` and `(event_type, ts)`. **`session_id` is a |
| soft reference — no FK, no cascade.** Tier events fire intra-turn |
| and can land before `session.created` reaches the daemon; the |
| soft reference guarantees no event is dropped. |
| - Per-feature `meta.tier_events_schema_version = 1`. The |
| cross-language sqlite_vec contract continues to live on |
| `meta.schema_version = 1` (still M2-compatible with the TS |
| `@opencoti/memory` reader); the new key is independent. |
| |
| - **TS plumbing.** `@opencoti/opencoti-server-client` gains |
| `appendTierEvent` + `listTierEvents` (camelCase→snake_case wire |
| mapping mirroring `mapSessionMessage`). `healthz()` surfaces |
| `tierEventsSchemaVersion?: number`; plugins use it as their |
| feature gate. |
| |
| - **Design pivot continues: plugin path > surgical hook.** The |
| `Telemetry` interface in `@opencoti/tiers` (designed for this |
| milestone — its docstring at `telemetry.ts:1-4` said so) is |
| already threaded end-to-end through `runtime.ts:99 → |
| executor.ts → escalator.ts → fanout.ts`. M5 plugs a real sink |
| into the existing seam via a new module-level |
| `registered-telemetry.ts` slot (mirrors `active-config.ts`). |
| Precedence chain in `runtime.ts:99` becomes: |
| |
| ```ts |
| const telemetry = |
| hook.telemetry ?? |
| execOpts.telemetry ?? |
| getRegisteredTelemetry() ?? |
| noopTelemetry |
| ``` |
| |
| The new `@opencoti/tier-events-plugin` calls |
| `setRegisteredTelemetry(impl)` during plugin boot when the |
| daemon's healthz reports `tierEventsSchemaVersion >= 1`. **Zero |
| new opencoti-hook source markers** — count unchanged from F3 M4 |
| (9 source + 4 JSON deps + 3 prose). See the F1 page for where |
| the Telemetry interface itself was added. |
| |
| - **Fire-and-forget posture.** The plugin's Telemetry impl maps |
| each method to a `client.appendTierEvent(...).catch(() => {})` |
| — synchronous to the caller, HTTP swallowed on failure. The |
| tier engine never blocks on the daemon. Cf. M4's |
| `@opencoti/session-events-plugin`, same design. |
| |
| - **Auto-wire via defaults.** `@opencoti/tier-events-plugin` is in |
| `OPENCOTI_DEFAULT_PLUGINS` alongside the M4 session plugin. Any |
| user with `opencoti: {...}` in their config gets the audit log |
| for free; if the daemon is unreachable, the healthz gate fails |
| and the runtime stays on noop. |
| |
| - **Cross-version compat.** An M5 plugin against an M4 daemon |
| sees `sessions_schema_version: 1` but no |
| `tier_events_schema_version` in healthz, so |
| `setRegisteredTelemetry` is never called and the runtime stays |
| on noop. An M5 daemon against M4 clients keeps shipping the |
| same `/v1/sessions/*` surface untouched. |
|
|
| - **Explicitly deferred to later:** aggregation endpoints |
| (`/v1/tier-events/stats?...`), real-time push (WS/SSE), |
| server-side retention policy, client-side batching/coalescing, |
| FK on `session_id`. M5 ships raw events + filters; everything |
| on top of that is composable later. |
|
|
| ### M6 — Tool registry + invocation (shipped 2026-05-23) |
|
|
| - **Daemon side: `/v1/tools`.** |
| - `GET /v1/tools` lists every registered tool sorted by name: |
| `{ tools: [{ name, description, params_schema, handler, |
| created_at, updated_at }] }`. `params_schema` is a |
| string-serialised JSON Schema; `handler` is a stable opaque |
| dispatch ID (debug-only — invocation always goes by name). |
| - `GET /v1/tools/{name}` fetches one (404 tool_not_found). |
| - `POST /v1/tools/{name}/invoke` runs it. Body |
| `{ arguments: {...}, session_id? }` → `{ result: {...} }` (200) |
| or `{ error: { code, message } }` (400 invalid_arguments / 404 |
| tool_not_found / 500 handler_failed / 413 payload_too_large at |
| the same 8 MiB cap as `/v1/sessions/*` and `/v1/tier-events`). |
| - One `tools` table: `(name PK, description, params_schema, |
| handler, created_at, updated_at)`. Process-scoped, no |
| session FK. Dispatch is a daemon-internal |
| `map[string]ToolHandler` in `toolhandlers.go`; **no external |
| registration in M6** (no POST/DELETE on `/v1/tools` itself) — |
| the catalog is seeded into the binary at boot. |
| - Per-feature `meta.tools_schema_version = 1`, independent of the |
| cross-language `meta.schema_version = 1` (still M2-compatible |
| with the TS `@opencoti/memory` reader). |
| |
| - **Two seed tools** are `UpsertTool`'d at startup (idempotent — |
| `created_at` fixed on first insert, `updated_at` bumps on |
| re-seed): |
| - **`opencoti_episodic_search`** — lexical `LIKE` search across |
| the M4 `session_messages` table joined with `sessions`, |
| newest-first. Params `{ query, limit?, session_id_excludes? }`. |
| The exclude list lets the plugin pass the live session ID so |
| the model recalls *other* sessions, not echoes of the current |
| one. The integrated analog of claude-hooks's `episodic_server` |
| — same SQLite file the sessions API writes, no second daemon, |
| no shell-out. FTS5/vector ranking is an M7+ optimisation. |
| - **`opencoti_recall`** — vector recall over the M2 memory store, |
| delegating to the same `Store.Recall` path `/v1/memory/search` |
| uses. Params `{ embedding, collection?, k?, session_id? }`. |
| **Takes a pre-computed embedding, not a query string** — the |
| daemon bundles no embedder, so the caller (the plugin, or an |
| operator via curl) embeds the query and POSTs the float vector; |
| length must equal the daemon's `embedding_dim`. `session_id` |
| drives the M2 per-session ACL check. Coexists intentionally |
| with `@opencoti/memory`'s in-process `__memory_recall`: that |
| one survives a daemon being down; `opencoti_recall` gives the |
| same surface to plugins without direct DB access. |
| |
| - **TS plumbing.** `@opencoti/opencoti-server-client` gains |
| `listTools` + `getTool` + `invokeTool`; `healthz()` surfaces |
| `toolsSchemaVersion?: number` as the plugin's feature gate. |
| `ServerTool` is camelCase (`paramsSchema` ← `params_schema`), |
| `paramsSchema` left as `unknown` (interpreted in the plugin). |
|
|
| - **Design pivot continues: plugin path > surgical hook.** A new |
| module-level `registered-server-tools.ts` slot in |
| `@opencoti/tiers` (mirrors M5's `registered-telemetry.ts`) holds |
| the daemon tools as a `Record<string, Tool>`; `runtime.ts` merges |
| it **last** into the tier tool list, next to synthetic-tier and |
| memory-bridge tools: |
|
|
| ```ts |
| const tools = mergeTools(hook.prepared.tools, { |
| ...syntheticTools(hook.input.sessionID, telemetry), |
| ...memoryTools, |
| ...(getRegisteredServerTools() ?? {}), |
| }) |
| ``` |
|
|
| Server tools merge last so user-explicit + memory tools win on a |
| name collision (none expected — server tools are `opencoti_*`, |
| memory `__memory_*`, synthetic-tier `__tier_*`). **Zero new |
| opencoti-hook source markers** — count unchanged from F3 M4/M5 |
| (9 source + 4 JSON deps + 3 prose). |
|
|
| - **The `@opencoti/server-tools-plugin`** probes `/v1/healthz` |
| (gated on `toolsSchemaVersion >= 1`), fetches `/v1/tools` once on |
| boot, and builds an AI SDK `dynamicTool` per entry: |
| `params_schema` round-trips through `ai`'s `jsonSchema()` (no |
| JSON-Schema-to-Zod reimplementation); each `execute` forwards to |
| `/v1/tools/{name}/invoke`. Tool names are prefixed (`opencoti_`, |
| idempotently). The session ID arrives via `experimental_context` |
| on the single `openStream` `streamText` call and is forwarded as |
| `session_id`. Invocations are awaited (not fire-and-forget — tool |
| calls are model-blocking), but errors crash the *tool call*, not |
| the session: 404→`tool_not_found`, 400→`invalid_arguments`, |
| 5xx/network→`tool_handler_failed`, deadline→`timeout` |
| (plugin-owned `Promise.race`, separate `invoke_timeout_ms`). |
| Args over 1 MiB are pre-rejected client-side. |
|
|
| - **Auto-wire via defaults.** `@opencoti/server-tools-plugin` is in |
| `OPENCOTI_DEFAULT_PLUGINS`. No reachable daemon → healthz gate |
| fails → `setRegisteredServerTools` never called → runtime tool |
| set unchanged. |
|
|
| - **Cross-version compat.** An M6 plugin against an M5 daemon sees |
| no `tools_schema_version` in healthz, so the gate fails and the |
| tool set is unchanged — a model query expecting the tool gets |
| "I don't have that tool", no crash. An M6 daemon serves M5 |
| clients the `/v1/sessions/*` and `/v1/tier-events` surfaces |
| untouched. |
|
|
| - **Explicitly deferred:** external tool registration (POST/DELETE |
| on `/v1/tools` — needs a per-tool ACL/owner model); streaming |
| tool results; tool invocations as `tier_events` audit rows; an |
| MCP wrapper for the registry; FTS5/vector ranking in |
| `episodic_search`; an in-daemon embedder so `opencoti_recall` |
| can take a `query` string directly. |
|
|
| ### M7 — pgvector backend parity (shipped 2026-05-23) |
|
|
| opencoti-server gains its **second storage backend**: PostgreSQL + |
| the pgvector extension, via `github.com/jackc/pgx/v5` (pure Go, no new |
| cgo). `*pgvector.Store` implements the **full** interface family — |
| `Store` + `SessionStore` + `TierEventStore` + `ToolStore` + |
| `SchemaInspector` — so the HTTP layer, `/v1/healthz`, and the TS client |
| are **untouched**. sqlite-vec stays the zero-dependency default; |
| pgvector is for hosts that already run Postgres. |
|
|
| **Schema-phrasing correction.** The original M7 stub said *"/v1/memory |
| works against pgvector with the same schema as claude-hooks."* That |
| predated the M4–M6 buildout. opencoti's binding contract is now its |
| **own** `Store` interface (collections + per-session ACL + sessions + |
| tier_events + tools), which is a different data model from |
| claude-hooks's flat `memories` + `kg_*` schema. So M7 implements |
| **opencoti's own model on a dedicated `opencoti` Postgres database**, |
| isolated from claude-hooks. "Same as claude-hooks" is reread as *same |
| storage technology (Postgres + pgvector)*, not the same tables. |
| |
| - **`internal/store/pgvector/`** mirrors `internal/store/sqlitevec/` |
| table-for-table in PG dialect, with three dialect differences: |
| - the embedding lives **inline** on `memories` as a `vector(dim)` |
| column (no separate vec0 virtual table); an HNSW `vector_l2_ops` |
| index is created when `dim ≤ 2000` (pgvector's HNSW ceiling) — |
| correctness-neutral, perf-positive. |
| - `Recall` pushes the ACL filter into the query |
| (`WHERE collection_name = ANY($readable) ORDER BY embedding <-> $q |
| LIMIT k`) — Postgres can filter + rank in one statement where |
| sqlite-vec must overscan + join-back + post-filter. The `<->` |
| operator is **L2**, matching sqlite-vec's vec0 default, so |
| `MemoryHit.Distance` stays comparable across backends. |
| - `Store` dedups via `INSERT … ON CONFLICT (collection_name, |
| content_hash) DO NOTHING RETURNING id` in one round trip. |
| - **Per-feature schema versions** (`schema_version`, |
| `sessions_/tier_events_/tools_schema_version`, all `1`) live in a |
| `meta` key/value table, validated on open exactly as the sqlite-vec |
| backend does — but they are pgvector's own contract (no cross-language |
| reader, since pgvector is daemon-only). |
| - **Shared conformance harness** `internal/store/storetest/` is the |
| parity guarantee: a single `RunConformance` body (Memory, Sessions, |
| TierEvents, Tools sub-suites) that **both** backends opt into via a |
| thin `conformance_test.go`. Either backend drifting from the contract |
| fails the same assertions. sqlite-vec runs it with `t.TempDir`; |
| pgvector with a testcontainers fixture. |
| - **Pure-logic lifted to `internal/store/common.go`** (package `store`): |
| the ACL resolver (`ResolveAccessMode`/`CanRead`/`CanWrite`), the |
| collection/session validators, `Sha256Hex`, and the dim bounds — one |
| cgo-free source of truth both backends share. sqlite-vec keeps its |
| exported names as thin delegating wrappers, so the HTTP layer's |
| `sqlitevec.ValidateCollectionName`/`ValidateSessionID` call sites are |
| byte-identical. |
| - **Backend selection** is a serve-time flag, not a build flag: |
| `--backend sqlite|pgvector` (default `sqlite`) + `--pg-dsn` (falling |
| back to `$OPENCOTI_PG_DSN`). `--db-path` stays sqlite-only; |
| `--embedding-dim` applies to both (locks the `vector(dim)` column). |
| The interface var is assigned only on a successful open (avoids the |
| typed-nil trap), so a pgvector open failure degrades to |
| `store_ok=false` + a clear `store_error` rather than crashing. The |
| Windows `install` command bakes `--backend`/`--pg-dsn` into the |
| service args alongside the existing flags. |
| - **healthz** reports a **redacted** `store_path` for pgvector |
| (`pg://user@host:port/db`, never the password). |
| - **Tests**: `pgtest` spins an ephemeral `pgvector/pgvector:pg17` |
| container via testcontainers-go and skips cleanly when Docker is |
| absent (`testcontainers.SkipIfProviderIsNotHealthy`), so |
| `make test` stays green on a Docker-less host. Both backends pass the |
| full shared conformance suite; an end-to-end smoke (daemon → |
| pgvector container) confirms collection create, store, and `<->` - |
| ranked recall over HTTP. |
| - **Toolchain**: the modern pgx / pgvector-go / testcontainers-go |
| releases require **Go 1.25**, so the module's `go` directive and the |
| host toolchain moved to go1.25 (latest stable). No build-posture |
| regression — pgx is pure Go; sqlite-vec's existing cgo requirement is |
| unchanged. |
| - **Zero new surgical hooks.** pgvector is additive Go inside |
| `opencoti/server/`; no TS changes. The surgical-hook grep count |
| stays at 18 (see `docs/protocols/UPSTREAM_SYNC.md`). |
| |
| **Deferred from M7:** data migration between backends (a `migrate` |
| subcommand — "Migration tools later"); a TS pgvector backend for the |
| in-process `@opencoti/memory` (pgvector is daemon-only); external tool |
| registration over HTTP (still security-gated, from M6); pgxpool tuning |
| / read replicas. |
| |
| ### M8 — Multi-session coordination (shipped 2026-05-23) |
| |
| Goal **G3**: opencoti instances/sessions on one host see each other and |
| cooperate. M8 ships the *live-coordination* half as an **in-memory, |
| ephemeral** hub — deliberately store-independent (neither the `Store` |
| interface nor either backend is touched), because presence and locks are |
| runtime state that should not survive a daemon restart. |
|
|
| - **`internal/coord.Hub`** — backend-agnostic, one `sync.RWMutex`: |
| - **Presence**: `RegisterPeer` / `Heartbeat` / `DeregisterPeer` / |
| `ListPeers`, with a background TTL sweep that reaps peers whose |
| heartbeat lapsed (default 45s) and emits `peer.left`. `ListPeers` |
| also filters expired peers lazily. |
| - **Broadcasts**: a pub/sub bus — `Publish` assigns a monotonic `seq`, |
| appends to a bounded replay ring, and fans out non-blockingly to |
| subscribers (a full subscriber channel is dropped + closed so the |
| client reconnects with its last seq). `Subscribe(since)` atomically |
| snapshots the replay backlog and registers for live events. |
| - **Advisory locks**: try-only `AcquireLock` (reentrant-by-holder |
| refresh; expired locks reclaimable) / `ReleaseLock` (holder-checked) |
| / `ListLocks`. Lock transitions emit `lock.acquired` / `lock.released`. |
| - **HTTP** (`internal/server/coord.go`), hub injected via `Options.Hub`, |
| 503 `coord_unavailable` when absent: |
| |
| | Method + path | Purpose | |
| | --- | --- | |
| | `POST /v1/coord/peers` | register/upsert presence → PeerInfo | |
| | `POST /v1/coord/peers/{id}/heartbeat` | refresh TTL | |
| | `DELETE /v1/coord/peers/{id}` | deregister | |
| | `GET /v1/coord/peers` | list live peers | |
| | `POST /v1/coord/broadcast` | publish `{peer_id,topic,payload}` → `{seq}` | |
| | `GET /v1/coord/events?since=&peer_id=` | **SSE** event stream (the daemon's first) | |
| | `POST /v1/coord/locks/{name}` | acquire `{holder,ttl_ms?}`; 200 or 409 `lock_held` | |
| | `DELETE /v1/coord/locks/{name}` | release `{holder}`; 200 / 404 `lock_not_held` / 409 `lock_not_holder` | |
| | `GET /v1/coord/locks` | list held locks | |
|
|
| SSE is viable because the `http.Server` sets no `WriteTimeout`; the |
| handler exits on request-context cancellation so graceful shutdown |
| releases it within the grace window. `/v1/healthz` gains |
| `coord_ok` + `coord_peers`. |
| - **TS client**: `registerPeer` / `heartbeatPeer` / `deregisterPeer` / |
| `listPeers` / `broadcast` / `acquireLock` / `releaseLock` / |
| `listLocks`, plus `subscribeCoordEvents` — the client's first |
| streaming method (reads `response.body`, parses `data:` frames). |
| - **`@opencoti/coordination-plugin`**: registers each session as a peer, |
| heartbeats while active, deregisters on delete, subscribes to the |
| event stream (SSE) to keep a live peer view, and advertises |
| "N other active opencoti session(s)" in the system prompt. Auto-wired |
| via the `@opencoti/tiers` default-plugins list — **no surgical hook** |
| (grep count stays 18). |
|
|
| **Deferred to F3 M9:** opt-in *shared context* (a thin convention atop |
| existing M2 global collections + per-session ACL — a session opts to |
| expose a collection to peers); the sqlite↔pgvector `migrate` subcommand |
| (open from M7); blocking/queued lock acquire (M8 is try-only). Cross-host |
| federation remains an explicit F3 non-goal. |
|
|
| ### M9 — Opt-in shared context (shipped 2026-05-23) |
|
|
| Goal **G3**, the *persisted-context* half: a session exposes one of its |
| session-scoped memory collections to peer sessions on the same host. The |
| storage model from M2 already supports the grant, so M9 adds **no new |
| `Store` method, no new schema, no new surgical hook** (grep count stays |
| 18). It is a thin convention bridging two things that already exist — the |
| persisted per-session ACL (`SetSessionACL`) and M8's ephemeral coord bus. |
|
|
| - **`internal/share.Manager`** — in-memory registry + one goroutine: |
| - `Share(collection, owner, mode)` records the share, grants read-ACL |
| to every live peer `!= owner` via `SetSessionACL`, and publishes |
| `collection.shared` on the hub. |
| - It **subscribes to the hub** and, on `peer.joined`, grants every |
| active share to the newcomer — so peers that join *after* a share |
| still get access (auto-grant via the bus, dogfooding M8's SSE). |
| - `Unshare(collection, owner)` revokes live peers (writes mode `none`, |
| the non-owner default) and publishes `collection.unshared`. |
| - **Persisted vs ephemeral:** the ACL grants persist (they survive a |
| restart); the "keep auto-granting new joiners" intent is in-memory |
| and lost on restart by design — existing grants remain, but the owner |
| must re-share to resume auto-granting. This keeps M9 storage-free. |
| - **Recall "just works":** once a peer holds an `r` grant, its no-filter |
| `Recall` includes the shared collection automatically (M2's |
| readable-collections resolution), so no recall-path change is needed. |
| - **HTTP** (`internal/server/share.go`), Manager constructed in |
| `server.New()` when `Store`+`Hub` are present, released on `Shutdown`; |
| 503 `share_unavailable` when the hub is absent: |
| |
| | Method + path | Purpose | |
| | --- | --- | |
| | `POST /v1/memory/collections/{name}/share` | owner-only share `{owner_session, mode?}` (mode default `r`) → `{granted}` | |
| | `DELETE /v1/memory/collections/{name}/share` | withdraw `{owner_session}` → 200 | |
| | `GET /v1/memory/shares` | list active shares | |
|
|
| **Owner-only:** the handler looks the collection up via |
| `ListCollections` (no filter) and requires `scope == session` and |
| `session_id == owner_session`; global collections (already `r`-for-all) |
| are rejected `not_shareable`, a different owner `not_owner` (403). |
| - **TS client**: `shareCollection` / `unshareCollection` / `listShares` |
| + `SharedCollectionInfo`. |
| - **`@opencoti/coordination-plugin`** (extended in place — no new plugin): |
| consumes `collection.shared` / `collection.unshared` to advertise |
| peer-shared collections in the system prompt, and gains an opt-in |
| `share_session_collection` flag (default **false**) that shares the |
| session's own collection (`sessionCollectionName(id)`) on |
| `session.created` and unshares it on `session.deleted` (best-effort; a |
| not-yet-created collection's 404 is swallowed). |
|
|
| **Deferred to F3 M10:** the sqlite↔pgvector `migrate` subcommand — **not** |
| a cutover: both backends are first-class and may run in parallel; migrate |
| is an idempotent, re-runnable, either-direction copy (`--from`/`--to`, |
| dedupe on `content_hash`, upsert PKs) that tops up a parallel target, |
| never abandoning the source. Also deferred: revoke-on-peer-leave cleanup |
| (M9 keeps grants on `peer.left`, since peers may return) and wildcard / |
| group ACLs. |
|
|
| ### M10 — sqlite↔pgvector `migrate` (shipped 2026-05-24) |
|
|
| A daemon-internal `migrate` subcommand that copies data between the two |
| first-class backends. It is **not a cutover**: both backends stay |
| first-class and may run in parallel (two daemons, or alternating |
| `--backend`). `migrate` is an **idempotent, re-runnable, either-direction, |
| selectable copy** that *tops up* a target (dedupe on natural keys), never |
| "move then abandon source". Additive Go only — **no plugin, no HTTP |
| surface, no TS client, no surgical hook** (grep count stays 18). |
|
|
| **Use cases (maximum flexibility):** |
|
|
| 1. **Scale-up (primary).** Start on sqlite; migrate *everything* |
| (collections, memories+embeddings, ACL, sessions, session_events, |
| session_messages, tier_events, tools) into pgvector; then **switch the |
| primary backend** by changing the serve flag to `--backend pgvector`. |
| "Switch primary backend" is operational, no extra code: run the full |
| migrate, then change `--backend`. |
| 2. **Way back.** Same command with `--from`/`--to` swapped. |
| 3. **Single memory container / additional partial backend.** Copy only |
| specific collection(s) with `--collection`, so e.g. pgvector holds just |
| certain memories while sqlite keeps the rest. The two backends coexist, |
| each holding different data. |
| 4. **Additive top-up.** Re-running, or copying into a populated target, |
| merges idempotently (natural-key dedupe). |
| |
| **Selection model — two orthogonal selectors:** |
| |
| - `--include <csv>` of sections: `memory,sessions,tier-events,tools`. |
| Default (unset) = **all four** (full dataset). The `memory` section |
| carries collections + their memories + their ACL rows. |
| - `--collection <name>` (repeatable) restricts the `memory` section to |
| those collections only. When `--collection` is given and `--include` is |
| unset, the default narrows to **memory-only** (the single-container |
| case). |
| |
| So: full switch = no selectors; single container = `--collection notes`; |
| logs-only = `--include sessions,tier-events`. |
| |
| **Copy phases (FK-respecting; each gated by the selection):** |
| |
| | Section | Phase | Export | Import | |
| | --- | --- | --- | --- | |
| | memory | collections | `ListCollections` (filtered) | `PutCollection` | |
| | memory | memories | `ListMemories` (per coll, keyset-paged) | `PutMemory` | |
| | memory | session_acl | `ListSessionACLs` | `SetSessionACL` | |
| | sessions | sessions | `ListSessions(+deleted)` | `UpsertSession` | |
| | sessions | session_events | `ListSessionEvents`/sess | `PutSessionEvent` | |
| | sessions | session_messages | `ListSessionMessages`/sess | `UpsertSessionMessages` | |
| | tier-events | tier_events | `ListTierEvents` (batched) | `PutTierEvent` | |
| | tools | tools | `ListTools` | `UpsertTool` | |
| |
| Both backends are the same concrete `*Store` implementing every feature |
| interface, so migrate opens each side as `storepkg.Store`, type-asserts |
| the optional `MigrationStore`, and copies a section only if **both** sides |
| implement it. Per-phase counts (scanned / inserted / skipped) print at the |
| end; `--dry-run` reads sources and reports would-copy counts, no writes. |
| |
| **Idempotency / dedup.** Memories dedupe on `(collection_name, |
| content_hash)`. The append-only audit logs (`session_events`, |
| `tier_events`) have **no natural key** (autoinc id only), so their |
| idempotent import dedups on an **insert-if-no-identical-row** check (all |
| business columns match) — re-running produces no duplicates. A second full |
| run reports 0 inserts everywhere. |
|
|
| **Two backend-specific wrinkles (both resolved):** |
|
|
| - **sqlite-vec embeddings are an opaque BLOB.** vec0 `memory_vecs` stores |
| the embedding as a blob and the Go binding ships `SerializeFloat32` but |
| no deserialize. The on-disk format is plain little-endian `float32` |
| (`binary.Write(buf, LittleEndian, vector)`), so `deserializeFloat32` |
| reverses it with `binary.Read` at the locked dim — **byte-faithful, no |
| re-embedding**. pgvector exports via `pgvector.Vector.Scan` + `.Slice()`. |
| - **The public `Store()` write path enforces ACL** (a global collection is |
| `r`-for-all → `ErrWriteDenied`), so migrate cannot reuse it for imports. |
| The new `MigrationStore.PutMemory` is **ACL-free and timestamp- |
| preserving**. |
|
|
| **Fidelity contract (documented caveat).** Preserved exactly: collection |
| `created_at`, memory `ts`+`content`+embedding (byte-identical), session |
| `created_at`/`deleted_at`, message `created_at`/`finished_at`, event |
| `occurred_at`/`ts`, all ACL modes. Rewritten to migrate-time (reused |
| upserts stamp `now()`): `sessions.updated_at`, `tools.created_at`/ |
| `updated_at` — acceptable "last-written" fields (tools are also re-seeded |
| at startup). A byte-identical `PutSession`/`PutTool` is deferred. |
|
|
| The new `MigrationStore` optional interface (in `store.go`, mirroring the |
| `SchemaInspector`/`SessionStore` convention) is exercised by the |
| `storetest` conformance harness on **both** backends — the round-trip |
| asserts the exported embedding is byte-identical, validating the sqlite LE |
| deserialize and the pgvector scan together. Verified end-to-end across a |
| `sqlite→pgvector→sqlite` hop: recall on the round-tripped file returns |
| `distance 0` for the exact source vector. |
|
|
| **Deferred to F3 M11+:** continuous-sync / daemon mode (migrate is |
| one-shot CLI only), cross-host federation (explicit F3 non-goal), |
| per-session selective log copy (selective granularity is per-collection |
| for memory + section toggles for the rest), and the byte-identical |
| `updated_at`/tool-timestamp imports noted above. |
|
|
| ## Open questions |
|
|
| - **Do we ship the server inside the opencoti binary or alongside it?** |
| Alongside (separate binary) is cleaner; explore a single-fat-binary |
| option as an opt-in. |
| - **Authentication.** Local UDS owner is the user; over loopback TCP |
| we need a shared secret. Default to UDS. |
| - **claude-hooks coexistence.** If a user runs both Claude Code (with |
| claude-hooks) and opencoti (with opencoti-server) on the same host, |
| the two memories should *not* collide. They use different stores |
| and different schemas by default; a separate migration tool can |
| bridge them if the user wants. |
|
|
| ## Risks |
|
|
| - Two persistence backends doubles the test matrix. Mitigation: a |
| storage trait + a shared conformance test. |
| - Drift from claude-hooks features. Mitigation: don't try to replicate |
| feature-for-feature; replicate goals. claude-hooks remains the |
| reference for Claude Code; opencoti-server can diverge where it |
| makes opencoti better. |
|
|