File size: 52,977 Bytes
9cee049 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 838 839 840 841 842 843 844 845 846 847 848 849 850 851 852 853 854 855 856 857 858 859 860 861 862 863 864 865 866 867 868 869 870 871 872 873 874 875 876 877 878 879 880 881 882 883 884 885 886 887 888 889 890 891 892 893 894 895 896 897 898 899 900 901 902 903 904 905 906 907 908 909 910 911 912 913 914 915 916 917 918 919 920 921 922 923 924 925 926 927 928 929 930 931 932 933 934 935 936 937 938 939 940 941 942 943 944 945 946 947 948 949 950 951 952 953 954 955 956 957 958 959 960 961 962 963 964 965 966 967 968 969 970 971 972 973 974 975 976 977 978 979 980 981 982 983 984 985 986 987 988 989 990 991 992 993 994 995 996 997 998 999 1000 1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 | # F3 — opencoti-server (Go companion daemon)
> Parent: [../MASTER_PLAN.md](../MASTER_PLAN.md)
> Status: **planning**
> Owner: TBD
> Reference: `/shared/dev/claude-hooks` (the inspiration; mirror its
> goals, then go further)
## Problem
`claude-hooks` is a Python daemon that wraps Claude Code with a hook
API and adds persistent memory, multi-session coordination, tool
glue, and a small zoo of integrations. It works very well *for Claude
Code*. opencoti has the same shape of needs — and a few extra
because opencoti has its own tier engine, its own local llamafile,
and its own session model.
We want a Go daemon that is to opencoti what claude-hooks is to
Claude Code, **but deeper** because we control opencoti's source.
Instead of bolting onto a hook API from outside, opencoti-server can
be a first-class peer of opencoti's runtime.
## Goals
- **G1.** Single Go binary, `opencoti-server`, that exposes a small,
well-versioned API to opencoti.
- **G2.** Persistent memory (vector + KG) shared across opencoti
sessions on a host.
- **G3.** Multi-session coordination: opencoti CLI instances on the
same host see each other, share context where the user wants it.
- **G4.** Tool glue: a place to register external tools (search,
filesystem extensions, MCP-like) once and have them appear in all
opencoti sessions.
- **G5.** Event sink for tier-engine routing decisions (so we can
log, audit, and later learn from them).
- **G6.** Hook fan-out compatible with the existing claude-hooks
ecosystem where it makes sense — but not bound by it.
## Non-goals (for now)
- Replacing claude-hooks. opencoti-server is opencoti-specific.
- Becoming a generic "MCP host". MCP integration is a feature, not
the architecture.
- Multi-host federation. Single-host first.
## Design sketch
### Where it lives
- `opencoti/server/` — top-level non-bun source root. Go module.
Putting it outside `packages/` keeps the bun workspace clean.
- `packages/opencoti-server-client/` — TypeScript client used by
opencoti to talk to the server.
- **Surgical hooks** in opencode session lifecycle (session start,
session end, turn complete, tool invocation, model call result):
each hook is a one-line "if a client is available, notify it".
Listed in `docs/protocols/UPSTREAM_SYNC.md`.
### Transport
Unix domain socket on Linux/macOS, named pipe on Windows. Loopback
TCP as a fallback. HTTP/JSON wire format for simplicity (no gRPC
build dependency for the client).
**Daemon-launch policy (applies to opencoti-server and every
opencoti daemon)**:
- Listen port MUST come from opencoti's reserved
**47000-48000 range** — never 38000-39000 or 18790-18811
(claude-hooks territory). Tentative reservation table to
keep adjacent ports clear of accidental collision:
- **47092** — `@opencoti/embedder` (F4)
- **47190** — opencoti-server daemon (proposed default; revisit
when F3 M1 ships)
- **47191** — opencoti-server dashboard (proposed default)
- First-run setup MUST ask the user explicitly where the
daemon should bind: `127.0.0.1` (default), all interfaces,
or a specific IP. The default port from the reservation
table is proposed; the user may override either field within
the 47000-48000 range. Setup writes the validated choice to
the user's config.
### Persistence
Two backends, picked per-deployment:
- **pgvector** — same as solidPC's `claude-hooks` setup. Best when a
Postgres is already on the host.
- **sqlite-vec** — single-file, no external dependency. Default for
fresh installs.
The backend choice is config; the memory API is identical.
### Surface (illustrative)
```
POST /v1/sessions # register a session
POST /v1/sessions/:id/turns # record a turn
POST /v1/memory # store a memory (M2)
POST /v1/memory/search # vector recall (M2 — POST because the embedding is multi-KB)
POST /v1/memory/collections # collections CRUD (M2)
PUT /v1/memory/collections/:name/acl # per-session ACL (M2)
GET /v1/memory/count # count (M2)
POST /v1/memory/kg/entities # KG ops
POST /v1/memory/kg/relations
POST /v1/tier-events # log a tier-engine decision
GET /v1/tools # list registered tools
POST /v1/tools/:name/invoke # invoke (server-side glue)
GET /v1/healthz # M1; M2 extended with store_ok/store_path/embedding_dim/store_error
```
The server is **embedder-agnostic** at M2: clients pass vectors,
not text. The text→vector convenience (e.g.,
`GET /v1/memory?q=<text>` proxying to the embedder daemon) is a
deferred follow-up — adding it requires a runtime dep on
`@opencoti/embedder` (F4 M5) and is most naturally landed once
the TS client (M3) demonstrates the convenience is wanted at
the HTTP layer vs done client-side.
API is **versioned** (`/v1`) and the wire schema is owned by
`packages/opencoti-server-client/` (so opencoti can rev independently
of the server binary, within compatible versions).
### Mirroring claude-hooks goals
Same goals, mapped:
| claude-hooks | opencoti-server |
| --- | --- |
| `claude-hooks` recall hooks | implicit recall on `/v1/memory` query, surfaced by opencoti's prompt builder |
| `mcp__pgvector__*` MCP tools | `/v1/memory` (and a thin MCP shim if external MCP clients want it) |
| Stop hook auto-ingest | server records turn-complete events, applies same heuristics |
| `claude-hooks` companion tools (Episodic etc.) | first-class tools registered via `/v1/tools` |
### Deeper integration than claude-hooks
Because we own opencoti, we can:
- Receive **tier-engine** decisions directly (provider chosen,
escalation reasons, cost) without scraping logs.
- Receive **diff and tool-output** events as structured payloads, not
parsed from a transcript.
- Push back: server can suggest a memory recall payload that
opencoti **injects directly** into the prompt builder, instead of
appending to a transcript.
## Milestones
### M1 — Server skeleton + healthz *(2026-05-23 — shipped)*
- Go module under `opencoti/server/` (module
`github.com/mann1x/opencoti/server`, Go 1.22). Single dep:
`golang.org/x/sys` for the Windows `svc` packages.
- `opencoti-server serve` binds a UDS by default on Unix
(`$XDG_RUNTIME_DIR/opencoti/server.sock` or
`~/.opencoti/server.sock` fallback) or `tcp://127.0.0.1:47190`
on Windows. `--addr unix:///path` and `--addr tcp://host:port`
override.
- One HTTP endpoint: `GET /v1/healthz` returns 200 + JSON
`{status, version, started_at, uptime_seconds}`. All other
paths return a 404 envelope with code + message so the wire
format is consistent for M2+. `POST /v1/healthz` returns 405
with the same envelope.
- Graceful shutdown on SIGTERM/SIGINT (Unix) or SCM
Stop/Shutdown (Windows). UDS file unlinked on Unix shutdown.
- **Windows service support shipped in M1 (per user directive
2026-05-23):**
- `install` subcommand wraps SCM (`golang.org/x/sys/windows/svc/mgr`):
`--name`, `--display-name`, `--description`,
`--start-type=auto|manual|disabled`, `--addr`. Registers
the Event Log source under the same name.
- `uninstall` removes the SCM entry AND the Event Log source.
- `start` / `stop` wrap `mgr.Service.Start` /
`Service.Control(svc.Stop)`. `stop` waits up to 10s for the
state to become Stopped.
- When `svc.IsWindowsService()` returns true, `serve` enters
`svc.Run` with a handler that translates SCM Stop /
Shutdown into a graceful `http.Server.Shutdown`. Log
records flow to the Windows Application Event Log via
`eventlog.Open(name)`.
- Single source tree, build-tagged platform splits
(`*_windows.go` / `*_nonwindows.go`). Same binary surface
on every platform — `install` / `uninstall` / `start` /
`stop` on Unix print "Windows-only command" and exit 2.
- 7 cross-platform tests (parser + listener + healthz/404/405
+ UDS roundtrip + UDS unlink-on-cleanup + double-shutdown
safety) + 3 Windows-tagged tests for the `BuildMgrConfig`
install-config builder. All green on Linux; Windows
cross-compile clean.
- **Hook footprint: ZERO.** M1 is purely additive — no
`opencoti-hook:` markers, no UPSTREAM_SYNC.md registry rows.
The autostart hook lands in M3.
- **Deferred from M1:** Windows named-pipe TRANSPORT (use TCP
on Windows in M1; named-pipe support is a follow-up);
systemd/launchd unit files on Unix; CI workflows (the repo
has no CI yet — separate concern); the first-run setup
wizard step ("install as Windows service?" — comes when the
setup-flow picks up F3 concerns).
### M2 — Memory backend (sqlite-vec) + recall API *(2026-05-23 — shipped)*
- Go SQLite stack: `github.com/mattn/go-sqlite3` (cgo) +
`github.com/asg017/sqlite-vec-go-bindings/cgo`, sqlite-vec
statically linked via `sqlite_vec.Auto()` so the daemon ships
as a single binary (no `vec0.so` to bundle). CGO becomes a
hard build requirement at M2; cross-compile to Windows from
Linux needs MinGW-w64 (`apt install gcc-mingw-w64-x86-64`).
- `internal/store/store.go` defines a `Store` interface
mirroring the TS `MemoryStore` in
`packages/opencoti-memory/src/types.ts` 1:1. M7's pgvector
backend will implement the same interface, so the HTTP layer
doesn't change.
- `internal/store/sqlitevec/` is the M2 backend. DDL mirrors
`packages/opencoti-memory/src/schema.ts` byte-functionally:
four tables (`meta`, `collections`, `memories`, `session_acl`)
plus the `memory_vecs` vec0 virtual table; `SCHEMA_VERSION=1`;
`embedding_dim` parameter-substituted into the vec0 DDL at
first creation and stored in `meta` for validation on later
opens. Content idempotency: SHA256-hex with
`UNIQUE(collection_name, content_hash)`. Pure `ResolveAccessMode`
mirrors the TS resolver (global=r, session-owner=rw,
others=none; explicit `session_acl` rows override). Recall
is vec0 MATCH + overscan
(`k * min(8, max(2, len(candidates)))`) + join-back +
ACL filter, truncated to k.
- **DB schema is wire-compatible with the in-process
`@opencoti/memory` package** — a DB created by either side
opens cleanly under the other. This is the load-bearing
invariant for M3's path-B refactor.
- HTTP surface (eight new routes, all under `/v1/memory/*`):
- `POST /v1/memory/collections` — create (201 / 409
collection_exists / 400 invalid_collection_name).
- `GET /v1/memory/collections?session_id=...` — list,
optionally filtered through the ACL resolver.
- `DELETE /v1/memory/collections/{name}` — cascade-deletes
memories + vec0 rows + session_acl entries.
- `PUT /v1/memory/collections/{name}/acl` — set per-session
ACL (`r`/`w`/`rw`/`none`).
- `GET /v1/memory/collections/{name}/acl?session_id=...` —
read the resolved access mode.
- `POST /v1/memory` — store. 422 dim_mismatch on length
mismatch; 403 write_denied on ACL deny.
- `POST /v1/memory/search` — vector recall (POST because
a 1024-element Float32 array doesn't fit a query string).
- `GET /v1/memory/count?collection=...` — count, optionally
scoped to one collection.
- All error responses use M1's `{error: {code, message}}`
envelope. Embeddings on the wire are JSON `[float32, ...]`
arrays (verbose but trivial for the M3 TS client; base64 raw
bytes is a follow-up if wire size measurably matters).
- `/v1/healthz` extended with `store_ok`, `store_path`,
`embedding_dim`, and `store_error` (omitempty). When the
store fails to open, the daemon still serves `/v1/healthz`
with `store_ok=false` and `/v1/memory/*` returns 503
`store_unavailable` — partial degradation beats refuse-to-start
so the daemon stays observable when something's wrong with the DB.
- `serve` gets `--db-path` (default
`$HOME/.opencoti/memory/state.db` — matches the TS
`defaultDbPath()` exactly so both implementations point at the
same file by default) and `--embedding-dim` (default 1024,
matches `DEFAULT_EMBEDDING_DIM`). `install` bakes both into
the SCM service-start arguments.
- 30 tests cover the surface: 18 unit tests on the store
conformance (collection lifecycle, ACL resolver, idempotent
store, recall ordering, count, close+reopen, dim-mismatch
paths) + 12 HTTP integration tests via `httptest.NewServer`
(every endpoint, happy + 4xx + 5xx envelopes).
- `testdata/crosslang/` ships a Go↔Bun schema-compat probe:
Go writes a DB → optionally invokes a Bun script that opens
the same DB via `@opencoti/memory` and confirms readback.
Skips cleanly when bun or the workspace package isn't on
PATH. `make test-cross-lang` is always safe to run.
- **Hook footprint: ZERO** (unchanged from M1). M2 is purely
additive — the API exists but no surgical hooks into opencode
yet.
- **Deferred from M2:** text→vector convenience
(`GET /v1/memory?q=<text>`); streaming recall (SSE);
pagination on list-collections; authentication for TCP
transport (UDS owner-only on Unix is the M2 security
boundary); pgvector backend (M7); migration tools (schemas
are identical — the DB just opens); server-side embedder
cache; bulk-store / batch endpoints; base64 raw-byte
embedding wire format.
### M3 — TS client + opencoti hook to start the server *(2026-05-23 — shipped)*
- `@opencoti/opencoti-server-client` published in-workspace under
`packages/opencoti-server-client/`. Single class
`OpencotiServerClient` implements `MemoryStore` over HTTP — all
eight `/v1/memory/*` routes from M2 plus a `healthz()` probe.
Wire format is M2's verbatim: JSON `[float32, ...]` for embeddings;
M1/M2's `{error: {code, message}}` envelope decoded into a
sentinel `OpencotiServerError({status, code, message})` so callers
can match on `code === "store_unavailable"`, `dim_mismatch`, etc.
Per-request `AbortController` + 5 s default timeout. `close()` is
a documented no-op (no per-instance handle to close).
- **Transport** uses Bun's native `fetch` — `tcp://host:port` is
rewritten to `http://...` in `toFetchBase`; `unix:///path` passes
through verbatim (Bun supports UDS fetch natively, no custom
adapter needed). `defaultAddress(platform)` mirrors the Go
`DefaultAddress()` exactly: `$XDG_RUNTIME_DIR/opencoti/server.sock`
on Linux (`$HOME/.opencoti/server.sock` fallback), and
`tcp://127.0.0.1:47190` on Windows.
- **Autostart** lives in `src/autostart.ts`:
`startAutostart({address, binaryPath?, dbPath?, embeddingDim?, ...})`
probes `/v1/healthz` first (200 → `already_running`), locates the
binary via `Bun.which` if none was passed (→ `no_binary` when
missing), spawns with `stdio: "ignore"` (mirroring the
`@opencoti/embedder` manager pattern), and polls `/v1/healthz`
every 200 ms until the deadline (default 5 s; →
`ready_timeout` if never recovers). Returns a discriminated
`AutostartOutcome` so consumers can log five distinct states
(`already_running | spawned | no_binary | spawn_failed | ready_timeout`)
without exception handling. Test seams: `spawnImpl`, `fetchImpl`,
`whichImpl`, `sleepImpl`.
- **Surgical hook footprint: 3 markers + 1 package.json dep.** All
three TS markers land in `packages/opencode/src/config/config.ts`
(the same file that already carries `opencoti-default-plugins`):
the import next to it, the `opencoti.server.*` schema struct
(`autostart: boolean`, `address: string`, `binary_path: string`,
`db_path: string`, `embedding_dim: PositiveInt`), and the
fire-and-forget `void maybeStartOpencotiServer(...)` call site
right after the existing `applyDefaultPlugins(...)` call. The
hook's full substance lives in
`@opencoti/opencoti-server-client/autostart-hook` so the
opencode-side surface stays minimal — three lines plus the
package.json dep (no anchor comment in JSON, registered in the
table).
- **Plugin swap.** Both `@opencoti/memory-plugin` and
`@opencoti/tui-memory` accept a new optional
`server_address?: string`. The default `StoreFactory`:
if `serverAddress` is set, instantiate `OpencotiServerClient`,
call `healthz()`, and return the client when
`storeOk === true`; otherwise silently fall through to the
existing in-process `SqliteVecStore.open(...)`. The
`__setStoreFactory` test seam is unchanged — existing tests
continue to inject stubs that ignore `serverAddress`.
- **Health envelope mismatch detection.** Client validates that
`embedding_dim` from the server matches the configured dim;
on mismatch, `healthz()` returns `storeOk: false` with a
`dim mismatch` error so the plugin falls back to the
in-process store rather than corrupting the wire format.
This is **client-side**; the server's own dim check still
fires on `POST /v1/memory`.
- **Tests.** 56 new tests in the client package: 33 mocked-fetch
client conformance (every endpoint, happy + 4xx + 5xx envelopes,
dedup, dim mismatch, store_unavailable propagation), 11 pure
tests for `defaultAddress` / `toFetchBase` / `joinURL` /
`probeReady`, 6 mocked-spawn `startAutostart` cases (already
running, no binary, spawned-then-ready, ready timeout,
spawn failed, --addr passthrough). Plus 2 new cases in the
memory-plugin tests and 3 new in the tui-memory tests for the
`server_address` swap path. All 23 workspace packages
typecheck clean; 82 tests pass across the three affected
packages.
- **Live verification.** Manual end-to-end smoke against the
F3 M2 binary (TCP loopback): `healthz` returns the expected
shape (`store_ok: true`, `embedding_dim: 64`), `createCollection`
+ `setSessionAcl(rw)` + `store(...)` (with `deduplicated:
false` then `true` on idempotency) + `recall(...)` (distance
0 on exact match) + `count(...)` (returns 1) + `deleteCollection`
all return the expected shapes. UDS path tested at the
discover / autostart layer; full UDS smoke deferred to
M4 when session-event hooks bring it under day-to-day use.
- **Path-B loop closed.** With M3 in, the F4 M5
in-process-then-daemon migration path described in
[memory_embedder.md](memory_embedder.md) is mechanically
complete: setting `opencoti.server.address` (or `autostart:
true`) in `opencode.jsonc` is the only change a user makes
to switch from the in-process store to the Go daemon. No
code paths change in `@opencoti/memory` itself.
- **Deferred from M3:** detached / setsid daemon lifecycle
(M3's spawn dies with opencode — cross-session sharing
requires running the daemon externally, e.g. the M1 Windows
service installer or a future systemd unit); the first-run
"install as a service?" wizard step (a follow-up once the
setup-flow picks up F3 concerns); a text-`q=` convenience
endpoint (`GET /v1/memory?q=<text>`) — every M3 caller has
the embedder in-process and can pre-embed; retry / backoff
on transient HTTP errors (single attempt, the plugin's
silent fallback handles the failure case); bulk-store /
batch endpoints; TLS / shared-secret authentication for TCP
transport (UDS owner-only remains the M3 security boundary).
### M4 — Session + turn events flowing *(2026-05-23 — shipped)*
**Design pivot vs the original spec.** M4 was originally specced
as *surgical hooks at session start/end/turn complete*. Phase-1
exploration confirmed opencode's `@opencode-ai/plugin` API
already exposes those events fully typed via `Hooks.event`
(session.created/updated/deleted/error/idle plus
session.status). Following the precedent set by M5-D2
(`@opencoti/memory-plugin`), M4 ships as a plugin with **zero
opencoti-hook footprint** — strictly richer than a surgical hook
for this use case (typed payloads, zero upstream-source touch,
no UPSTREAM_SYNC.md row to maintain on every sync). Hook count
stays at 9 source markers + 4 JSON deps (unchanged from F3 M3).
- **Per-feature schema versioning.** `meta.schema_version` stays
at 1 (memory tables — kept stable so M2/M3 TypeScript
`@opencoti/memory` clients still open M4 DBs cleanly). New
`meta.sessions_schema_version = 1` is written by M4+ daemons
and reported on `/v1/healthz`. M3 clients without the key
treat the daemon as pre-M4; M4 plugins disable forwarding
silently when `sessionsSchemaVersion < 1`.
- **Three new tables** in the existing `state.db` (no new
`--flag`, no migration tool):
- `sessions` — soft-delete via `deleted_at`. Indices on
`parent_id` (fork-tree queries) and `deleted_at`
(cheap `WHERE deleted_at IS NULL`). Upsert is idempotent
on `id`; `INSERT ... ON CONFLICT(id) DO UPDATE SET ...`
overwrites all columns from the latest snapshot.
- `session_events` — append-only audit log. `FOREIGN KEY ...
ON DELETE CASCADE` would normally wipe history on
`DELETE FROM sessions`; we use soft-delete instead so the
audit trail outlives the session row.
- `session_messages` — UPSERT on opencode `MessageID`. The
plugin re-sends the full latest snapshot on every turn
boundary; SQLite UPSERT handles dedup.
- **SessionStore interface** (`internal/store/store.go`): 8
methods + sentinel errors (`ErrSessionNotFound`,
`ErrInvalidSessionID`, `ErrSessionPayloadTooLarge`). The
same `*sqlitevec.Store` implements both `Store` (memory) and
`SessionStore`; server.go type-asserts at each entry point.
M7 pgvector will implement the same interface.
- **HTTP surface** under `/v1/sessions/*`:
- `POST /v1/sessions` — upsert (201 on insert, 200 on update,
400 invalid_session_id, 413 payload_too_large).
- `GET /v1/sessions[?parent_id=&include_deleted=&limit=&offset=]`
— list, default newest-first by `updated_at`, omitting
soft-deleted rows.
- `GET /v1/sessions/{id}` / `DELETE /v1/sessions/{id}` —
fetch / soft-delete.
- `POST /v1/sessions/{id}/events` /
`GET /v1/sessions/{id}/events[?type=&limit=&offset=]` —
append + list audit entries.
- `POST /v1/sessions/{id}/messages` /
`GET /v1/sessions/{id}/messages[?limit=&offset=&since=]` —
bulk-upsert (idempotent on message id) + list.
- `GET /v1/sessions/{id}/turns` — coarse server-side
projection grouping messages by user-message boundaries.
Authoritative turn-boundary logic stays in opencode's
`compaction.ts`; this view is for UI / debugging.
- **healthz extension**: `sessions_schema_version` field
added; M3 clients without the field treat the daemon as
pre-M4.
- **Payload size guards**: 8 MiB hard cap on
`session_events.payload` and the bulk-upsert messages body
(413 `payload_too_large`); the plugin chunks at a softer
1 MiB default before posting.
- **TS client extension** (`@opencoti/opencoti-server-client`):
8 new `OpencotiServerClient` methods (`upsertSession`,
`listSessions`, `getSession`, `deleteSession`,
`appendSessionEvent`, `listSessionEvents`,
`upsertSessionMessages`, `listSessionMessages`,
`listSessionTurns`). Wire format snake_case → camelCase via
private mappers (same pattern as M2's `mapHit` /
`mapCollectionInfo`). `healthz()` return type +
`ProbeResult` gain `sessionsSchemaVersion?: number`.
- **New plugin `@opencoti/session-events-plugin`**:
subscribes to `session.created/updated/deleted/error/idle`
and `session.status` (idle transition) via opencode's
typed `Hooks.event`; does **not** subscribe to
`message.updated` (fires thousands of times per turn —
text deltas, tool calls, reasoning chunks — and forwarding
every delta would melt the daemon). On idle, the plugin
fetches the session's latest messages via
`input.client.session.messages.list` and bulk-upserts in
byte-bounded chunks. Maintains a per-session merge cache so
opencode's partial `session.updated` payloads are reconciled
into a full snapshot before reaching the daemon.
Best-effort, never-throws: a forwarding plugin must not
propagate daemon failures into the opencode session.
- **session.error before sessionID exists** lands on a
synthetic `__pre_session_errors__` session (lazily upserted
on first such error) so the audit log captures pre-creation
failures without losing context.
- **Auto-wire**: one-line addition to
`@opencoti/tiers/default-plugins` `OPENCOTI_DEFAULT_PLUGINS`.
Any user with opencoti config in `opencode.jsonc` gets the
plugin automatically; without an `opencoti.server.address`
the plugin's healthz probe fails, the internal client stays
`undefined`, and every event handler early-returns — same
no-op-when-unconfigured ergonomic the memory-plugin uses.
- **Tests**: 26 new Go tests (16 store unit + 10 HTTP
integration) — total Go test count climbs from 30 → 63.
14 new TS client tests (33 → 47). 16 new plugin tests in
`@opencoti/session-events-plugin`. All 24 workspace
packages typecheck clean.
- **Live verification**: end-to-end smoke against a real M4
daemon (TCP loopback) confirmed healthz reports
`sessions_schema_version: 1`, full upsert → update → get →
list → events → messages → turns → soft-delete cycle
works, soft-deleted session's events + messages remain
queryable through `include_deleted=true`.
- **Deferred from M4**:
- **`message.updated` real-time forwarding** — coalescing on
idle is the right granularity; per-delta forwarding would
mean O(text-deltas) HTTP roundtrips per turn.
- **Per-tool-invocation events** as first-class rows —
captured today inside the message's `parts` JSON; can be
promoted to a `session_tool_calls` table later if needed.
- **WebSocket / SSE push from daemon → client** —
request-response only in M4.
- **TLS / authentication for `/v1/sessions/*` TCP transport**
— UDS owner-only on Unix remains the M4 security boundary.
### M5 — Tier-engine event sink (shipped 2026-05-23)
- **Daemon side: `/v1/tier-events`.**
- `POST /v1/tier-events` appends a row (201; 400 invalid_tier_event;
413 payload_too_large at the same 8 MiB cap as `/v1/sessions/*`).
Body: `{event_type, ts?, session_id?, payload}` where `payload`
is opaque JSON — the daemon does not introspect it.
- `GET /v1/tier-events?session_id=&event_type=&since=&limit=&offset=`
returns the audit log newest-first.
- One new `tier_events` table: `(id PK AUTO, session_id TEXT,
event_type TEXT, payload TEXT, ts INTEGER)` + two indices
`(session_id, ts)` and `(event_type, ts)`. **`session_id` is a
soft reference — no FK, no cascade.** Tier events fire intra-turn
and can land before `session.created` reaches the daemon; the
soft reference guarantees no event is dropped.
- Per-feature `meta.tier_events_schema_version = 1`. The
cross-language sqlite_vec contract continues to live on
`meta.schema_version = 1` (still M2-compatible with the TS
`@opencoti/memory` reader); the new key is independent.
- **TS plumbing.** `@opencoti/opencoti-server-client` gains
`appendTierEvent` + `listTierEvents` (camelCase→snake_case wire
mapping mirroring `mapSessionMessage`). `healthz()` surfaces
`tierEventsSchemaVersion?: number`; plugins use it as their
feature gate.
- **Design pivot continues: plugin path > surgical hook.** The
`Telemetry` interface in `@opencoti/tiers` (designed for this
milestone — its docstring at `telemetry.ts:1-4` said so) is
already threaded end-to-end through `runtime.ts:99 →
executor.ts → escalator.ts → fanout.ts`. M5 plugs a real sink
into the existing seam via a new module-level
`registered-telemetry.ts` slot (mirrors `active-config.ts`).
Precedence chain in `runtime.ts:99` becomes:
```ts
const telemetry =
hook.telemetry ??
execOpts.telemetry ??
getRegisteredTelemetry() ??
noopTelemetry
```
The new `@opencoti/tier-events-plugin` calls
`setRegisteredTelemetry(impl)` during plugin boot when the
daemon's healthz reports `tierEventsSchemaVersion >= 1`. **Zero
new opencoti-hook source markers** — count unchanged from F3 M4
(9 source + 4 JSON deps + 3 prose). See the F1 page for where
the Telemetry interface itself was added.
- **Fire-and-forget posture.** The plugin's Telemetry impl maps
each method to a `client.appendTierEvent(...).catch(() => {})`
— synchronous to the caller, HTTP swallowed on failure. The
tier engine never blocks on the daemon. Cf. M4's
`@opencoti/session-events-plugin`, same design.
- **Auto-wire via defaults.** `@opencoti/tier-events-plugin` is in
`OPENCOTI_DEFAULT_PLUGINS` alongside the M4 session plugin. Any
user with `opencoti: {...}` in their config gets the audit log
for free; if the daemon is unreachable, the healthz gate fails
and the runtime stays on noop.
- **Cross-version compat.** An M5 plugin against an M4 daemon
sees `sessions_schema_version: 1` but no
`tier_events_schema_version` in healthz, so
`setRegisteredTelemetry` is never called and the runtime stays
on noop. An M5 daemon against M4 clients keeps shipping the
same `/v1/sessions/*` surface untouched.
- **Explicitly deferred to later:** aggregation endpoints
(`/v1/tier-events/stats?...`), real-time push (WS/SSE),
server-side retention policy, client-side batching/coalescing,
FK on `session_id`. M5 ships raw events + filters; everything
on top of that is composable later.
### M6 — Tool registry + invocation (shipped 2026-05-23)
- **Daemon side: `/v1/tools`.**
- `GET /v1/tools` lists every registered tool sorted by name:
`{ tools: [{ name, description, params_schema, handler,
created_at, updated_at }] }`. `params_schema` is a
string-serialised JSON Schema; `handler` is a stable opaque
dispatch ID (debug-only — invocation always goes by name).
- `GET /v1/tools/{name}` fetches one (404 tool_not_found).
- `POST /v1/tools/{name}/invoke` runs it. Body
`{ arguments: {...}, session_id? }` → `{ result: {...} }` (200)
or `{ error: { code, message } }` (400 invalid_arguments / 404
tool_not_found / 500 handler_failed / 413 payload_too_large at
the same 8 MiB cap as `/v1/sessions/*` and `/v1/tier-events`).
- One `tools` table: `(name PK, description, params_schema,
handler, created_at, updated_at)`. Process-scoped, no
session FK. Dispatch is a daemon-internal
`map[string]ToolHandler` in `toolhandlers.go`; **no external
registration in M6** (no POST/DELETE on `/v1/tools` itself) —
the catalog is seeded into the binary at boot.
- Per-feature `meta.tools_schema_version = 1`, independent of the
cross-language `meta.schema_version = 1` (still M2-compatible
with the TS `@opencoti/memory` reader).
- **Two seed tools** are `UpsertTool`'d at startup (idempotent —
`created_at` fixed on first insert, `updated_at` bumps on
re-seed):
- **`opencoti_episodic_search`** — lexical `LIKE` search across
the M4 `session_messages` table joined with `sessions`,
newest-first. Params `{ query, limit?, session_id_excludes? }`.
The exclude list lets the plugin pass the live session ID so
the model recalls *other* sessions, not echoes of the current
one. The integrated analog of claude-hooks's `episodic_server`
— same SQLite file the sessions API writes, no second daemon,
no shell-out. FTS5/vector ranking is an M7+ optimisation.
- **`opencoti_recall`** — vector recall over the M2 memory store,
delegating to the same `Store.Recall` path `/v1/memory/search`
uses. Params `{ embedding, collection?, k?, session_id? }`.
**Takes a pre-computed embedding, not a query string** — the
daemon bundles no embedder, so the caller (the plugin, or an
operator via curl) embeds the query and POSTs the float vector;
length must equal the daemon's `embedding_dim`. `session_id`
drives the M2 per-session ACL check. Coexists intentionally
with `@opencoti/memory`'s in-process `__memory_recall`: that
one survives a daemon being down; `opencoti_recall` gives the
same surface to plugins without direct DB access.
- **TS plumbing.** `@opencoti/opencoti-server-client` gains
`listTools` + `getTool` + `invokeTool`; `healthz()` surfaces
`toolsSchemaVersion?: number` as the plugin's feature gate.
`ServerTool` is camelCase (`paramsSchema` ← `params_schema`),
`paramsSchema` left as `unknown` (interpreted in the plugin).
- **Design pivot continues: plugin path > surgical hook.** A new
module-level `registered-server-tools.ts` slot in
`@opencoti/tiers` (mirrors M5's `registered-telemetry.ts`) holds
the daemon tools as a `Record<string, Tool>`; `runtime.ts` merges
it **last** into the tier tool list, next to synthetic-tier and
memory-bridge tools:
```ts
const tools = mergeTools(hook.prepared.tools, {
...syntheticTools(hook.input.sessionID, telemetry),
...memoryTools,
...(getRegisteredServerTools() ?? {}),
})
```
Server tools merge last so user-explicit + memory tools win on a
name collision (none expected — server tools are `opencoti_*`,
memory `__memory_*`, synthetic-tier `__tier_*`). **Zero new
opencoti-hook source markers** — count unchanged from F3 M4/M5
(9 source + 4 JSON deps + 3 prose).
- **The `@opencoti/server-tools-plugin`** probes `/v1/healthz`
(gated on `toolsSchemaVersion >= 1`), fetches `/v1/tools` once on
boot, and builds an AI SDK `dynamicTool` per entry:
`params_schema` round-trips through `ai`'s `jsonSchema()` (no
JSON-Schema-to-Zod reimplementation); each `execute` forwards to
`/v1/tools/{name}/invoke`. Tool names are prefixed (`opencoti_`,
idempotently). The session ID arrives via `experimental_context`
on the single `openStream` `streamText` call and is forwarded as
`session_id`. Invocations are awaited (not fire-and-forget — tool
calls are model-blocking), but errors crash the *tool call*, not
the session: 404→`tool_not_found`, 400→`invalid_arguments`,
5xx/network→`tool_handler_failed`, deadline→`timeout`
(plugin-owned `Promise.race`, separate `invoke_timeout_ms`).
Args over 1 MiB are pre-rejected client-side.
- **Auto-wire via defaults.** `@opencoti/server-tools-plugin` is in
`OPENCOTI_DEFAULT_PLUGINS`. No reachable daemon → healthz gate
fails → `setRegisteredServerTools` never called → runtime tool
set unchanged.
- **Cross-version compat.** An M6 plugin against an M5 daemon sees
no `tools_schema_version` in healthz, so the gate fails and the
tool set is unchanged — a model query expecting the tool gets
"I don't have that tool", no crash. An M6 daemon serves M5
clients the `/v1/sessions/*` and `/v1/tier-events` surfaces
untouched.
- **Explicitly deferred:** external tool registration (POST/DELETE
on `/v1/tools` — needs a per-tool ACL/owner model); streaming
tool results; tool invocations as `tier_events` audit rows; an
MCP wrapper for the registry; FTS5/vector ranking in
`episodic_search`; an in-daemon embedder so `opencoti_recall`
can take a `query` string directly.
### M7 — pgvector backend parity (shipped 2026-05-23)
opencoti-server gains its **second storage backend**: PostgreSQL +
the pgvector extension, via `github.com/jackc/pgx/v5` (pure Go, no new
cgo). `*pgvector.Store` implements the **full** interface family —
`Store` + `SessionStore` + `TierEventStore` + `ToolStore` +
`SchemaInspector` — so the HTTP layer, `/v1/healthz`, and the TS client
are **untouched**. sqlite-vec stays the zero-dependency default;
pgvector is for hosts that already run Postgres.
**Schema-phrasing correction.** The original M7 stub said *"/v1/memory
works against pgvector with the same schema as claude-hooks."* That
predated the M4–M6 buildout. opencoti's binding contract is now its
**own** `Store` interface (collections + per-session ACL + sessions +
tier_events + tools), which is a different data model from
claude-hooks's flat `memories` + `kg_*` schema. So M7 implements
**opencoti's own model on a dedicated `opencoti` Postgres database**,
isolated from claude-hooks. "Same as claude-hooks" is reread as *same
storage technology (Postgres + pgvector)*, not the same tables.
- **`internal/store/pgvector/`** mirrors `internal/store/sqlitevec/`
table-for-table in PG dialect, with three dialect differences:
- the embedding lives **inline** on `memories` as a `vector(dim)`
column (no separate vec0 virtual table); an HNSW `vector_l2_ops`
index is created when `dim ≤ 2000` (pgvector's HNSW ceiling) —
correctness-neutral, perf-positive.
- `Recall` pushes the ACL filter into the query
(`WHERE collection_name = ANY($readable) ORDER BY embedding <-> $q
LIMIT k`) — Postgres can filter + rank in one statement where
sqlite-vec must overscan + join-back + post-filter. The `<->`
operator is **L2**, matching sqlite-vec's vec0 default, so
`MemoryHit.Distance` stays comparable across backends.
- `Store` dedups via `INSERT … ON CONFLICT (collection_name,
content_hash) DO NOTHING RETURNING id` in one round trip.
- **Per-feature schema versions** (`schema_version`,
`sessions_/tier_events_/tools_schema_version`, all `1`) live in a
`meta` key/value table, validated on open exactly as the sqlite-vec
backend does — but they are pgvector's own contract (no cross-language
reader, since pgvector is daemon-only).
- **Shared conformance harness** `internal/store/storetest/` is the
parity guarantee: a single `RunConformance` body (Memory, Sessions,
TierEvents, Tools sub-suites) that **both** backends opt into via a
thin `conformance_test.go`. Either backend drifting from the contract
fails the same assertions. sqlite-vec runs it with `t.TempDir`;
pgvector with a testcontainers fixture.
- **Pure-logic lifted to `internal/store/common.go`** (package `store`):
the ACL resolver (`ResolveAccessMode`/`CanRead`/`CanWrite`), the
collection/session validators, `Sha256Hex`, and the dim bounds — one
cgo-free source of truth both backends share. sqlite-vec keeps its
exported names as thin delegating wrappers, so the HTTP layer's
`sqlitevec.ValidateCollectionName`/`ValidateSessionID` call sites are
byte-identical.
- **Backend selection** is a serve-time flag, not a build flag:
`--backend sqlite|pgvector` (default `sqlite`) + `--pg-dsn` (falling
back to `$OPENCOTI_PG_DSN`). `--db-path` stays sqlite-only;
`--embedding-dim` applies to both (locks the `vector(dim)` column).
The interface var is assigned only on a successful open (avoids the
typed-nil trap), so a pgvector open failure degrades to
`store_ok=false` + a clear `store_error` rather than crashing. The
Windows `install` command bakes `--backend`/`--pg-dsn` into the
service args alongside the existing flags.
- **healthz** reports a **redacted** `store_path` for pgvector
(`pg://user@host:port/db`, never the password).
- **Tests**: `pgtest` spins an ephemeral `pgvector/pgvector:pg17`
container via testcontainers-go and skips cleanly when Docker is
absent (`testcontainers.SkipIfProviderIsNotHealthy`), so
`make test` stays green on a Docker-less host. Both backends pass the
full shared conformance suite; an end-to-end smoke (daemon →
pgvector container) confirms collection create, store, and `<->` -
ranked recall over HTTP.
- **Toolchain**: the modern pgx / pgvector-go / testcontainers-go
releases require **Go 1.25**, so the module's `go` directive and the
host toolchain moved to go1.25 (latest stable). No build-posture
regression — pgx is pure Go; sqlite-vec's existing cgo requirement is
unchanged.
- **Zero new surgical hooks.** pgvector is additive Go inside
`opencoti/server/`; no TS changes. The surgical-hook grep count
stays at 18 (see `docs/protocols/UPSTREAM_SYNC.md`).
**Deferred from M7:** data migration between backends (a `migrate`
subcommand — "Migration tools later"); a TS pgvector backend for the
in-process `@opencoti/memory` (pgvector is daemon-only); external tool
registration over HTTP (still security-gated, from M6); pgxpool tuning
/ read replicas.
### M8 — Multi-session coordination (shipped 2026-05-23)
Goal **G3**: opencoti instances/sessions on one host see each other and
cooperate. M8 ships the *live-coordination* half as an **in-memory,
ephemeral** hub — deliberately store-independent (neither the `Store`
interface nor either backend is touched), because presence and locks are
runtime state that should not survive a daemon restart.
- **`internal/coord.Hub`** — backend-agnostic, one `sync.RWMutex`:
- **Presence**: `RegisterPeer` / `Heartbeat` / `DeregisterPeer` /
`ListPeers`, with a background TTL sweep that reaps peers whose
heartbeat lapsed (default 45s) and emits `peer.left`. `ListPeers`
also filters expired peers lazily.
- **Broadcasts**: a pub/sub bus — `Publish` assigns a monotonic `seq`,
appends to a bounded replay ring, and fans out non-blockingly to
subscribers (a full subscriber channel is dropped + closed so the
client reconnects with its last seq). `Subscribe(since)` atomically
snapshots the replay backlog and registers for live events.
- **Advisory locks**: try-only `AcquireLock` (reentrant-by-holder
refresh; expired locks reclaimable) / `ReleaseLock` (holder-checked)
/ `ListLocks`. Lock transitions emit `lock.acquired` / `lock.released`.
- **HTTP** (`internal/server/coord.go`), hub injected via `Options.Hub`,
503 `coord_unavailable` when absent:
| Method + path | Purpose |
| --- | --- |
| `POST /v1/coord/peers` | register/upsert presence → PeerInfo |
| `POST /v1/coord/peers/{id}/heartbeat` | refresh TTL |
| `DELETE /v1/coord/peers/{id}` | deregister |
| `GET /v1/coord/peers` | list live peers |
| `POST /v1/coord/broadcast` | publish `{peer_id,topic,payload}` → `{seq}` |
| `GET /v1/coord/events?since=&peer_id=` | **SSE** event stream (the daemon's first) |
| `POST /v1/coord/locks/{name}` | acquire `{holder,ttl_ms?}`; 200 or 409 `lock_held` |
| `DELETE /v1/coord/locks/{name}` | release `{holder}`; 200 / 404 `lock_not_held` / 409 `lock_not_holder` |
| `GET /v1/coord/locks` | list held locks |
SSE is viable because the `http.Server` sets no `WriteTimeout`; the
handler exits on request-context cancellation so graceful shutdown
releases it within the grace window. `/v1/healthz` gains
`coord_ok` + `coord_peers`.
- **TS client**: `registerPeer` / `heartbeatPeer` / `deregisterPeer` /
`listPeers` / `broadcast` / `acquireLock` / `releaseLock` /
`listLocks`, plus `subscribeCoordEvents` — the client's first
streaming method (reads `response.body`, parses `data:` frames).
- **`@opencoti/coordination-plugin`**: registers each session as a peer,
heartbeats while active, deregisters on delete, subscribes to the
event stream (SSE) to keep a live peer view, and advertises
"N other active opencoti session(s)" in the system prompt. Auto-wired
via the `@opencoti/tiers` default-plugins list — **no surgical hook**
(grep count stays 18).
**Deferred to F3 M9:** opt-in *shared context* (a thin convention atop
existing M2 global collections + per-session ACL — a session opts to
expose a collection to peers); the sqlite↔pgvector `migrate` subcommand
(open from M7); blocking/queued lock acquire (M8 is try-only). Cross-host
federation remains an explicit F3 non-goal.
### M9 — Opt-in shared context (shipped 2026-05-23)
Goal **G3**, the *persisted-context* half: a session exposes one of its
session-scoped memory collections to peer sessions on the same host. The
storage model from M2 already supports the grant, so M9 adds **no new
`Store` method, no new schema, no new surgical hook** (grep count stays
18). It is a thin convention bridging two things that already exist — the
persisted per-session ACL (`SetSessionACL`) and M8's ephemeral coord bus.
- **`internal/share.Manager`** — in-memory registry + one goroutine:
- `Share(collection, owner, mode)` records the share, grants read-ACL
to every live peer `!= owner` via `SetSessionACL`, and publishes
`collection.shared` on the hub.
- It **subscribes to the hub** and, on `peer.joined`, grants every
active share to the newcomer — so peers that join *after* a share
still get access (auto-grant via the bus, dogfooding M8's SSE).
- `Unshare(collection, owner)` revokes live peers (writes mode `none`,
the non-owner default) and publishes `collection.unshared`.
- **Persisted vs ephemeral:** the ACL grants persist (they survive a
restart); the "keep auto-granting new joiners" intent is in-memory
and lost on restart by design — existing grants remain, but the owner
must re-share to resume auto-granting. This keeps M9 storage-free.
- **Recall "just works":** once a peer holds an `r` grant, its no-filter
`Recall` includes the shared collection automatically (M2's
readable-collections resolution), so no recall-path change is needed.
- **HTTP** (`internal/server/share.go`), Manager constructed in
`server.New()` when `Store`+`Hub` are present, released on `Shutdown`;
503 `share_unavailable` when the hub is absent:
| Method + path | Purpose |
| --- | --- |
| `POST /v1/memory/collections/{name}/share` | owner-only share `{owner_session, mode?}` (mode default `r`) → `{granted}` |
| `DELETE /v1/memory/collections/{name}/share` | withdraw `{owner_session}` → 200 |
| `GET /v1/memory/shares` | list active shares |
**Owner-only:** the handler looks the collection up via
`ListCollections` (no filter) and requires `scope == session` and
`session_id == owner_session`; global collections (already `r`-for-all)
are rejected `not_shareable`, a different owner `not_owner` (403).
- **TS client**: `shareCollection` / `unshareCollection` / `listShares`
+ `SharedCollectionInfo`.
- **`@opencoti/coordination-plugin`** (extended in place — no new plugin):
consumes `collection.shared` / `collection.unshared` to advertise
peer-shared collections in the system prompt, and gains an opt-in
`share_session_collection` flag (default **false**) that shares the
session's own collection (`sessionCollectionName(id)`) on
`session.created` and unshares it on `session.deleted` (best-effort; a
not-yet-created collection's 404 is swallowed).
**Deferred to F3 M10:** the sqlite↔pgvector `migrate` subcommand — **not**
a cutover: both backends are first-class and may run in parallel; migrate
is an idempotent, re-runnable, either-direction copy (`--from`/`--to`,
dedupe on `content_hash`, upsert PKs) that tops up a parallel target,
never abandoning the source. Also deferred: revoke-on-peer-leave cleanup
(M9 keeps grants on `peer.left`, since peers may return) and wildcard /
group ACLs.
### M10 — sqlite↔pgvector `migrate` (shipped 2026-05-24)
A daemon-internal `migrate` subcommand that copies data between the two
first-class backends. It is **not a cutover**: both backends stay
first-class and may run in parallel (two daemons, or alternating
`--backend`). `migrate` is an **idempotent, re-runnable, either-direction,
selectable copy** that *tops up* a target (dedupe on natural keys), never
"move then abandon source". Additive Go only — **no plugin, no HTTP
surface, no TS client, no surgical hook** (grep count stays 18).
**Use cases (maximum flexibility):**
1. **Scale-up (primary).** Start on sqlite; migrate *everything*
(collections, memories+embeddings, ACL, sessions, session_events,
session_messages, tier_events, tools) into pgvector; then **switch the
primary backend** by changing the serve flag to `--backend pgvector`.
"Switch primary backend" is operational, no extra code: run the full
migrate, then change `--backend`.
2. **Way back.** Same command with `--from`/`--to` swapped.
3. **Single memory container / additional partial backend.** Copy only
specific collection(s) with `--collection`, so e.g. pgvector holds just
certain memories while sqlite keeps the rest. The two backends coexist,
each holding different data.
4. **Additive top-up.** Re-running, or copying into a populated target,
merges idempotently (natural-key dedupe).
**Selection model — two orthogonal selectors:**
- `--include <csv>` of sections: `memory,sessions,tier-events,tools`.
Default (unset) = **all four** (full dataset). The `memory` section
carries collections + their memories + their ACL rows.
- `--collection <name>` (repeatable) restricts the `memory` section to
those collections only. When `--collection` is given and `--include` is
unset, the default narrows to **memory-only** (the single-container
case).
So: full switch = no selectors; single container = `--collection notes`;
logs-only = `--include sessions,tier-events`.
**Copy phases (FK-respecting; each gated by the selection):**
| Section | Phase | Export | Import |
| --- | --- | --- | --- |
| memory | collections | `ListCollections` (filtered) | `PutCollection` |
| memory | memories | `ListMemories` (per coll, keyset-paged) | `PutMemory` |
| memory | session_acl | `ListSessionACLs` | `SetSessionACL` |
| sessions | sessions | `ListSessions(+deleted)` | `UpsertSession` |
| sessions | session_events | `ListSessionEvents`/sess | `PutSessionEvent` |
| sessions | session_messages | `ListSessionMessages`/sess | `UpsertSessionMessages` |
| tier-events | tier_events | `ListTierEvents` (batched) | `PutTierEvent` |
| tools | tools | `ListTools` | `UpsertTool` |
Both backends are the same concrete `*Store` implementing every feature
interface, so migrate opens each side as `storepkg.Store`, type-asserts
the optional `MigrationStore`, and copies a section only if **both** sides
implement it. Per-phase counts (scanned / inserted / skipped) print at the
end; `--dry-run` reads sources and reports would-copy counts, no writes.
**Idempotency / dedup.** Memories dedupe on `(collection_name,
content_hash)`. The append-only audit logs (`session_events`,
`tier_events`) have **no natural key** (autoinc id only), so their
idempotent import dedups on an **insert-if-no-identical-row** check (all
business columns match) — re-running produces no duplicates. A second full
run reports 0 inserts everywhere.
**Two backend-specific wrinkles (both resolved):**
- **sqlite-vec embeddings are an opaque BLOB.** vec0 `memory_vecs` stores
the embedding as a blob and the Go binding ships `SerializeFloat32` but
no deserialize. The on-disk format is plain little-endian `float32`
(`binary.Write(buf, LittleEndian, vector)`), so `deserializeFloat32`
reverses it with `binary.Read` at the locked dim — **byte-faithful, no
re-embedding**. pgvector exports via `pgvector.Vector.Scan` + `.Slice()`.
- **The public `Store()` write path enforces ACL** (a global collection is
`r`-for-all → `ErrWriteDenied`), so migrate cannot reuse it for imports.
The new `MigrationStore.PutMemory` is **ACL-free and timestamp-
preserving**.
**Fidelity contract (documented caveat).** Preserved exactly: collection
`created_at`, memory `ts`+`content`+embedding (byte-identical), session
`created_at`/`deleted_at`, message `created_at`/`finished_at`, event
`occurred_at`/`ts`, all ACL modes. Rewritten to migrate-time (reused
upserts stamp `now()`): `sessions.updated_at`, `tools.created_at`/
`updated_at` — acceptable "last-written" fields (tools are also re-seeded
at startup). A byte-identical `PutSession`/`PutTool` is deferred.
The new `MigrationStore` optional interface (in `store.go`, mirroring the
`SchemaInspector`/`SessionStore` convention) is exercised by the
`storetest` conformance harness on **both** backends — the round-trip
asserts the exported embedding is byte-identical, validating the sqlite LE
deserialize and the pgvector scan together. Verified end-to-end across a
`sqlite→pgvector→sqlite` hop: recall on the round-tripped file returns
`distance 0` for the exact source vector.
**Deferred to F3 M11+:** continuous-sync / daemon mode (migrate is
one-shot CLI only), cross-host federation (explicit F3 non-goal),
per-session selective log copy (selective granularity is per-collection
for memory + section toggles for the rest), and the byte-identical
`updated_at`/tool-timestamp imports noted above.
## Open questions
- **Do we ship the server inside the opencoti binary or alongside it?**
Alongside (separate binary) is cleaner; explore a single-fat-binary
option as an opt-in.
- **Authentication.** Local UDS owner is the user; over loopback TCP
we need a shared secret. Default to UDS.
- **claude-hooks coexistence.** If a user runs both Claude Code (with
claude-hooks) and opencoti (with opencoti-server) on the same host,
the two memories should *not* collide. They use different stores
and different schemas by default; a separate migration tool can
bridge them if the user wants.
## Risks
- Two persistence backends doubles the test matrix. Mitigation: a
storage trait + a shared conformance test.
- Drift from claude-hooks features. Mitigation: don't try to replicate
feature-for-feature; replicate goals. claude-hooks remains the
reference for Claude Code; opencoti-server can diverge where it
makes opencoti better.
|