| # PolyKV Control-Plane API β Design (F5 M6-S5 / #712) |
|
|
| **Status:** design (2026-07-16). Implementation is phased (Β§10); nothing here |
| is shipped yet. |
| **Inputs:** [polykv_api_research.md](polykv_api_research.md) (design-space |
| survey), [poly_kv.md](poly_kv.md) (the KV mechanism this sits on top of), |
| [llamafile-usage.md](../llamafile-usage.md) Β§3.4 (SharedKVPool user surface). |
| **Single rule:** additive / opt-in. Every endpoint and field lives under the |
| existing `opencoti` JSON namespace or new `/polykv/*` routes; no upstream |
| behavior changes when PolyKV control is unused. |
|
|
| --- |
|
|
| ## 1. Goals |
|
|
| A rich, opt-in control plane that satisfies every multi-agent serving journey |
| on top of the existing SharedKVPool (`shared_pool_slot` + |
| `shared_prefix_n_tokens`, server-context.cpp:3085). Four requirements, plus |
| two the requirements imply: |
|
|
| 1. **Full status** of pools + the sessions attached to each. |
| 2. **tps reporting** β aggregate system throughput *and* per-session |
| generation tps. (All-new: upstream llama.cpp reports per-*response* |
| `timings` but has no per-session/slot running instrumentation.) |
| 3. **Autonomous capacity/admission loop** β a main session sets a per-session |
| gen-tps target (e.g. 5 tps); when the average across active sessions |
| approaches the floor the orchestrator holds off spawning sub-agents. |
| Behavioral knobs: reject a new session vs. warn. |
| 4. **LangGraph extensions** β a custom `BaseChatModel`, the pool-control API |
| exposed as LangGraph tools, and PolyKV-aware agent-state routing. |
| 5. **Nesting** (implied by 3): a sub-agent that discovers fan-out work can |
| become a pool parent for its own sub-agents, so parallelizable tasks run |
| in parallel instead of being forced sequential. |
| 6. **Fork / branch** (COW): a sub-agent group that shares an *ancestor* prefix |
| but then diverges (different toolset/context) gets its own shared pool |
| branched at the divergence point. |
|
|
| ## 2. Decisions locked (2026-07-16) |
|
|
| | # | Decision | Choice | |
| |---|----------|--------| |
| | API home | where the control plane lives | **Self-contained in the C++ engine** β extends `/props`, `/slots`, adds `/polykv/*`. Works standalone; no `opencoti-server` in the path. | |
| | Admission | enforce the tps floor or advise | **Both** β advisory (report) by default; optional per-pool enforced mode with `reject`\|`warn`\|`queue`. | |
| | Pool model | how pools are created | **Explicit first-class objects** β `create`/`fork`/`attach`/`release`, immutable from birth (Β§3.3). Contiguous-prefix contract checked at fork. Boilerplate hidden by the opencoti plugin. | |
| | Scope | deliverable shape | **Design the full surface now; implement in gated phases** (Β§10). | |
| | Fork | divergent child prefix | **First-class in v1** (copy-free COW, Β§3.2 / Β§7) β *not* deferred. | |
|
|
| **Defaulted (override any):** transport = HTTP on the engine + SSE for the |
| live tps stream (poll fallback); tenancy = single trust domain (no per-pool |
| ACL); tps = per-session *experienced* decode tps (EWMA) + aggregate decode |
| throughput, floor = mean of active sessions' experienced tps (prefill |
| excluded); nesting depth uncapped; interior pools pinned while any descendant |
| is attached + grace TTL; seq-ids for pools+leaves sized at boot. |
|
|
| --- |
|
|
| ## 3. Data model |
|
|
| ### 3.1 A pool is a tree node |
|
|
| ``` |
| Pool { |
| id : int32 // seq_id in the unified cache |
| parent : int32 | ROOT // -1 for a root pool |
| branch_pos D : int32 // shared-ancestor length inherited from parent |
| own_range : [D, L) // tokens this pool materialized itself |
| prefix_len L : int32 // total shared prefix = D + |own_range| |
| tokens : [llama_token] // full prefix token array [0, L), host-side (auto-P + fork validation) |
| prefix_hash : [u64] // cumulative rolling hash per position (validates any D; ~8 B/token host RAM) |
| pin : bool // resident-pinned (auto while descendants attached) |
| children : [Pool] // interior pools branched/extended from this one |
| sessions : [Session] // leaf sharers attached at this pool's prefix |
| last_access : ts |
| } |
| Session { // a leaf = an ordinary decoding slot |
| id : int32 // seq_id |
| pool : int32 // the pool it attached to |
| suffix_range : [L, L+S) // its own private, mutable continuation |
| tps_ewma : f32 // experienced decode tps |
| ttft_ms, state, ... |
| } |
| ``` |
|
|
| The **root** pool (`parent = ROOT`, `D = 0`) is what today's |
| `shared_pool_slot` request already produces. Everything else is new. |
|
|
| ### 3.2 Three edge types, one mechanism |
|
|
| A child pool inherits `[0, D)` from its parent and owns `[D, L)`. `D` selects |
| the edge type: |
|
|
| - **extend** β `D = parent.L`. Child = parent prefix **+** appended shared |
| tokens. This is *nesting*: a sub-agent adds its own shared context and hands |
| it to its sub-agents. |
| - **branch (COW fork)** β `D < parent.L`. Child shares the common ancestor |
| `[0, D)` and diverges with its own `[D, L)`. This is the fork: same system |
| prefix, different toolset after `D`. |
| - **root** β `parent = ROOT`, `D = 0`. |
|
|
| **Why the fork is copy-free.** `seq_cp` in the unified cache is additive |
| membership: for each cell where `seq_has(cell, parent)` in `[0, D)` it calls |
| `seq_add(cell, child)` (llama-kv-cache.cpp:2007-2044). No cell is duplicated β |
| the ancestor cells simply gain the child in their sequence set. Because |
| prefixes are **immutable** (FROZEN, Β§3.3), the shared range is never mutated, |
| so the "copy" a classic COW would eventually pay never happens. The child only |
| computes new KV for `[D, L)`, prefilled at RoPE position `D` attending to the |
| shared `[0, D)` already in cache β exactly how attaching with a shorter |
| `shared_prefix_n_tokens` works today. **Extend and branch are the same code |
| path parameterized by `D`.** |
|
|
| ### 3.3 Immutability & versioning |
|
|
| **Pools are immutable from birth.** A pool's prefix is fully materialized at |
| `create`/`fork` and never changes afterwards β there is no OPEN/extend-in-place |
| state. "Extending" a pool *always* means creating a child (`fork` with |
| `D = parent.L`). This is simpler than a freeze-on-first-attach state machine |
| and loses nothing: the "build context incrementally" journey is covered by the |
| main agent decoding in its **own session** and then forking `from_session` |
| (Β§4.1), which snapshots its current context as a new pool in one call β no |
| in-place mutation ever needed. |
|
|
| This makes "add a tool to the shared prefix after agents attached" |
| well-defined β it is a *new versioned pool* (a fork), and the old one keeps |
| serving its sharers unchanged. (This is the precise answer to the HuggingFace |
| question: the prefix is static per pool; a tool/MCP added later is a new pool |
| version, valid only for sessions that attach to it β never mutated under |
| sessions already on the old one.) |
|
|
| ### 3.4 Lifetime |
|
|
| Cell freeing is implicit-refcounted: a cell is reclaimed only when its sequence |
| set empties, so an ancestor's cells physically survive as long as *any* |
| descendant references them β even if the ancestor's own slot is torn down. On |
| top of that: |
|
|
| - **Pin policy.** Interior pools are auto-pinned (`--no-clear-idle` semantics, |
| per-node) while any descendant is attached. Without this a new leaf attaching |
| to an idle-cleared interior pool hits the residency guard |
| (server-context.cpp:3116) and silently falls back to full reprocess β |
| correct but slow. A **grace TTL** delays reclaim after the last descendant |
| detaches (cheap re-attach for bursty fan-out). |
| - **seq-id budget.** Every pool *and* every leaf consumes one `n_seq_max` slot. |
| A deep/wide tree spends seq-ids on interior nodes too; the boot-time reserve |
| accounts for both. Exposed in `/capacity`. |
|
|
| ### 3.5 Materialization β the slot/seq decoupling (P2's real work) |
|
|
| Today a "pool" is just an ordinary server **slot** that happened to serve a |
| request: `shared_pool_slot` names a slot index, the pool's tokens live in |
| `slots[pool].prompt.tokens`, and the pool seq **is** the slot's seq. First-class |
| pools break that coupling, and this is the largest single implementation lift |
| in the milestone: |
|
|
| - **seq-ids beyond slots.** Pool seq-ids come from a boot-time reserve above |
| the slot range: `n_seq_max = n_parallel + --polykv-max-pools` (default |
| proposed: 16). A pool occupies a seq-id but **no slot** once materialized. |
| - **Prefill-only internal task.** `create`/`fork` materializes `[D, L)` via an |
| internal task that runs through the normal batching path (so it composes |
| with every KV feature) but emits **no logits and no generation**; it |
| transiently borrows a slot for the prefill, then the KV stays on the pool's |
| seq and the slot is returned. `seq_cp(parent β pool, 0, D)` runs first for |
| fork/extend. |
| - **Host-side pool state.** The Pool object (tokens, cumulative hash, tree |
| links, pin/TTL) lives in server memory beside the slot table β *not* in a |
| slot β and is what `/pools` reports and attach validates against. |
| - **Cache layout (REVISED c6, bug-2248).** Originally pools required |
| `--kv-unified` (the split-stream `seq_cp` overload hard-asserts |
| `is_full`, so a partial cross-stream share was undefined β this was a |
| boot-time refusal). Since patch `0145` pools work on BOTH layouts: |
| unified shares stay zero-copy set-membership; on the split cache |
| (`--parallel` without `--kv-unified` β the layout the rolling-KV |
| position window requires) every prefix share runs as a PHYSICAL copy |
| via `oc_seq_share_prefix()` (full-stream `seq_cp` + trim to `[0,P)`, |
| positions preserved; cross-stream copies include the window's host |
| tail). Trade-off: no VRAM dedup off unified β each sharer stream |
| holds its own prefix copy (window mode caps device use at |
| `window_cells`/stream). The boot check is now an INFO stating the |
| copy-share semantics. `/capacity` is STREAM-AWARE on the split cache |
| (patch `0146`): the verdict checks the per-stream budget and free |
| streams ("no free stream (split cache)" hard-stop; headroom clamped |
| to free streams) and the payload adds `kv_streams_total` / |
| `kv_streams_free` / `kv_cells_per_stream` (null on unified). |
|
|
| --- |
|
|
| ## 4. API surface |
|
|
| All additive. Reuses the `opencoti` JSON namespace (`get_opencoti_props()` |
| :3787) and the `/slots` accumulator precedent (:183). Endpoint shapes follow |
| research Β§6. |
|
|
| ### 4.1 Pool lifecycle (imperative, explicit) |
|
|
| | Verb | Endpoint | Body β result | |
| |------|----------|---------------| |
| | create root | `POST /polykv/pools` | `{tokens? \| from_session?}` β `{pool_id, prefix_len}`. `from_session` snapshots a live session's current context as the pool prefix β the **primary** creation path for agents (no token round-trip). | |
| | fork (extend or COW branch) | `POST /polykv/pools/{id}/fork` | `{branch_pos D, tokens[D:L] \| from_session?}` β new child `{pool_id, parent:id, branch_pos, prefix_len}`. `D == parent.L` β extend (nesting); `D < parent.L` β COW branch. | |
| | attach | *(request path, unchanged mechanism)* | per-request `pool_id` (legacy `shared_pool_slot` kept). **`shared_prefix_n_tokens` becomes optional**: the server auto-computes `P` as the longest hash-match between the request prompt and the pool's stored tokens (**auto-P**) β killing the mis-set-P footgun and the HF-question confusion. An explicit `P` is still accepted and validated. | |
| | pin/evict/flush | `POST /polykv/pools/{id}/{pin\|evict\|flush}` | retention control (research Β§6 Req 1) | |
| | release | `DELETE /polykv/pools/{id}` | detach + reclaim when refcount hits 0 (respects grace TTL) | |
| |
| Contiguous-prefix contract is checked at `fork`: the child's declared |
| `tokens[0:D]` must hash-match the parent's `[0:D)` (cumulative `prefix_hash` |
| at `D`), else `409 Conflict` β the caller then either re-roots or |
| full-prefills. All routes live under a `/polykv/` prefix so a future upstream |
| llamafile/llama.cpp bump can never collide with an unprefixed `/pools`. |
| |
| ### 4.2 Status (research Β§6 Req 1) |
| |
| - `GET /polykv/pools` β `{pools:[{pool_id, parent, branch_pos, prefix_len, |
| own_kv_bytes, subtree_kv_bytes, residency, pinned, last_access_ts, |
| children:[pool_id], sessions:[{session_id, suffix_n_tokens, |
| marginal_kv_bytes, tps_ewma, ttft_ms, state}]}], tree_depth, seq_ids_used, |
| seq_ids_max}`. The `parent`/`children` fields make the **tree** explicit. |
| **KV attribution is by ownership, never by membership**: a shared cell |
| belongs to many seqs, so summing per-seq bytes would double-count β a pool |
| reports `own_kv_bytes` = its `[D, L)` range only, plus a `subtree_kv_bytes` |
| rollup (own + all descendants + attached sessions' suffixes). Ξ£ own over the |
| tree == physical bytes, by construction. |
| - `GET /polykv/pools/{id}` β one subtree. |
| - `/slots` gains `pool_id` + `tps_ewma` per slot (extends #677). |
| - `/props` `opencoti` block gains a `polykv` sub-object (config: floor, |
| admission mode, pin/TTL policy). |
|
|
| ### 4.3 tps (research Β§6 Req 2) |
|
|
| - **Per-response** (already partly present): `timings{predicted_per_second, |
| prompt_per_second, cache_n}`. |
| - **Headers** (TGI model): `x-session-tps`, `x-cached-prefix-tokens`, |
| `x-pool-id`. **Deferred to P2+**: HTTP headers must be emitted before the |
| body, but the slot (and thus the live tps value) is assigned only after the |
| task is queued β a header set at request time would always be stale, and |
| trailers aren't portable. The per-response `timings` JSON already carries the |
| authoritative per-response rate; headers add value only once pool attach |
| happens at request routing (P2), where `x-pool-id`/`x-cached-prefix-tokens` |
| are known up front. |
| - **Live stream:** `GET /polykv/tps` (SSE) β periodic |
| `{ts, aggregate_tps, sessions:[{session_id, pool_id, tps_ewma}], mean_active_tps}`. |
| This is the signal the admission loop subscribes to. |
| - **`/metrics`** (Prometheus, low-cardinality per research Β§4): gauges |
| `opencoti_polykv:gen_throughput_tps{pool}`, `pool_kv_bytes{pool}`, |
| `pool_sessions{pool}`; counters `prefix_cache_hits_total{pool}`, |
| `queries_total{pool}`. Label = pool only (never session β cardinality). |
|
|
| ### 4.4 Admission (research Β§6 Req 3; decision: both modes) |
|
|
| - `POST /polykv/pools/{id}/admission` `{target_tps_per_session, mode:"advisory"| |
| "enforced", on_saturation:"reject"|"warn"|"queue", guarantee_min_sessions, |
| settle_tokens, settle_max_ms}` β per-pool policy (last three = P7 Β§14, |
| defaults 1 / 48 / 5000). |
| - `GET /polykv/pools/{id}/capacity` β |
| `{can_admit, projected_mean_tps_if_admitted, headroom_sessions, kv_headroom_pct, reason}` |
| + P7 (Β§14): `{settling, settle_remaining_ms, n_warming, n_pool_sessions, |
| guaranteed, projected_mean_tps_model, projected_mean_tps_measured, |
| drop_per_admit_ewma, known_session}`. The projection is measured-drop |
| based once the pool has absorbed β₯1 admit; the raw mean excludes |
| warming (EWMA-cold) sessions. |
| `headroom_sessions` is computed against **marginal** per-session KV |
| (`S_iΒ·bpt`), not full context β the basis of the ~7Γ reduction (research Β§5). |
| - **Advisory (default):** endpoints report; the orchestrator decides. Nothing |
| is rejected server-side. |
| - **Enforced (opt-in):** on a new attach with projected `mean_active_tps < |
| floor` OR `kv_headroom` exhausted β `reject` = `429`/`503` + `Retry-After` |
| (RFC 6585/9110); `warn` = admit + `X-Sessions-Remaining` header; `queue` = |
| hold until headroom returns. P7 (Β§14): the gate applies to **NEW sessions |
| only** (an existing sessionβslot affinity = continuation, never gated); |
| `"overcommit": true` in the request body bypasses it explicitly; and |
| `reason:"measurement settling"` maps to a bounded HOLD (β€ `settle_max_ms` |
| + 1 s) rather than a 429. |
| - **Throttle-toward-floor is explicitly NOT v1.** Andes' insight (over-floor |
| generation is wasted GPU β throttle fast sessions back toward the floor to |
| reclaim capacity) requires per-slot rate control inside the continuous- |
| batching decode loop β i.e. skipping a slot in some decode batches. That is |
| scheduler surgery with its own correctness/perf gates, not an API feature; |
| it is banked as a P4-follow candidate, and v1 `warn` is headers-only. |
| |
| --- |
| |
| ## 5. tps instrumentation (all-new, C++ server-side) |
| |
| Upstream has no per-session running tps. Design: |
| |
| - Extend the `/slots` lifetime accumulators (server-context.cpp:183) with a |
| per-slot decode-token counter + wall-clock, updated at each decode-batch |
| completion for the tokens attributable to that slot. |
| - **Experienced tps** per session = EWMA over a rolling window (default: last |
| `W` decoded tokens or `T` ms, whichever first) of that slot's decode |
| throughput. This is the quantity that *falls as concurrency rises* even while |
| aggregate climbs (attention doesn't batch-amortize β research Β§5), so it is |
| the correct SLO signal. |
| - **Aggregate** = system decode throughput (all slots' decode tokens / wall). |
| - **`mean_active_tps`** = mean of experienced tps across currently-decoding |
| sessions β the value the floor compares against (matches "average sessions |
| gen tps"). |
| - Prefill tokens are excluded from "gen tps" (counted separately as |
| `prompt_per_second`). |
| - Cost: a handful of integer counters + one EWMA update per slot per batch β |
| negligible; gated behind the PolyKV block so off-path is byte-identical. |
| |
| --- |
| |
| ## 6. Admission control |
| |
| Two coupled signals (research Β§5): |
| |
| 1. **Soft / low-lag:** `mean_active_tps` vs the floor. Directly measures the |
| SLO; leads `num_requests_waiting` (which lags). |
| 2. **Hard wall:** pool KV-occupancy %. Reject *before* this forces |
| preempt-oldest eviction. |
| |
| `projected_mean_tps_if_admitted` estimates the post-admit floor from the |
| `B_sat` knee model (below the knee admitting is ~free; above it every admit |
| costs everyone's TPOT ~linearly). On high-VRAM cards (RTX 6000 96 GB) the tps |
| knee is hit **before** the KV wall, so signal (1) dominates on opencoti's |
| hardware β which is exactly why the floor, not occupancy, is the headline. |
|
|
| Framing: "5 tok/s/session" = **TPOT SLO 200 ms/token; maximize goodput s.t. |
| TPOT β₯ floor** (a floor, not a maximize β Andes). The default is advisory: the |
| LangGraph capacity-gate node (Β§8) reads `/capacity` and decides. Enforced mode |
| is the same math with server-side `reject`/`warn`/`queue`. |
|
|
| --- |
|
|
| ## 7. Nesting & COW-fork β worked journeys |
|
|
| **Journey A β sub-agent parallelization (nesting / extend).** Main agent M |
| runs on root pool `P0 = [system+tools]` (len N0, FROZEN). M builds task context |
| and `POST /pools/P0/fork {branch_pos:N0, tokens:[N0:N1]}` β interior pool `P1 = |
| [system+tools+task]`. M spawns k sub-agents that each attach to `P1` and decode |
| their own suffix in parallel. Physically, `P1`'s `[0:N0)` cells are P0's cells |
| (now in seq-set {P0,P1,leafβ¦}); `[N0:N1)` is prefilled once. The k sub-agents |
| cost only their marginal suffixes. The fan-out work that used to run |
| sequentially now runs concurrently under one shared branch. |
|
|
| **Journey B β divergent toolset (COW fork).** Two sub-agent groups share the |
| system preamble `[0:D)` but need different toolsets. Group B does |
| `POST /pools/P0/fork {branch_pos:D, tokens:[D:D']}` (D < N0) β `P1' = |
| [system+toolsetB]`, sharing `[0:D)` zero-copy, prefilling `[D:D')` fresh. Its |
| sub-agents attach to `P1'`. No physical copy of the shared preamble. |
|
|
| **Invariants enforced.** (a) contiguous-prefix hash check at `fork`; (b) |
| immutability from birth β a pool's prefix is fully materialized before any |
| descendant can attach (Β§3.3), so no attach ever races a prefix change; (c) |
| recursive pin β interior pools stay resident while descendants attached; (d) |
| seq-id budget counts interior nodes. Divergence that is *not* a clean prefix of any |
| existing pool β new root (or full prefill) β reported, never silently wrong. |
|
|
| **Hybrid/recurrent guard (bug-2203) is recursive.** Every attach/fork on a |
| recurrent or hybrid model still hits the exact-full-state guard |
| (server-context.cpp:3116) at every level. |
|
|
| --- |
|
|
| ## 8. LangGraph integration (research Β§6 Req 4) |
|
|
| Package home: a new **top-level `opencoti/` non-bun root** (per the repo's |
| "new non-bun roots" rule) β Python package `opencoti-langgraph` (pip). Kept out |
| of the bun workspace. Pin `BaseChatModel` signatures to a specific |
| `langchain-core` version tag (the API is mid-migration β research caveat). |
|
|
| - **`OpencotiChatModel(BaseChatModel)`** β implement `_generate` + `_llm_type` |
| (required), `_stream`/`_agenerate`/`_astream`; override `bind_tools`. |
| Populate `usage_metadata` and |
| `response_metadata{pool_id, session_tps, backpressure}` from the headers/Β§4.3. |
| - **Pool-control tools** β `create_pool`, `fork_pool`, `pool_status`, |
| `session_tps`, `capacity` exposed as LangChain tools via `bind_tools` / |
| `ToolNode`, so an agent can manage its own pool tree. |
| - **Capacity-gate node** β reads `/capacity`; returns |
| `Command(update={"capacity_ok":ok, "headroom_sessions":n}, goto=...)` to |
| route the graph (spawn vs. hold). `interrupt()` for block-until-free. |
| `InMemoryRateLimiter` is explicitly insufficient β it can't see per-session |
| tps or KV headroom. |
| - **Agent-state routing** β a reducer surfaces `headroom_sessions` / |
| `mean_active_tps` into graph state so conditional edges branch on live PolyKV |
| capacity. |
|
|
| --- |
|
|
| ## 9. Composition with the KV stack |
|
|
| PolyKV control is orthogonal to the KV *mechanics* and must not perturb them: |
|
|
| - **rolling-KV / DCA / turbo / TCQ / quant-KV** β pool sharing is `seq_cp` |
| membership on whatever cell type is configured; the tps/admission plane is |
| read-only over the existing decode loop. Off-path byte-identical (gate). |
| - **MTP / speculative** β per-session tps counts *accepted* tokens (the user- |
| visible generation rate), not draft steps. |
| - **`--parallel β₯ 2`** β the multi-slot requirement is already the |
| SharedKVPool baseline; the tree just adds interior seq-ids. |
| `--kv-unified` is optional since c6 (zero-copy shares with it, |
| copy-shares without it β see the layout note in Β§3.5). |
| - **hybrid/recurrent** β bug-2203 guard, recursive (Β§7). |
|
|
| ## 10. Phased implementation plan (gated) |
|
|
| Full surface designed here; implementation lands in gated phases, each with a |
| correctness + off-path-identity gate before the next: |
|
|
| - **P1 β tps instrumentation** (Β§5). Per-slot counters + EWMA + `/slots`/`/props` |
| exposure + `/polykv/tps` SSE + `/metrics`. Prereq for everything. |
| **β
IMPLEMENTED 2026-07-16** (live in vendored tree, patch capture pending): |
| gates green β sustained EWMA β3.2% vs `timings` ground truth; 2-session |
| concurrency shows correct `aggregate_tps` β Ξ£ sessions + `mean_active_tps`; |
| `/metrics` 3 series render. bug-2205: aggregate MUST come from |
| Ξ£ `n_decoded_cum` (per-token), not `n_tokens_predicted_total` |
| (completion-time only). |
| - **P2 β pool objects + status API** (Β§3, Β§4.1-4.2). The slot/seq decoupling |
| (Β§3.5): seq-id reserve, prefill-only materialization task, host-side Pool |
| state, **auto-P attach**; `GET /polykv/pools`. Root pools only. |
| **β
IMPLEMENTED 2026-07-16** (live in vendored tree, patch capture pending): |
| gate 23/23 on Qwen3-4B β boot check (`--polykv-max-pools` without |
| `--kv-unified` fails clean; NOTE `--parallel` must be explicit, unset |
| auto-sets `kv_unified=true`), create-from-prompt prefill-only P=95, |
| auto-P exact attach `cache_n=95`, divergence attach `cache_n=42<95`, |
| `from_session` zero-copy `seq_cp` snapshot (affinity is evict-on-reuse β |
| only the most-recent session per slot resolves), pin/unpin/GET-one, |
| release + released-id clean fallback, disabled path inert, pooled vs |
| unpooled greedy text IDENTICAL (bit-shared cells). Mutations only on the |
| server thread via `SERVER_TASK_TYPE_POLYKV`; recurrent guard per bug-2203. |
| - **P3 β nesting & COW-fork** (Β§7). `fork` with arbitrary `D` (extend + COW |
| branch), cumulative-hash contract check, recursive pin, seq-id accounting. |
| **β
IMPLEMENTED 2026-07-16** (live in vendored tree, patch capture pending): |
| gate 18/18 substantive on Qwen3-4B β extend fork (D=parent.L, own=22), |
| COW branch (D=47<95, zero-copy `seq_cp` of `[0,D)` + fresh prefill), |
| contract violation β clean 4xx, auto-P attach to a child (cache_n=117), |
| tree reporting (parent/children/branch_pos/own_len/tree_depth), release |
| ordering enforced (parent-with-children refused, leaves-first drains), |
| seq exhaustion at max_pools, `from_session` fork. The Β§4.1 prefix |
| contract is enforced **token-exact** (`std::equal` against the parent's |
| stored tokens β strictly subsumes the cumulative-hash check; `prefix_hash` |
| kept as identity fingerprint). Pooled-vs-unpooled greedy divergence on the |
| branch was classified benign by first-token probe (same top-5 ranking, |
| max |Ξlogprob| 0.112 β batch-shape numerics, #495 class). Recurrent |
| bug-2203 guard is recursive: hybrid/recurrent forks must extend at |
| D == parent.L. |
| - **P4 β admission** (Β§6). `/capacity`, advisory metrics, then enforced mode |
| (`reject`/`warn`/`queue`). |
| **β
IMPLEMENTED 2026-07-16** (live in vendored tree, patch capture pending; |
| `queue` deferred per Β§11 default): gate 18/18 on Qwen3-4B β /capacity shape |
| (idle: can_admit, projected null, kv cells by OWNERSHIP: pools' own ranges + |
| slots' marginal suffixes via new `slot.n_pool_shared`), advisory floor never |
| rejects, **enforced+reject β 429 + Retry-After** under an active decoding |
| session (tps-floor arm needs n_active β₯ 1), **enforced+warn β 200 + |
| X-Sessions-Remaining**, off-path (no pool_id / unknown pool_id) untouched. |
| Projection = conservative fully-saturated knee: `mean_active Β· n/(n+1)` |
| (never over-admits below the real B_sat knee). Fast path: an atomic |
| enforced-pool count β ordinary traffic pays zero; the gate itself is a |
| high-priority OP_CAPACITY control task from `handle_completions_impl` |
| (same pattern as /slots' METRICS task). |
| - **P5 β LangGraph package** (Β§8). `opencoti-langgraph` under top-level |
| `opencoti/`. Runs in the first-class **`opencoti` conda env** (py3.11, |
| langchain-core 1.4.x + langgraph 1.2.x pinned; the same env will later |
| host the consultants council alongside opencoti-server). Includes the P6d |
| `CompactionNode` (Β§13). |
| **P5 β
IMPLEMENTED 2026-07-16**: offline 21/21 (pytest, httpx MockTransport |
| FakeServer), live gate **19/19** on Qwen3-4B `:8231 --parallel 2 --kv-unified |
| --polykv-max-pools 4` β pool tree (token-path fork D=65), `OpencotiChatModel` |
| invoke + streaming with pool attach proven (`cache_n=68 β₯ root prefix 65`), |
| `session_tps`/usage metadata, capacity context arm + gate-node routing, |
| enforced 429 β `PoolSaturatedError(retry_after)` β advisory recovery, |
| **P6c compact-by-refork e2e** (old pool released, migrated session coherent |
| on the new pool, `cache_n=71`), tps `ctx_*` fields. Contract lessons |
| (bug-2208): fork `prompt`/`tokens` = the child's **FULL prefix [0,L)** |
| token-exact vs the parent (never suffix-only β build via `/tokenize`: |
| parent tokens + suffix `add_special=false`); a pool prefix meant for |
| `/v1/chat/completions` attach must be the **chat-templated** system block |
| (raw text shares 0 tokens β `cache_n=0`); Qwen3 thinking models need |
| `chat_template_kwargs: {enable_thinking: false}` for content-based gates. |
| Client accordingly ships `tokenize()` + `compact_by_refork(compacted_tokens=β¦)`. |
| - **P6 β context exhaustion & compaction** (Β§13). **P6a (ctx-shift pool |
| guard) + P6b (context visibility + /capacity context arm) land BEFORE P5** |
| per user decision 2026-07-16; P6c/P6d ship with P5. |
| **P6a+P6b β
IMPLEMENTED 2026-07-16** (live in vendored tree, patch capture |
| pending): gate 23/23 on Qwen3-4B at `-c 4096 --parallel 1 --context-shift` |
| (the exact Β§13.1-regime-3 configuration) β boot WARN; pooled session forced |
| past n_ctx = graceful `truncated=true` stop at 4021 tokens (position wall |
| fires one token before unified-cell exhaustion when the prefix is |
| pool-shared), NO shift, pool provably coherent afterwards (fresh attach |
| cache_n=60, needle answered); non-pooled ctx-shift still engages and |
| completes after pools released (off-path intact); `?expected_tokens` context |
| arm rejects with `"context headroom exhausted"`; `compaction_pressure` |
| 0 at idle, β[0,1]; `orphaned_pin` true for a pinned never-attached leaf; |
| `/polykv/tps` sessions carry `ctx_used/ctx_total/ctx_headroom_tokens`. |
| Guard bookkeeping: `slot.n_pool_shared` is clamped to the retained prefix at |
| prompt-cache decision time (`min(n_pool_shared, n_past)`) β NOT blind-reset, |
| because cache reuse keeps pool-shared cells across later non-pool requests |
| on the same session. NOTE the geometry lesson: with `--kv-unified` the slot |
| n_ctx is the FULL `-c` (not divided by `--parallel`), and multi-slot cell |
| exhaustion precedes the positional wall β gates must use `--parallel 1`. |
| |
| ## 11. Open sub-decisions (defaulted; flag to change) |
| |
| - EWMA window `W`/`T` defaults (proposed: 128 tokens or 2 s). |
| - Grace TTL for interior-pool reclaim (proposed: reuse `--slot-shrink-idle-ms`). |
| - `/metrics` opt-in flag name + default-off. |
| - Whether `queue` (admission) is v1 or P4-follow (proposed: `reject`+`warn` v1, |
| `queue` follow). |
| - `--polykv-max-pools` default (proposed: 16; each reserved seq-id costs KV |
| bookkeeping but no cells until materialized). |
| - Hash algorithm for the cumulative `prefix_hash` (proposed: xxhash64 rolling; |
| 8 B/token host RAM β ~800 KB for a 100k-token prefix, negligible). |
| - SSE tps-stream cadence (proposed: 500 ms, configurable; stream is opt-in so |
| off-path cost is zero). |
|
|
| ## 12. Verification |
|
|
| - Off-path identity: PolyKV-control-unused boot is byte-identical (DSO + decode). |
| - tps accuracy: instrumented per-session tps vs. an external wall-clock |
| token-rate measurement, Β±few %. |
| - Fork correctness: a branched pool's decode is logit-equivalent to the same |
| session full-prefilled (real_frac=0 / KLD), never greedy-needle. |
| - Tree lifetime: ancestor cells survive descendant-only references; reclaim on |
| refcount 0; recursive pin prevents the residency-miss fallback. |
| - Admission: projected vs. realized `mean_active_tps` under a spawn ramp; |
| `reject`/`warn` behavior + `Retry-After`/`X-Sessions-Remaining` headers. |
| - Composition gates: rolling-KV / DCA / turbo / MTP / `--parallel 2` unchanged. |
| |
| ## 13. Context exhaustion & compaction orchestration (P6) |
| |
| *Added 2026-07-16 on user review: the design above admits sessions by tps floor |
| and marginal KV, but had no story for what happens when the shared context |
| actually fills β and continuous agentic frameworks WILL fill it. Decision: |
| P6a+P6b land BEFORE P5 (the LangGraph capacity gate builds on the context arm); |
| P6c/P6d ship with/after P5.* |
| |
| ### 13.1 What the engine does today (verified, three regimes) |
| |
| 1. **Prompt too long at admission** β clean `ERROR_TYPE_EXCEED_CONTEXT_SIZE`. |
| 2. **Generation reaches `n_ctx`, `--ctx-shift` off (default)** β clean stop |
| with `truncated=true` (`server-context.cpp:1898`). Not corrupting, but |
| silent mid-thought truncation is exactly the "fails badly" mode for agents: |
| no advance warning, no orchestration signal. |
| 3. **`--ctx-shift` on + pool-attached session β CORRUPTION.** The shift does |
| `seq_rm` + `seq_add(β¦, -n_discard)` (`server-context.cpp:3261-3262`); in |
| the unified cache `seq_add` mutates the **per-cell position**, and |
| pool-shared cells are members of many seqs β the shift re-RoPEs the pool's |
| prefix for every other session and every descendant pool. Upstream guards |
| this for parent/child shared prompts (`server-context.cpp:3192`) but a |
| pool attach is neither, so the guard does not fire. The rest-kv-eviction |
| variant uses the same primitives and has the same hazard. |
|
|
| The "safe" naive alternative β COW-copy shared cells before shifting β |
| allocates a duplicate of the prefix at the exact moment the cache is full. |
| Both intuited failure modes (corruption / usage inflation) are real. |
|
|
| ### 13.2 Design principle: compaction is a prompt REWRITE, never an in-place KV op |
|
|
| LangGraph (and every agentic framework) compacts at the prompt-assembly |
| level: `trim_messages`, `RemoveMessage` reducers, or a summarization |
| `pre_model_hook` that replaces old messages with a running summary. The |
| compacted context is **new text** β its tokens cannot match the old cells, so |
| cell-level "compaction" is not even meaningful. Therefore the server must |
| never mutate pooled cells; the correct primitive already exists in the tree: |
|
|
| > **Compaction = re-root via fork.** The system prompt + tool definitions |
| > survive compaction verbatim β that is the stable ancestor pool |
| > `[0, D_sys)`. Compacted context = `fork(ancestor, D=D_sys, |
| > tokens = summary + recent tail)` β a new sibling branch, prefilled once. |
| > Sessions migrate to the new pool; the old working subtree is `release`d. |
| > Pool immutability is preserved by construction, no cell position is ever |
| > touched, and net cells go **down** (old subtree reclaimed) instead of up |
| > (COW duplication). |
|
|
| Pinned pools under this model are a **leak hazard, not a corruption hazard**: |
| a pin left on the abandoned subtree blocks reclaim. Discipline is |
| unpin-after-migrate; the server reports orphans (pinned + zero sessions + |
| zero children) so the orchestrator can sweep. |
|
|
| ### 13.3 Phases |
|
|
| - **P6a β corruption guard (before P5).** Extend the ctx-shift refusal to |
| pool-attached slots (`slot.n_pool_shared > 0` β clean error, same as the |
| `:3192` shared-prompt refusal) + boot WARN when `--ctx-shift` is combined |
| with `--polykv-max-pools`. Rest-kv-eviction path included. Off-path inert. |
| - **P6b β visibility + spawn-gate parity (before P5).** Per-session |
| `ctx_used`/`ctx_headroom_tokens` in `/polykv/tps` + `/slots`; `/capacity` |
| gains `?expected_tokens=N` and a **context arm** to `can_admit` |
| (reason `"context headroom exhausted"`) so sub-agent spawns are gated on |
| context exactly like the tps floor; plus a `compaction_pressure` field |
| (0..1, driven by free-cell fraction and largest-session share) so the |
| orchestrator compacts BEFORE the wall. |
| - **P6c β compact-by-refork protocol (with P5).** Documented two-step: |
| summarize via normal completion β `fork` the stable ancestor at `D_sys` |
| with the compacted suffix β migrate sessions β `release` old subtree. |
| Orphaned-pin flag in `/polykv/pools`. Atomic convenience endpoint only if |
| the two-step proves racy in practice. |
| - **P6d β LangGraph side (inside opencoti-langgraph).** `CompactionNode` |
| (pre_model_hook pattern: watch `response_metadata` headroom / |
| `compaction_pressure` β summarize β re-root β swap `pool_id` in graph |
| state); the capacity-gate node checks the context arm from day 1. |
|
|
| ### 13.4 Non-goals |
|
|
| - In-place KV compaction/merging of pooled cells (meaningless under rewrite |
| semantics; corrupting under sharing). |
| - Server-side self-summarization (the orchestrator owns the summary β it has |
| the conversation semantics; the server only has tokens). |
| - Growing `n_ctx` at runtime (allocation is boot-time; capacity planning is |
| the admission plane's job). |
|
|
| --- |
|
|
| ## 14. P7 β measurement-settled admission, measured-drop forecast, guaranteed minimum |
|
|
| > **Status: design locked (2026-07-23, user).** Driven by the |
| > `courier-qwopus9b-mtp-s55-p10` run: floor 15 tps, spawns every 5 s on a |
| > tps EWMA (tau 2 s) that had not yet absorbed the previous admit |
| > (38β31.7β21.7β15.9 at successive ticks), overshoot to n=6 at 11.6 tps, |
| > and the corrective shrink actuated **22 minutes later** (workers retire |
| > only at episode end). Decisions: pacing lives in BOTH layers (server-side |
| > hold + orchestrator pacing); overflow valve = per-pool |
| > `guarantee_min_sessions` (default 1) + per-request `overcommit` honored. |
|
|
| ### 14.1 Root causes (from the run + code audit) |
|
|
| 1. **Admit-on-stale-measurement.** Nothing marks a pool "still absorbing the |
| last admit"; every 5 s tick admitted again before the EWMA settled. |
| 2. **Warming bias.** `mean_active_tps` averages `tps_ewma` over all |
| processing slots, including just-admitted ones still at `tps_ewma == 0` |
| β right after a spawn the mean is dragged toward 0, corrupting both the |
| floor comparison and the projection. |
| 3. **Optimistic projection.** `n/(n+1)` (flat-aggregate knee model) under- |
| predicted the real drop at every step of the run (e.g. predicted 17.4 |
| post-admit, reality 15.9). No measured feedback. |
| 4. **Courier bypassed the projection.** `scheduler()` compares raw |
| `mean_active_tps` to the floor instead of |
| `projected_mean_tps_if_admitted` (bug β fixed in P7's courier pass). |
| 5. **Enforced gate cannot tell admission from continuation.** The gate at |
| request routing fires on ANY request carrying `pool_id`, so a pool under |
| floor 429s its own running sessions' next turns β livelock risk. The gate |
| must gate only NEW sessions (no `session_to_slot` affinity yet). |
| 6. **Shrink actuation lag is structural** (episode-end retire). P7 does not |
| change it; it makes overshoot rare instead (prevention, not cure). |
|
|
| ### 14.2 Server design |
|
|
| **Settle window (per pool).** On each NEW-session admit record |
| `{t_admit_us, admit_slot, n_decoded_at_admit, mean_at_admit}`. The pool is |
| **settling** until the admitted slot has decoded `settle_tokens` (default 48) |
| more tokens OR `settle_max_ms` (default 5000) elapsed. While settling: |
| - `/capacity` β `settling: true`, `settle_remaining_ms`, `can_admit: false`, |
| `reason: "measurement settling"` (advisory clients wait, not shrink). |
| - Enforced mode β the gate HOLDS the attach (bounded sleep-poll β€ |
| `settle_max_ms`, then re-evaluates) instead of 429 β "slow the spawn just |
| enough", never a hard reject for settling alone. |
|
|
| **Warming-aware mean.** Slots with `tps_ewma == 0` while processing are |
| **warming**: excluded from `mean_active_tps`/projection; reported as |
| `n_warming` in `/capacity` and `/polykv/tps`. |
|
|
| **Measured-drop forecast.** When a settle completes, fold |
| `drop = max(0, mean_at_admit β mean_settled_now)` into a per-pool EWMA |
| (`drop_ewma`, alpha 0.3). Projection: |
| `projected_measured = mean_settled β drop_ewma` (once β₯1 sample), |
| `projected_model = meanΒ·n/(n+1)` (fallback / always reported). The floor arm |
| uses `projected_measured` when available, else the model. |
| `/capacity` reports both plus `drop_per_admit_ewma`. |
|
|
| **Idle-gap estimate.** When every session is between turns (tool calls in |
| flight) `n_active` is 0 and "no projection" reads as free capacity β the |
| hole that still ramped the first P7 gate run to n=8. If the pool has live |
| sessions and a FRESH last settled mean (< 30 s), `/capacity` projects from |
| it instead (`projected_idle_estimate: true`; measured-drop variant when |
| available, else `last_settled_meanΒ·n_pool/(n_pool+1)`), and the floor arm |
| applies (`reason: "projected mean tps below floor (idle estimate)"`). A |
| long-idle pool (stale sample) genuinely has capacity and stays |
| unprojected. |
|
|
| **Guaranteed minimum (the anti-deadlock valve).** Admission policy gains |
| `guarantee_min_sessions` (default **1**, settable via |
| `POST /polykv/pools/{id}/admission`). In `/capacity`, when the pool's |
| currently-attached session count `< guarantee_min_sessions`, `can_admit` is |
| **true** with `reason:"guaranteed minimum"`, `guaranteed:true` β bypassing |
| the tps-floor and kv-headroom arms. The **context arm stays hard** |
| (`expected_tokens > free_cells` still refuses β cells are physical). A brand |
| -new or freshly forked (nested) pool therefore ALWAYS gets its first agent, |
| even oversubscribed past the target; deadlock is impossible by construction. |
| Once at/over the minimum, replacements follow the normal floor/target checks |
| β an oversubscribed pool converges back down as episodes end. |
|
|
| **Per-request `overcommit`.** `"overcommit": true` in the completion body |
| (threaded like `pool_id`) skips the enforced gate for that request β |
| explicit caller-controlled oversubscription, logged. |
|
|
| **Continuation bypass.** The enforced gate resolves the request's |
| `session_id` against `session_to_slot` (via the capacity control task, which |
| runs on the server thread); a known session is a continuation β never gated. |
|
|
| ### 14.3 Courier/orchestrator pass (both-layer pacing) |
|
|
| - `scheduler()` gates spawns on `projected_mean_tps_if_admitted` (not raw |
| mean) AND `settling == false`; a settling tick is a no-op (never a shrink |
| signal). `n_warming > 0` also defers judgment. |
| - After `spawn_worker`, the scheduler does not evaluate again until the |
| server reports the pool settled (one poll of `/capacity` suffices β the |
| server owns the settle clock). |
| - Report gains settle/guarantee/overcommit event rows + a settling band on |
| the tps chart. |
|
|
| ### 14.4 Compatibility & phases |
|
|
| All fields additive; pools without a policy behave exactly as before except |
| the warming exclusion (a pure measurement fix) and the continuation bypass |
| (strictly less rejection). Off-path (no PolyKV) byte-identical. |
|
|
| - **P7a** β warming-aware mean + settle state + measured-drop forecast + |
| `/capacity` fields (server). **DONE 2026-07-23.** |
| - **P7b** β `guarantee_min_sessions` + `overcommit` + continuation bypass + |
| settling hold in the enforced gate (server). **DONE 2026-07-23.** |
| - **P7c** β courier scheduler/report pass + rerun the courier gate. |
| **PASS 2026-07-23** (solidPC 3090, Qwopus3.5-9B MTP, 10 pkg Γ 55 steps, |
| floor 15): vs the baseline run, wall 2187β1919 s (β12%), time-under-floor |
| 1648β790 s (β52%), deep sub-floor (<7.5 tps) 706β122 s (β83%), delivery |
| p50 1350β764 s (β43%), same 100% score. Two gate iterations: the first |
| (settle+guarantee+projection only) got β23% under-floor but still ramped |
| to n=8 through idle-gap "no projection = free capacity" spawns; the |
| idle-estimate arm closed that hole. |
|
|
