docs/features/polykv_api.md

PolyKV Control-Plane API — Design (F5 M6-S5 / #712)

Status: design (2026-07-16). Implementation is phased (§10); nothing here is shipped yet. Inputs: polykv_api_research.md (design-space survey), poly_kv.md (the KV mechanism this sits on top of), llamafile-usage.md §3.4 (SharedKVPool user surface). Single rule: additive / opt-in. Every endpoint and field lives under the existing opencoti JSON namespace or new /polykv/* routes; no upstream behavior changes when PolyKV control is unused.

---

1. Goals

A rich, opt-in control plane that satisfies every multi-agent serving journey on top of the existing SharedKVPool (shared_pool_slot + shared_prefix_n_tokens, server-context.cpp:3085). Four requirements, plus two the requirements imply:

1. Full status of pools + the sessions attached to each.
2. tps reporting — aggregate system throughput and per-session generation tps. (All-new: upstream llama.cpp reports per-response timings but has no per-session/slot running instrumentation.)
3. Autonomous capacity/admission loop — a main session sets a per-session gen-tps target (e.g. 5 tps); when the average across active sessions approaches the floor the orchestrator holds off spawning sub-agents. Behavioral knobs: reject a new session vs. warn.
4. LangGraph extensions — a custom BaseChatModel, the pool-control API exposed as LangGraph tools, and PolyKV-aware agent-state routing.
5. Nesting (implied by 3): a sub-agent that discovers fan-out work can become a pool parent for its own sub-agents, so parallelizable tasks run in parallel instead of being forced sequential.
6. Fork / branch (COW): a sub-agent group that shares an ancestor prefix but then diverges (different toolset/context) gets its own shared pool branched at the divergence point.

2. Decisions locked (2026-07-16)

#	Decision	Choice
API home	where the control plane lives	Self-contained in the C++ engine — extends /props, /slots, adds /polykv/*. Works standalone; no opencoti-server in the path.
Admission	enforce the tps floor or advise	Both — advisory (report) by default; optional per-pool enforced mode with reject|warn|queue.
Pool model	how pools are created	Explicit first-class objects — create/fork/attach/release, immutable from birth (§3.3). Contiguous-prefix contract checked at fork. Boilerplate hidden by the opencoti plugin.
Scope	deliverable shape	Design the full surface now; implement in gated phases (§10).
Fork	divergent child prefix	First-class in v1 (copy-free COW, §3.2 / §7) — not deferred.

Defaulted (override any): transport = HTTP on the engine + SSE for the live tps stream (poll fallback); tenancy = single trust domain (no per-pool ACL); tps = per-session experienced decode tps (EWMA) + aggregate decode throughput, floor = mean of active sessions' experienced tps (prefill excluded); nesting depth uncapped; interior pools pinned while any descendant is attached + grace TTL; seq-ids for pools+leaves sized at boot.

---

3. Data model

3.1 A pool is a tree node

Pool {
  id            : int32          // seq_id in the unified cache
  parent        : int32 | ROOT   // -1 for a root pool
  branch_pos D  : int32          // shared-ancestor length inherited from parent
  own_range     : [D, L)         // tokens this pool materialized itself
  prefix_len L  : int32          // total shared prefix = D + |own_range|
  tokens        : [llama_token]  // full prefix token array [0, L), host-side (auto-P + fork validation)
  prefix_hash   : [u64]          // cumulative rolling hash per position (validates any D; ~8 B/token host RAM)
  pin           : bool           // resident-pinned (auto while descendants attached)
  children      : [Pool]         // interior pools branched/extended from this one
  sessions      : [Session]      // leaf sharers attached at this pool's prefix
  last_access   : ts
}
Session {                        // a leaf = an ordinary decoding slot
  id            : int32          // seq_id
  pool          : int32          // the pool it attached to
  suffix_range  : [L, L+S)       // its own private, mutable continuation
  tps_ewma      : f32            // experienced decode tps
  ttft_ms, state, ...
}

The root pool (parent = ROOT, D = 0) is what today's shared_pool_slot request already produces. Everything else is new.

3.2 Three edge types, one mechanism

A child pool inherits [0, D) from its parent and owns [D, L). D selects the edge type:

• extend — D = parent.L. Child = parent prefix + appended shared tokens. This is nesting: a sub-agent adds its own shared context and hands it to its sub-agents.
• branch (COW fork) — D < parent.L. Child shares the common ancestor [0, D) and diverges with its own [D, L). This is the fork: same system prefix, different toolset after D.
• root — parent = ROOT, D = 0.

Why the fork is copy-free. seq_cp in the unified cache is additive membership: for each cell where seq_has(cell, parent) in [0, D) it calls seq_add(cell, child) (llama-kv-cache.cpp:2007-2044). No cell is duplicated — the ancestor cells simply gain the child in their sequence set. Because prefixes are immutable (FROZEN, §3.3), the shared range is never mutated, so the "copy" a classic COW would eventually pay never happens. The child only computes new KV for [D, L), prefilled at RoPE position D attending to the shared [0, D) already in cache — exactly how attaching with a shorter shared_prefix_n_tokens works today. Extend and branch are the same code path parameterized by D.

3.3 Immutability & versioning

Pools are immutable from birth. A pool's prefix is fully materialized at create/fork and never changes afterwards — there is no OPEN/extend-in-place state. "Extending" a pool always means creating a child (fork with D = parent.L). This is simpler than a freeze-on-first-attach state machine and loses nothing: the "build context incrementally" journey is covered by the main agent decoding in its own session and then forking from_session (§4.1), which snapshots its current context as a new pool in one call — no in-place mutation ever needed.

This makes "add a tool to the shared prefix after agents attached" well-defined — it is a new versioned pool (a fork), and the old one keeps serving its sharers unchanged. (This is the precise answer to the HuggingFace question: the prefix is static per pool; a tool/MCP added later is a new pool version, valid only for sessions that attach to it — never mutated under sessions already on the old one.)

3.4 Lifetime

Cell freeing is implicit-refcounted: a cell is reclaimed only when its sequence set empties, so an ancestor's cells physically survive as long as any descendant references them — even if the ancestor's own slot is torn down. On top of that:

• Pin policy. Interior pools are auto-pinned (--no-clear-idle semantics, per-node) while any descendant is attached. Without this a new leaf attaching to an idle-cleared interior pool hits the residency guard (server-context.cpp:3116) and silently falls back to full reprocess — correct but slow. A grace TTL delays reclaim after the last descendant detaches (cheap re-attach for bursty fan-out).
• seq-id budget. Every pool and every leaf consumes one n_seq_max slot. A deep/wide tree spends seq-ids on interior nodes too; the boot-time reserve accounts for both. Exposed in /capacity.

3.5 Materialization — the slot/seq decoupling (P2's real work)

Today a "pool" is just an ordinary server slot that happened to serve a request: shared_pool_slot names a slot index, the pool's tokens live in slots[pool].prompt.tokens, and the pool seq is the slot's seq. First-class pools break that coupling, and this is the largest single implementation lift in the milestone:

• seq-ids beyond slots. Pool seq-ids come from a boot-time reserve above the slot range: n_seq_max = n_parallel + --polykv-max-pools (default proposed: 16). A pool occupies a seq-id but no slot once materialized.
• Prefill-only internal task. create/fork materializes [D, L) via an internal task that runs through the normal batching path (so it composes with every KV feature) but emits no logits and no generation; it transiently borrows a slot for the prefill, then the KV stays on the pool's seq and the slot is returned. seq_cp(parent → pool, 0, D) runs first for fork/extend.
• Host-side pool state. The Pool object (tokens, cumulative hash, tree links, pin/TTL) lives in server memory beside the slot table — not in a slot — and is what /pools reports and attach validates against.
• Cache layout (REVISED c6, bug-2248). Originally pools required --kv-unified (the split-stream seq_cp overload hard-asserts is_full, so a partial cross-stream share was undefined — this was a boot-time refusal). Since patch 0145 pools work on BOTH layouts: unified shares stay zero-copy set-membership; on the split cache (--parallel without --kv-unified — the layout the rolling-KV position window requires) every prefix share runs as a PHYSICAL copy via oc_seq_share_prefix() (full-stream seq_cp + trim to [0,P), positions preserved; cross-stream copies include the window's host tail). Trade-off: no VRAM dedup off unified — each sharer stream holds its own prefix copy (window mode caps device use at window_cells/stream). The boot check is now an INFO stating the copy-share semantics. /capacity is STREAM-AWARE on the split cache (patch 0146): the verdict checks the per-stream budget and free streams ("no free stream (split cache)" hard-stop; headroom clamped to free streams) and the payload adds kv_streams_total / kv_streams_free / kv_cells_per_stream (null on unified).

---

4. API surface

All additive. Reuses the opencoti JSON namespace (get_opencoti_props() :3787) and the /slots accumulator precedent (:183). Endpoint shapes follow research §6.

4.1 Pool lifecycle (imperative, explicit)

Verb	Endpoint	Body → result
create root	POST /polykv/pools	{tokens? | from_session?} → {pool_id, prefix_len}. from_session snapshots a live session's current context as the pool prefix — the primary creation path for agents (no token round-trip).
fork (extend or COW branch)	POST /polykv/pools/{id}/fork	{branch_pos D, tokens[D:L] | from_session?} → new child {pool_id, parent:id, branch_pos, prefix_len}. D == parent.L ⇒ extend (nesting); D < parent.L ⇒ COW branch.
attach	(request path, unchanged mechanism)	per-request pool_id (legacy shared_pool_slot kept). shared_prefix_n_tokens becomes optional: the server auto-computes P as the longest hash-match between the request prompt and the pool's stored tokens (auto-P) — killing the mis-set-P footgun and the HF-question confusion. An explicit P is still accepted and validated.
pin/evict/flush	POST /polykv/pools/{id}/{pin|evict|flush}	retention control (research §6 Req 1)
release	DELETE /polykv/pools/{id}	detach + reclaim when refcount hits 0 (respects grace TTL)

Contiguous-prefix contract is checked at fork: the child's declared tokens[0:D] must hash-match the parent's [0:D) (cumulative prefix_hash at D), else 409 Conflict — the caller then either re-roots or full-prefills. All routes live under a /polykv/ prefix so a future upstream llamafile/llama.cpp bump can never collide with an unprefixed /pools.

4.2 Status (research §6 Req 1)

• GET /polykv/pools → {pools:[{pool_id, parent, branch_pos, prefix_len, own_kv_bytes, subtree_kv_bytes, residency, pinned, last_access_ts, children:[pool_id], sessions:[{session_id, suffix_n_tokens, marginal_kv_bytes, tps_ewma, ttft_ms, state}]}], tree_depth, seq_ids_used, seq_ids_max}. The parent/children fields make the tree explicit. KV attribution is by ownership, never by membership: a shared cell belongs to many seqs, so summing per-seq bytes would double-count — a pool reports own_kv_bytes = its [D, L) range only, plus a subtree_kv_bytes rollup (own + all descendants + attached sessions' suffixes). Σ own over the tree == physical bytes, by construction.
• GET /polykv/pools/{id} → one subtree.
• /slots gains pool_id + tps_ewma per slot (extends #677).
• /props opencoti block gains a polykv sub-object (config: floor, admission mode, pin/TTL policy).

4.3 tps (research §6 Req 2)

• Per-response (already partly present): timings{predicted_per_second, prompt_per_second, cache_n}.
• Headers (TGI model): x-session-tps, x-cached-prefix-tokens, x-pool-id. Deferred to P2+: HTTP headers must be emitted before the body, but the slot (and thus the live tps value) is assigned only after the task is queued — a header set at request time would always be stale, and trailers aren't portable. The per-response timings JSON already carries the authoritative per-response rate; headers add value only once pool attach happens at request routing (P2), where x-pool-id/x-cached-prefix-tokens are known up front.
• Live stream: GET /polykv/tps (SSE) — periodic {ts, aggregate_tps, sessions:[{session_id, pool_id, tps_ewma}], mean_active_tps}. This is the signal the admission loop subscribes to.
• /metrics (Prometheus, low-cardinality per research §4): gauges opencoti_polykv:gen_throughput_tps{pool}, pool_kv_bytes{pool}, pool_sessions{pool}; counters prefix_cache_hits_total{pool}, queries_total{pool}. Label = pool only (never session — cardinality).

4.4 Admission (research §6 Req 3; decision: both modes)

• POST /polykv/pools/{id}/admission {target_tps_per_session, mode:"advisory"| "enforced", on_saturation:"reject"|"warn"|"queue", guarantee_min_sessions, settle_tokens, settle_max_ms} — per-pool policy (last three = P7 §14, defaults 1 / 48 / 5000).
• GET /polykv/pools/{id}/capacity → {can_admit, projected_mean_tps_if_admitted, headroom_sessions, kv_headroom_pct, reason}
  • P7 (§14): {settling, settle_remaining_ms, n_warming, n_pool_sessions, guaranteed, projected_mean_tps_model, projected_mean_tps_measured, drop_per_admit_ewma, known_session}. The projection is measured-drop based once the pool has absorbed ≥1 admit; the raw mean excludes warming (EWMA-cold) sessions. headroom_sessions is computed against marginal per-session KV (S_i·bpt), not full context — the basis of the ~7× reduction (research §5).
• Advisory (default): endpoints report; the orchestrator decides. Nothing is rejected server-side.
• Enforced (opt-in): on a new attach with projected mean_active_tps < floor OR kv_headroom exhausted → reject = 429/503 + Retry-After (RFC 6585/9110); warn = admit + X-Sessions-Remaining header; queue = hold until headroom returns. P7 (§14): the gate applies to NEW sessions only (an existing session→slot affinity = continuation, never gated); "overcommit": true in the request body bypasses it explicitly; and reason:"measurement settling" maps to a bounded HOLD (≤ settle_max_ms
  • 1 s) rather than a 429.
• Throttle-toward-floor is explicitly NOT v1. Andes' insight (over-floor generation is wasted GPU → throttle fast sessions back toward the floor to reclaim capacity) requires per-slot rate control inside the continuous- batching decode loop — i.e. skipping a slot in some decode batches. That is scheduler surgery with its own correctness/perf gates, not an API feature; it is banked as a P4-follow candidate, and v1 warn is headers-only.

---

5. tps instrumentation (all-new, C++ server-side)

Upstream has no per-session running tps. Design:

• Extend the /slots lifetime accumulators (server-context.cpp:183) with a per-slot decode-token counter + wall-clock, updated at each decode-batch completion for the tokens attributable to that slot.
• Experienced tps per session = EWMA over a rolling window (default: last W decoded tokens or T ms, whichever first) of that slot's decode throughput. This is the quantity that falls as concurrency rises even while aggregate climbs (attention doesn't batch-amortize — research §5), so it is the correct SLO signal.
• Aggregate = system decode throughput (all slots' decode tokens / wall).
• mean_active_tps = mean of experienced tps across currently-decoding sessions — the value the floor compares against (matches "average sessions gen tps").
• Prefill tokens are excluded from "gen tps" (counted separately as prompt_per_second).
• Cost: a handful of integer counters + one EWMA update per slot per batch — negligible; gated behind the PolyKV block so off-path is byte-identical.

---

6. Admission control

Two coupled signals (research §5):

1. Soft / low-lag: mean_active_tps vs the floor. Directly measures the SLO; leads num_requests_waiting (which lags).
2. Hard wall: pool KV-occupancy %. Reject before this forces preempt-oldest eviction.

projected_mean_tps_if_admitted estimates the post-admit floor from the B_sat knee model (below the knee admitting is ~free; above it every admit costs everyone's TPOT ~linearly). On high-VRAM cards (RTX 6000 96 GB) the tps knee is hit before the KV wall, so signal (1) dominates on opencoti's hardware — which is exactly why the floor, not occupancy, is the headline.

Framing: "5 tok/s/session" = TPOT SLO 200 ms/token; maximize goodput s.t. TPOT ≥ floor (a floor, not a maximize — Andes). The default is advisory: the LangGraph capacity-gate node (§8) reads /capacity and decides. Enforced mode is the same math with server-side reject/warn/queue.

---

7. Nesting & COW-fork — worked journeys

Journey A — sub-agent parallelization (nesting / extend). Main agent M runs on root pool P0 = [system+tools] (len N0, FROZEN). M builds task context and POST /pools/P0/fork {branch_pos:N0, tokens:[N0:N1]} → interior pool P1 = [system+tools+task]. M spawns k sub-agents that each attach to P1 and decode their own suffix in parallel. Physically, P1's [0:N0) cells are P0's cells (now in seq-set {P0,P1,leaf…}); [N0:N1) is prefilled once. The k sub-agents cost only their marginal suffixes. The fan-out work that used to run sequentially now runs concurrently under one shared branch.

Journey B — divergent toolset (COW fork). Two sub-agent groups share the system preamble [0:D) but need different toolsets. Group B does POST /pools/P0/fork {branch_pos:D, tokens:[D:D']} (D < N0) → P1' = [system+toolsetB], sharing [0:D) zero-copy, prefilling [D:D') fresh. Its sub-agents attach to P1'. No physical copy of the shared preamble.

Invariants enforced. (a) contiguous-prefix hash check at fork; (b) immutability from birth — a pool's prefix is fully materialized before any descendant can attach (§3.3), so no attach ever races a prefix change; (c) recursive pin — interior pools stay resident while descendants attached; (d) seq-id budget counts interior nodes. Divergence that is not a clean prefix of any existing pool → new root (or full prefill) — reported, never silently wrong.

Hybrid/recurrent guard (bug-2203) is recursive. Every attach/fork on a recurrent or hybrid model still hits the exact-full-state guard (server-context.cpp:3116) at every level.

---

8. LangGraph integration (research §6 Req 4)

Package home: a new top-level opencoti/ non-bun root (per the repo's "new non-bun roots" rule) — Python package opencoti-langgraph (pip). Kept out of the bun workspace. Pin BaseChatModel signatures to a specific langchain-core version tag (the API is mid-migration — research caveat).

• OpencotiChatModel(BaseChatModel) — implement _generate + _llm_type (required), _stream/_agenerate/_astream; override bind_tools. Populate usage_metadata and response_metadata{pool_id, session_tps, backpressure} from the headers/§4.3.
• Pool-control tools — create_pool, fork_pool, pool_status, session_tps, capacity exposed as LangChain tools via bind_tools / ToolNode, so an agent can manage its own pool tree.
• Capacity-gate node — reads /capacity; returns Command(update={"capacity_ok":ok, "headroom_sessions":n}, goto=...) to route the graph (spawn vs. hold). interrupt() for block-until-free. InMemoryRateLimiter is explicitly insufficient — it can't see per-session tps or KV headroom.
• Agent-state routing — a reducer surfaces headroom_sessions / mean_active_tps into graph state so conditional edges branch on live PolyKV capacity.

---

9. Composition with the KV stack

PolyKV control is orthogonal to the KV mechanics and must not perturb them:

• rolling-KV / DCA / turbo / TCQ / quant-KV — pool sharing is seq_cp membership on whatever cell type is configured; the tps/admission plane is read-only over the existing decode loop. Off-path byte-identical (gate).
• MTP / speculative — per-session tps counts accepted tokens (the user- visible generation rate), not draft steps.
• --parallel ≥ 2 — the multi-slot requirement is already the SharedKVPool baseline; the tree just adds interior seq-ids. --kv-unified is optional since c6 (zero-copy shares with it, copy-shares without it — see the layout note in §3.5).
• hybrid/recurrent — bug-2203 guard, recursive (§7).

10. Phased implementation plan (gated)

Full surface designed here; implementation lands in gated phases, each with a correctness + off-path-identity gate before the next:

• P1 — tps instrumentation (§5). Per-slot counters + EWMA + /slots//props exposure + /polykv/tps SSE + /metrics. Prereq for everything. ✅ IMPLEMENTED 2026-07-16 (live in vendored tree, patch capture pending): gates green — sustained EWMA −3.2% vs timings ground truth; 2-session concurrency shows correct aggregate_tps ≈ Σ sessions + mean_active_tps; /metrics 3 series render. bug-2205: aggregate MUST come from Σ n_decoded_cum (per-token), not n_tokens_predicted_total (completion-time only).
• P2 — pool objects + status API (§3, §4.1-4.2). The slot/seq decoupling (§3.5): seq-id reserve, prefill-only materialization task, host-side Pool state, auto-P attach; GET /polykv/pools. Root pools only. ✅ IMPLEMENTED 2026-07-16 (live in vendored tree, patch capture pending): gate 23/23 on Qwen3-4B — boot check (--polykv-max-pools without --kv-unified fails clean; NOTE --parallel must be explicit, unset auto-sets kv_unified=true), create-from-prompt prefill-only P=95, auto-P exact attach cache_n=95, divergence attach cache_n=42<95, from_session zero-copy seq_cp snapshot (affinity is evict-on-reuse — only the most-recent session per slot resolves), pin/unpin/GET-one, release + released-id clean fallback, disabled path inert, pooled vs unpooled greedy text IDENTICAL (bit-shared cells). Mutations only on the server thread via SERVER_TASK_TYPE_POLYKV; recurrent guard per bug-2203.
• P3 — nesting & COW-fork (§7). fork with arbitrary D (extend + COW branch), cumulative-hash contract check, recursive pin, seq-id accounting. ✅ IMPLEMENTED 2026-07-16 (live in vendored tree, patch capture pending): gate 18/18 substantive on Qwen3-4B — extend fork (D=parent.L, own=22), COW branch (D=47<95, zero-copy seq_cp of [0,D) + fresh prefill), contract violation → clean 4xx, auto-P attach to a child (cache_n=117), tree reporting (parent/children/branch_pos/own_len/tree_depth), release ordering enforced (parent-with-children refused, leaves-first drains), seq exhaustion at max_pools, from_session fork. The §4.1 prefix contract is enforced token-exact (std::equal against the parent's stored tokens — strictly subsumes the cumulative-hash check; prefix_hash kept as identity fingerprint). Pooled-vs-unpooled greedy divergence on the branch was classified benign by first-token probe (same top-5 ranking, max |Δlogprob| 0.112 — batch-shape numerics, #495 class). Recurrent bug-2203 guard is recursive: hybrid/recurrent forks must extend at D == parent.L.
• P4 — admission (§6). /capacity, advisory metrics, then enforced mode (reject/warn/queue). ✅ IMPLEMENTED 2026-07-16 (live in vendored tree, patch capture pending; queue deferred per §11 default): gate 18/18 on Qwen3-4B — /capacity shape (idle: can_admit, projected null, kv cells by OWNERSHIP: pools' own ranges + slots' marginal suffixes via new slot.n_pool_shared), advisory floor never rejects, enforced+reject → 429 + Retry-After under an active decoding session (tps-floor arm needs n_active ≥ 1), enforced+warn → 200 + X-Sessions-Remaining, off-path (no pool_id / unknown pool_id) untouched. Projection = conservative fully-saturated knee: mean_active · n/(n+1) (never over-admits below the real B_sat knee). Fast path: an atomic enforced-pool count — ordinary traffic pays zero; the gate itself is a high-priority OP_CAPACITY control task from handle_completions_impl (same pattern as /slots' METRICS task).
• P5 — LangGraph package (§8). opencoti-langgraph under top-level opencoti/. Runs in the first-class opencoti conda env (py3.11, langchain-core 1.4.x + langgraph 1.2.x pinned; the same env will later host the consultants council alongside opencoti-server). Includes the P6d CompactionNode (§13). P5 ✅ IMPLEMENTED 2026-07-16: offline 21/21 (pytest, httpx MockTransport FakeServer), live gate 19/19 on Qwen3-4B :8231 --parallel 2 --kv-unified --polykv-max-pools 4 — pool tree (token-path fork D=65), OpencotiChatModel invoke + streaming with pool attach proven (cache_n=68 ≥ root prefix 65), session_tps/usage metadata, capacity context arm + gate-node routing, enforced 429 → PoolSaturatedError(retry_after) → advisory recovery, P6c compact-by-refork e2e (old pool released, migrated session coherent on the new pool, cache_n=71), tps ctx_* fields. Contract lessons (bug-2208): fork prompt/tokens = the child's FULL prefix [0,L) token-exact vs the parent (never suffix-only — build via /tokenize: parent tokens + suffix add_special=false); a pool prefix meant for /v1/chat/completions attach must be the chat-templated system block (raw text shares 0 tokens → cache_n=0); Qwen3 thinking models need chat_template_kwargs: {enable_thinking: false} for content-based gates. Client accordingly ships tokenize() + compact_by_refork(compacted_tokens=…).
• P6 — context exhaustion & compaction (§13). P6a (ctx-shift pool guard) + P6b (context visibility + /capacity context arm) land BEFORE P5 per user decision 2026-07-16; P6c/P6d ship with P5. P6a+P6b ✅ IMPLEMENTED 2026-07-16 (live in vendored tree, patch capture pending): gate 23/23 on Qwen3-4B at -c 4096 --parallel 1 --context-shift (the exact §13.1-regime-3 configuration) — boot WARN; pooled session forced past n_ctx = graceful truncated=true stop at 4021 tokens (position wall fires one token before unified-cell exhaustion when the prefix is pool-shared), NO shift, pool provably coherent afterwards (fresh attach cache_n=60, needle answered); non-pooled ctx-shift still engages and completes after pools released (off-path intact); ?expected_tokens context arm rejects with "context headroom exhausted"; compaction_pressure 0 at idle, ∈[0,1]; orphaned_pin true for a pinned never-attached leaf; /polykv/tps sessions carry ctx_used/ctx_total/ctx_headroom_tokens. Guard bookkeeping: slot.n_pool_shared is clamped to the retained prefix at prompt-cache decision time (min(n_pool_shared, n_past)) — NOT blind-reset, because cache reuse keeps pool-shared cells across later non-pool requests on the same session. NOTE the geometry lesson: with --kv-unified the slot n_ctx is the FULL -c (not divided by --parallel), and multi-slot cell exhaustion precedes the positional wall — gates must use --parallel 1.

11. Open sub-decisions (defaulted; flag to change)

• EWMA window W/T defaults (proposed: 128 tokens or 2 s).
• Grace TTL for interior-pool reclaim (proposed: reuse --slot-shrink-idle-ms).
• /metrics opt-in flag name + default-off.
• Whether queue (admission) is v1 or P4-follow (proposed: reject+warn v1, queue follow).
• --polykv-max-pools default (proposed: 16; each reserved seq-id costs KV bookkeeping but no cells until materialized).
• Hash algorithm for the cumulative prefix_hash (proposed: xxhash64 rolling; 8 B/token host RAM — ~800 KB for a 100k-token prefix, negligible).
• SSE tps-stream cadence (proposed: 500 ms, configurable; stream is opt-in so off-path cost is zero).

12. Verification

• Off-path identity: PolyKV-control-unused boot is byte-identical (DSO + decode).
• tps accuracy: instrumented per-session tps vs. an external wall-clock token-rate measurement, ±few %.
• Fork correctness: a branched pool's decode is logit-equivalent to the same session full-prefilled (real_frac=0 / KLD), never greedy-needle.
• Tree lifetime: ancestor cells survive descendant-only references; reclaim on refcount 0; recursive pin prevents the residency-miss fallback.
• Admission: projected vs. realized mean_active_tps under a spawn ramp; reject/warn behavior + Retry-After/X-Sessions-Remaining headers.
• Composition gates: rolling-KV / DCA / turbo / MTP / --parallel 2 unchanged.

13. Context exhaustion & compaction orchestration (P6)

Added 2026-07-16 on user review: the design above admits sessions by tps floor and marginal KV, but had no story for what happens when the shared context actually fills — and continuous agentic frameworks WILL fill it. Decision: P6a+P6b land BEFORE P5 (the LangGraph capacity gate builds on the context arm); P6c/P6d ship with/after P5.

13.1 What the engine does today (verified, three regimes)

1. Prompt too long at admission → clean ERROR_TYPE_EXCEED_CONTEXT_SIZE.
2. Generation reaches n_ctx, --ctx-shift off (default) → clean stop with truncated=true (server-context.cpp:1898). Not corrupting, but silent mid-thought truncation is exactly the "fails badly" mode for agents: no advance warning, no orchestration signal.
3. --ctx-shift on + pool-attached session → CORRUPTION. The shift does seq_rm + seq_add(…, -n_discard) (server-context.cpp:3261-3262); in the unified cache seq_add mutates the per-cell position, and pool-shared cells are members of many seqs — the shift re-RoPEs the pool's prefix for every other session and every descendant pool. Upstream guards this for parent/child shared prompts (server-context.cpp:3192) but a pool attach is neither, so the guard does not fire. The rest-kv-eviction variant uses the same primitives and has the same hazard.

The "safe" naive alternative — COW-copy shared cells before shifting — allocates a duplicate of the prefix at the exact moment the cache is full. Both intuited failure modes (corruption / usage inflation) are real.

13.2 Design principle: compaction is a prompt REWRITE, never an in-place KV op

LangGraph (and every agentic framework) compacts at the prompt-assembly level: trim_messages, RemoveMessage reducers, or a summarization pre_model_hook that replaces old messages with a running summary. The compacted context is new text — its tokens cannot match the old cells, so cell-level "compaction" is not even meaningful. Therefore the server must never mutate pooled cells; the correct primitive already exists in the tree:

Compaction = re-root via fork. The system prompt + tool definitions survive compaction verbatim — that is the stable ancestor pool [0, D_sys). Compacted context = fork(ancestor, D=D_sys, tokens = summary + recent tail) — a new sibling branch, prefilled once. Sessions migrate to the new pool; the old working subtree is released. Pool immutability is preserved by construction, no cell position is ever touched, and net cells go down (old subtree reclaimed) instead of up (COW duplication).

Pinned pools under this model are a leak hazard, not a corruption hazard: a pin left on the abandoned subtree blocks reclaim. Discipline is unpin-after-migrate; the server reports orphans (pinned + zero sessions + zero children) so the orchestrator can sweep.

13.3 Phases

• P6a — corruption guard (before P5). Extend the ctx-shift refusal to pool-attached slots (slot.n_pool_shared > 0 ⇒ clean error, same as the :3192 shared-prompt refusal) + boot WARN when --ctx-shift is combined with --polykv-max-pools. Rest-kv-eviction path included. Off-path inert.
• P6b — visibility + spawn-gate parity (before P5). Per-session ctx_used/ctx_headroom_tokens in /polykv/tps + /slots; /capacity gains ?expected_tokens=N and a context arm to can_admit (reason "context headroom exhausted") so sub-agent spawns are gated on context exactly like the tps floor; plus a compaction_pressure field (0..1, driven by free-cell fraction and largest-session share) so the orchestrator compacts BEFORE the wall.
• P6c — compact-by-refork protocol (with P5). Documented two-step: summarize via normal completion → fork the stable ancestor at D_sys with the compacted suffix → migrate sessions → release old subtree. Orphaned-pin flag in /polykv/pools. Atomic convenience endpoint only if the two-step proves racy in practice.
• P6d — LangGraph side (inside opencoti-langgraph). CompactionNode (pre_model_hook pattern: watch response_metadata headroom / compaction_pressure → summarize → re-root → swap pool_id in graph state); the capacity-gate node checks the context arm from day 1.

13.4 Non-goals

• In-place KV compaction/merging of pooled cells (meaningless under rewrite semantics; corrupting under sharing).
• Server-side self-summarization (the orchestrator owns the summary — it has the conversation semantics; the server only has tokens).
• Growing n_ctx at runtime (allocation is boot-time; capacity planning is the admission plane's job).

---

14. P7 — measurement-settled admission, measured-drop forecast, guaranteed minimum

Status: design locked (2026-07-23, user). Driven by the courier-qwopus9b-mtp-s55-p10 run: floor 15 tps, spawns every 5 s on a tps EWMA (tau 2 s) that had not yet absorbed the previous admit (38→31.7→21.7→15.9 at successive ticks), overshoot to n=6 at 11.6 tps, and the corrective shrink actuated 22 minutes later (workers retire only at episode end). Decisions: pacing lives in BOTH layers (server-side hold + orchestrator pacing); overflow valve = per-pool guarantee_min_sessions (default 1) + per-request overcommit honored.

14.1 Root causes (from the run + code audit)

1. Admit-on-stale-measurement. Nothing marks a pool "still absorbing the last admit"; every 5 s tick admitted again before the EWMA settled.
2. Warming bias. mean_active_tps averages tps_ewma over all processing slots, including just-admitted ones still at tps_ewma == 0 — right after a spawn the mean is dragged toward 0, corrupting both the floor comparison and the projection.
3. Optimistic projection. n/(n+1) (flat-aggregate knee model) under- predicted the real drop at every step of the run (e.g. predicted 17.4 post-admit, reality 15.9). No measured feedback.
4. Courier bypassed the projection. scheduler() compares raw mean_active_tps to the floor instead of projected_mean_tps_if_admitted (bug — fixed in P7's courier pass).
5. Enforced gate cannot tell admission from continuation. The gate at request routing fires on ANY request carrying pool_id, so a pool under floor 429s its own running sessions' next turns — livelock risk. The gate must gate only NEW sessions (no session_to_slot affinity yet).
6. Shrink actuation lag is structural (episode-end retire). P7 does not change it; it makes overshoot rare instead (prevention, not cure).

14.2 Server design

Settle window (per pool). On each NEW-session admit record {t_admit_us, admit_slot, n_decoded_at_admit, mean_at_admit}. The pool is settling until the admitted slot has decoded settle_tokens (default 48) more tokens OR settle_max_ms (default 5000) elapsed. While settling:

• /capacity → settling: true, settle_remaining_ms, can_admit: false, reason: "measurement settling" (advisory clients wait, not shrink).
• Enforced mode → the gate HOLDS the attach (bounded sleep-poll ≤ settle_max_ms, then re-evaluates) instead of 429 — "slow the spawn just enough", never a hard reject for settling alone.

Warming-aware mean. Slots with tps_ewma == 0 while processing are warming: excluded from mean_active_tps/projection; reported as n_warming in /capacity and /polykv/tps.

Measured-drop forecast. When a settle completes, fold drop = max(0, mean_at_admit − mean_settled_now) into a per-pool EWMA (drop_ewma, alpha 0.3). Projection: projected_measured = mean_settled − drop_ewma (once ≥1 sample), projected_model = mean·n/(n+1) (fallback / always reported). The floor arm uses projected_measured when available, else the model. /capacity reports both plus drop_per_admit_ewma.

Idle-gap estimate. When every session is between turns (tool calls in flight) n_active is 0 and "no projection" reads as free capacity — the hole that still ramped the first P7 gate run to n=8. If the pool has live sessions and a FRESH last settled mean (< 30 s), /capacity projects from it instead (projected_idle_estimate: true; measured-drop variant when available, else last_settled_mean·n_pool/(n_pool+1)), and the floor arm applies (reason: "projected mean tps below floor (idle estimate)"). A long-idle pool (stale sample) genuinely has capacity and stays unprojected.

Guaranteed minimum (the anti-deadlock valve). Admission policy gains guarantee_min_sessions (default 1, settable via POST /polykv/pools/{id}/admission). In /capacity, when the pool's currently-attached session count < guarantee_min_sessions, can_admit is true with reason:"guaranteed minimum", guaranteed:true — bypassing the tps-floor and kv-headroom arms. The context arm stays hard (expected_tokens > free_cells still refuses — cells are physical). A brand -new or freshly forked (nested) pool therefore ALWAYS gets its first agent, even oversubscribed past the target; deadlock is impossible by construction. Once at/over the minimum, replacements follow the normal floor/target checks — an oversubscribed pool converges back down as episodes end.

Per-request overcommit. "overcommit": true in the completion body (threaded like pool_id) skips the enforced gate for that request — explicit caller-controlled oversubscription, logged.

Continuation bypass. The enforced gate resolves the request's session_id against session_to_slot (via the capacity control task, which runs on the server thread); a known session is a continuation → never gated.

14.3 Courier/orchestrator pass (both-layer pacing)

• scheduler() gates spawns on projected_mean_tps_if_admitted (not raw mean) AND settling == false; a settling tick is a no-op (never a shrink signal). n_warming > 0 also defers judgment.
• After spawn_worker, the scheduler does not evaluate again until the server reports the pool settled (one poll of /capacity suffices — the server owns the settle clock).
• Report gains settle/guarantee/overcommit event rows + a settling band on the tps chart.

14.4 Compatibility & phases

All fields additive; pools without a policy behave exactly as before except the warming exclusion (a pure measurement fix) and the continuation bypass (strictly less rejection). Off-path (no PolyKV) byte-identical.

• P7a — warming-aware mean + settle state + measured-drop forecast + /capacity fields (server). DONE 2026-07-23.
• P7b — guarantee_min_sessions + overcommit + continuation bypass + settling hold in the enforced gate (server). DONE 2026-07-23.
• P7c — courier scheduler/report pass + rerun the courier gate. PASS 2026-07-23 (solidPC 3090, Qwopus3.5-9B MTP, 10 pkg × 55 steps, floor 15): vs the baseline run, wall 2187→1919 s (−12%), time-under-floor 1648→790 s (−52%), deep sub-floor (<7.5 tps) 706→122 s (−83%), delivery p50 1350→764 s (−43%), same 100% score. Two gate iterations: the first (settle+guarantee+projection only) got −23% under-floor but still ramped to n=8 through idle-gap "no projection = free capacity" spawns; the idle-estimate arm closed that hole.