GPU-share — zero-conf coordination between opencoti-llamafile peers on one GPU

Status: G1+G2 IMPLEMENTED (2026-07-23, c5, patch 0141) — shm peer registry + --gpu-share-weight + GET /gpu/peers + /props opencoti.gpu_share + duty-cycle pacing, all in tools/server/server-context.cpp (oc_gpu_share namespace). G3 (Windows/macOS live validation) + CUDA-UUID DSO keying remain (c6). Original design brief: Multiple opencoti-llamafile processes share one GPU (the multi-server pattern opencoti-server manages, plus standalone uses as an agentic-framework engine). They need direct, zero-conf, port-less discovery + a weight that governs each process's compute share. Decisions (user): weight = compute share (decode pacing; full speed when alone); transport = named shared-memory segment; Linux/macOS/Windows from the ONE APE binary.

1. Semantics

New flag --gpu-share-weight W (float > 0, default 1.0).
Share of process i = w_i / Σ w_j over alive, active peers on the same physical GPU — a plain ratio. Example (the spec case): weights 2, 1, 1 → 50% / 25% / 25%. The endpoint always shows each peer's resolved percentage so there is no guessing.
Idle peers cost nothing: a peer that hasn't decoded/prefilled within the activity window (default 2 s) is excluded from the denominator — a lone active process always runs at 100% regardless of registered peers.
Enforcement is cooperative duty-cycle pacing (see §4) — approximate by design; the goal is proportional sharing, not hard isolation (that would need MPS/MIG, which is Linux-only / non-portable and breaks zero-conf).

2. Why a named shared-memory registry (research result)

Requirements: zero-conf (no broker, no config, no ports), sub-ms reads from the decode loop, crash-safe (a SIGKILLed peer must not wedge the others), one APE binary for Linux/macOS/Windows.

Chosen: POSIX shm_open + mmap(MAP_SHARED) via cosmopolitan. cosmo libc (vendored .cosmocc/4.0.2) declares shm_open/shm_unlink (libc/calls/calls.h:161) and its mmap implements shared file mappings on Windows over CreateFileMapping/MapViewOfFile — the same C source compiles and runs on all three OSes inside the APE. Linux backs it with /dev/shm, macOS with POSIX shm, Windows with the cosmo shim.
Rejected: unix sockets / named pipes — message passing + broker election we don't need for a shared counter table; 3 platform code paths.
Rejected: filesystem lock/registry files — stale-file cleanup, dir permission variance, and mtime-based liveness are all worse than pid+ heartbeat words in shm.
Rejected: CUDA MPS CUDA_MPS_ACTIVE_THREAD_PERCENTAGE — Linux-only, requires an MPS control daemon (not zero-conf), static percentages (no idle-peer reflow), nothing on Windows/macOS.
No locks in the segment: fixed-size slots, single-writer-per-slot (each process writes only its own slot), seq/heartbeat words with atomic stores; readers tolerate torn strings (name is display-only). A crashed peer simply stops heartbeating and ages out — nothing to clean.

3. Registry layout

Segment name: /opencoti-gpu-<gpu_key> (one per physical GPU), 4 KiB.

header: { magic "OCGPU", version u32, slot_count u32 = 16 }
slot:   { pid u32 (0 = free, CAS-claimed), boot_us i64,
          weight f32, active u32 (decoded within window),
          heartbeat_us i64 (atomic store, ~500 ms cadence),
          busy_ppm u32 (measured busy fraction, parts-per-million),
          name char[48] (alias/model, display only) }

gpu_key: CUDA device UUID via a new DSO export ggml_backend_cuda_get_device_uuid (we own the published .so/.dll builds — additive symbol; when absent in an older side-loaded DSO, fall back to a hash of device description + PCI ordinal). Metal → metal-0 (Apple GPUs are single-device). CPU-only runs register nothing.
Liveness: heartbeat age > 3 s ⇒ dead; its slot is reclaimable by CAS. Last liver does NOT shm_unlink (races the next joiner); a 4 KiB segment persisting until reboot is harmless and self-heals via the heartbeat rule.

4. Pacing (compute-share enforcement)

Since c6 (patch 0143) pacing is deterministic weighted time-division (TDM) over a host-shared wall clock — not feedback control:

Each peer derives the SAME schedule from the shm registry + the host clock: a repeating period carved into one contiguous window per alive+active peer, proportional to its (10%-floored, renormalized) share, ordered by slot index. The period is adaptive for latency: a peer's worst token stall is (1−share)·period, so the period is the smallest that still gives the smallest share a contiguous-effective window (40 ms — bug-2243's floor): period = clamp(40ms / min_share, 80ms, 400ms). 2:1 weights → a 120 ms period (worst stalls 80/40 ms); 1:1 → 80 ms (40 ms); only extreme splits reach 400 ms.
In the decode loop, after each step: if the current wall-clock position is inside my window → keep decoding; otherwise sleep exactly until my next window starts. Overshoot into a peer's window is ≤1 decode step.
Alone (or all peers idle) → no schedule, never sleep. Idle peers leave the schedule via the 2 s activity window; kill -9 peers via the 3 s heartbeat age.

Exactly one peer runs at any instant, so the pacer is work-conserving (aggregate ≈ solo throughput) and the split equals the weight ratio by construction — there is no control loop to converge or oscillate.

Why not feedback? Three control laws failed in sequence:

Proportional sleep dt·(busy/share−1) has no fixed point (bug-2243).
Integral debt on absolute busy (debt += dt − share·wall, the c5 pacer) over-throttles the minority peer once busy is measured honestly (2.45:1 on the 3090 / 3:1 on the 5080 at 2:1 weights): contended, every peer's llama_decode wall includes waiting on the OTHER peer's kernels, so busy reads ≈ wall for everyone (bug-2244 measurement is correct — the signal itself is contention-inflated).
Integral debt on relative busy (my busy / Σ peers' busy) fixes the bias but lets both peers sleep in overlapping windows (500 ms busy-publish lag → oscillation): GPU idled ~half the time, aggregate 73 vs 116 tok/s.

Two clock/measurement invariants remain load-bearing:

Shared epoch (bug-2245): ggml_time_us() is per-process-epoch on Windows (native timer_start, and cosmo's CLOCK_MONOTONIC alike), so shm heartbeat ages misread by the peers' launch delta — a pair started >3 s apart never saw each other alive (latent in c5), and TDM windows landed at random offsets (5080 aggregate 53 vs 146). All shm timestamps + the schedule use wall_us() = CLOCK_REALTIME (Unix epoch, shared by every process on the host, on every platform).
Busy through llama_synchronize (bug-2244): llama_decode is async-submit; timing the call alone reported 4.7% busy at 93% real utilization. Busy (published for /gpu/peers and the activity flag) spans decode through a sync, which is imminent at sampling anyway.

Measured (TDM, c6, adaptive period):

solidPC RTX 3090, Qwen3-4B Q4_K_M batch-1: solo 121 tok/s; 2:1 → 75.6/38.4 (1.97:1, aggregate 114.0 vs the c5 feedback pacer's 107.9); with a deliberate 20 s start skew → 1.99:1; 1:1 → 58.9/59.0 (aggregate 117.9 vs 100.8, +17%); 2/1/1 → 56.5/27.3/29.5 (1.92:0.92:1, agg 113.3); solo-after-release 120.9.
Inter-token latency (streaming, 2:1): w2 p50 9 ms / p95 45 / max 49 ms; w1 p50 9 ms / p95 86 / max 89 ms — exactly the (1−share)·period bound. (The initial fixed 400 ms period stalled the w1 stream up to 267 ms; the c5 pacer's blocks were 100–150 ms.)
--parallel multi-slot (auto n_parallel, 4 concurrent gens each), 2:1: w2 agg 217.1 / w1 agg 125.6 (1.73:1, total 342.7 vs the c5 pacer's ~211–246 total; solo 4-way is 327.5 — TDM total is at solo level, i.e. fully work-conserving). Ratio softens at multi-slot because a batched iteration overshoots the window boundary by more than one token — accepted trade-off.
MTP (Qwopus3.5-9B NextN, --spec-type draft-mtp, -c 8192), 2:1: 74.8/35.2 (2.13:1, aggregate 110.0 ≈ 96% of the 114.7 solo); draft acceptance unharmed (0.91/0.82 vs 0.87 solo).
pandorum RTX 5080 (Windows, side-load DLL), 2:1 → 97.7/48.9 (2.00:1, aggregate 146.6 ≈ 97% of the 151 tok/s solo; the c5 fixed-block pacer gave 3:1 at aggregate ~99). Windows' 15.6 ms timer granularity is immaterial at 40–80 ms sleeps.
Gate discipline (bug-2246): on a 16 GB card set an explicit -c — VRAM auto-fit filled 15.8/16 GB and WDDM paging collapsed the aggregate to 53 tok/s (an artifact of the gate setup, not the pacer). Also gate with AUTO n_parallel: an explicit --parallel 4 takes a non-unified-KV path that decodes at ~28 tok/s aggregate regardless of pacing (bug-2247, pre-existing, out of gpu-share scope).

Prefill batches count as busy time; the 10% share floor guarantees no peer can be starved by misconfigured weights.

Driver-level alternatives evaluated (2026-07-23)

Could the sleeps be replaced by CUDA-native compute partitioning? Findings (CUDA 13.x):

MPS static SM partitioning (CUDA 13.1+, Linux, Ampere+): the control daemon (nvidia-cuda-mps-control -d -S) can carve exclusive SM partitions per MPS client (sm_partition add), deterministic and concurrent — no sleeps, no stall windows. This is the real long-term candidate for Linux fleets (bs2-class); requires daemon lifecycle management (a natural opencoti-server F3 feature) and an r590+ driver (solidPC's 580 predates it; bs2's 610 has it). Blackwell adds MLOPart (memory-locality-optimized partition devices).
Classic MPS + CUDA_MPS_ACTIVE_THREAD_PERCENTAGE (Volta+, Linux), measured on the 3090 (driver 580): 67/33 caps → 64.4/37.5 (1.72:1, aggregate 101.9 — ~11% BELOW TDM's 114.0, because batch-1 decode is bandwidth-bound, and concurrent kernels contend on memory bandwidth while SM caps only bound compute). Latency is smoother (max gap 50/53 ms vs TDM's 49/89) but p50 worsens for the small partition (16 ms vs 9). Critically not work-conserving: the caps are static — with the peer idle the 67% client does 110.9 tok/s vs TDM's full 120.9, and ratios track SM fraction only loosely.
Green contexts (CUDA 13.1 runtime "execution context" API): SM partitioning within a single process only — inapplicable to cross-process gpu-share (relevant someday for multi-model-in-one- process serving). Isolation is best-effort, not guaranteed.
Windows: no MPS, no cross-process partitioning at all — WDDM time-slicing is the only primitive, so wall-clock TDM remains the only portable mechanism there.

Conclusion: TDM stays the default (portable, zero-setup, work-conserving, exact ratios, instant idle-release). An opt-in Linux MPS mode (static SM partitioning managed by opencoti-server) is a future milestone for latency-critical fleets on r590+ drivers.

5. Visibility

GET /gpu/peers → { gpu_key, self: {pid, name, weight, share_pct, busy_pct, active}, peers: [{pid, name, weight, share_pct, busy_pct, active, heartbeat_age_ms}] }
/props opencoti block gains gpu_share carrying the same {enabled, gpu_key, self, peers} snapshot as /gpu/peers, so existing dashboards see it without a new call.
Boot log: gpu-share: joined /opencoti-gpu-<key> slot 2 (weight 1.5, 3 peers alive).

6. Phases

G1 — registry + flag + visibility (no throttle). shm module (llamafile/gpu_share.c, cosmo-portable), --gpu-share-weight, join/ heartbeat/reap, /gpu/peers + /props block. Gate: 3 instances on one GPU (solidPC 3090), each sees the other two with correct resolved percentages; kill -9 one → peers age it out ≤ 3 s.
G2 — duty-cycle pacing. Busy-window measurement + inter-batch sleep. Gate: 2 instances, weights 2:1, concurrent decode workloads → measured tps ratio ≈ 2:1 (±20%); solo instance regains ~full solo tps ≤ 2 s after peer goes idle.
G3 — Windows/macOS validation (pandorum APE run for the shm path on Windows; macOS opportunistic) + docs + opencoti-server fleet surfacing (daemon shows per-engine share/busy from /gpu/peers).

7. Non-goals

Hard isolation (MPS/MIG) — not zero-conf, not portable.
VRAM partitioning by weight — allocation is boot-time (-c, -ngl); the endpoint may later report VRAM per peer but never enforces it.
Cross-GPU balancing — the registry is strictly per-GPU; multi-GPU processes join one segment per device they occupy (v1: primary device only).