docs/features/gpu_share.md

# GPU-share — zero-conf coordination between opencoti-llamafile peers on one GPU

> **Status: G1+G2 IMPLEMENTED (2026-07-23, c5, patch `0141`)** — shm peer
> registry + `--gpu-share-weight` + `GET /gpu/peers` + `/props`
> `opencoti.gpu_share` + duty-cycle pacing, all in
> `tools/server/server-context.cpp` (`oc_gpu_share` namespace). G3
> (Windows/macOS live validation) + CUDA-UUID DSO keying remain (c6).
> Original design brief: Multiple opencoti-llamafile
> processes share one GPU (the multi-server pattern opencoti-server manages,
> plus standalone uses as an agentic-framework engine). They need direct,
> zero-conf, port-less discovery + a **weight** that governs each process's
> **compute share**. Decisions (user): weight = compute share (decode
> pacing; full speed when alone); transport = named shared-memory segment;
> Linux/macOS/Windows from the ONE APE binary.

## 1. Semantics

- New flag `--gpu-share-weight W` (float > 0, default `1.0`).
- Share of process *i* = `w_i / Σ w_j` over **alive, active** peers on the
  same physical GPU — a plain ratio. Example (the spec case): weights
  `2, 1, 1` → `50% / 25% / 25%`. The endpoint always shows each peer's
  resolved percentage so there is no guessing.
- **Idle peers cost nothing**: a peer that hasn't decoded/prefilled within
  the activity window (default 2 s) is excluded from the denominator — a
  lone active process always runs at 100% regardless of registered peers.
- Enforcement is **cooperative duty-cycle pacing** (see §4) — approximate
  by design; the goal is proportional sharing, not hard isolation (that
  would need MPS/MIG, which is Linux-only / non-portable and breaks
  zero-conf).

## 2. Why a named shared-memory registry (research result)

Requirements: zero-conf (no broker, no config, no ports), sub-ms reads from
the decode loop, crash-safe (a SIGKILLed peer must not wedge the others),
one APE binary for Linux/macOS/Windows.

- **Chosen: POSIX `shm_open` + `mmap(MAP_SHARED)` via cosmopolitan.**
  cosmo libc (vendored `.cosmocc/4.0.2`) declares `shm_open`/`shm_unlink`
  (`libc/calls/calls.h:161`) and its `mmap` implements shared file mappings
  on Windows over CreateFileMapping/MapViewOfFile — the same C source
  compiles and runs on all three OSes inside the APE. Linux backs it with
  `/dev/shm`, macOS with POSIX shm, Windows with the cosmo shim.
- **Rejected: unix sockets / named pipes** — message passing + broker
  election we don't need for a shared counter table; 3 platform code paths.
- **Rejected: filesystem lock/registry files** — stale-file cleanup, dir
  permission variance, and mtime-based liveness are all worse than pid+
  heartbeat words in shm.
- **Rejected: CUDA MPS `CUDA_MPS_ACTIVE_THREAD_PERCENTAGE`** — Linux-only,
  requires an MPS control daemon (not zero-conf), static percentages (no
  idle-peer reflow), nothing on Windows/macOS.
- **No locks in the segment**: fixed-size slots, single-writer-per-slot
  (each process writes only its own slot), seq/heartbeat words with atomic
  stores; readers tolerate torn strings (name is display-only). A crashed
  peer simply stops heartbeating and ages out — nothing to clean.

## 3. Registry layout

Segment name: `/opencoti-gpu-<gpu_key>` (one per physical GPU), 4 KiB.

```
header: { magic "OCGPU", version u32, slot_count u32 = 16 }
slot:   { pid u32 (0 = free, CAS-claimed), boot_us i64,
          weight f32, active u32 (decoded within window),
          heartbeat_us i64 (atomic store, ~500 ms cadence),
          busy_ppm u32 (measured busy fraction, parts-per-million),
          name char[48] (alias/model, display only) }
```

- **gpu_key**: CUDA device UUID via a new DSO export
  `ggml_backend_cuda_get_device_uuid` (we own the published `.so`/`.dll`
  builds — additive symbol; when absent in an older side-loaded DSO, fall
  back to a hash of device description + PCI ordinal). Metal → `metal-0`
  (Apple GPUs are single-device). CPU-only runs register nothing.
- **Liveness**: heartbeat age > 3 s ⇒ dead; its slot is reclaimable by CAS.
  Last liver does NOT `shm_unlink` (races the next joiner); a 4 KiB
  segment persisting until reboot is harmless and self-heals via the
  heartbeat rule.

## 4. Pacing (compute-share enforcement)

Since c6 (patch `0143`) pacing is **deterministic weighted time-division
(TDM)** over a host-shared wall clock — not feedback control:

1. Each peer derives the SAME schedule from the shm registry + the host
   clock: a repeating period carved into one **contiguous window per
   alive+active peer**, proportional to its (10%-floored, renormalized)
   share, ordered by slot index. The period is **adaptive for
   latency**: a peer's worst token stall is `(1−share)·period`, so the
   period is the smallest that still gives the smallest share a
   contiguous-effective window (40 ms — bug-2243's floor):
   `period = clamp(40ms / min_share, 80ms, 400ms)`. 2:1 weights → a
   120 ms period (worst stalls 80/40 ms); 1:1 → 80 ms (40 ms); only
   extreme splits reach 400 ms.
2. In the decode loop, after each step: if the current wall-clock
   position is inside my window → keep decoding; otherwise sleep exactly
   until my next window starts. Overshoot into a peer's window is ≤1
   decode step.
3. Alone (or all peers idle) → no schedule, never sleep. Idle peers
   leave the schedule via the 2 s activity window; kill -9 peers via the
   3 s heartbeat age.

Exactly one peer runs at any instant, so the pacer is **work-conserving**
(aggregate ≈ solo throughput) and the split equals the weight ratio **by
construction** — there is no control loop to converge or oscillate.

Why not feedback? Three control laws failed in sequence:
- Proportional sleep `dt·(busy/share−1)` has no fixed point (bug-2243).
- Integral debt on **absolute** busy (`debt += dt − share·wall`, the c5
  pacer) over-throttles the minority peer once busy is measured honestly
  (2.45:1 on the 3090 / 3:1 on the 5080 at 2:1 weights): contended,
  every peer's `llama_decode` wall includes waiting on the OTHER peer's
  kernels, so busy reads ≈ wall for everyone (bug-2244 measurement is
  correct — the signal itself is contention-inflated).
- Integral debt on **relative** busy (my busy / Σ peers' busy) fixes the
  bias but lets both peers sleep in overlapping windows (500 ms
  busy-publish lag → oscillation): GPU idled ~half the time, aggregate
  73 vs 116 tok/s.

Two clock/measurement invariants remain load-bearing:
- **Shared epoch (bug-2245):** `ggml_time_us()` is per-process-epoch on
  Windows (native `timer_start`, and cosmo's `CLOCK_MONOTONIC` alike),
  so shm heartbeat ages misread by the peers' launch delta — a pair
  started >3 s apart never saw each other alive (latent in c5), and TDM
  windows landed at random offsets (5080 aggregate 53 vs 146). All shm
  timestamps + the schedule use `wall_us()` = `CLOCK_REALTIME` (Unix
  epoch, shared by every process on the host, on every platform).
- **Busy through `llama_synchronize` (bug-2244):** `llama_decode` is
  async-submit; timing the call alone reported 4.7% busy at 93% real
  utilization. Busy (published for `/gpu/peers` and the activity flag)
  spans decode through a sync, which is imminent at sampling anyway.

**Measured (TDM, c6, adaptive period):**
- solidPC RTX 3090, Qwen3-4B Q4_K_M batch-1: solo 121 tok/s; 2:1 →
  75.6/38.4 (**1.97:1, aggregate 114.0** vs the c5 feedback pacer's
  107.9); with a deliberate 20 s start skew → 1.99:1; 1:1 → 58.9/59.0
  (**aggregate 117.9** vs 100.8, +17%); 2/1/1 → 56.5/27.3/29.5
  (1.92:0.92:1, agg 113.3); solo-after-release 120.9.
- **Inter-token latency (streaming, 2:1):** w2 p50 9 ms / p95 45 / max
  49 ms; w1 p50 9 ms / p95 86 / max 89 ms — exactly the
  `(1−share)·period` bound. (The initial fixed 400 ms period stalled
  the w1 stream up to 267 ms; the c5 pacer's blocks were 100–150 ms.)
- **`--parallel` multi-slot (auto n_parallel, 4 concurrent gens each),
  2:1:** w2 agg 217.1 / w1 agg 125.6 (**1.73:1, total 342.7** vs the
  c5 pacer's ~211–246 total; solo 4-way is 327.5 — TDM total is at
  solo level, i.e. fully work-conserving). Ratio softens at multi-slot
  because a batched iteration overshoots the window boundary by more
  than one token — accepted trade-off.
- **MTP (Qwopus3.5-9B NextN, `--spec-type draft-mtp`, `-c 8192`), 2:1:**
  74.8/35.2 (**2.13:1, aggregate 110.0 ≈ 96% of the 114.7 solo**);
  draft acceptance unharmed (0.91/0.82 vs 0.87 solo).
- pandorum RTX 5080 (Windows, side-load DLL), 2:1 → 97.7/48.9
  (**2.00:1, aggregate 146.6 ≈ 97% of the 151 tok/s solo**; the c5
  fixed-block pacer gave 3:1 at aggregate ~99). Windows' 15.6 ms timer
  granularity is immaterial at 40–80 ms sleeps.
- Gate discipline (bug-2246): on a 16 GB card set an explicit `-c` —
  VRAM auto-fit filled 15.8/16 GB and WDDM paging collapsed the
  aggregate to 53 tok/s (an artifact of the gate setup, not the pacer).
  Also gate with AUTO `n_parallel`: an explicit `--parallel 4` takes a
  non-unified-KV path that decodes at ~28 tok/s aggregate regardless of
  pacing (bug-2247, pre-existing, out of gpu-share scope).

Prefill batches count as busy time; the 10% share floor guarantees no
peer can be starved by misconfigured weights.

### Driver-level alternatives evaluated (2026-07-23)

Could the sleeps be replaced by CUDA-native compute partitioning?
Findings (CUDA 13.x):

- **MPS static SM partitioning (CUDA 13.1+, Linux, Ampere+):** the
  control daemon (`nvidia-cuda-mps-control -d -S`) can carve
  **exclusive SM partitions** per MPS client (`sm_partition add`),
  deterministic and concurrent — no sleeps, no stall windows. This is
  the real long-term candidate for Linux fleets (bs2-class); requires
  daemon lifecycle management (a natural opencoti-server F3 feature)
  and an r590+ driver (solidPC's 580 predates it; bs2's 610 has it).
  Blackwell adds MLOPart (memory-locality-optimized partition devices).
- **Classic MPS + `CUDA_MPS_ACTIVE_THREAD_PERCENTAGE` (Volta+, Linux),
  measured on the 3090 (driver 580):** 67/33 caps → 64.4/37.5
  (**1.72:1, aggregate 101.9** — ~11% BELOW TDM's 114.0, because
  batch-1 decode is bandwidth-bound, and concurrent kernels contend on
  memory bandwidth while SM caps only bound compute). Latency is
  smoother (max gap 50/53 ms vs TDM's 49/89) but p50 worsens for the
  small partition (16 ms vs 9). Critically **not work-conserving**: the
  caps are static — with the peer idle the 67% client does 110.9 tok/s
  vs TDM's full 120.9, and ratios track SM fraction only loosely.
- **Green contexts (CUDA 13.1 runtime "execution context" API):**
  SM partitioning **within a single process only** — inapplicable to
  cross-process gpu-share (relevant someday for multi-model-in-one-
  process serving). Isolation is best-effort, not guaranteed.
- **Windows:** no MPS, no cross-process partitioning at all — WDDM
  time-slicing is the only primitive, so wall-clock TDM remains the
  only portable mechanism there.

Conclusion: TDM stays the default (portable, zero-setup,
work-conserving, exact ratios, instant idle-release). An opt-in
Linux MPS mode (static SM partitioning managed by opencoti-server) is
a future milestone for latency-critical fleets on r590+ drivers.

## 5. Visibility

- `GET /gpu/peers` →
  `{ gpu_key, self: {pid, name, weight, share_pct, busy_pct, active},
     peers: [{pid, name, weight, share_pct, busy_pct, active,
              heartbeat_age_ms}] }`
- `/props` `opencoti` block gains `gpu_share` carrying the same
  `{enabled, gpu_key, self, peers}` snapshot as `/gpu/peers`, so existing
  dashboards see it without a new call.
- Boot log: `gpu-share: joined /opencoti-gpu-<key> slot 2 (weight 1.5, 3
  peers alive)`.

## 6. Phases

- **G1 — registry + flag + visibility (no throttle).** shm module
  (`llamafile/gpu_share.c`, cosmo-portable), `--gpu-share-weight`, join/
  heartbeat/reap, `/gpu/peers` + `/props` block. Gate: 3 instances on one
  GPU (solidPC 3090), each sees the other two with correct resolved
  percentages; kill -9 one → peers age it out ≤ 3 s.
- **G2 — duty-cycle pacing.** Busy-window measurement + inter-batch sleep.
  Gate: 2 instances, weights 2:1, concurrent decode workloads → measured
  tps ratio ≈ 2:1 (±20%); solo instance regains ~full solo tps ≤ 2 s after
  peer goes idle.
- **G3 — Windows/macOS validation** (pandorum APE run for the shm path on
  Windows; macOS opportunistic) + docs + opencoti-server fleet surfacing
  (daemon shows per-engine share/busy from `/gpu/peers`).

## 7. Non-goals

- Hard isolation (MPS/MIG) — not zero-conf, not portable.
- VRAM partitioning by weight — allocation is boot-time (`-c`, `-ngl`);
  the endpoint may later *report* VRAM per peer but never enforces it.
- Cross-GPU balancing — the registry is strictly per-GPU; multi-GPU
  processes join one segment per device they occupy (v1: primary device
  only).