code/offload/launch/windows/WIN2.md

WIN2 — fairer, deeper measurement and back-to-back windows

NDM 2026-10-27: "Fairer win rule is essential", "Do better deeper testing and measuring", "go window after window". Method source: Fable review (roadmap section "Independent review 2026-10-27 (Fable)", e8e3e3a).

Rule win2-v1, revision 3 (decision_v2.py)

Revision 2 (WIN2-R2, S92 FIX 3) and revision 3 (WIN2-R3 item 0, coordinator decision on the R2 report: the primary LOSS also needs its interval below zero) keep the name win2-v1 so runner code keeps working; every decision records rule_revision: 3.

Question	Answer
What decides	At most two decision cells per window; every other cell is reported only.
Primary metric	First series per cell. Prefer pooled server ms per round (server_decode_ms / spec_request_rounds) for decode cells and TTFT for prefill cells. Accepted/round and client tok/s are guards (they can only cause a LOSS).
Repeats	≥ 3 per arm in every decision cell (refused below).
WIN	point gain > 2 % and the 95 % interval excludes 0 (paired Student t; Welch t for unpaired arms; floored df).
Loss-repeat test	the candidate is worse than the loss threshold in ≥ ceil(2n/3) repeats (2 of 3). Control spread = 1.69 × the unbiased control SD (s / c4(n)) / mean, the expected range of three values whatever the number of control repeats (rev 1 used the range of all controls, which widened with more repeats: a true 5 % loss was caught 74 % of the time at 3 v 3 but 28 % at 8 v 4). Threshold = that spread unpaired, √2 × it paired (a paired difference has √2 × the SD of one value).
Primary LOSS	the loss-repeat test fires and the whole 95 % interval is below zero (revision 3; revision 2 needed the loss-repeat test only).
Guard LOSS	the guard's loss-repeat test fires and its whole 95 % interval is below zero.
Window	any decision-cell LOSS → NOT A CANDIDATE; ≥ 1 WIN and no LOSS → CANDIDATE; else INCONCLUSIVE. Optional groups give separate single-user / multi-user verdicts.
Prompt identity (DP-2)	every value of a decision series carries its prompt identity (Series.control_prompts / candidate_prompts, decision_v2.prompt_identity of the request's stream prompt_sha256; at B ≥ 2 the sha256 of the sorted member hashes). A decision cell refuses values without identities, values of two prompts in one arm, arms on different prompts, and a guard on another prompt than its primary. Values of different prompts are never pooled or paired.

Why prompt identity (DP-2, offload-x86host 98aa385, DP2.report.md): acceptance is dominated by the prompt. On the same day two prompts differ by up to 0.59 accepted per round (bench2 summary-32K 1.589 against corpus 32k-0 0.949 at 1,500 tokens), while each prompt holds steady over three days and across every configuration (legal-short 1.03 / 0.99 / 1.04 on 10-22 / 23 / 24). A comparison across prompts measures the prompts, so the rule refuses it.

When to pair. Series.paired defaults to True, but pair only true adjacent pairs: K/I/K/I alternation, one control and one candidate repeat per pair in run order (or a bracket where control repeat r is the mean of the two bracketing starts' repeat r). C/X/C with all control repeats pooled, and any layout with unequal repeats, is unpaired (paired=False, Welch). Unpaired, schedule ≥ 4 control repeats (C/X/C with 2 per C): at 3 v 3 the no-effect false loss per cell is 8 %.

No-effect rates (the candidate has no real effect; two decision cells, primary ms/round + both guards; S92's sim_corr.py model, seed 5, 10,000 windows: ms/round cv 1.5 %, acceptance cv 12 %, tok/s derived from both; scratch/WIN2R3/sims/sim_rev3.py, revision 2 in scratch/WIN2R2/sims/). "Rejected" = NOT A CANDIDATE, "accepted" = CANDIDATE:

design	rev 3 primary only: rejected / accepted	rev 3 with both guards	rev 2 with both guards	rev 1 with both guards
paired 3 v 3	1.7 % / 2.0 %	4.0 % / 1.9 %	11.4 % / 1.8 %	32.5 % / 1.4 %
unpaired 4 v 3 (C/X/C)	2.7 % / 2.1 %	6.2 % / 2.0 %	14.6 % / 1.9 %	16.0 % / 1.8 %

A no-effect candidate is now about as likely to be accepted as rejected on the primary alone; the guards add the rest of the rejections (they have their own two chances per window).

Power per decision cell (one cell, primary ms/round, cv 2 %, seed 9, 20,000 trials; LOSS % for a true loss, WIN % for a true gain; revision 2's LOSS % in brackets; WIN is unchanged by revision 3):

design	true loss 5 %	true loss 3 %	no effect: LOSS / WIN	true gain 3 %	true gain 5 %
unpaired 3 v 3	41.6 % [74.6]	16.8 % [45.3]	1.3 % [8.4] / 1.2 %	17.3 %	45.1 %
unpaired 4 v 3	53.3 % [76.7]	22.2 % [43.6]	1.4 % [5.8] / 1.4 %	26.4 %	62.1 %
unpaired 6 v 3	55.7 % [80.1]	22.6 % [42.7]	1.0 % [3.7] / 1.9 %	35.7 %	69.8 %
unpaired 8 v 3	55.3 % [81.5]	21.7 % [42.0]	0.7 % [2.8] / 2.1 %	37.8 %	70.0 %
unpaired 8 v 4	67.5 % [72.7]	22.6 % [27.9]	0.4 % [0.8] / 1.8 %	51.6 %	90.6 %
paired 3 v 3	29.3 % [55.4]	11.3 % [30.0]	0.8 % [4.8] / 1.8 %	19.5 %	42.6 %
paired 4 v 4	37.6 % [42.7]	12.2 % [16.8]	0.4 % [1.0] / 1.7 %	32.3 %	69.3 %
paired 6 v 6	44.4 % [44.7]	12.6 % [13.6]	0.2 % [0.3] / 1.3 %	55.1 %	93.8 %
paired 8 v 8	28.8 % [28.8]	4.6 % [4.6]	0.0 % [0.0] / 1.1 %	71.8 %	99.1 %

What the change costs: at three candidate repeats a real 5 % loss is now caught about half the time (unpaired 4–8 controls) instead of about three quarters; the remainder is INCONCLUSIVE, not CANDIDATE (a WIN needs the interval above zero, which a true loss essentially never gives: 0.0 % in every row). With ≥ 4 candidate repeats the interval condition rarely binds (8 v 4: 67.5 % vs 72.7 %). More controls stop helping past 6 because the Welch interval is then limited by the three candidate values; adding candidate repeats is what raises power. LOSS and WIN are now symmetric in their evidence: both need the 95 % interval to exclude zero, so a no-effect cell's chances split about evenly (≈ 1–2 % each). At paired 8 v 8 the 2-of-3 count needs six of eight differences beyond √2 × spread, which is why its LOSS power is lower than at 6 v 6.

Why ms/round: across the control archive (WIN2's 540 control rows from 10 windows), ms/round moves 0.7–5.0 % from day to day within one cell definition. Client tok/s moves up to 11 % and accepted/round up to 23 %, because acceptance per round depends on the sampled text (T = 1). Threshold2048's short cell is the example: ms/round was 3.4 % better while tok/s was 13.5 % worse, because that run's acceptance happened to be 28 % lower.

Seeded paired cells, rule win2-paired-v1 (WIN2-R3; F-METHOD §3.1)

Prompts are pinned (DP-2): window_cells.PROMPT_SHA256 registers every seeded prompt's sha256 (the corpus cuts hashed identically in all eight cuts.json files of 10-26..27, plus DP-2's two bench2 prompts). run_seeded_cell refuses any other text before its first request and records the prompt identity per pair and per cell; seeded_series refuses arms or pairs on different prompts (or without hashes) and decide_paired_window refuses a decision cell without the one proved identity. Report-only seeded cells (SeededCell(report_only=True), >= 3 pairs; the pilot's bridge) never decide.

Cells (window_cells.SEEDED_CELLS, run_seeded_cell): one prompt per member, ONE cold prefill per prompt under the cell's cache salt (proved cold: cached_tokens == 0), then n >= 6 seeded continuations of 256 tokens on the warm prefix (proved warm: cached_tokens > 0; ignore_eos, so every request is exactly 256 tokens). Seeds are registered per cell (seed_base + 10 x pair + member), identical in every arm. B1 cells scrape the server counters around EACH request (rounds, accepted drafts, decode seconds per request; the request-success counter must move by exactly one, so a foreign request refuses the cell); B >= 2 cells send groups of B concurrent requests and the group is the pair unit. Each pair records decode tok/s, ms/round, accepted/round, TTFT and a sha256 of every output (no text).

cell	members	profile	ESTIMATED time per arm (6 pairs)
seeded-short-b1	short	single	~38 s
seeded-32k-b1	32k-0	single	~56 s
seeded-32k-b2	32k-1, 32k-2	multi	~88 s
seeded-32k-b3	32k-1..3	multi	~120 s

Rule (decision_v2.decide_paired_cell / decide_paired_window; seeded_series builds the inputs from C1, X, C2):

• d_i = oriented relative difference of candidate pair i against control pair i (C/X/C: the mean of C1_i and C2_i).
• Paired t 95 % interval AND the exact two-sided sign-flip p (n = 6: the smallest p is 0.031, so all six pairs must agree; 3 pairs can never reach 0.05). n >= 6 is refused below.
• Primary WIN: point > 2 %, lower bound > 0, p < 0.05. LOSS (primary or guard): upper bound < 0, p < 0.05. Guards never win. The rule is symmetric apart from the 2 % a WIN must also clear.
• Built-in A/A (C1 vs C2): A/A drift (interval excludes 0 with p < 0.05) makes the cell INCONCLUSIVE; a target-side lever (expect_identical) also needs identical token streams and round counts in both control starts (window_cells.streams_identical), else INCONCLUSIVE.
• Start drift. Every pair shares ONE candidate start, so a start offset looks exactly like an effect. Register the between-start SD the seeded A/A pilot measures (start_sd={'<cell>/<metric>': sd}); it widens the interval by start_sd^2 x (1 + 1/k). Until the pilot has run, the rule assumes 0.
• Profiles: every cell is single (B1) or multi (B >= 2); the window reports one verdict per profile (at most two decision cells per profile); promotions state which profile they rely on.
• Primary metric: decode tok/s for single-stream decode cells (seed luck cancels in the pairs), ms/round reported; TTFT for prefill cells (no seeds needed). PP4's drafter has its own RNG: its pairs are only partly correlated, so report the measured diff CV, never assume the target-side figures.

Operating characteristics (per decision cell, simulated: seed luck 7 % shared by the arms, per-request residual 1 % or 2 %; scratch/WIN2R3/sims/sim_paired.txt, sim_paired_drift.txt; WIN / LOSS %):

design, residual, between-start SD	n	no effect	true +3 %	true +5 %	true -3 %
C/X/C, 1 %, 0	6	0.0 / 1.3	91.0 / 0	97.3 / 0	0 / 93.7
C/X/C, 1 %, 0	12	0.0 / 2.2	95.1 / 0	95.3 / 0	0 / 95.3
C/X/C, 2 %, 0	6	0.6 / 1.3	44.5 / 0	83.7 / 0	0 / 48.4
C/X/C, 2 %, 0	12	0.2 / 2.2	87.0 / 0	95.3 / 0	0 / 92.7
C/X/C, 1 %, 1 % (not registered)	6	4.5 / 10.0	53.6 / 0.1	70.8 / 0	0 / 61.1
C/X/C, 1 %, 1 % (registered start_sd 1 %)	6	0.4 / 0.4	27.9 / 0	63.1 / 0	—
C/X/C, 2 %, 1 % (registered start_sd 1 %)	12	1.0 / 1.0	32.6 / 0	63.7 / 0	—
C/X (one control start), 1 %, 1 %	6	8.2 / 13.5	69.8 / 0.2	95.6 / 0	0 / 77.2

Read: with no start drift the rule is fair (false WIN <= 0.6 %, false LOSS 1.3-2.2 %) and resolves 3 % at n = 12; the A/A check costs ~5 % power (it is itself a 5 % test). Unregistered start drift makes false calls common, and C/X (no A/A) is worse: keep C/X/C and register start_sd from the pilot.

Process rules as code (WIN2-R3; F-METHOD §3.2; adoption is NDM's decision 1)

1. Smoke start (smoke_start.py). Per candidate arm: start_exp.sh, /health + /v1/models, one greedy prompt, one 64-token seeded request whose speculative rounds must move, then the arm is ALWAYS stopped (a stop failure raises: HOLD stays, restore). ~8.5 min per arm (6.8 min MEASURED start), bound 15 min. chain_head(live, arms) smokes every candidate at a chain's head and drops failures before their windows; x_first(live, arm) makes the smoke the window's first start, and FAIL ends the window before any control start. The host side is the runner's Live backend (smoke_start.Backend); python -m offload.launch.windows.smoke_start <experiment> <arm> [--mode chain-head] prints the plan, and --execute --backend module:factory --out DIR runs it (coordinator only). Smoke windows never count toward the cap and never write the archive. No false-call rate: it is a pass/fail plumbing check (as are the cap and the filter below: neither judges a measurement).
2. Two-window cap (window_cap.py <LANE> [--prefix P]; launchers call it; exit 0 ALLOW, 2 REFUSE). It counts the lane's windows since adoption (<campaign>/WINDOW-CAP.json {"adopted_utc": ...}, or --since) whose summary status is COMPLETE, INCONCLUSIVE, NOT A CANDIDATE or CANDIDATE. Windows that stopped (INCOMPLETE, whatever their pre-filled decision says), admission windows and smoke starts do not count. From the third window on, it needs a new hypothesis dated after the latest counted window: experiments/<LANE>/hypothesis-<n>.md with a UTC timestamp in its first lines, a register line HYPOTHESIS <LANE> ... <UTC> in LANES.md, or NDM's WINDOW-CAP OVERRIDE <LANE> <UTC>. On the 2026-10-27 history (since 2026-10-26), PC1 and UP3 would already need a hypothesis; NBW, PP4 and DEP1 would not.
3. 2x resolution filter (resolution.py --cell --metric --design unpaired:AvB|pairs:N|seeded:N --expected-gain --max-tokens --clock --commit [--diff-cv]; exit 0 QUEUE, 2 REFUSE). The half-width comes from the archive per definition (cell, metric, max_tokens, clock cap, commit): the within-window CV of one repeat pooled over windows (df-weighted, all control repeats of a window). Seeded designs take the pilot's measured paired-difference CV (--diff-cv), else the ESTIMATE ms/round CV x sqrt(2). On today's archive (512 tokens, 2150 MHz, 596ba1c) the CVs are ms/round 2.55 % short-b1 and 1.07 % 32k-b1, decode tok/s 6.8 % and 6.7 % (F-METHOD's table used 2.1 % / 1.0 % / 7.1 % / 6.5 %). So the half-widths are a little wider than F-METHOD's: seeded 6 pairs ±3.8 % short-b1 and ±1.6 % 32k-b1. A +3 % lever on 32k-b1 needs 8 seeded pairs (±1.3 %), not 6.

Distributional gate for candidates (WIN2-R3; F-METHOD §3.4; dist_gate.py)

• Probe. 64 registered public-corpus prompts (32 short, 24 at 32K, 8 at 150K; dist_gate.DIST_PROMPTS), cut once (dist_gate cuts), then the top-20 next-token logprobs at 8 fixed positions: the prompt's last 8 tokens are the fixed continuation (/v1/completions, token-id prefixes, max_tokens 1, logprobs 20, one cache salt per run). Only token ids, prefix hashes and logprob vectors are written, on Controller, never git.

• Compare. Per position: KL(control || candidate) over the shared top-20 + OTHER (greedy_kl.top20_kl; positions with common mass < 0.99 are counted) and top-1 agreement.

• A/A band. Register it BEFORE any candidate use (dist_gate band, sha256-sealed) from production-vs-production comparisons. The seeded A/A pilot provides one; register more as pilots accumulate.

• Gate. dist_gate gate, exit 2 on LOSS. Candidate quality LOSS when:

  • its mean KL exceeds kl_ceiling = 1.5 × the 99th percentile of the A/A mean KL (seeded bootstrap over the A/A positions); or
  • its top-1 disagreements exceed the A/A's by an exact conditional binomial test at one-sided p < 0.01.
  • it has significantly more positions with top-20 common mass < 0.99 than the A/A (same exact test; the coarse KL is not trustworthy there);
  • Decision recorded (rule 27): F-METHOD's literal top-1 rule ("agreement < A/A minimum − 1") is reported (fmethod_top1_rule) but does not decide. Against one A/A run it rejects a no-effect candidate 13–24 % of the time once the A/A shows ≥ 1 expected near-tie flip.

• Greedy identity stays the W8 restore gate, unchanged.

• Simulated (512 positions, band from one A/A run, scratch/WIN2R3/sims/sim_dist.txt):

  expected A/A flips	no-effect false LOSS: this gate	no-effect false LOSS: F-METHOD literal	LOSS at 4× / 10× the flip rate
  0.25	0.1 %	2.1 %	0.2 % / 1.1 %
  1	0.0 %	13.7 %	4.9 % / 54.8 %
  3	0.0 %	24.2 %	38.8 % / —

  The mean-KL part alone gives 0.0–0.2 % false LOSS. The top-1 test's power grows with the number of registered A/A comparisons (q = 1/(k+1)).

• Cost (ESTIMATED): one cold prefill per prompt per arm. That is ~15.5 min per arm for all 64 prompts, or ~6.5 min for the short,32k subset that the pilot registers. F-METHOD's "+2 min" covers the short prompts only. The band and the gate must cover the same subset.

Seeded A/A pilot recipe (WIN2-R3; end-of-restore add-on, no extra HOLD; aa_pilot.py)

1. Start A: at the end of the window's last control start (production configuration, experiment port), after its own cells, run aa_pilot run --execute --url http://127.0.0.1:18001 ...: seeded short-b1, 32k-b1, 32k-b2 and 32k-b3, then the distributional probe on the short,32k subset.

2. Start B: the same on restored production, between W8 check (PASS) and W8 close, still inside the window's HOLD, so no legal traffic reaches it. W8's steps and its greedy-identity gate are unchanged; the pilot only reads. Start A (--start A) also runs DP-2's bridging cell (98aa385, DP2.report.md §5), report-only, so it needs no extra HOLD:

   cell	prompt (sha256)	tokens	DP-2 reference, accepted/round
   bridge-bench2-legal-1500	bench2 legal-short (dbc88a4f5274)	1,500	1.002 [0.986, 1.018], 10-22..24
   bridge-bench2-summary32k-1500	bench2 summary-32K (82a1d8a1d7e7)	1,500	1.589 [1.563, 1.615], 10-22..24
   bridge-short-512 / -1500	corpus short (932db59f8e4c)	512 / 1,500	1.294 (10-27) / 1.146 (10-26)
   bridge-32k0-512 / -1500	corpus 32k-0 (0ac74617c153)	512 / 1,500	1.048 (10-27) / 0.949 (10-26)

   Three seeded pairs each, production settings (T 1, thinking high, priority 10, ignore_eos), ESTIMATED 8.6 min of cells plus up to ~1 min rebuilding summary-32K. The bench2 prompts are rebuilt by bench2_texts with bench2's own algorithm (WORKLOADS read from offload/bench/bench2.py; build_real + make_prompt, filler real, seed 42, from /opt/dsv41/data/bench-data) and refused unless they hash to DP-2's values; an offline proof with Controller's local DeepSeek tokenizer reproduces summary-32K byte for byte (tests/test_aa_pilot.py). bridge_report gives each cell's accepted/round beside DP-2's reference, the same-day 512 − 1,500 difference per corpus prompt (length separated from date), and the H4 line: a summary-32K reading nearer 1.06 than 1.589 flags a production change between 10-24 and 10-26. It never decides anything.

3. Compare: aa_pilot compare A B (the bridge ran in A only and is not compared).

   • PASS = every B1 seeded cell has identical output digests and per-request round counts in both starts. B ≥ 2 cells are reported only.
   • The result also gives each cell and metric's measured paired-difference CV (resolution.py --diff-cv), a 1-df between-start SD estimate (accumulate over pilots before registering start_sd), and the distributional A/A comparison (dist_gate band).
   • FAIL means seeded pairing at B1 cannot assume identical streams: runners report measured diff CVs only, and no target-side lever uses expect_identical.

4. Cost (ESTIMATED): ~5 min of seeded cells plus ~6.5 min of probe per start, plus the bridge in start A, about 33 min in all.

Control archive

/opt/dsv41/campaigns/DSV41-OFFLOAD/control-archive.jsonl: append-only and sha-chained, schema dsv41-control-archive-v4 (WIN2-R3, DP-2: v3 + the prompt identity). Use archive_append to write, and archive_spread / archive_rows / archive_head to read. Every window appends every control repeat. Every field below is required (WIN2-R2, S92 FIX 2; DP-2); a row without one is refused, and a runner on older code refuses to append to the v4 file:

field	value
window, cell, metric, value	as before
max_tokens	positive integer; part of the cell definition
commit	the 40-hex production commit. Take the summary's production_commit, then production, then rollback.json (production_commit or commit): win2_history.production_commit(out). Never experiment_commit
clock_cap_mhz	the ONE cap of the row's own start (positive int): cap, source = win2_history.start_clock_cap(out, start). Proof is that start's log cmd/NNNN-<start>-start.log: start_exp.sh prints <host> clock cap N MHz only after nvidia-smi -lgc 0,N succeeded; both Sparks must print it and agree, else the start is refused. A start with no start log is classified by its checkpoint times against scratch/CLOCK.log (2150, except 2300 from 13:27:18 to 13:36:27Z and from 19:23:25Z on); a start that straddles a change is refused. Write clock_source ('start-log' or 'clock-log') beside it
clock_caps_mhz	optional window-level audit field: win2_history.clock_caps_mhz(out) (list of MHz, or 'unknown')
date	YYYYMMDD of the window (win2_history.window_date(out))
start, repeat	the control start's name and the repeat index inside it, so between-start variance can be watched
prompt_sha256	the repeat's prompt identity: decision_v2.prompt_identity([stream prompt_sha256, ...]) (B1: the stream's sha256; B ≥ 2: sha256 of the sorted member hashes). prompt_members (optional, checked) lists the stream hashes

max_tokens is part of the cell. NBW ran short-b1, 32k-b1, 32k-b2 and 32k-b3 at 1,500 tokens; every other runner used 512. At B3 that alone moves ms/round from about 72 to about 81 ms (r = −0.93 over 23 repeats): shorter answers spend more of their decode beside the other streams' 32K prefills. A runner whose cell differs in anything else (prompts, batch, late arrival) gives it a new name.

Spreads never pool definitions. archive_spread refuses a cell/metric whose rows mix max_tokens, production commit, clock cap or prompt identity unless where= picks one of each, e.g. where={'max_tokens': 512, 'commit': <40-hex>, 'clock_cap_mhz': 2300, 'prompt_sha256': <64-hex>}; archive_spreads(path, cell, metric) reports one spread per (max_tokens, clock cap, commit, prompt). The clock ceiling was 2150 MHz (dsv41.env default) for every start of every archived window through PC1-threshold512-W-20261027T181314Z (each start's log proves it) and is 2300 MHz from 2026-10-27 19:23:25Z (GPU_CLOCK_CAP=2300 in /etc/environment; PP4-941-admission at 19:23:39Z is the first 2300 window, and its start log says so). Busy clocks rose about 7 % (2125 → 2275 MHz), so a pooled spread would mostly measure the clock step. The PP4 promotion will change the commit.

Report archive_spread next to the in-window spread. Keep the 2 % required effect unless the archive shows larger real variation; if it does, say so in the report and never raise the bar silently. Record archive_head(path) (row count + head sha) in the window summary before and after appending: the chain detects an edited line, not a truncated tail. To backfill history, run python -m offload.launch.windows.win2_history backfill --archive <archive> <window dirs>. It is idempotent per (window, cell, metric) and reads a window named twice in one call once.

WIN2-R2 rebuilt the archive (596 rows: the 540 v1 rows with their production commit, per-start clock cap, start and repeat filled in, plus PC1-threshold512's 56). The v1 file is kept at scratch/WIN2R2/control-archive.v1-before-win2r2.jsonl; the proof (multiset equality) is in WIN2-R2.report.md.

v3 → v4 (DP-2): coordinator step, NOT yet applied. scratch/WIN2R3/rebuild_archive_v4.py rebuilds the 596 rows from the windows with their prompt identity (dry run: multiset equal on every v3 field, one identity per window and cell; scratch/WIN2R3/rebuild-v4-dry.txt). Apply it (--apply, in place under the archive's flock, v3 copy kept) only once every runner that appends has merged WIN2-R3: a runner on v3 code refuses to append to a v4 chain, and v4 code refuses a v3 chain.

New shared cells (gb10_budget/window_cells.py, opt-in by name, not in ORDER)

• ctl-mixed150k-4user: three 32K decoders; the 150K document arrives only after all three are decoding and must be admitted beside them (running gauge ≥ 4 while every decoder is alive; else the cell refuses). ~300 s.
• ctl-long-5k: one B1 request with a 5,000-token answer. ~180 s.

Back-to-back mode (chain.py)

• Coordinator sets WINDOW_CHAIN=1 for every window of a chain and WINDOW_CHAIN_LAST=1 on the last one.
• Hold placeholders stay up for the whole chain (coordinator rule, from R-PC1 / R-PP4; WIN2-R3 root fix of the defect R-NBW proved in scratch/RNBW/placeholder-proof.txt). Between chained windows there is no HOLD file, so the placeholders are what keep NVO drama3 and the local-law queue off the Sparks while production is down.
  • Two kinds. A window's own placeholder (made by hold_admission.open_script inside its -hold scope) lives on that scope's PRIVATE tmux socket (the scoped_remote tmux shim) and dies when the scope is fenced: it covers one window only. The chain placeholder is a tmux dsv41-hold session on the Spark's DEFAULT socket running vllm serve [DSV41-OFFLOAD hold] owner=<chain owner>; it outlives every window scope.
  • Owner. owner = chain.chain_owner() (the marker's while the window ran chained, else a new chain-<UTC>-<8 hex>); call it once per window and keep it. The owner is in the process title on each Spark and in the marker (placeholder_owner). Start, probe and stop refuse a placeholder of any other owner, including window.sh open's (no owner), and never touch it.
  • Where they are looked for. Every step uses /usr/bin/tmux -L default (chain.PLACEHOLDER_TMUX), never the PATH tmux, so it reaches the default socket from anywhere, including inside a scope.
  • Start chain.placeholders_up(owner): from the runner's own unscoped process over plain ssh (refused inside any dsv41w-*.scope, where a new default server would die with the fence), at a 'chained' end, BEFORE the first fence of the window's scopes. Raises unless both Sparks are UP for this owner. Idempotent: a chained window keeps the same processes.
  • Close chain.chained_close(owner) replaces R2's CHAINED_CLOSE (which looked at the scope's private socket and always failed): run inside the window's -restore-close scope; it proves this owner's placeholders on both Sparks through the scope's ssh shim, then removes only the HOLD (the rm shim records it and refuses another window's HOLD; commit-close removes it).
  • Probe chain.placeholders_present() (owner from the marker): the chained preflight and chain.gate() (so chain_prev_ok.sh) require both UP. A marker without an owner (written before this fix) stops the chain.
  • Stop chain.end_chain(owner=None): the chain's last step, ONLY after the strict W8 restore verified AND the post-close watch PASSED. It takes this owner's placeholders down (from an unscoped process; refused inside a scope), then clears the marker; a foreign or unreachable placeholder is left alone and reported (ENDED WITH ALERT). chain.clear() refuses while the marker records an owner. window.sh close (W8 close under a scope) cannot remove them: its kill reaches the scope's private socket.
  • systemd ≥ 258 expands ${...} in a systemd-run --scope command line (Controller runs 259), so the placeholder scripts and chained_close use no braced expansions.
• Runner start (S92 FIX 1). mode = chain.preflight(production_serving, hold_exists, placeholders_present=chain.placeholders_present()) (or chain.gate(), which probes this chain's placeholders only while the marker is present) replaces the "production serving K1" preflight. A chained preflight without the placeholder result refuses. In mode 'chained' only:
  • open the admission HOLD with hold_admission.open_script(..., allow_unreachable=chain.allow_unreachable(mode)). The default require-serving admission raises when :8000 does not answer, so a chained window would fail before its HOLD;
  • prove both Sparks quiet with spark_quiet.command(<marker>, deadline=...) on sparkhead and sparkworker-wired before start_exp.sh (the gate infers "production down" only from a failed /v1/models probe; it never proves quiet);
  • do not capture production (it is down): path, rollback = chain.production_snapshot() returns the chain's capture (the rollback.json of the window that started from serving production; its sha256 is checked against the marker), whatever runner wrote it. Copy it into the window's own rollback.json with chained_from / chained_from_sha256 and take the window's own first control start as the measurement baseline.
• Runner end. Decide with chain.end_action(stopped=<any stop>, hold_written=<own HOLD written>), never with hold_written alone:
  • 'chained': owner = chain.chain_owner(); chain.placeholders_up(owner) before the first fence; stop the experiment, prove the Sparks quiet, close the own HOLD with chain.chained_close(owner) inside the window's -restore-close scope (owner.command('launch', f'{tag}-restore-close', ['bash', '-c', chain.chained_close(owner)], ...), require chain.CHAINED_CLOSE_OK in its output, then the usual fenced commit-close; window.sh close would refuse without :8000), chain.finish_chained(out, quiet_proof, placeholder_owner=owner) (the marker records the owner and the chain's production snapshot: inherited from the previous marker, or this window's own rollback.json), write "chain": "CHAINED" and "restore": {"status": "CHAINED: production left down"} into the summary, and skip the W8 restore and the post-close watch;
  • 'restore': the normal strict W8 restore + post-close watch inside the own HOLD, then chain.end_chain() (only after RESTORED + watch PASS; chain.end_chain(owner) when this window started chain placeholders and failed before finish_chained). Until then the placeholders stay up;
  • 'abort' (no own HOLD, chain marker present: e.g. a pre-HOLD failure of a chained window; production is down from the previous window): summary['restore'] = chain.abort(out / 'chain-abort'). With no HOLD present it runs the strict W8 start|check|close under the reviewed post_close_watch.ScopedRecovery (its own HOLD, Sparks proved quiet, six distinct IDENTICAL greedy) plus the 15-minute post-close watch, and runs end_chain() (placeholders down, marker cleared) only on RESTORED + watch PASS. With a HOLD present it alerts and touches nothing; so it does while production is serving (S96 FIX 2a: the recovery would drain it, open a HOLD that pauses K24 and refuse at the guarded stop), and when an earlier automatic abort of the same stopped chain did not end RESTORED + watch PASS (S96 FIX 2b: ONE automatic recovery per stopped chain, the post-close watch rule; record <marker>.abort, removed by the chain's end). Never write "NOT NEEDED: production untouched" in this case;
  • 'none': no HOLD and no marker; production was not touched. chain.restore_required(stopped, hold_written) is end_action(...) in ('restore', 'abort').
• Coordinator gate: offload/launch/windows/chain_prev_ok.sh replaces campaign launch/prev-ok.sh for chained windows. It is coordinator-only (S96 FIX 1): without WINDOW_COORDINATOR=Deepseek/coordinator it exits 2 at once, before any wait or probe, because its chain abort needs it; every RUN template exports WINDOW_COORDINATOR before the gate (campaign launch/PC1-W2-RUN.sh and R-UPF/R-UP3 offload/upfix/launch/UPF-RUN.sh export it after the gate today and must move the export up). Exit 0 admits; exit 2 is a refusal with no chain marker (as prev-ok.sh); exit 3 is CHAIN STOPPED: PRODUCTION DOWN: any refusal (previous summary, gate, HOLD, stale container) while the marker is present. The gate then runs chain_abort.sh itself before exiting, so a stopped chain never leaves production down unattended. RUN scripts call the gate under set -e, so exit 3 also ends that RUN script; the abort's summary.json (in $C/CHAIN-ABORT-<UTC>/) is a valid PREV_FILE target for the next gate.
• Coordinator abort by hand: offload/launch/windows/chain_abort.sh [OUT] (= python -m offload.launch.windows.chain abort --out OUT, needs WINDOW_COORDINATOR). Exit 0 restored + watch PASS (placeholders down, marker cleared) or no marker; 3 failed (marker and placeholders kept, production down); 4 alert only (a HOLD exists, production is serving, or this chain's one automatic abort already ran). abort --again (by hand only) forgets the previous attempt and runs once more. By hand: python -m offload.launch.windows.chain placeholders [--owner X] (probe both Sparks) and, after a verified restore + watch PASS, WINDOW_COORDINATOR=... python -m offload.launch.windows.chain end [--owner X].
• The legal pipeline is down for the whole chain (NDM accepts this while building). The last window, or any stop, brings production back with the strict six-IDENTICAL W8.
• Window re-scoring (python -m offload.launch.windows.win2_history reanalyze --decision C1 [--decision C2] <window>) uses decide_window's cell rule, refuses more than two decision cells, treats roles control, K, off (TV1) and H (PF2) as controls, and judges every other role as its own arm (a two-arm window reports arms, never a pooled candidate).
• Test-only overrides (never set in production): DSV41_CHAIN_MARKER, DSV41_CHAIN_PRODUCTION_URL (chain.py), DSV41_CHAIN_CAMPAIGN, DSV41_CHAIN_PY, DSV41_CHAIN_ABORT (the two scripts). chain status prints the marker path and production URL in use.

Greedy refinement (greedy_kl.py, opt-in)

Candidate arm: nbw_cells.greedy(..., near_tie_nats=<registered>, record_distribution=True, probe_salt_prefix=<window tag>) keeps the top-20 and prefix at every divergence. On the command line: python -m offload.launch.windows.nbw_cells --execute --mode greedy ... --near-tie-nats <registered> [--record-distribution --probe-salt-prefix <window tag>] (defaults unchanged: 0.501 nat, no recording). Following control arm: greedy_kl.compare_recorded(api, candidate_rows, salt_prefix, bound=<registered>) re-probes those prefixes with a fresh cache salt and measures KL(control || candidate) over shared top-20 + OTHER (UP2's estimator); common mass must be ≥ 0.99. The bound is each runner's registration (UP2 used max(0.005, 2 × A/A floor)).

Adoption checklist for each runner fork

1. git merge offload/win2 (needs the "WIN2 API" commit ce1bcf2 or later).
2. Schedule ≥ 3 repeats per arm in the decision cells (C/X/C with ≥ 2 repeats per start gives ≥ 4 control and needs ≥ 3 candidate repeats: run the candidate start with 3 repeats or use C/X/C/X/C).
3. Build decision_v2.Series per decision cell, primary metric first; call decide_window(series, decision_cells, groups=...); write its dict as the summary decision (rule win2-v1).
4. archive_append every control repeat at the end of the window, including after a later stop for the starts that finished. Each row carries every field of the archive table above (commit = production, clock_cap_mhz of its own start from start_clock_cap, YYYYMMDD date, start, repeat, prompt_sha256); record archive_head before and after. Every decision Series carries control_prompts / candidate_prompts (one identity per value, from the streams' prompt_sha256): decision cells refuse without them (DP-2).
5. Chain mode as above; summary chain field. Tests (fake transport, tests/test_chain_stop.py shows the shared parts): a chained non-last end (marker, no restore); a last or stopped window (strict restore, chain.end_chain()); a chained start with production down (admission with allow_unreachable, quiet proof before start_exp.sh); a pre-HOLD failure in the chained state, which must restore (end_action == 'abort' -> chain.abort); and a gate refusal in the chained state, which exits 3 and takes the abort path; placeholders (WIN2-R3): a chained end calls placeholders_up(owner) before its first fence and chained_close(owner) in its -restore-close scope, finish_chained(..., placeholder_owner=owner); a chained start refuses without this chain's placeholders; a final restore ends with end_chain(); a chained start that follows another runner's window takes its production capture from chain.production_snapshot(). The real-scope proof is tests/test_chain_placeholder_rehearsal.py (real user scopes, scoped_remote shims, tmux, fake per-host HOME).
6. WIN2-R3 (seeded cells; adopt after its independent check). Swap the decode decision cells for seeded cells (run_seeded_cell on every arm, C/X/C; seeded_series + decide_paired_window), keep the existing gates, tag every cell's profile and state in the promotion which profile it relies on; call window_cap from the launcher and resolution on the pre-registration (expected_gain per decision cell); add the smoke start (x_first in a single window, chain_head at a chain's head); register start_sd / --diff-cv once the seeded A/A pilot has measured them.
7. Greedy: keep the runner's registered near-tie limit (0.9 general, 1.1 PP4 and DEP1 fixed2 per NDM) and, where the window has a following control start, the opt-in KL check.

Rehearsal before the first Spark window (design note; not built)

The real target runs only on the two Sparks (RUNBOOK-offload.md lists no other configuration of it), so no rehearsal off the Sparks can measure speed, acceptance or numerics. A rehearsal exists to catch runner faults (sequencing, parsing, timeouts, refusals, chain and archive writes) before they cost a window. Three levels:

1. Required for every new or changed runner: fake transport on Controller CPU. Real runner code over inert recording remote commands and an in-process HTTP fake (tests/test_nbw_rehearsal.py, tests/test_pf2_rehearsal.py, UP2-REHEARSAL.md). Add the WIN2 cases of checklist step 5: a chained non-last end (marker, no restore), a stop inside a chained window (restore, chain.end_chain()), a chained start with production down, a pre-HOLD failure in the chained state (chain.abort), and a gate refusal in the chained state (exit 3).
2. Proposed: a stub "target-only" HTTP server (CPU only; Controller or x86host CPU, loopback port, no GPU). One small threaded process that answers the endpoints the runners touch with vLLM's formats: /v1/models (one model, max_model_len), /tokenize (count from the text length), /v1/chat/completions (SSE with reasoning/content deltas, final usage, [DONE]), /v1/completions (top-20 logprobs, token ids), /metrics (the vllm:num_requests_running and vllm:num_requests_waiting gauges and the speculative counters bench2 parse_metrics reads), and /start_profile + /stop_profile. It needs a configurable admission limit and decode delay, so that ctl-mixed150k-4user's late-arrival check passes at limit 4 and refuses at limit 3. The runner points at it via its target URL with the remote side still faked. This proves real sockets, timeouts, SSE parsing, concurrency and admission. It proves nothing about performance. Not built here: matching vLLM's SSE and metrics formats exactly is a lane of its own, and level 1 already gates every runner.
3. Drafter/worker code changes only: an owned x86host window (as the PP4 server probe did). The coordinator books it, because every x86host 3090 is assigned to DSV41 work. It checks that the worker loads and answers; Spark rounds are still needed for any number. Config-only Spark arms skip this level.

Then, in the first chained window, the coordinator runs the runner's --plan / dry-run against the live read-only probes before the HOLD opens.