File size: 40,662 Bytes
1e251f3 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 | # WIN2 — fairer, deeper measurement and back-to-back windows
NDM 2026-10-27: "Fairer win rule is essential", "Do better deeper testing and measuring", "go window after window".
Method source: Fable review (roadmap section "Independent review 2026-10-27 (Fable)", e8e3e3a).
## Rule `win2-v1`, revision 3 (`decision_v2.py`)
Revision 2 (WIN2-R2, S92 FIX 3) and revision 3 (WIN2-R3 item 0, coordinator decision on the R2 report: the primary LOSS
also needs its interval below zero) keep the name `win2-v1` so runner code keeps working; every decision records
`rule_revision: 3`.
| Question | Answer |
|---|---|
| What decides | At most **two decision cells** per window; every other cell is reported only. |
| Primary metric | First series per cell. Prefer **pooled server ms per round** (`server_decode_ms / spec_request_rounds`) for decode cells and **TTFT** for prefill cells. Accepted/round and client tok/s are guards (they can only cause a LOSS). |
| Repeats | ≥ 3 per arm in every decision cell (refused below). |
| WIN | point gain > 2 % **and** the 95 % interval excludes 0 (paired Student t; Welch t for unpaired arms; floored df). |
| Loss-repeat test | the candidate is worse than the loss threshold in ≥ ceil(2n/3) repeats (2 of 3). Control spread = **1.69 × the unbiased control SD (s / c4(n)) / mean**, the expected range of three values whatever the number of control repeats (rev 1 used the range of all controls, which widened with more repeats: a true 5 % loss was caught 74 % of the time at 3 v 3 but 28 % at 8 v 4). Threshold = that spread unpaired, **√2 × it paired** (a paired difference has √2 × the SD of one value). |
| Primary LOSS | the loss-repeat test fires **and** the whole 95 % interval is below zero (revision 3; revision 2 needed the loss-repeat test only). |
| Guard LOSS | the guard's loss-repeat test fires **and** its whole 95 % interval is below zero. |
| Window | any decision-cell LOSS → NOT A CANDIDATE; ≥ 1 WIN and no LOSS → CANDIDATE; else INCONCLUSIVE. Optional `groups` give separate single-user / multi-user verdicts. |
| Prompt identity (DP-2) | every value of a decision series carries its prompt identity (`Series.control_prompts` / `candidate_prompts`, `decision_v2.prompt_identity` of the request's stream `prompt_sha256`; at B ≥ 2 the sha256 of the sorted member hashes). A decision cell **refuses** values without identities, values of two prompts in one arm, arms on different prompts, and a guard on another prompt than its primary. Values of different prompts are never pooled or paired. |
**Why prompt identity** (DP-2, offload-x86host 98aa385, `DP2.report.md`): acceptance is dominated by the prompt. On the
same day two prompts differ by up to 0.59 accepted per round (bench2 summary-32K 1.589 against corpus 32k-0 0.949 at
1,500 tokens), while each prompt holds steady over three days and across every configuration (legal-short 1.03 / 0.99 /
1.04 on 10-22 / 23 / 24). A comparison across prompts measures the prompts, so the rule refuses it.
**When to pair.** `Series.paired` defaults to True, but pair only true adjacent pairs: K/I/K/I alternation, one control
and one candidate repeat per pair in run order (or a bracket where control repeat r is the mean of the two bracketing
starts' repeat r). C/X/C with all control repeats pooled, and any layout with unequal repeats, is unpaired
(`paired=False`, Welch). Unpaired, schedule ≥ 4 control repeats (C/X/C with 2 per C): at 3 v 3 the no-effect false
loss per cell is 8 %.
**No-effect rates** (the candidate has no real effect; two decision cells, primary ms/round + both guards; S92's
`sim_corr.py` model, seed 5, 10,000 windows: ms/round cv 1.5 %, acceptance cv 12 %, tok/s derived from both;
`scratch/WIN2R3/sims/sim_rev3.py`, revision 2 in `scratch/WIN2R2/sims/`). "Rejected" = NOT A CANDIDATE, "accepted" =
CANDIDATE:
| design | rev 3 primary only: rejected / accepted | **rev 3 with both guards** | rev 2 with both guards | rev 1 with both guards |
|---|---|---|---|---|
| paired 3 v 3 | 1.7 % / 2.0 % | **4.0 % / 1.9 %** | 11.4 % / 1.8 % | 32.5 % / 1.4 % |
| unpaired 4 v 3 (C/X/C) | 2.7 % / 2.1 % | **6.2 % / 2.0 %** | 14.6 % / 1.9 % | 16.0 % / 1.8 % |
A no-effect candidate is now about as likely to be accepted as rejected on the primary alone; the guards add the rest of
the rejections (they have their own two chances per window).
**Power per decision cell** (one cell, primary ms/round, cv 2 %, seed 9, 20,000 trials; LOSS % for a true loss, WIN %
for a true gain; revision 2's LOSS % in brackets; WIN is unchanged by revision 3):
| design | true loss 5 % | true loss 3 % | no effect: LOSS / WIN | true gain 3 % | true gain 5 % |
|---|---|---|---|---|---|
| unpaired 3 v 3 | 41.6 % [74.6] | 16.8 % [45.3] | 1.3 % [8.4] / 1.2 % | 17.3 % | 45.1 % |
| unpaired 4 v 3 | 53.3 % [76.7] | 22.2 % [43.6] | 1.4 % [5.8] / 1.4 % | 26.4 % | 62.1 % |
| unpaired 6 v 3 | 55.7 % [80.1] | 22.6 % [42.7] | 1.0 % [3.7] / 1.9 % | 35.7 % | 69.8 % |
| unpaired 8 v 3 | 55.3 % [81.5] | 21.7 % [42.0] | 0.7 % [2.8] / 2.1 % | 37.8 % | 70.0 % |
| unpaired 8 v 4 | 67.5 % [72.7] | 22.6 % [27.9] | 0.4 % [0.8] / 1.8 % | 51.6 % | 90.6 % |
| paired 3 v 3 | 29.3 % [55.4] | 11.3 % [30.0] | 0.8 % [4.8] / 1.8 % | 19.5 % | 42.6 % |
| paired 4 v 4 | 37.6 % [42.7] | 12.2 % [16.8] | 0.4 % [1.0] / 1.7 % | 32.3 % | 69.3 % |
| paired 6 v 6 | 44.4 % [44.7] | 12.6 % [13.6] | 0.2 % [0.3] / 1.3 % | 55.1 % | 93.8 % |
| paired 8 v 8 | 28.8 % [28.8] | 4.6 % [4.6] | 0.0 % [0.0] / 1.1 % | 71.8 % | 99.1 % |
What the change costs: at three candidate repeats a real 5 % loss is now caught about half the time (unpaired 4–8
controls) instead of about three quarters; the remainder is INCONCLUSIVE, not CANDIDATE (a WIN needs the interval above
zero, which a true loss essentially never gives: 0.0 % in every row). With ≥ 4 candidate repeats the interval condition
rarely binds (8 v 4: 67.5 % vs 72.7 %). More controls stop helping past 6 because the Welch interval is then limited by
the three candidate values; adding candidate repeats is what raises power. LOSS and WIN are now symmetric in their
evidence: both need the 95 % interval to exclude zero, so a no-effect cell's chances split about evenly (≈ 1–2 % each).
At paired 8 v 8 the 2-of-3 count needs six of eight differences beyond √2 × spread, which is why its LOSS power is
lower than at 6 v 6.
Why ms/round: across the control archive (WIN2's 540 control rows from 10 windows), ms/round moves **0.7–5.0 %** from day to
day within one cell definition. Client tok/s moves up to 11 % and accepted/round up to 23 %, because acceptance per
round depends on the sampled text (T = 1). Threshold2048's short cell is the example: ms/round was 3.4 % better while
tok/s was 13.5 % worse, because that run's acceptance happened to be 28 % lower.
## Seeded paired cells, rule `win2-paired-v1` (WIN2-R3; F-METHOD §3.1)
**Prompts are pinned** (DP-2): `window_cells.PROMPT_SHA256` registers every seeded prompt's sha256 (the corpus cuts
hashed identically in all eight cuts.json files of 10-26..27, plus DP-2's two bench2 prompts). `run_seeded_cell` refuses
any other text before its first request and records the prompt identity per pair and per cell; `seeded_series` refuses
arms or pairs on different prompts (or without hashes) and `decide_paired_window` refuses a decision cell without the one
proved identity. Report-only seeded cells (`SeededCell(report_only=True)`, >= 3 pairs; the pilot's bridge) never decide.
**Cells** (`window_cells.SEEDED_CELLS`, `run_seeded_cell`): one prompt per member, ONE cold prefill per prompt under the
cell's cache salt (proved cold: `cached_tokens == 0`), then n >= 6 seeded continuations of 256 tokens on the warm prefix
(proved warm: `cached_tokens > 0`; `ignore_eos`, so every request is exactly 256 tokens). Seeds are registered per cell
(`seed_base + 10 x pair + member`), identical in every arm. B1 cells scrape the server counters around EACH request
(rounds, accepted drafts, decode seconds per request; the request-success counter must move by exactly one, so a foreign
request refuses the cell); B >= 2 cells send groups of B concurrent requests and the group is the pair unit. Each pair
records decode tok/s, ms/round, accepted/round, TTFT and a sha256 of every output (no text).
| cell | members | profile | ESTIMATED time per arm (6 pairs) |
|---|---|---|---|
| `seeded-short-b1` | short | single | ~38 s |
| `seeded-32k-b1` | 32k-0 | single | ~56 s |
| `seeded-32k-b2` | 32k-1, 32k-2 | multi | ~88 s |
| `seeded-32k-b3` | 32k-1..3 | multi | ~120 s |
**Rule** (`decision_v2.decide_paired_cell` / `decide_paired_window`; `seeded_series` builds the inputs from C1, X, C2):
- d_i = oriented relative difference of candidate pair i against control pair i (C/X/C: the mean of C1_i and C2_i).
- Paired t 95 % interval AND the exact two-sided sign-flip p (n = 6: the smallest p is 0.031, so all six pairs must
agree; 3 pairs can never reach 0.05). n >= 6 is refused below.
- Primary WIN: point > 2 %, lower bound > 0, p < 0.05. LOSS (primary or guard): upper bound < 0, p < 0.05. Guards never
win. The rule is symmetric apart from the 2 % a WIN must also clear.
- Built-in A/A (C1 vs C2): A/A drift (interval excludes 0 with p < 0.05) makes the cell INCONCLUSIVE; a target-side lever
(`expect_identical`) also needs identical token streams and round counts in both control starts
(`window_cells.streams_identical`), else INCONCLUSIVE.
- **Start drift.** Every pair shares ONE candidate start, so a start offset looks exactly like an effect. Register the
between-start SD the seeded A/A pilot measures (`start_sd={'<cell>/<metric>': sd}`); it widens the interval by
start_sd^2 x (1 + 1/k). Until the pilot has run, the rule assumes 0.
- Profiles: every cell is `single` (B1) or `multi` (B >= 2); the window reports one verdict per profile (at most two
decision cells per profile); promotions state which profile they rely on.
- Primary metric: decode tok/s for single-stream decode cells (seed luck cancels in the pairs), ms/round reported;
TTFT for prefill cells (no seeds needed). PP4's drafter has its own RNG: its pairs are only partly correlated, so
report the measured diff CV, never assume the target-side figures.
**Operating characteristics** (per decision cell, simulated: seed luck 7 % shared by the arms, per-request residual
1 % or 2 %; `scratch/WIN2R3/sims/sim_paired.txt`, `sim_paired_drift.txt`; WIN / LOSS %):
| design, residual, between-start SD | n | no effect | true +3 % | true +5 % | true -3 % |
|---|---|---|---|---|---|
| C/X/C, 1 %, 0 | 6 | 0.0 / 1.3 | 91.0 / 0 | 97.3 / 0 | 0 / 93.7 |
| C/X/C, 1 %, 0 | 12 | 0.0 / 2.2 | 95.1 / 0 | 95.3 / 0 | 0 / 95.3 |
| C/X/C, 2 %, 0 | 6 | 0.6 / 1.3 | 44.5 / 0 | 83.7 / 0 | 0 / 48.4 |
| C/X/C, 2 %, 0 | 12 | 0.2 / 2.2 | 87.0 / 0 | 95.3 / 0 | 0 / 92.7 |
| C/X/C, 1 %, 1 % (not registered) | 6 | **4.5 / 10.0** | 53.6 / 0.1 | 70.8 / 0 | 0 / 61.1 |
| C/X/C, 1 %, 1 % (registered start_sd 1 %) | 6 | 0.4 / 0.4 | 27.9 / 0 | 63.1 / 0 | — |
| C/X/C, 2 %, 1 % (registered start_sd 1 %) | 12 | 1.0 / 1.0 | 32.6 / 0 | 63.7 / 0 | — |
| C/X (one control start), 1 %, 1 % | 6 | **8.2 / 13.5** | 69.8 / 0.2 | 95.6 / 0 | 0 / 77.2 |
Read: with no start drift the rule is fair (false WIN <= 0.6 %, false LOSS 1.3-2.2 %) and resolves 3 % at n = 12;
the A/A check costs ~5 % power (it is itself a 5 % test). Unregistered start drift makes false calls common, and C/X
(no A/A) is worse: keep C/X/C and register start_sd from the pilot.
## Process rules as code (WIN2-R3; F-METHOD §3.2; adoption is NDM's decision 1)
1. **Smoke start** (`smoke_start.py`). Per candidate arm: start_exp.sh, /health + /v1/models, one greedy prompt, one
64-token seeded request whose speculative rounds must move, then the arm is ALWAYS stopped (a stop failure raises:
HOLD stays, restore). ~8.5 min per arm (6.8 min MEASURED start), bound 15 min. `chain_head(live, arms)` smokes every
candidate at a chain's head and drops failures before their windows; `x_first(live, arm)` makes the smoke the
window's first start, and FAIL ends the window before any control start. The host side is the runner's Live
backend (`smoke_start.Backend`); `python -m offload.launch.windows.smoke_start <experiment> <arm> [--mode chain-head]`
prints the plan, and `--execute --backend module:factory --out DIR` runs it (coordinator only). Smoke windows never
count toward the cap and never write the archive. No false-call rate: it is a pass/fail plumbing check (as are the
cap and the filter below: neither judges a measurement).
2. **Two-window cap** (`window_cap.py <LANE> [--prefix P]`; launchers call it; exit 0 ALLOW, 2 REFUSE). It counts the
lane's windows since adoption (`<campaign>/WINDOW-CAP.json` {"adopted_utc": ...}, or `--since`) whose summary
`status` is COMPLETE, INCONCLUSIVE, NOT A CANDIDATE or CANDIDATE. Windows that stopped (INCOMPLETE, whatever their
pre-filled decision says), admission windows and smoke starts do not count. From the third window on, it needs a new
hypothesis dated after the latest counted window: `experiments/<LANE>/hypothesis-<n>.md` with a UTC timestamp in its
first lines, a register line `HYPOTHESIS <LANE> ... <UTC>` in LANES.md, or NDM's `WINDOW-CAP OVERRIDE <LANE> <UTC>`.
On the 2026-10-27 history (since 2026-10-26), PC1 and UP3 would already need a hypothesis; NBW, PP4 and DEP1 would not.
3. **2x resolution filter** (`resolution.py --cell --metric --design unpaired:AvB|pairs:N|seeded:N --expected-gain
--max-tokens --clock --commit [--diff-cv]`; exit 0 QUEUE, 2 REFUSE). The half-width comes from the archive per
definition (cell, metric, max_tokens, clock cap, commit): the within-window CV of one repeat pooled over windows
(df-weighted, all control repeats of a window). Seeded designs take the pilot's measured paired-difference CV
(`--diff-cv`), else the ESTIMATE ms/round CV x sqrt(2). On today's archive (512 tokens, 2150 MHz, 596ba1c) the CVs are
ms/round 2.55 % short-b1 and 1.07 % 32k-b1, decode tok/s 6.8 % and 6.7 % (F-METHOD's table used 2.1 % / 1.0 % /
7.1 % / 6.5 %). So the half-widths are a little wider than F-METHOD's: seeded 6 pairs ±3.8 % short-b1 and ±1.6 %
32k-b1. A +3 % lever on 32k-b1 needs 8 seeded pairs (±1.3 %), not 6.
## Distributional gate for candidates (WIN2-R3; F-METHOD §3.4; `dist_gate.py`)
- **Probe.** 64 registered public-corpus prompts (32 short, 24 at 32K, 8 at 150K; `dist_gate.DIST_PROMPTS`), cut once
(`dist_gate cuts`), then the top-20 next-token logprobs at 8 fixed positions: the prompt's last 8 tokens are the fixed
continuation (`/v1/completions`, token-id prefixes, max_tokens 1, logprobs 20, one cache salt per run). Only token ids,
prefix hashes and logprob vectors are written, on Controller, never git.
- **Compare.** Per position: KL(control || candidate) over the shared top-20 + OTHER (`greedy_kl.top20_kl`; positions
with common mass < 0.99 are counted) and top-1 agreement.
- **A/A band.** Register it BEFORE any candidate use (`dist_gate band`, sha256-sealed) from production-vs-production
comparisons. The seeded A/A pilot provides one; register more as pilots accumulate.
- **Gate.** `dist_gate gate`, exit 2 on LOSS. Candidate quality LOSS when:
- its mean KL exceeds `kl_ceiling` = 1.5 × the 99th percentile of the A/A mean KL (seeded bootstrap over the A/A
positions); or
- its top-1 disagreements exceed the A/A's by an exact conditional binomial test at one-sided p < 0.01.
- it has significantly more positions with top-20 common mass < 0.99 than the A/A (same exact test; the coarse KL is
not trustworthy there);
- **Decision recorded (rule 27):** F-METHOD's literal top-1 rule ("agreement < A/A minimum − 1") is reported
(`fmethod_top1_rule`) but does not decide. Against one A/A run it rejects a no-effect candidate 13–24 % of the time
once the A/A shows ≥ 1 expected near-tie flip.
- **Greedy identity** stays the W8 restore gate, unchanged.
- **Simulated** (512 positions, band from one A/A run, `scratch/WIN2R3/sims/sim_dist.txt`):
| expected A/A flips | no-effect false LOSS: this gate | no-effect false LOSS: F-METHOD literal | LOSS at 4× / 10× the flip rate |
|---|---|---|---|
| 0.25 | 0.1 % | 2.1 % | 0.2 % / 1.1 % |
| 1 | 0.0 % | 13.7 % | 4.9 % / 54.8 % |
| 3 | 0.0 % | 24.2 % | 38.8 % / — |
The mean-KL part alone gives 0.0–0.2 % false LOSS. The top-1 test's power grows with the number of registered A/A
comparisons (q = 1/(k+1)).
- **Cost** (ESTIMATED): one cold prefill per prompt per arm. That is ~15.5 min per arm for all 64 prompts, or ~6.5 min for
the `short,32k` subset that the pilot registers. F-METHOD's "+2 min" covers the short prompts only. The band and the
gate must cover the same subset.
## Seeded A/A pilot recipe (WIN2-R3; end-of-restore add-on, no extra HOLD; `aa_pilot.py`)
1. **Start A:** at the end of the window's last control start (production configuration, experiment port), after its
own cells, run `aa_pilot run --execute --url http://127.0.0.1:18001 ...`: seeded short-b1, 32k-b1, 32k-b2 and 32k-b3,
then the distributional probe on the `short,32k` subset.
2. **Start B:** the same on restored production, between W8 `check` (PASS) and W8 `close`, still inside the window's
HOLD, so no legal traffic reaches it. W8's steps and its greedy-identity gate are unchanged; the pilot only reads.
Start A (`--start A`) also runs **DP-2's bridging cell** (98aa385, DP2.report.md §5), report-only, so it needs no
extra HOLD:
| cell | prompt (sha256) | tokens | DP-2 reference, accepted/round |
|---|---|---|---|
| `bridge-bench2-legal-1500` | bench2 legal-short (dbc88a4f5274) | 1,500 | 1.002 [0.986, 1.018], 10-22..24 |
| `bridge-bench2-summary32k-1500` | bench2 summary-32K (82a1d8a1d7e7) | 1,500 | 1.589 [1.563, 1.615], 10-22..24 |
| `bridge-short-512` / `-1500` | corpus short (932db59f8e4c) | 512 / 1,500 | 1.294 (10-27) / 1.146 (10-26) |
| `bridge-32k0-512` / `-1500` | corpus 32k-0 (0ac74617c153) | 512 / 1,500 | 1.048 (10-27) / 0.949 (10-26) |
Three seeded pairs each, production settings (T 1, thinking high, priority 10, `ignore_eos`), ESTIMATED 8.6 min of
cells plus up to ~1 min rebuilding summary-32K. The bench2 prompts are rebuilt by `bench2_texts` with bench2's own
algorithm (WORKLOADS read from `offload/bench/bench2.py`; `build_real` + `make_prompt`, filler real, seed 42, from
`/opt/dsv41/data/bench-data`) and refused unless they hash to DP-2's values; an offline proof with Controller's
local DeepSeek tokenizer reproduces summary-32K byte for byte (`tests/test_aa_pilot.py`). `bridge_report` gives each
cell's accepted/round beside DP-2's reference, the same-day 512 − 1,500 difference per corpus prompt (length separated
from date), and the H4 line: a summary-32K reading nearer 1.06 than 1.589 flags a production change between 10-24 and
10-26. It never decides anything.
3. **Compare:** `aa_pilot compare A B` (the bridge ran in A only and is not compared).
- **PASS** = every B1 seeded cell has identical output digests and per-request round counts in both starts. B ≥ 2
cells are reported only.
- The result also gives each cell and metric's measured paired-difference CV (`resolution.py --diff-cv`), a 1-df
between-start SD estimate (accumulate over pilots before registering `start_sd`), and the distributional A/A
comparison (`dist_gate band`).
- **FAIL** means seeded pairing at B1 cannot assume identical streams: runners report measured diff CVs only, and no
target-side lever uses `expect_identical`.
4. **Cost** (ESTIMATED): ~5 min of seeded cells plus ~6.5 min of probe per start, plus the bridge in start A, about
33 min in all.
## Control archive
`/opt/dsv41/campaigns/DSV41-OFFLOAD/control-archive.jsonl`: append-only and sha-chained, schema
`dsv41-control-archive-v4` (WIN2-R3, DP-2: v3 + the prompt identity). Use `archive_append` to write, and
`archive_spread` / `archive_rows` / `archive_head` to read. Every window appends every control repeat. **Every field
below is required** (WIN2-R2, S92 FIX 2; DP-2); a row without one is refused, and a runner on older code refuses to
append to the v4 file:
| field | value |
|---|---|
| `window`, `cell`, `metric`, `value` | as before |
| `max_tokens` | positive integer; part of the cell definition |
| `commit` | the 40-hex **production** commit. Take the summary's `production_commit`, then `production`, then `rollback.json` (`production_commit` or `commit`): `win2_history.production_commit(out)`. **Never** `experiment_commit` |
| `clock_cap_mhz` | the ONE cap of the row's **own start** (positive int): `cap, source = win2_history.start_clock_cap(out, start)`. Proof is that start's log `cmd/NNNN-<start>-start.log`: start_exp.sh prints `<host> clock cap N MHz` only after `nvidia-smi -lgc 0,N` succeeded; both Sparks must print it and agree, else the start is refused. A start with no start log is classified by its checkpoint times against `scratch/CLOCK.log` (2150, except 2300 from 13:27:18 to 13:36:27Z and from 19:23:25Z on); a start that straddles a change is refused. Write `clock_source` (`'start-log'` or `'clock-log'`) beside it |
| `clock_caps_mhz` | optional window-level **audit** field: `win2_history.clock_caps_mhz(out)` (list of MHz, or `'unknown'`) |
| `date` | `YYYYMMDD` of the window (`win2_history.window_date(out)`) |
| `start`, `repeat` | the control start's name and the repeat index inside it, so between-start variance can be watched |
| `prompt_sha256` | the repeat's prompt identity: `decision_v2.prompt_identity([stream prompt_sha256, ...])` (B1: the stream's sha256; B ≥ 2: sha256 of the sorted member hashes). `prompt_members` (optional, checked) lists the stream hashes |
**max_tokens is part of the cell.** NBW ran short-b1, 32k-b1, 32k-b2 and 32k-b3 at 1,500 tokens; every other runner
used 512. At B3 that alone moves ms/round from about 72 to about 81 ms (r = −0.93 over 23 repeats): shorter answers
spend more of their decode beside the other streams' 32K prefills. A runner whose cell differs in anything else
(prompts, batch, late arrival) gives it a new name.
**Spreads never pool definitions.** `archive_spread` refuses a cell/metric whose rows mix max_tokens, production commit,
clock cap or prompt identity unless `where=` picks one of each, e.g. `where={'max_tokens': 512, 'commit': <40-hex>,
'clock_cap_mhz': 2300, 'prompt_sha256': <64-hex>}`; `archive_spreads(path, cell, metric)` reports one spread per
(max_tokens, clock cap, commit, prompt).
The clock ceiling was 2150 MHz (dsv41.env default) for every start of every archived window through
PC1-threshold512-W-20261027T181314Z (each start's log proves it) and is 2300 MHz from 2026-10-27 19:23:25Z
(`GPU_CLOCK_CAP=2300` in /etc/environment; PP4-941-admission at 19:23:39Z is the first 2300 window, and its start log
says so). Busy clocks rose about 7 % (2125 → 2275 MHz), so a pooled spread would mostly measure the clock step. The PP4
promotion will change the commit.
Report `archive_spread` next to the in-window spread. Keep the 2 % required effect unless the archive shows larger
real variation; if it does, say so in the report and never raise the bar silently. Record `archive_head(path)` (row
count + head sha) in the window summary before and after appending: the chain detects an edited line, not a truncated
tail. To backfill history, run `python -m offload.launch.windows.win2_history backfill --archive <archive> <window
dirs>`. It is idempotent per (window, cell, metric) and reads a window named twice in one call once.
WIN2-R2 rebuilt the archive (596 rows: the 540 v1 rows with their production commit, per-start clock cap, start and
repeat filled in, plus PC1-threshold512's 56). The v1 file is kept at
`scratch/WIN2R2/control-archive.v1-before-win2r2.jsonl`; the proof (multiset equality) is in `WIN2-R2.report.md`.
**v3 → v4 (DP-2): coordinator step, NOT yet applied.** `scratch/WIN2R3/rebuild_archive_v4.py` rebuilds the 596 rows
from the windows with their prompt identity (dry run: multiset equal on every v3 field, one identity per window and
cell; `scratch/WIN2R3/rebuild-v4-dry.txt`). Apply it (`--apply`, in place under the archive's flock, v3 copy kept) only
once every runner that appends has merged WIN2-R3: a runner on v3 code refuses to append to a v4 chain, and v4 code
refuses a v3 chain.
## New shared cells (`gb10_budget/window_cells.py`, opt-in by name, not in ORDER)
- `ctl-mixed150k-4user`: three 32K decoders; the 150K document arrives only after all three are decoding and must be
admitted beside them (running gauge ≥ 4 while every decoder is alive; else the cell refuses). ~300 s.
- `ctl-long-5k`: one B1 request with a 5,000-token answer. ~180 s.
## Back-to-back mode (`chain.py`)
- Coordinator sets `WINDOW_CHAIN=1` for every window of a chain and `WINDOW_CHAIN_LAST=1` on the last one.
- **Hold placeholders stay up for the whole chain** (coordinator rule, from R-PC1 / R-PP4; WIN2-R3 root fix of the
defect R-NBW proved in `scratch/RNBW/placeholder-proof.txt`). Between chained windows there is no HOLD file, so the
placeholders are what keep NVO drama3 and the local-law queue off the Sparks while production is down.
- **Two kinds.** A window's own placeholder (made by `hold_admission.open_script` inside its `-hold` scope) lives on
that scope's PRIVATE tmux socket (the scoped_remote tmux shim) and dies when the scope is fenced: it covers one
window only. The **chain placeholder** is a tmux `dsv41-hold` session on the Spark's DEFAULT socket running
`vllm serve [DSV41-OFFLOAD hold] owner=<chain owner>`; it outlives every window scope.
- **Owner.** `owner = chain.chain_owner()` (the marker's while the window ran chained, else a new
`chain-<UTC>-<8 hex>`); call it once per window and keep it. The owner is in the process title on each Spark and in
the marker (`placeholder_owner`). Start, probe and stop refuse a placeholder of any other owner, including
window.sh open's (no owner), and never touch it.
- **Where they are looked for.** Every step uses `/usr/bin/tmux -L default` (`chain.PLACEHOLDER_TMUX`), never the
PATH `tmux`, so it reaches the default socket from anywhere, including inside a scope.
- **Start** `chain.placeholders_up(owner)`: from the runner's own unscoped process over plain ssh (refused inside any
`dsv41w-*.scope`, where a new default server would die with the fence), at a `'chained'` end, BEFORE the first fence
of the window's scopes. Raises unless both Sparks are UP for this owner. Idempotent: a chained window keeps the same
processes.
- **Close** `chain.chained_close(owner)` replaces R2's `CHAINED_CLOSE` (which looked at the scope's private socket and
always failed): run inside the window's `-restore-close` scope; it proves this owner's placeholders on both Sparks
through the scope's ssh shim, then removes only the HOLD (the rm shim records it and refuses another window's HOLD;
commit-close removes it).
- **Probe** `chain.placeholders_present()` (owner from the marker): the chained preflight and `chain.gate()` (so
`chain_prev_ok.sh`) require both UP. A marker without an owner (written before this fix) stops the chain.
- **Stop** `chain.end_chain(owner=None)`: the chain's last step, ONLY after the strict W8 restore verified AND the
post-close watch PASSED. It takes this owner's placeholders down (from an unscoped process; refused inside a scope),
then clears the marker; a foreign or unreachable placeholder is left alone and reported (`ENDED WITH ALERT`).
`chain.clear()` refuses while the marker records an owner. `window.sh close` (W8 close under a scope) cannot
remove them: its kill reaches the scope's private socket.
- systemd ≥ 258 expands `${...}` in a `systemd-run --scope` command line (Controller runs 259), so the placeholder
scripts and `chained_close` use no braced expansions.
- **Runner start** (S92 FIX 1). `mode = chain.preflight(production_serving, hold_exists,
placeholders_present=chain.placeholders_present())` (or `chain.gate()`, which probes this chain's placeholders only
while the marker is present) replaces the "production serving K1" preflight. A chained preflight without the placeholder
result refuses. In mode `'chained'` only:
- open the admission HOLD with `hold_admission.open_script(..., allow_unreachable=chain.allow_unreachable(mode))`.
The default `require-serving` admission raises when :8000 does not answer, so a chained window would fail before
its HOLD;
- prove both Sparks quiet with `spark_quiet.command(<marker>, deadline=...)` on sparkhead and sparkworker-wired **before**
`start_exp.sh` (the gate infers "production down" only from a failed `/v1/models` probe; it never proves quiet);
- do not capture production (it is down): `path, rollback = chain.production_snapshot()` returns the chain's
capture (the rollback.json of the window that started from serving production; its sha256 is checked against the
marker), whatever runner wrote it. Copy it into the window's own `rollback.json` with `chained_from` /
`chained_from_sha256` and take the window's own first control start as the measurement baseline.
- **Runner end.** Decide with `chain.end_action(stopped=<any stop>, hold_written=<own HOLD written>)`, never with
`hold_written` alone:
- `'chained'`: `owner = chain.chain_owner()`; `chain.placeholders_up(owner)` before the first fence; stop the
experiment, prove the Sparks quiet, close the own HOLD with `chain.chained_close(owner)` inside the window's
`-restore-close` scope (`owner.command('launch', f'{tag}-restore-close', ['bash', '-c', chain.chained_close(owner)],
...)`, require `chain.CHAINED_CLOSE_OK` in its output, then the usual fenced commit-close; `window.sh close` would
refuse without :8000), `chain.finish_chained(out, quiet_proof, placeholder_owner=owner)` (the marker records the
owner and the chain's production snapshot: inherited from the previous marker, or this window's own rollback.json),
write
`"chain": "CHAINED"` and `"restore": {"status": "CHAINED: production left down"}` into the summary, and **skip** the
W8 restore and the post-close watch;
- `'restore'`: the normal strict W8 restore + post-close watch inside the own HOLD, then `chain.end_chain()` (only
after RESTORED + watch PASS; `chain.end_chain(owner)` when this window started chain placeholders and failed before
`finish_chained`). Until then the placeholders stay up;
- `'abort'` (no own HOLD, chain marker present: e.g. a pre-HOLD failure of a chained window; production is down from
the previous window): `summary['restore'] = chain.abort(out / 'chain-abort')`. With no HOLD present it runs the strict
W8 start|check|close under the reviewed `post_close_watch.ScopedRecovery` (its own HOLD, Sparks proved quiet, six
distinct IDENTICAL greedy) plus the 15-minute post-close watch, and runs `end_chain()` (placeholders down, marker
cleared) only on RESTORED + watch PASS.
With a HOLD present it alerts and touches nothing; so it does while production is serving (S96 FIX 2a: the recovery
would drain it, open a HOLD that pauses K24 and refuse at the guarded stop), and when an earlier automatic abort of
the same stopped chain did not end RESTORED + watch PASS (S96 FIX 2b: ONE automatic recovery per stopped chain, the
post-close watch rule; record `<marker>.abort`, removed by the chain's end). Never write "NOT NEEDED: production
untouched" in this case;
- `'none'`: no HOLD and no marker; production was not touched.
`chain.restore_required(stopped, hold_written)` is `end_action(...) in ('restore', 'abort')`.
- **Coordinator gate**: `offload/launch/windows/chain_prev_ok.sh` replaces campaign `launch/prev-ok.sh` for chained windows.
It is coordinator-only (S96 FIX 1): without `WINDOW_COORDINATOR=Deepseek/coordinator` it exits 2 at once, before any
wait or probe, because its chain abort needs it; **every RUN template exports WINDOW_COORDINATOR before the gate**
(campaign `launch/PC1-W2-RUN.sh` and R-UPF/R-UP3 `offload/upfix/launch/UPF-RUN.sh` export it after the gate today and
must move the export up). Exit 0 admits; exit 2 is a refusal with no chain marker (as prev-ok.sh); **exit 3 is `CHAIN STOPPED: PRODUCTION DOWN`**:
any refusal (previous summary, gate, HOLD, stale container) while the marker is present. The gate then runs
`chain_abort.sh` itself before exiting, so a stopped chain never leaves production down unattended. RUN scripts call
the gate under `set -e`, so exit 3 also ends that RUN script; the abort's `summary.json` (in
`$C/CHAIN-ABORT-<UTC>/`) is a valid PREV_FILE target for the next gate.
- **Coordinator abort by hand**: `offload/launch/windows/chain_abort.sh [OUT]` (= `python -m offload.launch.windows.chain
abort --out OUT`, needs `WINDOW_COORDINATOR`). Exit 0 restored + watch PASS (placeholders down, marker cleared) or no
marker; 3 failed (marker and placeholders kept, production down); 4 alert only (a HOLD exists, production is
serving, or this chain's one automatic abort already ran). `abort --again` (by hand only) forgets the previous attempt
and runs once more. By hand:
`python -m offload.launch.windows.chain placeholders [--owner X]` (probe both Sparks) and, after a verified restore +
watch PASS, `WINDOW_COORDINATOR=... python -m offload.launch.windows.chain end [--owner X]`.
- The legal pipeline is down for the whole chain (NDM accepts this while building). The last window, or any stop,
brings production back with the strict six-IDENTICAL W8.
- Window re-scoring (`python -m offload.launch.windows.win2_history reanalyze --decision C1 [--decision C2] <window>`)
uses decide_window's cell rule, refuses more than two decision cells, treats roles `control`, `K`, `off` (TV1) and
`H` (PF2) as controls, and judges every other role as its own arm (a two-arm window reports `arms`, never a pooled
candidate).
- Test-only overrides (never set in production): `DSV41_CHAIN_MARKER`, `DSV41_CHAIN_PRODUCTION_URL` (chain.py),
`DSV41_CHAIN_CAMPAIGN`, `DSV41_CHAIN_PY`, `DSV41_CHAIN_ABORT` (the two scripts). `chain status` prints the marker path
and production URL in use.
## Greedy refinement (`greedy_kl.py`, opt-in)
Candidate arm: `nbw_cells.greedy(..., near_tie_nats=<registered>, record_distribution=True,
probe_salt_prefix=<window tag>)` keeps the top-20 and prefix at every divergence. On the command line:
`python -m offload.launch.windows.nbw_cells --execute --mode greedy ... --near-tie-nats <registered>
[--record-distribution --probe-salt-prefix <window tag>]` (defaults unchanged: 0.501 nat, no recording). Following control arm:
`greedy_kl.compare_recorded(api, candidate_rows, salt_prefix, bound=<registered>)` re-probes those prefixes with a fresh
cache salt and measures KL(control || candidate) over shared top-20 + OTHER (UP2's estimator); common mass must be
≥ 0.99. The bound is each runner's registration (UP2 used max(0.005, 2 × A/A floor)).
## Adoption checklist for each runner fork
1. `git merge offload/win2` (needs the "WIN2 API" commit ce1bcf2 or later).
2. Schedule ≥ 3 repeats per arm in the decision cells (C/X/C with ≥ 2 repeats per start gives ≥ 4 control and needs
≥ 3 candidate repeats: run the candidate start with 3 repeats or use C/X/C/X/C).
3. Build `decision_v2.Series` per decision cell, primary metric first; call `decide_window(series, decision_cells,
groups=...)`; write its dict as the summary `decision` (rule `win2-v1`).
4. `archive_append` every control repeat at the end of the window, including after a later stop for the starts that
finished. Each row carries every field of the archive table above (commit = production, `clock_cap_mhz` of its own
start from `start_clock_cap`, `YYYYMMDD` date, start, repeat, `prompt_sha256`); record `archive_head` before and after.
Every decision `Series` carries `control_prompts` / `candidate_prompts` (one identity per value, from the streams'
`prompt_sha256`): decision cells refuse without them (DP-2).
5. Chain mode as above; summary `chain` field. Tests (fake transport, `tests/test_chain_stop.py` shows the shared
parts): a chained non-last end (marker, no restore); a last or stopped window (strict restore, `chain.end_chain()`);
**a chained start with production down** (admission with `allow_unreachable`, quiet proof before `start_exp.sh`);
**a pre-HOLD failure in the chained state, which must restore** (`end_action == 'abort'` -> `chain.abort`); and
**a gate refusal in the chained state**, which exits 3 and takes the abort path; **placeholders** (WIN2-R3): a
chained end calls `placeholders_up(owner)` before its first fence and `chained_close(owner)` in its
`-restore-close` scope, `finish_chained(..., placeholder_owner=owner)`; a chained start refuses without this
chain's placeholders; a final restore ends with `end_chain()`; a chained start that follows another runner's window
takes its production capture from `chain.production_snapshot()`. The real-scope proof is
`tests/test_chain_placeholder_rehearsal.py` (real user scopes, scoped_remote shims, tmux, fake per-host HOME).
6. **WIN2-R3 (seeded cells; adopt after its independent check).** Swap the decode decision cells for seeded cells
(`run_seeded_cell` on every arm, C/X/C; `seeded_series` + `decide_paired_window`), keep the existing gates, tag every
cell's profile and state in the promotion which profile it relies on; call `window_cap` from the launcher and
`resolution` on the pre-registration (`expected_gain` per decision cell); add the smoke start (`x_first` in a single
window, `chain_head` at a chain's head); register `start_sd` / `--diff-cv` once the seeded A/A pilot has measured them.
7. Greedy: keep the runner's registered near-tie limit (0.9 general, 1.1 PP4 and DEP1 fixed2 per NDM) and, where the
window has a following control start, the opt-in KL check.
## Rehearsal before the first Spark window (design note; not built)
The real target runs only on the two Sparks (RUNBOOK-offload.md lists no other configuration of it), so no rehearsal
off the Sparks can measure speed, acceptance or numerics. A rehearsal exists to catch runner faults (sequencing,
parsing, timeouts, refusals, chain and archive writes) before they cost a window. Three levels:
1. **Required for every new or changed runner: fake transport on Controller CPU.** Real runner code over inert recording
remote commands and an in-process HTTP fake (`tests/test_nbw_rehearsal.py`, `tests/test_pf2_rehearsal.py`,
`UP2-REHEARSAL.md`). Add the WIN2 cases of checklist step 5: a chained non-last end (marker, no restore), a stop
inside a chained window (restore, `chain.end_chain()`), a chained start with production down, a pre-HOLD failure in the
chained state (`chain.abort`), and a gate refusal in the chained state (exit 3).
2. **Proposed: a stub "target-only" HTTP server (CPU only; Controller or x86host CPU, loopback port, no GPU).** One small
threaded process that answers the endpoints the runners touch with vLLM's formats: `/v1/models` (one model,
`max_model_len`), `/tokenize` (`count` from the text length), `/v1/chat/completions` (SSE with reasoning/content
deltas, final usage, `[DONE]`), `/v1/completions` (top-20 logprobs, token ids), `/metrics` (the
`vllm:num_requests_running` and `vllm:num_requests_waiting` gauges and the speculative counters bench2
`parse_metrics` reads), and `/start_profile` + `/stop_profile`. It needs a configurable admission limit and decode
delay, so that `ctl-mixed150k-4user`'s late-arrival check passes at limit 4 and refuses at limit 3. The runner points
at it via its target URL with the remote side still faked. This proves real sockets, timeouts, SSE parsing,
concurrency and admission. It proves nothing about performance. Not built here: matching vLLM's SSE and metrics
formats exactly is a lane of its own, and level 1 already gates every runner.
3. **Drafter/worker code changes only: an owned x86host window** (as the PP4 server probe did). The coordinator books it,
because every x86host 3090 is assigned to DSV41 work. It checks that the worker loads and answers; Spark rounds are
still needed for any number. Config-only Spark arms skip this level.
Then, in the first chained window, the coordinator runs the runner's `--plan` / dry-run against the live read-only probes
before the HOLD opens.
|