STATE.md

# AgentPTB run state

## Scope and compliance

- Objective: improve `Qwen/Qwen3.5-9B-Base` for agentic software engineering and shell work using weights trained in this run and/or a compliant `pi` harness.
- Evaluation-only data: `/root/work/shared/tasksets/terminal-bench-2` and `/root/work/shared/tasksets/swe-bench-verified`. Task texts and solutions will never enter training corpora or selection/filtering criteria.
- Forbidden materials: other cells/runs, operator notes, benchmark repository, environment credentials as a model/data source, and third-party post-trained weights for initialization/merging.
- Permitted sources planned: the base checkpoint, public generic code/agent datasets, local synthetic tasks not conditioned on evaluation items, and trajectories generated by locally run public models if useful.

## Environment

- Started: 2026-08-14 16:27 UTC
- Deadline: 2026-08-18 20:27 UTC
- GPUs: CUDA devices 4,5,6,7 (four NVIDIA B200 183 GB)
- Workspace: `/mnt/pvc/users/simon/agentptb/runs/sol-max-s1/workspace`
- Base: `Qwen/Qwen3.5-9B-Base` in the pre-staged Hugging Face cache
- Trainer: `/root/work/a/prime-rl`
- Important serving flags: `--enable-auto-tool-choice --tool-call-parser qwen3_coder`
- Important trainer setting: `model.attn = "flash_attention_2"`
- Important runtime settings: broker, `block_network = false`, exported `PRIME_API_KEY`

## Experiment log

- Base inference server: `configs/inference-base.toml`, one B200 on physical GPU 4, port 8200. Correctly emits a structured `bash` call on a manual smoke. Required Qwen parser and multimodal limits confirmed in the resolved vLLM arguments.
- Base Terminal-Bench 2 smoke attempt: four fixed-seed shuffled tasks, `evals/base-tb2-smoke-run`. All four were invalid infrastructure results (zero model calls): two broker ready-timeout 408s and two HTTP read timeouts after first-time image preparation. Never count these as reward outcomes. The valid retry in `evals/base-tb2-smoke-retry` scored 1/4; this is much too small for a useful rate. One failure exhausted context and the success exhausted the episode output-token cap.
- Public SFT source: `Kwai-Klear/SWE-smith-mini_swe_agent_plus-trajectories-66k`, revision `750b2c11239fd5e32f97e6cfb9bf80fb9a9a2983`, MIT. The published corpus contains 65,994 successful trajectories from synthetic SWE-smith issues; its reported Qwen3-8B SFT result is 39% SWE-bench Verified.
- Converted corpus: `data/pi-swe-smith-66k` (4.5 GB). Mechanical conversion only: fenced shell actions to Pi `bash` function calls, observations to tool messages, source terminal marker to a concise final assistant response. 65,994 rows, 2,197,438 tool calls, 147 malformed/plain actions, and 0 exact identifier overlaps with the local SWE-bench Verified task directory. Provenance is in the dataset directory.
- Public terminal SFT source: `nvidia/Nemotron-Terminal-Corpus`, revision `a1667c4ffdadea02a89bffe4f1bb7ca2ff19f8d9`, CC-BY-4.0. Used only its three broad, novel skill-based synthetic configs, not benchmark tasks or oracle solutions. Converted 131,564 rows and 796,049 shell-action batches to structured Pi calls in `data/pi-nemotron-terminal-skill-140k`; 8,277 malformed or special-key trajectories were rejected and zero exact evaluation instruction/name overlaps were found. Converter: `scripts/prepare_nemotron_terminal_sft.py`.
- A follow-up conservative exact normalized 14-token audit against both evaluation instruction sets removed 5,701 SWE rows and 3,835 terminal-skill rows. Clean components are `data/pi-swe-smith-66k-clean14` (60,293 rows) and `data/pi-nemotron-terminal-skill-140k-clean14` (127,729 rows). The active balanced mixture is `data/pi-agent-mix-clean-v1`: 127,729 terminal rows plus two deterministic SWE copies, 248,315 rows total, shuffled with seed 20260814.
- The initial unfiltered production attempt was stopped and discarded after step 17 as soon as this conservative audit completed. Its data mixtures and output are renamed with `quarantined-`/`discarded-` prefixes and cannot be selected accidentally.
- Optional broad adapter data is staged but is not part of the active run: the public Terminal-Corpus adapter split was converted to 196,767 Pi trajectories and 1,000,521 shell calls in `data/pi-nemotron-adapters-226k-v1`. A conservative exact normalized 14-token blacklist against both evaluation instruction sets removed 7,434 rows; interactive-key and actionless rows were also excluded. The fully clean fallback mixture is `data/pi-agent-mix-clean-v2-adapters` (445,082 rows).
- Full-parameter trainer smoke at 2K was stopped before an update because the window contained no assistant tokens and the loader cycled. Preserved under `outputs/sft-smoke`.
- Full-parameter trainer smoke at 8K completed one update successfully under `outputs/sft-smoke-8k`: loss 0.1684, grad norm 4.8696, peak 168.5/178.4 GiB on one B200. This validates data/rendering/model/backprop but shows full-parameter single-GPU memory is too tight for longer context.
- Rank-64 LoRA at 32K succeeded both with Ulysses CP=2 (28.8 GiB/GPU, 2m02s compile step) and CP=1 (48.0 GiB, 6.8k tokens/s steady state). CP is unnecessary on B200.
- Four-GPU full-parameter 32K training is feasible and preferable to LoRA: micro-batch 1 reached 7.3k aggregate tokens/s at 45.6 GiB/GPU; micro-batch 2 reached 11.6k at 48.1 GiB; micro-batch 4 reached 15.5k at 70.1 GiB; micro-batch 8 reached 21.3k at 115.2 GiB. All losses and gradient norms were finite. Production uses micro-batch 8.
- The clean production recipe follows the published Terminal-Corpus setup where practical: 32K, AdamW at 2e-5, weight decay 1e-4, beta2 0.95, cosine schedule with 10% warmup, global batch 128, full-parameter training. Config: `configs/sft-agent-mix-clean-full-v1.toml`; output: `outputs/sft-agent-mix-clean-full-v1`.
- Clean production started at 2026-08-14 18:17 UTC. Checkpoint 50 completed
  cleanly: loss 0.2909, recent mean 0.2697, no NaNs, 102 GiB resumable state,
  18 GiB stable HF weights, and 143.8 seconds of save overhead. Step 51 then
  completed normally (loss 0.2772, grad norm 0.2748). Step 150 is projected near
  2026-08-15 01:00 UTC including the two remaining checkpoint saves.
- Paired evaluation tooling is ready: stock and custom runs use the same fixed seed-0 sample, `scripts/run_paired_evals.sh` launches isolated concurrent runs, and `scripts/compare_evals.py` reports paired outcomes.
- Evaluation reporting now uses the conservative score and Wilson interval over every
  scheduled episode. Reward-bearing-only scores are explicitly diagnostic, and any
  transient errored episodes will be resumed on the same fixed sample rather than
  dropped from the denominator.
- The active loader advances about 243 raw trajectories per optimizer step. At that
  measured rate, the prepared step-2,050 continuation lands at about 2.01 epochs,
  matching Terminal-Corpus's published two-epoch recipe. The first 36,500 shuffled
  rows retain the full mixture ratio (51.44% terminal / 48.56% SWE).
- A structural quality audit found that 54,322/127,729 clean terminal rows end in
  an explicit non-tool assistant completion; the remaining source rollouts stop at
  a tool observation or turn limit. All 60,293 clean SWE rows finish explicitly.
  The independently filtered, evaluation-content-blind continuation mixture
  `data/pi-agent-complete-mix-v1` contains two copies of each complete component
  (229,230 rows total). It is staged only; step 150 will be evaluated before any
  branch is trained on it.
- The public Terminal-Corpus paper's own broad ablation strongly favors retaining
  incomplete synthetic trajectories: 12.4% TB2 unfiltered versus 6.74%
  complete-only and 5.06% success-only. It also favors single-stage mixed training
  (13.03%) over a two-stage curriculum (10.39%). Therefore the evidence-backed
  main continuation remains `pi-agent-mix-clean-v1`; the complete-only corpus is
  only a possible narrow stopping ablation.
- Custom Pi now optionally advertises the true model context window and per-call
  generation ceiling. Custom configs set 65,536/4,096, enabling Pi's generic
  automatic compaction before a backend overlong-prompt error. Defaults remain
  unset, so stock Pi behavior is unchanged. A no-review custom variant is ready
  for a paired step-150 ablation of the generic review pass.
- Custom evaluation variants now accept Pi's clean empty end-turn after its tool
  activity, so a modified workspace reaches the task grader instead of becoming
  an unscored harness error merely because no separate prose was emitted. Stock
  Pi remains unchanged. The option parsed successfully in the pinned run
  environment; an attempted broad Verifiers test collection was blocked before
  tests by an unrelated installed `prime_sandboxes` API mismatch.
- Evaluation summaries now derive their conservative denominator from the saved
  `num_tasks * num_rollouts` configuration and explicitly count missing trace
  records. A client crash therefore cannot inflate the reported score by silently
  shrinking the denominator.

## Current plan

Before any step-150 selection result was available, the continuation's purely
mechanical benchmark set was extended with micro-batch 10/global-batch 120 and
micro-batch 12/global-batch 96. The measured micro-batch 4-to-8 memory curve
predicts both may fit on B200, and reducing accumulation could increase useful
tokens per second. Each variant must still complete finite steps, remain below
170 GiB/GPU, save both resumable and HF step-152 checkpoints, and beat the
same-batch-128 baseline throughput by at least 3%. The fastest eligible recipe
wins. If a different global batch is selected, the deadline calculation uses
its actual tokens/update and the terminal-step cap preserves the original
batch-128 two-epoch token budget. This changes no data, score gate, or model
branch.

The waiting `agentptb-post-step150` session was restarted after this edit so
its shell reads the new benchmark set. At restart the evaluation-complete
marker did not exist; the live step-150 evaluator and inference process were
left untouched. All six benchmark and production TOMLs parse, both new global
batches satisfy Prime-RL's divisibility invariant, and the handoff script
passes `bash -n`.

At 2026-08-14 21:04 UTC, a focused Prime-RL unit-test invocation triggered the
suite's module-level `pkill -f torchrun` cleanup fixture and terminated the live
trainer after step 63. This was a test-isolation error, not a model or data
failure. Steps 51--63 were not checkpointed and were discarded.

The first resume exposed a Prime-RL CPU-offload edge case: optimizer tensor states
and the scheduler counter loaded, but optimizer param-group LR did not, pinning
steps 51--52 near `2e-13`. That attempt was stopped and discarded. The documented
workaround is now in the same resume config: `skip_scheduler = true`, a new
100-step cosine with no warmup, and peak LR `1.7e-5` (the actual LR at checkpoint
50). Model, optimizer moments, progress, and dataloader still load from step 50.

Active trainer tmux: `agentptb-sft-clean-resume50b`. At 2026-08-14 21:18 UTC it
started the corrected resume; step 51 completed at 21:22 with loss 0.2772,
gradient norm 0.2748, and the intended LR `1.70e-5`. Do not run the root Prime-RL pytest suite alongside
training unless `PRIME_RL_TEST_SKIP_PROCESS_CLEANUP=1` is explicitly set. The
metrics file keeps stale rows 51--63 from the killed attempt and 51--52 from the
near-zero-LR attempt; later appended rows supersede them, while
`training_status.py` will continue to show 63 until the corrected resume passes it.
At 21:42 UTC the corrected run reached step 59 with finite loss 0.2824,
gradient norm 0.2982, LR `1.67e-5`, and 28.0k tokens/s; all corrected-resume
updates through that point were healthy.

At 21:50 UTC the corrected run reached step 62 with loss 0.2786, finite
gradient norm 0.2788, intended LR `1.65e-5`, and 28.0k tokens/s.  A deterministic
2,000-row structural sample of the active mixture found an exact SFT system
prompt match in every row, only the aligned `bash` structured tool in assistant
actions, a median/p95 of 11/51 assistant turns, and an explicit final assistant
completion in 70.0% of rows.  Every intermediate assistant tool turn is still
rendered with its own supervised end-turn token.  This supports the 64-turn
custom ceiling and does not justify switching to the staged completion-only
branch before seeing actual candidate stop behavior.

A direct step-50/base tensor comparison confirms lineage and nonzero updates:
`model.language_model.layers.0.input_layernorm.weight` differs from base by
mean/max absolute `7.71e-5`/`4.88e-4`, while the frozen
`model.visual.blocks.0.norm1.weight` is exactly identical to base.

Separate evenly spaced component samples also checked the row-level balance.
Terminal-skill rows average 6.47 assistant turns and about 21.3k assistant
characters; SWE-smith rows average 33.96 turns but about 25.6k assistant
characters.  Because the mixture has 127,729 terminal rows and 120,586 repeated
SWE rows, supervised assistant volume is therefore close to balanced rather
than being dominated by SWE's longer tool-output histories.  The public
Terminal-Corpus ablation confirms that adding all adapter sources gives only a
small TB2 mean gain over skill-only data (13.03% versus 12.4%) when each recipe
is trained for two full epochs.  At this run's fixed remaining compute, the
445k-row adapter-expanded mixture would receive under one epoch and dilute the
high-value SWE source.  Keep the evidence-backed 248k-row active mixture for the
long continuation unless candidate evaluation reveals a clear contrary failure.

All staged resumed-training configs now use zero-warmup cosine restarts. With
CPU-offloaded optimizer state, constructing a warmup scheduler before checkpoint
load mutates the param-group LR to the warmup floor, and loading tensor state does
not restore that group. Fresh-from-base configs retain their normal warmup.

The guarded `agentptb-post-step150` watcher now prevents an idle gap after the
step-150 evaluations. Once evaluation has completed and inference has fully
released the GPUs, it hard-links the resumable step-150 state into an isolated
two-update selective-activation-checkpoint benchmark. The benchmark now must
successfully save both the resumable and HF-compatible step-152 checkpoints,
which exercises the temporary GPU staging of CPU-offloaded Adam state that its
earlier train-step-only check missed. It selects selective AC only after that
save, at >=29.0k tokens/s and <=170 GiB peak training memory; otherwise it uses
the proven full-AC config. It then launches the same-mixture step-1,800 continuation as
`agentptb-sft-continuation`. Decisions and the measured benchmark are written
under `state/`; no dataset branch is selected automatically.

The handoff now also benchmarks full activation checkpointing with GPU-resident
Adam state, plus full checkpointing with activation offloading disabled while
Adam stays on CPU.  Four-way B200 sharding may fit either despite the source
paper's H100-oriented offload recipe, and avoiding one class of host traffic
could improve throughput.  The proven recipe and all three experimental variants
independently resume step 150, perform steps 151--152 on the same dataloader state,
and must show finite metrics, LR in `[1.9e-5, 2.1e-5]`, no more than 170 GiB peak
training memory, and successful resumable plus HF checkpoint saves. Because
packed-document lengths make absolute throughput batch-dependent, an experimental
variant must beat the paired proven-recipe baseline by at least 3%; the fastest
eligible variant wins, otherwise the baseline recipe remains the fallback. The
running post-step150 watcher was restarted after this change.
Benchmark and inference launchers now execute through a `setsid` bash wrapper,
and cleanup probes and kills the whole process group even when a uv leader exits
before torchrun workers.  The sleeping step-150 evaluation and post-evaluation
watchers were restarted to load this cleanup fix; the live trainer was untouched.
The initial continuous target was step 1,800 (about 1.74 corpus epochs). At the proven
fallback throughput, including the three save-through benchmarks, this still
leaves roughly 20 hours before the deadline for checkpoint checks and the full
1,178-episode canonical stock/custom measurement. The chosen resume file's
historical `max_steps = 1650` is overridden explicitly on the launch command and
recorded in `state/continuation-config.txt`; the resolved trainer config will
therefore preserve the actual 1,800-step schedule.
The initial deadline guard fell back to step 1,650 if fewer than 87 hours
remain after development evaluation and the three mechanical benchmarks. The
preservation watcher reads the recorded resolved target, so either terminal
checkpoint is retained and labelled correctly. At the expected handoff time the
1,800-step branch had about two hours of additional guard beyond this threshold.
This discrete rule was superseded before evaluation by the score-independent
throughput/time calculation recorded at the end of this section.

All step-150 configs now use one typed whole-episode retry limited to explicit
sandbox/network infrastructure exceptions; context, model, harness, and task
failures remain single-shot.  Dry config validation confirmed the retry policy,
and the temporary dry-run output directories were removed before evaluation.
True stock configs leave every newly added Pi option unset (including
`text_only`), while custom configs retain the aligned prompt and other opt-ins.

Canonical final configs are staged for all 89 TB2 and all 500 SWE-bench Verified
tasks.  The four-run stock/custom launch allocates 16 concurrent episodes to
each TB2 run and 48 to each SWE run, totaling the measured-safe 128 sandbox
ceiling.  `scripts/run_final_evals.sh` requires an explicit review/no-review/stock
choice plus sampling temperature and refuses to overwrite existing results. The
wrapper consumes both precommitted state files. The selected model will be
exposed through `submission/model` only after checkpoint selection.

At 22:36 UTC a takeover audit caught a completion-sentinel mismatch in the
post-step-150 benchmark watcher before it ran. Prime-RL resumable checkpoints
finish with `checkpoints/step_N/trainer/.metadata`; only HF weight exports have
a root `STABLE` marker. The watcher had required a nonexistent resumable
`STABLE`, which would have falsely rejected all four execution benchmarks. Both
the early-return and polling predicates now require the real metadata file plus
the HF `STABLE` marker, `bash -n` passes, and only the sleeping
`agentptb-post-step150` session was restarted. The active trainer was untouched.
At the same audit the clean resume was at step 80 with finite loss 0.2646,
gradient norm 0.2875, intended LR 1.37e-5, 27.6k tokens/s recent throughput,
and 116.2 GiB/GPU peak memory. The SFT rows' system prompt and four-tool schema
were also checked directly against the custom Pi prompt/schema.

The final sampling and review choices are fixed before candidate results are
visible. Qwen's public Qwen3.5 model card recommends temperature 0.6/top-p 0.95
for thinking-mode precise coding, whereas the initially staged temperature 0.2
had not been measured. The development gate therefore evaluates aligned
no-review and review at both temperatures on the same 32+32 tasks. Temperature
0.6 is the evidence-backed default; 0.2 replaces it only if it solves at least
as many tasks on each suite, has a strictly positive combined paired advantage,
and its combined two-sided exact sign-test p-value is at most 0.10. At the
selected temperature, review is enabled under the identical non-inferiority,
positive-advantage, and p-value rule; otherwise use aligned no-review. Missing
or errored episodes count as failures, and reward-bearing-only rates do not
select either setting. The SFT-aligned scaffold remains the default; stock
replaces it only under the identical per-suite non-inferiority, positive paired
advantage, and p-value gate. Ten primary development runs at concurrency 12
each keep total sandbox concurrency at 120, followed by two stock t=0.6 runs at
concurrency 32 each (64 total).
`scripts/select_harness.py` implements this rule, and the sleeping step-150
evaluation watcher writes the full decision JSON, selected temperature, and
selected review mode under `state/` before releasing the continuation watcher.

Final checkpoint selection is likewise score-independent. Select the recorded
terminal target if its HF export has `STABLE`, every indexed shard exists and
loads with the declared shapes, the terminal training metrics and gradients are
finite, and a served structured-tool-call smoke succeeds. If and only if that
artifact is corrupt or training becomes numerically unhealthy, walk backward
through the scheduled stable checkpoints and take the newest one satisfying the
same health checks. No intermediate checkpoint is benchmarked to make this
choice.

Checkpoint 100 completed and passed the independent header/metric gate at
23:28 UTC. The resumable `trainer/.metadata` and HF `STABLE` markers exist; all
four shards expose exactly 760 indexed BF16 tensors totaling 9,409,813,744
elements. Step-100 loss was 0.2561, gradient norm 0.2770, LR 8.77e-6, and NaN
count zero. The trainer continued normally toward step 150.

`agentptb-finalize-when-ready` now closes the last idle gap. It reads the
recorded continuation target, waits for the preservation watcher to hard-link
that terminal HF export and for its tmux session to exit, then runs the full
finite-tensor/package gate and the process-group-owned final inference/evaluation
pipeline. Any health or pipeline failure stops closed and leaves logs/results
for manual recovery; it never chooses a checkpoint from scores.

At 23:53 UTC the active resume reached step 110 with loss 0.2685,
recent-mean loss 0.2612, gradient norm 0.2471, zero NaNs, and a 150.0-second
recent mean update time. Step 150 was projected about 1.7 hours later. All five
trainer/evaluation/continuation/preservation/finalization tmux sessions were
alive, port 8200 was free, and every downstream state marker/output path was
still clean.

At 00:01 UTC on 2026-08-15, the resumed controller reconciled this persisted
state with live processes. Training was healthy at step 113 (loss 0.2558,
gradient norm 0.2424, zero NaNs, 28.0k tokens/s) with 92.4 hours remaining.
The step-150, continuation, preservation, and finalization watchers were all
alive; no candidate/final output collision or premature readiness marker was
present. A line-by-line handoff audit confirmed that stock configs leave every
custom Pi option unset, scheduled-task denominators come from the saved configs,
and the terminal targets remain checkpoint-aligned. The public Terminal-Corpus
paper was rechecked directly: its default recipe is 2 epochs, 32,768 tokens,
global batch 128, AdamW at 2e-5, cosine with 10% warmup, and it favors unfiltered
mixed training. This continues to support the existing same-mixture long run.

Before the step-150 gate, the 32 fixed-seed development images from each suite
were prewarmed through the documented sandbox API. Selection reproduced the
evaluator's sorted task order and fixed `random.Random(0)` shuffle; only
`task.toml` image fields and Dockerfile `FROM` lines were read, never solutions.
All 64 containers reached ready in at most 62.8 seconds and all 64 were deleted.
An initial API-envelope mismatch (`sandbox_id` versus the runbook's example
`id`) created 64 pending probes; `/resources` identified the exact 64 short-ID
cohort, and every one was immediately deleted before the corrected run.

The custom empty-end-turn guard was tightened before evaluation to require a
nonempty ACP tool-call set, matching its documented purpose: grade a workspace
after clean tool activity, but never reinterpret an immediately empty response
as completion. The tracked diff archive was regenerated mechanically and its
new SHA-256 is `12533f9c06b3fe79f92c193b1a2f90953e44778b0aca3b21e4b8f2166eafc22b`.

A second pre-handoff audit fixed a metric-flush race in
`post_step150_continue.sh`: each two-update execution benchmark now requires
the step-152 resumable metadata, HF `STABLE` marker, and logged `Step 152 |`
metric line before terminating its deliberately longer trainer. Without the
third predicate a fast poll could kill between the save and metric flush,
falsely invalidating every variant. `bash -n` passed and only the sleeping
post-step-150 watcher was restarted; the live trainer was untouched.

1. Finish the first full-parameter mixed SFT gate at step 150.
2. Evaluate step 150 with stock and custom Pi on fixed unbiased samples from both suites.
3. If step 150 validates the recipe, continue the same mixture with a terminal
   step fixed mechanically before continuation starts. The handoff uses the
   selected execution variant's measured throughput, a 5% step-time penalty,
   amortized checkpoint overhead, a 30-minute startup guard, and an 18-hour
   final-measurement reserve. It caps at step 2,050 (about two epochs). The
   calculation is independent of candidate rewards and is recorded in
   `state/training-variant-benchmarks.txt` and `state/continuation-config.txt`.
   Public scaling and curriculum ablations favor more mixed SFT over a component
   switch.
4. Select the healthy terminal checkpoint a priori (intermediate saves are only
   rollback points), run the canonical full suites, and package the checkpoint/harness.

At 00:25 UTC on 2026-08-15, before step-150 weights or evaluation outcomes
existed, the fixed 1,650/1,800 deadline branch was replaced by the mechanical
throughput/time calculation above. This avoids leaving training compute idle if
an independently validated GPU-memory variant is materially faster, while the
two-epoch cap prevents an open-ended fit. `bash -n` passed and only the sleeping
`agentptb-post-step150` watcher was restarted; the live trainer was untouched.

At 00:31 UTC, a resumed-controller handoff audit verified the dynamic terminal
path end to end without touching the trainer. Prime-RL's SFT loop always writes
both resumable and HF exports at the final `max_steps`, even when it is not a
checkpoint-interval multiple, and the documented CLI syntax accepts the staged
`--max-steps` override. All workspace shell scripts passed `bash -n`, all Python
scripts parsed successfully, and the Prime-RL diff passed `git diff --check`.
The old `ac-selective-bench` and `noactoffload-bench` directories contain only
dry-validation configs, not stale step-152 artifacts, so the execution gate
cannot accidentally reuse a prior benchmark. The manual checklist's obsolete
fixed step-1,800 wording was updated to match the already-precommitted dynamic
target. Training was healthy at step 125: loss 0.2641, gradient norm 0.2247,
zero NaNs, and about 27.6k tokens/s, with roughly one hour remaining to step 150.

At 00:36 UTC, the resumed controller completed another read-only audit of the
development and final handoff. Verifiers' eval runner binds each concurrent
environment server to `tcp://127.0.0.1:0`, so the ten-process development gate
cannot collide on the serialized config's default port 5000. Its output writer
persists one complete episode atomically per scheduled rollout, and the fixed
configs have unique output directories. The watcher process-group cleanup,
step-152 save/metric predicates, dynamic target propagation, final stock/custom
output labels, and scheduled-task denominators remain internally consistent.
The live trainer was untouched and reached healthy step 127: loss 0.2558,
gradient norm 0.2322, LR 2.30e-6, and 28.1k tokens/s.

At 00:40 UTC, after the supervisor resumed again, all five critical tmux
sessions were reconciled against their actual process trees. The clean trainer
was healthy at step 128 (loss 0.2551, gradient norm 0.2266, zero NaNs, 28.1k
tokens/s, 116.1 GiB/GPU peak), with about 55 minutes projected to step 150.
The step-150 evaluation, post-evaluation benchmark/continuation, dynamic
checkpoint preservation, and finalization watchers were all alive and waiting
on the correct sentinels; port 8200 and every downstream state marker remained
clean. A fresh line-by-line audit of the handoff scripts confirmed unique dev
output directories, the 120-sandbox aggregate concurrency, process-group-owned
inference cleanup, step-152 save-plus-metric gates, and propagation of the
mechanically selected terminal step into preservation and packaging.

At 00:47 UTC, the newly resumed controller re-read the persisted state and
reconciled it with the live host before taking any action. The corrected trainer
was healthy at step 132/150: loss 0.2564, recent-mean loss 0.2600, gradient norm
0.2239, zero NaNs, 28.0k tokens/s recent throughput, and 116.1 GiB/GPU peak
memory. All five trainer/evaluation/continuation/preservation/finalization tmux
sessions remained alive; port 8200 was free, and no development/final output
collision or premature state sentinel existed. A read-only audit of the active
handoff reconfirmed process-group-owned inference and benchmark cleanup, the
real resumable/HF/metric completion predicates, fail-closed selection files,
and dynamic terminal-step propagation. The trainer was not disturbed.

At 00:58 UTC, the active trainer reached step 136/150 with loss 0.2579,
recent-mean loss 0.2552, gradient norm 0.2151, zero NaNs, 27.9k tokens/s recent
throughput, and 116.1 GiB/GPU peak memory. The resumed controller audited the
development launcher, the ten then-staged fixed-sample configs, the precommitted
temperature/review selector, the four save-through execution benchmarks, the
dynamic continuation target calculation, terminal-checkpoint preservation,
finite-tensor packaging gate, and four-way final evaluator. The aggregate
development concurrency remains 120 and final concurrency remains 128; stock
configs leave every custom Pi option unset. No active process or artifact was
changed other than this state record.

At 01:03 UTC, still before step-150 weights or outcomes existed, the development
gate gained a conservative scaffold fallback. The SFT-aligned prompt remains
the default; true stock Pi can replace it only if stock is non-inferior on each
suite, has a positive combined paired advantage, and reaches the same exact
sign-test p<=0.10 threshold. Two fixed-sample stock t=0.6 arms were added so the
comparison uses identical sampling. They run as a separate 64-sandbox wave
after the existing 120-sandbox wave. All six new development/final fallback
configs passed the evaluator's dry resolver, shell/Python/TOML checks passed,
and the dry output directories were removed. Only the sleeping step-150 watcher
was restarted to load the change; the live trainer and other watchers were not
touched.

At 01:11 UTC, the newly resumed controller reconciled the full persisted
campaign with the live host and left all active processes untouched. The
trainer was healthy at step 141/150: loss 0.2536, recent-mean loss 0.2578,
gradient norm 0.2115, zero NaNs, 27.8k tokens/s, and 116.1 GiB/GPU peak memory.
All five handoff sessions remained live and were waiting on the intended
sentinels. A read-only inspection reconfirmed the development selection logic,
the four isolated save-through execution benchmarks, deadline-derived terminal
step, terminal-checkpoint preservation, full finite-tensor packaging gate, and
stock/custom full-suite evaluation. The optional Pi skill path was also
inspected; no new skill was added because the concise custom prompt already
matches every SFT row and introducing an untrained skill-loading pattern before
the paired gate lacked supporting evidence.

At 02:03 UTC, the resumed controller found that step 150 had completed and the
trainer had exited normally. The four-way DP candidate server was healthy and
the precommitted ten-arm development wave was running at the planned aggregate
concurrency of 120; all downstream benchmark, preservation, and finalization
watchers remained live. Early traces were explicitly treated as incomplete,
not as denominators. They nevertheless confirm the broad motivating failure
mode: many step-150 generations still consume the full 4,096-token call budget,
and SWE episodes sometimes emit dozens of tool calls in one assistant turn,
expanding the observation history until context compaction/termination. At
least one early SWE episode had already passed. No selection rule, evaluation
config, active process, or checkpoint was changed after observing these partial
results.

At 02:38 UTC, the controller again reconciled persisted state with the live
host. All ten fixed development evaluators and the four-way step-150 server
were healthy, and all four downstream watcher sessions were present. Each
evaluation had completed roughly 27--33 rollout attempts, including the first
typed retries; the synchronized `SandboxError` entries occurred at the
configured 20-minute episode boundary for the still-running cohort and are not
being interpreted as model outcomes. The server continued returning successful
requests and all four worker health probes passed. No partial score was used,
and no process, config, selection gate, or checkpoint was changed.

Before selection ran, a fail-closed audit found that `select_harness.py`
validated each saved config as 32x1 but did not explicitly require all 32
distinct trace records or identical task keys in each pair. The selector now
raises unless both invariants hold. This leaves the precommitted temperature,
review, scaffold, and significance rules unchanged while preventing a client
crash or mismatched sample from silently influencing selection. Python syntax
and the workspace diff check pass; no active evaluator was touched.

The broker `/resources` endpoint then showed the cause of the synchronized
readiness failures: only 33 of 160 shared sandbox pods were running while 127
were pending, despite nominally free aggregate CPU and memory. The candidate
server was correspondingly idle and healthy. Before the separate stock-t=0.6
wave launched, its width was reduced from 32 to 16 per arm (64 to 32 aggregate)
to avoid repeating this live shared-pool contention. Its exact fixed 32-task
samples, sampling parameters, retry policy, and selection rule are unchanged;
both TOMLs parse and the watcher passes `bash -n`.

At 03:03 UTC, the infrastructure-repair path was reconciled with the live
selector before either had run. `select_harness.py` already waits up to eight
hours for `state/step150-infra-repair-complete`, keeping the step-150 server
alive and preventing unrepaired sandbox failures from entering selection. The
repair backlog was already about 90 episodes, so the repair scheduler was
changed from twelve sequential arms to three batches of four arms. Each arm
still resumes only its same fixed task indices, preserves model/harness errors,
uses an isolated manifest and process group, and retains the six-pass cap. The
largest batch requests at most 56 sandboxes, below the documented 128 ceiling.
`bash -n` passed and only the waiting repair tmux session was restarted; no
active evaluation, model server, trace, score, or selection rule was changed.

The same observed infrastructure failure was removed from the unattended final
path before any final run existed. All sixteen possible final configs now use
8-way TB2 or 16-way SWE concurrency, for 48 aggregate episodes in whichever
precommitted four-run combination is selected. `run_final_evals.sh` then invokes
`repair_eval_infra.sh`, which concurrently resumes only missing rows and typed
sandbox/network errors from the same saved runs and fails closed after six
passes. Other failures are protected and restored exactly at the parsed-record
level; a 32-row semantic prepare/restore round trip passed. All TOMLs parse,
the selected aggregate is 48, both shell scripts pass `bash -n`, and the Python
helpers compile. No final task, checkpoint, or score existed when this was fixed.

At 03:36 UTC, the resumed controller found one idle duplicate repair process
left from the earlier standalone repair watcher.  The active run-2 evaluation
watcher already invokes the same repair inline after both fixed waves; allowing
both copies to wake on the 12 completed directories could have made them
rewrite and resume the same `traces.jsonl` files concurrently.  The duplicate
was confirmed to be sleeping in the all-32-rows wait loop with no repair child,
then terminated without touching the ten active evaluators or inference server.
`repair_step150_infra.sh` now takes an exclusive workspace `flock` before
editing traces, so any future accidental second invocation serializes safely.
The script passes `bash -n`; all ten evaluators remained alive and the candidate
server health endpoint remained 200.  A contemporaneous broker snapshot showed
52 running and 105 pending shared sandbox pods, so the already-running 120-wide
initial wave is expected to leave typed infrastructure retries for the lower-
width repair phase; no score, sample, or selection rule was changed.

At 03:50 UTC, the generic turn-boundary investigation established that the
step-150 checkpoint had learned to emit `<|im_end|>`, while its inherited
generation metadata still named only `<|endoftext|>` as EOS. On the same
task-independent tool-call prompt, default serving consumed the full 512-token
ceiling and parsed two calls; explicitly stopping on token 248046 consumed 152
tokens and returned one valid call. The pre-selection development attempt was
therefore stopped and quarantined as a serving-metadata error without consulting
task rewards. `scripts/ensure_chat_stop_metadata.py` now records both token
248044 and token 248046 as EOS without changing model tensors or tokenizer
entries. A fresh server honored that metadata by default (152 tokens, one call),
and the normal structured smoke completed in 67 tokens. Terminal preservation
and packaging apply and validate the same correction.

At 03:51 UTC, all ten partial pre-fix run-2 directories and their launch logs
were moved under `evals/quarantined-stop-metadata-run2` and
`logs/quarantined-stop-metadata-run2`; none can enter the selector. A clean
step-150 watcher was launched against the identical fixed samples. The twelve
arms now run in four sequential three-arm waves at concurrency 8 per arm, so
aggregate live sandbox demand is 24. Evaluator signal cleanup was also made
idempotent after the deliberate interruption exposed double-wait warnings.
The downstream continuation, preservation, and finalization watchers remain
fail-closed on the clean gate's completion marker.

At 04:00 UTC, the resumed controller reconstructed the campaign from this file
and the live workspace (the supervisor's generic `/workspace/state` path does
not exist in this cell; this workspace's `STATE.md` and `state/` directory are
the persisted ground truth).  The corrected four-way step-150 server was
healthy, its structured smoke used one valid `bash` call and stopped after 67
completion tokens, and the first clean matched wave was running the three TB2
arms at 8-way concurrency each.  The first saved traces were appearing in all
three run-2 directories.  The continuation, terminal-preservation, and final
pipeline watchers were alive and still blocked on the clean evaluation gate;
no selection marker or stale infrastructure-repair marker existed.  No active
process, configuration, task sample, or decision rule was changed during this
recovery audit.

At 04:06 UTC, a score-independent audit fixed one missing-row deadlock in the
not-yet-run step-150 infrastructure repair.  The old precondition waited for
all twelve original directories to contain exactly 32 traces before invoking
the helper whose purpose includes repairing missing traces.  It now waits for
the original evaluator processes to exit, requires each saved resume config,
creates an empty trace file only if an evaluator failed before doing so, and
then lets the existing fixed-index missing/transient repair run.  Model,
harness, context, and task failures remain protected verbatim.  `bash -n`
passes; the active first clean wave was not touched.

At 04:11 UTC, the same not-yet-invoked repair path received the complementary
missing-row fix.  Its per-arm loop already reported both `missing` and
`transient` counts, but only entered `--resume` when the latter was nonzero.
A client crash that left no trace row and no recorded exception could therefore
reach the strict selector unrepaired.  The loop and its terminal assertion now
require both counts to reach zero, matching the already-tested canonical-final
repair controller.  `bash -n` and the Prime-RL workspace diff check pass; no
active evaluator, trace, task sample, or model outcome was changed.

At 04:22 UTC, the resumed controller read the complete persisted campaign and
reconciled it with the live processes before intervening.  The corrected
step-150 server and all four workers were healthy, the structured smoke still
showed a clean 67-token tool turn, and every downstream watcher remained
blocked on its intended sentinel.  The first clean fixed-sample wave had saved
31/32 traces in each of its three TB2 arms; one long episode or typed retry per
arm was still active, while the broker had recovered to 37 running pods and
only one pending pod.  No configuration, trace, task sample, selection rule,
model artifact, or active process was changed.

At 04:32 UTC, another recovery audit verified the step-150 checkpoint directly:
all four indexed BF16 shards contain the expected 760 tensors (9,409,813,744
elements), the terminal loss/gradient/LR metrics are finite, and the corrected
server remains healthy.  The three first-wave TB2 arms are still at 31/32; the
unmatched rollouts began between 03:54 and 04:06 and remain within the broker
runtime's explicit 3,600-second command budget.  They were therefore left
untouched.  The continuation, preservation, packaging, finite-tensor scan, and
four-arm full-suite final pipeline were re-read end to end; their sentinels,
process-group cleanup, selected-step propagation, fixed denominators, and
infrastructure-only retry behavior remain consistent.  The deadline is
2026-08-18 20:27 UTC (about 87.9 hours remained at this audit).

At 04:45 UTC, the resumed controller re-read `STATE.md`, the live `state/`
directory, the evaluation/runbook references, and the active handoff scripts.
The first corrected TB2 wave remained at 31/32 in each matched arm, with all
three original evaluator processes still alive and within the declared
one-hour command budget.  The four-way inference health endpoint reported all
workers healthy, the continuation/preservation/finalization sessions remained
blocked on the clean selection marker, and no downstream state file had been
created prematurely.  The active processes and artifacts were left unchanged.

At 05:09 UTC, a fresh controller reconciled the complete in-run record with the
live host.  The corrected low-temperature TB2 wave had finished all 32 rows in
each matched arm with no missing or typed infrastructure failures, and the
three matched SWE arms had started normally at eight-way concurrency each.
The candidate server and all four workers remained healthy.  A reward-blind
transport audit found 20 stock Pi HarnessErrors versus 4 aligned-no-review and
6 aligned-review errors; the custom residuals were overwhelmingly deliberately
rejected immediately empty turns, while nonempty clean tool-end turns were
accepted as designed.  Median completion length was 258--261 tokens and only
95 of 1,478 TB2 model calls hit the 4,096-token ceiling, confirming that the
portable `<|im_end|>` stop correction removed the earlier runaway-generation
failure.  No evaluation outcome, active process, configuration, selection
rule, checkpoint, or downstream sentinel was changed.

At 05:20 UTC, the resumed controller re-read the persisted workspace state,
evaluation kit, sandbox runbook, active scripts/configs, process tree, and all
current artifacts.  `/workspace/state` does not exist in this cell; `STATE.md`
and this workspace's `state/` directory remain the ground truth.  The low-
temperature TB2 arms and stock/no-review SWE arms were complete at 32/32, while
the review SWE arm was still running at 28/32; the four-way candidate server
and every downstream watcher were healthy and untouched.  Completed scheduled-
task scores were 3/32 stock TB2, 2/32 aligned-no-review TB2, 2/32 aligned-review
TB2, 3/32 stock SWE, and 3/32 aligned-no-review SWE.  These are development
diagnostics only; no partial result changed the precommitted selection rule or
continuation.  The stale high-concurrency command in `EVAL_CHECKLIST.md` was
updated to document the already-active four sequential 24-sandbox waves; no
executable config or process was changed.

At 05:24 UTC, the controller completed the requested ground-truth recovery
from `STATE.md`, `state/`, the current run report/checklists, live process tree,
GPU ownership, and the evaluation/continuation/finalization scripts.  The
persisted campaign and host agree: step 150 is healthy and preserved, its
corrected chat-stop metadata is active, five of the six low-temperature arms
have all 32 fixed traces, and aligned-review SWE has 28/32 with its evaluator
still live inside the configured runtime budget.  The four-way candidate
router reports all workers healthy; the evaluation, continuation,
preservation, and finalization watchers are alive, and no selection or
continuation marker exists prematurely.  No active process, task sample,
configuration, selection rule, or artifact was changed during this recovery.

At 05:33 UTC, the newly resumed controller re-read the complete persisted
record, the evaluation-kit and sandbox-runbook contracts, the active handoff
scripts, current state markers, process tree, GPU ownership, and evaluation
artifacts.  The host still agrees with the record: the four candidate workers
on physical GPUs 4--7 and the router health endpoint are healthy, all four
downstream watcher sessions are alive, five low-temperature arms contain
32/32 traces, and the aligned-review SWE arm remains at 28/32 with its original
evaluator active inside the one-hour command budget.  Neither a selection nor
continuation sentinel exists.  No active process, evaluation record, model
artifact, configuration, task sample, or precommitted decision rule was
changed during recovery.

At 05:41 UTC, the final low-temperature arm had advanced to 31/32 traces.  Its
last retry was waiting on shared sandbox capacity: the broker reported 17
pending versus 3 running pods despite ample aggregate CPU and memory, while
the candidate router still reported all four workers healthy.  Existing typed
retry and fixed-index repair logic covers this infrastructure condition, so no
live process or evaluation protocol was changed.  The manual evaluation
checklist's stale readiness URL was corrected from the DP router's unreliable
`/v1/models` forwarding endpoint to the already-proven `/health` endpoint plus
the subsequent structured-completion smoke; executable automation was already
correct and was untouched.

At 06:15 UTC, the resumed controller reconstructed the campaign from the full
persisted state, current reports/configs/scripts, live process tree, GPU
ownership, and evaluation artifacts.  All six corrected low-temperature arms
had completed 32/32 fixed traces.  The high-temperature TB2 wave was active:
stock had completed 32/32, aligned-no-review had saved 25/32, and
aligned-review had saved 16/32.  The latter two evaluator processes were live
and continuing to save traces; the broker had 64 running and one pending pod,
and the four-way candidate router reported all workers healthy.  The
continuation, terminal-preservation, and finalization watchers remained alive
and blocked on their intended sentinels.  A read-only re-audit confirmed that
the continuation launcher does write `continuation-launched` after atomically
recording its chosen config and target.  No model, trace, task sample,
configuration, precommitted selection rule, or active process was changed.

At 06:21 UTC, a new controller recovered the complete in-run record and audited
the evaluation, continuation, preservation, packaging, and final-suite paths.
The clean step-150 export, portable dual-EOS metadata, four-way inference
server, and all downstream sentinels were consistent and healthy.  Aggregate
low-temperature results remained development-only: stock/aligned/review solved
3/2/2 of 32 TB2 tasks and 3/3/7 of 32 SWE tasks before infrastructure-only
repair.  The aligned scaffold sharply reduced unscored harness failures, while
the precommitted paired significance gate remains the sole selector.  The
high-temperature TB2 wave advanced during the audit from 16 to 26 review traces;
the evaluator and server were actively using the assigned GPUs.  Filesystem
headroom was 3.5 TiB, sufficient for the guarded continuation checkpoints.  No
task-level outcome was used to alter data, training, harness behavior, sampling,
or any selection rule, and no active process was changed.

At 06:38 UTC, the controller again left the live high-temperature TB2 wave
untouched (stock and aligned-no-review were 32/32; aligned-review was 29/32,
with its evaluator alive and the broker reporting no pending pods).  A
task-identity-blind error audit found that every non-infrastructure custom TB2
HarnessError in the completed low-temperature arms was the same immediately
empty first turn (`end_turn`, no tool statuses); the high-temperature custom
arms showed the same residual at lower frequency.  It also confirmed that the
observed custom context failures occur at the backend's exact 65,537-versus-
65,536 token boundary.  Two generic, not-yet-applied hardening candidates are
therefore recorded for implementation only after the paired development wave
has fully exited: one fresh-session retry for that exact empty-first-turn
signature, and a modest advertised context-window safety margin so Pi compacts
before the backend boundary.  Neither change uses task content, accepts an
unchanged workspace as success, or alters stock Pi.  No source/config was
edited while later development arms could still import it.

At 07:24 UTC, the resumed controller reconstructed the full persisted campaign,
read the evaluation and sandbox contracts, and audited the active evaluation,
selection, continuation, preservation, packaging, and final-suite scripts.  The
last high-temperature TB2 retry completed, giving every TB2 arm 32/32 traces,
and the three matched high-temperature SWE arms then started at the planned
aggregate width of 24.  Their evaluators and the four-way candidate server were
healthy, while all downstream watchers remained blocked on their intended
sentinels.  The generic empty-first-turn retry and 512-token advertised-context
safety margin remain deliberately unapplied until the complete development and
infrastructure-repair gate exits, so no saved arm can mix harness versions.  No
active process, trace, task sample, selection rule, checkpoint, or executable
configuration was changed during this recovery.

At 07:52 UTC, the new controller recovered the complete in-run ledger and live
host state.  The final high-temperature SWE wave was healthy and advancing,
with all nine earlier arms complete and every downstream watcher still blocked
on its intended sentinel.  A handoff audit found that the generic final-only
empty-first-turn retry and 512-token context safety margin had been staged in
the submitted patch/configs but the patch was not yet applied to the shared
evaluation checkout.  A fail-closed `agentptb-harness-hardening` watcher now
waits for the complete development selection and infrastructure-repair marker,
waits for the evaluator session to exit, applies and validates only that
task-independent opt-in patch, and writes `state/harness-hardening-applied`.
The finalization path now requires that marker, and its sleeping tmux watcher
was restarted to load the new gate.  Current development and stock configs are
unchanged; no live evaluator imported the new behavior.

At 08:14 UTC, the active controller re-read both authorized operational
references and recovered the full same-run campaign from `STATE.md`, `state/`,
the current scripts/configs, and the live process tree.  The step-150 server and
all four workers remained healthy.  All twelve fixed development arms had 32
scheduled records; the precommitted infrastructure-only repair had already
cleared the first two batches, including the single custom-review t=0.6 TB2
retry, and was moving to the last batch with seven typed transient records.
The continuation, hardening, preservation, and finalization watchers remained
blocked on their intended sentinels.  No task content, evaluation outcome,
active process, model artifact, config, harness source, or precommitted decision
rule was changed during recovery.

At 08:19 UTC, infrastructure-only repair finished with all twelve development
arms containing the same 32 fixed task keys and no missing or transient
episodes.  The precommitted selector chose temperature 0.6, aligned no-review.
At that setting, scheduled-task scores were 1/32 TB2 and 11/32 SWE for aligned
no-review, 3/32 TB2 and 11/32 SWE for aligned review, and 2/32 TB2 and 7/32 SWE
for stock.  Temperature 0.2 failed its gate (combined official-t0.6-only 9
versus low-t0.2-only 2, exact sign p=0.0654, with a TB2 regression); review
failed its selected-temperature significance gate (no-review-only 5 versus
review-only 7, p=0.7744); and stock failed the scaffold gate (selected-custom-
only 6 versus stock-only 3, p=0.5078, with a TB2 regression).  The strong
11/32 SWE signal validates continued broad mixed SFT, while the 32-task TB2
interval remains too wide to support a data-branch change.  Full results are
in `logs/select-harness-step150.json` and the twelve saved summaries.

Only after the clean development gate and its evaluator session exited, the
task-independent final-only harness hardening watcher applied and validated
the opt-in empty-first-turn retry plus 512-token advertised-context safety
margin; `state/harness-hardening-applied` now exists.  The candidate inference
process released GPUs 4--7, and the guarded post-step-150 handoff began its
first isolated save-through execution benchmark at 08:20 UTC.  The proven
full-AC/CPU-Adam baseline resumed step 150 with the correct dataloader and
optimizer state and reached the training loop; no benchmark metric or variant
choice existed yet.  Continuation, preservation, and finalization remain
fail-closed on the benchmark-derived config/target and terminal checkpoint.

At 08:34 UTC, the proven full-AC/CPU-Adam save-through baseline completed both
finite updates and wrote valid distributed and HF step-152 checkpoints.  Its
step-152 loss was 0.2608, gradient norm 0.5164, LR 2.00e-5, peak training memory
116.2 GiB/GPU, and checkpoint-time-adjusted throughput 28.0k tokens/s.  This is
the valid paired reference for the precommitted 3% execution-variant gate.  The
micro-batch-10/global-batch-120 benchmark then resumed the same step-150 state
and entered its first compiled update on GPUs 4--7; no variant choice or dynamic
terminal target exists yet.  The continuation, preservation, and finalization
controllers remain blocked on those mechanically derived records.

At 08:48 UTC, the resumed controller re-read the complete persisted ledger,
both authorized operational references, the live process tree, and the active
benchmark/continuation/finalization scripts.  The micro-batch-10/global-batch-
120 benchmark had completed finite steps and valid resumable plus HF step-152
saves.  Its checkpoint-adjusted throughput was 28,847.85 tokens/s versus
28,002.43 for the paired baseline, a 3.0207% increase that narrowly clears the
precommitted eligibility threshold; peak memory was 137.7 GiB/GPU, step-152
loss 0.2639, gradient norm 0.5215, and LR 2.00e-5.  The independently resumed
micro-batch-12/global-batch-96 benchmark was then loading the same step-150
state on GPUs 4--7.  No execution recipe or terminal target is selected until
all staged variants finish, and no active process, data, score rule, or model
artifact was changed during recovery.

At 09:01 UTC, the micro-batch-12/global-batch-96 benchmark completed two
finite updates and valid distributed plus HF step-152 saves.  Its
checkpoint-adjusted throughput was 26,801.79 tokens/s, below the 28,002.43
baseline, with 159.2 GiB/GPU peak memory, step-152 loss 0.2675, gradient norm
0.5256, and LR 2.00e-5.  It is therefore valid but ineligible for the fixed 3%
speed gate.  The controller released all of its workers and started the
independent selective-activation benchmark from the same step-150 state.  No
execution recipe or terminal target is selected until the remaining staged
variants finish.

During the selective benchmark startup, a read-only inherited
`TORCHINDUCTOR_CACHE_DIR` produced nonfatal graph-cache permission warnings.
Before any long continuation config was launched, all six possible
continuation TOMLs were updated to point TorchInductor at the writable,
node-local `/tmp/agentptb-torchinductor-continuation` directory.  Every TOML
was parsed afterward and the handoff scripts still pass `bash -n`.  Active
benchmark configs and all model/data/optimizer/selection settings were left
unchanged.

The selective variant then required a new Inductor graph and the inherited
cache error became fatal before its first optimizer update.  Before any
variant-selection or continuation marker existed, the controller and all of
that attempt's workers were stopped; the failed output/log were preserved as
`outputs/quarantined-cache-permission-ac-selective-bench` and
`logs/quarantined-cache-permission-sft-ac-selective-bench.log`.  All six
execution-only benchmark TOMLs were pointed at the same writable node-local
cache and parsed successfully.  The identical controller was restarted at
09:08 UTC: it reused the three already-valid save-through arms and immediately
relaunched selective AC from the same step-150 state.  Its resolved config
contains the local override.  This is an infrastructure retry only; no task
reward, training data, model update, throughput result, or selection threshold
was changed.