# fastfill/v2/ops — operations and final-analysis scripts (2026-10-08)

Server-side scripts used to rebuild data, run the autopilot and analyse the final model. The isambard_* scripts and
run_plan.sh run from any checkout; the others assume the old server checkout `/home/jovyan/shanliantian/fastfill`.
Nothing here holds credentials — the LLM key lived only in `/home/jovyan/shanliantian/.fastfill_api.env` on that server.

Operations
- `rebuild_and_publish.sh <name>` (Mac): full data chain (legacy_build → legacy_verify → review_data → qualified_data
  → multisource_verify) from a pinned worktree (`SRC=`), HF upload, `NEXT_DATA.json`; `NO_HANDOFF=1` holds the server
  handoff until the reports are checked. Stops at the first failing verifier (set -euo pipefail).
- `start_autorun_retrain.sh`: retrain on new data with logged `--set` overrides (`model.max_objects=256`,
  `loss.yaw_cls=0.5`), `--name-suffix`, LLM step disabled (no paid calls).
- `run_checkpoint.sh <model-step> <tag>`: five-room RoomGenBench hand-off on CPU with `--require-placement`
  (the autopilot calls it; runs from any checkout). It rebuilds `runs/roomgenbench/requests` from the scenes and builder
  on disk each time (a differing old directory is set aside as `requests.stale-*`) and records them in `<tag>/requests.sha256`.
- Isambard-AI (aarch64 GH200, Slurm): from a checkout under `$PROJECTDIR`, `bash fastfill/v2/ops/isambard_submit.sh`
  submits `isambard_setup.sbatch` (conda env with the baseline's package versions, pinned data, backbone and LLM rows,
  tests) and, once it succeeds, two `isambard_autorun.sbatch` jobs (one node each): yaw_cls 0.5 and the 0.08 control,
  both from `configs/main7-cell05-main-20261007b-e5.json` (the baseline's configuration) with `max_objects` 256.
  Status in `runs/autorun-yawcls05/` and `runs/autorun-yawcls008/`, job logs in `runs/isambard/`. Resubmitting an
  autorun job resumes it.
- `run_structured.sh`, `rescore_llm.sh`: the OptiScene-style LLM modes on the frozen 300 rows and re-scoring of saved
  LLM predictions with the current evaluator (no API calls); `run_reference.sh`: reference check of a 5-room export.

Final analysis on Isambard-AI (after both autopilot arms; one node, 4 GPUs, at most 12 h)
- First `sacct -j <setup>,<yawcls05>,<yawcls008> -o JobID,JobName,State,ExitCode` and `git log -1`: the setup of
  2cd4f47 fails its test step on Isambard (fixed in 3806325), and arms queued with afterok on a failed setup never
  start; then pull main and run `isambard_submit.sh` again. Pull only while no arm's STATUS.json phase is F-* (a change
  of a fastfill/v2/*.py file during phase F stops it). Submit from the checkout with the job ids of the arms that will actually run (once
  both `runs/autorun-yawcls05/STATUS.json` and `runs/autorun-yawcls008/STATUS.json` say done / done-with-failures the
  dependency can be dropped):
  `sbatch --dependency=afterok:<yawcls05 job>:<yawcls008 job> fastfill/v2/ops/isambard_analysis.sbatch`
  Unattended only if both arm jobs exit 0 within their 24 h: an arm that fails or times out (no automatic resubmission;
  resubmitting resumes it) leaves the analysis job pending; after resubmitting that arm run
  `scontrol update JobId=<ff-analysis job> Dependency=afterok:<new ids>` (or scancel it and submit again).
  It runs `run_plan.sh all` (inputs → gpu → cpu → report) from the checkout; the report is
  `runs/analysis/final/report.html`, the job log `runs/isambard/ff-analysis-<job>.out`, step times
  `runs/analysis/walltime.tsv`. Resubmitting resumes (finished outputs are kept; all of them are redone if the chosen
  checkpoints changed). Single steps by hand only inside an allocation (`srun --nodes=1 --gpus=4 ...`), not on a login node.
- `isambard_inputs.py` (step inputs): refuses unless both arms are done / done-with-failures with the same select2 cohort
  bytes scored by the same fastfill/v2 code (implementation_sha256); final = the arm with the lower select2 score
  (STATUS best.score), control = the other. Downloads with the submitting shell's HF_HOME / login, every file sha256-pinned: the old baseline (`liantian/fastfill-v2-models`, used from
  `runs/baseline-hf/` whose model_config.json names this checkout's `models/Qwen3-8B`, recorded in BACKBONE_REWRITE.json),
  the saved LLM answers (prompt, harness → `runs/llm-{prompt,harness}-300/`; the structured modes only if
  `runs/llm-structured*-300` exist, else the report says they are missing) and the old data `data/main-20261007b`.
  Rebuilds the baseline's old selection cohorts with `Autopilot.eval_data`. Writes `runs/analysis/inputs.json`.
- The report compares final vs baseline (≤ 128-object rooms), final vs control (the evidence for the choice), spread vs
  argmax per model, final vs the LLM answers re-scored by the evaluator on disk (no API calls), the ablations and the
  RoomGenBench reference checks; validation decides, test is report-only.
- `isambard_dryrun.sh WORK PYTHON NEW_DATA OLD_DATA LLM_DIR ROOMGENBENCH`: CPU end-to-end dry run of the whole job in a
  throwaway checkout (tiny-backbone stand-ins, both arms made by the real autopilot, `DRYRUN=1`: no downloads).

Analysis scripts (`make_sample.sh` is the old server's CPU sample test of 2026-10-08, kept as a record)
- `stratify.py`: per-object errors split by target selection rule (old vs added targets), place, source and object
  count, reproducing `evaluate`'s report exactly; validator collision / clean-room rates per stratum.
- `compare.py`: paired room-level A−B (bootstrap CI of the mean per-room difference as the primary criterion, Holm
  sign test as robustness); refuses different rows or different decoding code.
- `llm_compare.py`: FastFill (raw / spread) vs the four LLM modes on the frozen 300 rows; transport failures,
  latency and token columns.
- `sample_validation.py`, `diag_ablation.py`, `selftest_shift.py`, `crosscheck.py`, `report.py` (self-contained HTML).
Decisions (spread vs argmax, replacing the baseline) read validation comparisons only; test sections are report-only.