code/fastfill/v2/ops/README.md

fastfill/v2/ops — operations and final-analysis scripts (2026-10-08)

Server-side scripts used to rebuild data, run the autopilot and analyse the final model. The isambard_* scripts and run_plan.sh run from any checkout; the others assume the old server checkout /home/jovyan/shanliantian/fastfill. Nothing here holds credentials — the LLM key lived only in /home/jovyan/shanliantian/.fastfill_api.env on that server.

Operations

• rebuild_and_publish.sh <name> (Mac): full data chain (legacy_build → legacy_verify → review_data → qualified_data → multisource_verify) from a pinned worktree (SRC=), HF upload, NEXT_DATA.json; NO_HANDOFF=1 holds the server handoff until the reports are checked. Stops at the first failing verifier (set -euo pipefail).
• start_autorun_retrain.sh: retrain on new data with logged --set overrides (model.max_objects=256, loss.yaw_cls=0.5), --name-suffix, LLM step disabled (no paid calls).
• run_checkpoint.sh <model-step> <tag>: five-room RoomGenBench hand-off on CPU with --require-placement (the autopilot calls it; runs from any checkout). It rebuilds runs/roomgenbench/requests from the scenes and builder on disk each time (a differing old directory is set aside as requests.stale-*) and records them in <tag>/requests.sha256.
• Isambard-AI (aarch64 GH200, Slurm): from a checkout under $PROJECTDIR, bash fastfill/v2/ops/isambard_submit.sh submits isambard_setup.sbatch (conda env with the baseline's package versions, pinned data, backbone and LLM rows, tests) and, once it succeeds, two isambard_autorun.sbatch jobs (one node each): yaw_cls 0.5 and the 0.08 control, both from configs/main7-cell05-main-20261007b-e5.json (the baseline's configuration) with max_objects 256. Status in runs/autorun-yawcls05/ and runs/autorun-yawcls008/, job logs in runs/isambard/. Resubmitting an autorun job resumes it.
• run_structured.sh, rescore_llm.sh: the OptiScene-style LLM modes on the frozen 300 rows and re-scoring of saved LLM predictions with the current evaluator (no API calls); run_reference.sh: reference check of a 5-room export.

Final analysis on Isambard-AI (after both autopilot arms; one node, 4 GPUs, at most 12 h)

• First sacct -j <setup>,<yawcls05>,<yawcls008> -o JobID,JobName,State,ExitCode and git log -1: the setup of 2cd4f47 fails its test step on Isambard (fixed in 3806325), and arms queued with afterok on a failed setup never start; then pull main and run isambard_submit.sh again. Pull only while no arm's STATUS.json phase is F-* (a change of a fastfill/v2/*.py file during phase F stops it). Submit from the checkout with the job ids of the arms that will actually run (once both runs/autorun-yawcls05/STATUS.json and runs/autorun-yawcls008/STATUS.json say done / done-with-failures the dependency can be dropped): sbatch --dependency=afterok:<yawcls05 job>:<yawcls008 job> fastfill/v2/ops/isambard_analysis.sbatch Unattended only if both arm jobs exit 0 within their 24 h: an arm that fails or times out (no automatic resubmission; resubmitting resumes it) leaves the analysis job pending; after resubmitting that arm run scontrol update JobId=<ff-analysis job> Dependency=afterok:<new ids> (or scancel it and submit again). It runs run_plan.sh all (inputs → gpu → cpu → report) from the checkout; the report is runs/analysis/final/report.html, the job log runs/isambard/ff-analysis-<job>.out, step times runs/analysis/walltime.tsv. Resubmitting resumes (finished outputs are kept; all of them are redone if the chosen checkpoints changed). Single steps by hand only inside an allocation (srun --nodes=1 --gpus=4 ...), not on a login node.
• isambard_inputs.py (step inputs): refuses unless both arms are done / done-with-failures with the same select2 cohort bytes scored by the same fastfill/v2 code (implementation_sha256); final = the arm with the lower select2 score (STATUS best.score), control = the other. Downloads with the submitting shell's HF_HOME / login, every file sha256-pinned: the old baseline (liantian/fastfill-v2-models, used from runs/baseline-hf/ whose model_config.json names this checkout's models/Qwen3-8B, recorded in BACKBONE_REWRITE.json), the saved LLM answers (prompt, harness → runs/llm-{prompt,harness}-300/; the structured modes only if runs/llm-structured*-300 exist, else the report says they are missing) and the old data data/main-20261007b. Rebuilds the baseline's old selection cohorts with Autopilot.eval_data. Writes runs/analysis/inputs.json.
• The report compares final vs baseline (≤ 128-object rooms), final vs control (the evidence for the choice), spread vs argmax per model, final vs the LLM answers re-scored by the evaluator on disk (no API calls), the ablations and the RoomGenBench reference checks; validation decides, test is report-only.
• isambard_dryrun.sh WORK PYTHON NEW_DATA OLD_DATA LLM_DIR ROOMGENBENCH: CPU end-to-end dry run of the whole job in a throwaway checkout (tiny-backbone stand-ins, both arms made by the real autopilot, DRYRUN=1: no downloads).

Analysis scripts (make_sample.sh is the old server's CPU sample test of 2026-10-08, kept as a record)

• stratify.py: per-object errors split by target selection rule (old vs added targets), place, source and object count, reproducing evaluate's report exactly; validator collision / clean-room rates per stratum.
• compare.py: paired room-level A−B (bootstrap CI of the mean per-room difference as the primary criterion, Holm sign test as robustness); refuses different rows or different decoding code.
• llm_compare.py: FastFill (raw / spread) vs the four LLM modes on the frozen 300 rows; transport failures, latency and token columns.
• sample_validation.py, diag_ablation.py, selftest_shift.py, crosscheck.py, report.py (self-contained HTML). Decisions (spread vs argmax, replacing the baseline) read validation comparisons only; test sections are report-only.