RUNBOOK.md

Runbook: Flint x ProgramBench (solo PIV vs team PIV)

How to reproduce this experiment from a fresh machine, and what actually happened when it ran (2026-10-22/23). The design is in plan/cohere-sec4-piv-plan.md; the layout is in README.md. All times are UTC. "Mirror" means this directory, which is a copy of the bucket hf://buckets/cmpatino/flint-programbench.

1. Prerequisites

• Accounts. A Hugging Face account that can run HF Jobs on flavor h200 (the serving job), cpu-upgrade (task and grade jobs) and cpu-xl (three big tasks). Write access to a bucket (this run used hf://buckets/cmpatino/flint-programbench).
• HF_TOKEN in the shell, with write access to that bucket. It is passed to every job by name (--secrets HF_TOKEN), is the API key for the served model (jobs proxy), and is what hf buckets sync uses. The hidden tests (hf://datasets/programbench/ProgramBench-Tests) are mounted only in grade jobs.
• Python 3.12 and uv. swarmlab needs Python >= 3.12. On this machine uv's default managed Python location has no exec bit, so UV_PYTHON_INSTALL_DIR=$AM_LOCAL/uv-python must be set, or the env and the wheel that swarmlab job run builds fail to install.
• Code. GitHub huggingface/swarmlab, branch exp/flint-programbench at 699846d21a8d499ba5fa2a194c805e4e49348de1 (main + PR #32 self-served models, + PR #33 guidance with the solo template and guidance: file:, + the ContextExceeded commit). The first serving job and the first 4 task jobs (step 2, see section 4) ran at 3a77725, which lacks only the ContextExceeded commit.
• Model. RecursiveSI/flint (public, Apache 2.0), revision 2a247be0c0aec895ccb0ac7ca92229ec6df37288. The repo has exactly one commit ("Initial release", 2026-10-21T14:24:03Z) and lastModified is still 2026-10-21, so this is the revision every serving job downloaded on 2026-10-22/23 (the serving jobs did not pass --revision; the ledger rows have "revision": null). It is served with the SGLang kit shipped in the model repo, image lmsysorg/sglang:v0.5.16-cu130, on one H200 (the kit needs >= 141 GB per GPU; the profile refuses other flavors). Context 262,144 tokens (probe/probe2-*.log: context_len=262144).
• Money. The agreed ceiling was $100 (plan, "Decisions 2026-10-22 (later)"). The actual bill was about $60 (section 4.2).

2. Step by step

Variables used below (adapt them):

export AM_LOCAL=/home/node/local                        # any local, executable disk
export EXP=/data/workspaces/swarmlab/experiments/flint-programbench   # the mirror
export ENV=$AM_LOCAL/envs/flint-exp
export UV_PYTHON_INSTALL_DIR=$AM_LOCAL/uv-python
export SWARMLAB_CMD=$ENV/bin/swarmlab                   # what sweep.py calls
BUCKET=hf://buckets/cmpatino/flint-programbench         # default in scripts/sweep.py (BUCKET)

(a) Rebuild the code env

uv python install 3.12
git clone https://github.com/huggingface/swarmlab $AM_LOCAL/git/wt-flint-exp
git -C $AM_LOCAL/git/wt-flint-exp checkout 699846d21a8d499ba5fa2a194c805e4e49348de1
cd $AM_LOCAL/git/wt-flint-exp
UV_PROJECT_ENVIRONMENT=$ENV uv sync --python 3.12 --extra dev --extra hub --extra anthropic --extra programbench
$ENV/bin/swarmlab --help            # sanity check

The env used for the run has programbench 1.2.5, huggingface_hub 2.1.1, anthropic 1.11.0. scripts/plot_scores.py also needs matplotlib, which is not in the env: run it with uv run --with matplotlib (step f). Docs to read in the checkout: docs/guide/real-models.md ("Self-served models in HF Jobs"), docs/guide/programbench.md ("Async team", "Running in HF Jobs", "Grading", "Documented deviations"), docs/guide/cli.md.

(b) Get the mirror and adapt its paths

hf buckets sync $BUCKET $EXP --exclude "jobs/*"    # the bucket also holds every job's stage dir under jobs/

The specs reference the prompt files by absolute path (file:/data/workspaces/swarmlab/experiments/flint-programbench/prompts/...), in specs/*.yaml and in all 60 sweep/specs/*.yaml. On another machine, rewrite them:

OLD=/data/workspaces/swarmlab/experiments/flint-programbench
grep -rl "$OLD" $EXP/specs $EXP/sweep/specs | xargs sed -i "s#$OLD#$EXP#g"

or regenerate sweep/specs/ from the condition specs with sweep.py plan (below). Note the difference: the condition specs now carry timeout_s: 900 on the served provider, but 54 of the 60 materialised specs that actually ran do not (only the 6 retried ones do; section 2e). To reproduce what ran, keep sweep/specs/ and only rewrite the paths; to run the corrected design, regenerate. If the bucket is not cmpatino/flint-programbench, change BUCKET in scripts/sweep.py (or the bucket field of a new manifest) and pass --bucket to every manual swarmlab job ... command.

sweep.py takes --exp-dir (default: the directory above scripts/) and runs every command with the experiment dir as cwd, so the launch ledger is $EXP/runs/jobs.jsonl.

(c) Serve Flint

cd $EXP
$ENV/bin/python -I scripts/sweep.py serve             # prints the command and the GPU ceiling (12 h x $5/h = $60)
$ENV/bin/python -I scripts/sweep.py serve --launch      # submits, records the job in sweep/manifest.json "serve"
$ENV/bin/python -I scripts/sweep.py endpoint            # repeat until "ready": true (about 3-4 min)

sweep.py imports swarmlab for serve and the analysis scripts import it too, so run them with the env's python ($ENV/bin/python). It runs swarmlab job serve RecursiveSI/flint --engine sglang --profile flint --flavor h200 --timeout 12h --bucket $BUCKET. To pin the model revision, launch by hand instead (sweep.py serve has no revision flag) and record the job id in the manifest:

$ENV/bin/swarmlab job serve RecursiveSI/flint --engine sglang --profile flint --flavor h200 \
  --timeout 12h --revision 2a247be0c0aec895ccb0ac7ca92229ec6df37288 --bucket $BUCKET --launch --json
# then set manifest["serve"] = {"job_id": "<id>", "launched_at": "<UTC>", "timeout": "12h"}

sweep.py serve --launch refuses when the manifest already has a serve job. To start a new one, move the old serve object into the serve_history list (add a stopped note) and set "serve": {}; this is how serve job 1 was retired.

The serving job bills $5/h from start to stop, busy or idle. It ends only at its --timeout or with swarmlab job stop <id>.

(d) Plan and run the sweep under the budget

$ENV/bin/python -I scripts/sweep.py plan                # 60 specs in sweep/specs/, manifest entries "planned"
$ENV/bin/python -I scripts/sweep.py run --dry-run       # prints the job commands only
$ENV/bin/python -I scripts/sweep.py run --max-concurrent 6 --budget-usd 100 --poll 120
$ENV/bin/python -I scripts/sweep.py status --refresh    # budget line, counts, serve cost

The main sweep ran exactly sweep.py run --max-concurrent 6 --budget-usd 100 --poll 120. Each task job is swarmlab job run sweep/specs/<condition>__<task>.yaml --endpoint <serve job> --bucket $BUCKET --arm <condition> --seeds 0 --launch --json on cpu-upgrade (7 h timeout). plan keeps the status of existing entries; run resumes from the manifest.

Manifest semantics (scripts/sweep.py docstring): statuses go planned -> running -> done|failed -> fetched -> graded. --stop-at N bounds the total number of entries ever launched (any status but planned), across invocations; --max-concurrent is the cap on jobs running at once. --retry-failed moves a failed, ungraded entry's job fields into its attempts list and sets it back to planned.

Budget guard: run, fetch and serve --launch refuse a launch when spent + worst case of running jobs + the next job's worst case > --budget-usd. Pitfall: hf jobs inspect gives no running time for a CANCELED job, so the manifest caches every stopped serve job with secs: 0 (see sweep/manifest.json jobs). After a serve job is stopped, its GPU cost drops out of "spent", and the guard undercounts by that amount. Keep the GPU total by hand (section 4.2) and lower --budget-usd accordingly.

Never run plan, run, fetch (or status --refresh) concurrently: they all rewrite sweep/manifest.json, and the last writer wins.

(e) Known failure modes, handled as they were

1. Provider request timeout (4 solo jobs: dsq, code-minimap, rust-sloth, zstd). The served provider's default request timeout is 90 s; Flint's long reasoning answers on a shared server exceeded it and the job ended ERROR. Fix: add timeout_s: 900 to the served provider params of the spec, then relaunch (below).
2. OOM on cpu-upgrade (team codesnap, team php-src; then solo rust-sloth on its first retry). Exit code 137, OOMKilled. Fix: relaunch with --flavor cpu-xl ($1.00/h in the ledger).
3. Context overflow. Before 699846d, a context-length error from the server crashed the job (solo shellharden at step 2). With 699846d it ends the agent's episode as ContextExceeded and the run stays gradable. No action needed.

The relaunches were done by hand, one job per entry, against a fresh serving job:

$ENV/bin/swarmlab job run $EXP/sweep/specs/solo-piv__multiprocessio__dsq.c3ae0ba.yaml \
  --endpoint <serve job id> --bucket $BUCKET --arm solo-piv --seeds 0 --launch --json
$ENV/bin/swarmlab job run $EXP/sweep/specs/team-piv__php__php-src.c891263.yaml \
  --endpoint <serve job id> --bucket $BUCKET --arm team-piv --seeds 0 --flavor cpu-xl --launch --json

and folded into the manifest by hand: the failed job's job_id, job_stage, tag go into the entry's attempts list; the entry gets the new job_id, tag, flavor, launched_at and status running (then done when the job completes; sweep.py run --once polls it). Each relaunch is also a line in sweep/retries.jsonl. (sweep.py run --retry-failed is the scripted equivalent for the timeout case, but it cannot pass --flavor.) Entries reset for other reasons keep the old fields under history with reason and reset_at (team cmatrix: re-run with the lead template variant; solo shellharden: crashed on context length before 699846d).

(f) Fetch, grade, analyse, plot

$ENV/bin/python -I scripts/sweep.py fetch --budget-usd 100     # every done entry: fetch, then grade --launch --wait
$ENV/bin/python -I scripts/analyze.py                          # results/summary.{md,json}
UV_PROJECT_ENVIRONMENT=$ENV uv run --project $AM_LOCAL/git/wt-flint-exp --with matplotlib \
  python -I scripts/plot_scores.py                              # results/scores.png

fetch runs swarmlab job fetch <tag>/<run_id> --out runs --bucket $BUCKET and then swarmlab programbench grade runs/<run_id> --bucket $BUCKET --launch --wait --json one entry at a time (a big task waits up to an hour or more). It is resumable: an interrupted fetch leaves the entry fetched and the next fetch grades it. A grade job that ends without a grade.json leaves grade_error on the entry and the status fetched. If a --wait was cut but the grade job finished, swarmlab programbench fetch-grade runs/<run_id> picks up the score (the manifest then needs score/passed/total/grade_job/status: graded by hand).

(g) Stop the server and sync

$ENV/bin/swarmlab job stop <serve job id>      # as soon as no task job needs it
$ENV/bin/swarmlab job status <serve job id>    # CANCELED
find $EXP -name __pycache__ -o -name .ruff_cache | xargs rm -rf
hf buckets sync $EXP $BUCKET                   # needs HF_TOKEN with write access

Do not sync the mirror to the bucket root with --delete. The bucket is also the jobs' bucket: jobs/<tag>/ holds every job's stage dir (and each grade job's grade.json), and runs/<tag>/ holds every task job's remote run dir. The mirror has neither (its runs/ holds the fetched copies under the run id). A --delete sync would delete them, including the only copies of runs not yet fetched. To remove a stale path from the bucket, sync an empty directory onto that prefix only.

3. Pitfalls (summary)

• Absolute prompt paths in the specs (step b).
• The bucket is a default in scripts/sweep.py (BUCKET), stored in the manifest's bucket.
• hf buckets sync needs HF_TOKEN with write access; never --delete onto the bucket root.
• The serving job bills $5/h until stopped; stop it by hand (swarmlab job stop). Its timeout (12 h) is only a ceiling.
• The budget guard forgets stopped serve jobs (secs: 0), step d.
• --stop-at counts every entry ever launched, across invocations; --retry-failed resets only failed, ungraded entries.
• Never run plan/run/fetch concurrently; serve --launch refuses while serve is set.
• Retries on cpu-xl and manifest resets were manual; sweep.py alone does not reproduce them.
• The 54 specs that ran first lack timeout_s: 900; regenerated specs have it (step b).

4. What actually happened

4.1 Timeline (UTC)

when	what	evidence
2026-10-21 14:24	RecursiveSI/flint released, commit 2a247be (its only commit)	HF API
2026-10-22 11:59, 12:03	H200 probe jobs 6ad9fa8d... (died on a huggingface_hub upgrade in the script) and 6ad9fb9c... (server ready, tool calls parsed, 514 tok/s at 4 streams). Not in the ledger.	probe/
10-22 13:37	Serve job 1 6ada117bc191903149a71704 (code 3a77725), ready 13:40	sweep/serve-ready.log, ledger
10-22 13:46-14:06	Step 2 (pipeline proof): solo cmatrix 6ada1390... (graded 0.947, kept), team cmatrix 6ada13f4... (graded 0.966, later reset), solo shellharden 6ada1834... (ERROR: context length; reset)	manifest history, commands.log
10-22 14:36-15:10	Team cmatrix with prompts/piv-lead-strict.txt and the official lead template (specs/team-piv-strict__cmatrix.yaml), job 6ada1f81..., grade job 6ada2819...; not a manifest entry, run dir runs/pb-team-piv-strict-cmatrix-3a77725-20261022-143558/	ledger, sweep/strict-status.log
10-22 ~15:35	Serve job 1 stopped by hand, moved to serve_history	manifest serve_history[0].stopped
10-22 16:01	Commit 699846d (ContextExceeded)	git
10-22 16:05	Serve job 2 6ada3417925af0e27d4f4e59 (12 h timeout), ready 16:09:45	sweep/serve2-ready.log
10-22 16:09-22:53	Main sweep sweep.py run --max-concurrent 6 --budget-usd 100 --poll 120: 59 task jobs launched (all entries but solo cmatrix), last launch 22:53 (team typst); last job done 00:12 (team rumdl)	ledger, commands.log
10-23 ~00:13	Serve job 2 stopped by hand: 29,212 s = 8.11 GPU h, $40.57 at the last check (00:12:52)	manifest serve
10-23 00:15	Serve job 3 6adaa715925af0e27d4f8e3b (3 h timeout) for the retries	ledger
10-23 00:20-00:29	6 relaunches: dsq, code-minimap, rust-sloth, zstd (solo, timeout_s: 900, cpu-upgrade); codesnap, php-src (team, cpu-xl)	sweep/retries.jsonl
10-23 00:34	rust-sloth relaunch 6adaa928... ERROR (OOM on cpu-upgrade); the other 5 completed by 01:12	sweep/retries-status.log
10-23 ~01:13	Serve job 3 stopped by hand (last poll 01:13:03)	commands.log
10-23 01:14-01:39	Serve job 4 6adab4d5c191903149a7886e (2 h timeout), ready 01:17:26; rust-sloth on cpu-xl 6adab5d5... 01:19-01:39 COMPLETED; serve job 4 stopped 01:39:17	sweep/retries-status.log
10-22 14:21 - 10-23 16:51	Grading with sweep.py fetch (step-2 runs on 10-22, the rest from 10-23 00:14). The log pauses at 01:39-07:52, 09:34-10:50 (marked fetch resumed 10:50) and after 16:51 (machine restarts; resumed each time). 57 grade jobs in the ledger by then.	commands.log, ledger
10-23 15:45-16:33	Grade job 6adb80fc... for team php-src ended ERROR with no grade.json (grade_error in the manifest)	manifest
10-23 16:51	Grading stopped at 54/60 graded	manifest
2026-11-02 16:21	commands.log says fetch resumed 2026-11-02T16:21; a swarmlab programbench grade runs/pb-team-piv-php--php-src.c891263__team-piv__s0 --launch --wait process started 16:21:28 and launched grade job 6ae8b9acc78863f9c630bae1 at 16:26 (RUNNING when this runbook was written). Its parent sweep.py process was no longer running, so the manifest will not record its result by itself.	commands.log, ledger, ps

4.2 Jobs and cost

From runs/jobs.jsonl (132 rows on 2026-11-02: 4 serve, 70 task, 58 grade, the last one running) and the manifest's jobs cache (running seconds x the ledger's list price). The serve jobs' hours come from the manifest and the logs, because canceled jobs report no running time.

item	jobs	hours	cost
Serve job 1 6ada117b... (13:37:02 - ~15:35)	1	~1.97	$9.8 at list price; the manifest note says "$6" (not resolved)
Serve job 2 6ada3417...	1	8.11	$40.57
Serve job 3 6adaa715... (00:15:30 - ~01:13)	1	~0.96	~$4.8 (estimate)
Serve job 4 6adab4d5... (01:14:01 - 01:39:17)	1	0.42	~$2.1
Task jobs, cpu-upgrade ($0.03/h), incl. failed and step-2/strict	67	43.5	$1.30
Task jobs, cpu-xl ($1.00/h)	3	1.58	$1.58
Grade jobs, cpu-upgrade (57 by 10-23; 6ae8b9ac... running)	57	7.53	$0.23
Probe jobs (2 x h200, minutes each)	2	-	not in the ledger
Total			~$56-60 (budget $100)

GPU time dominates. The billing page of the HF account is the authority for the serve jobs.

4.3 Deviations from the official ProgramBench run

• Solo prompt: the official solo template plus the PIV paragraph as guidance (prompts/piv-solo.txt; PR #33). Not the official solo run (specs/solo-piv.yaml comment).
• Lead template variant: all 30 team runs use prompts/lead-template-team.yaml, swarmlab's lead template with one rule changed for the lead only: "Your response MUST include AT LEAST ONE bash tool call" becomes "AT LEAST ONE tool call (bash, post, read_board or wait_for_workers)". The first team cmatrix run (6ada13f4..., 0.966) predates the variant and was reset and re-run.
• Team shape: async, 1 lead + 4 workers (fixed pool), pull delivery, rooms topology; lead 1000 steps, workers 300, rounds_per_agent: 120 (specs/team-piv.yaml). PIV texts are swarmlab's wording (prompts/), not the paper's App B.
• ContextExceeded (699846d): a context-length error ends the episode instead of crashing the job. Flint runs with a 262,144-token context, max_tokens 32768, temperature at the server default (1.0), and its reasoning is kept in the history (history_reasoning: true, which Flint requires), so long episodes fill the context.
• Provider timeout: 90 s default request timeout for 54 of the 60 runs; 900 s for the 6 relaunched ones (sweep/specs/* with timeout_s).
• Resources: cpu-upgrade (8 vCPU, 32 GB) vs the official 20 CPU / 60 GB; team codesnap, team php-src and solo rust-sloth ran on cpu-xl.
• Network: DNS blackhole + pinned hosts (Hub, Xet, the serving endpoint) + sudo removed, instead of --network none (docs/guide/programbench.md, "Documented deviations").
• Grading: the submitted code (git seed, compile.sh, eval/run.sh) runs as the unprivileged agent user; test branches share one job instead of a fresh container each; scores are not cheat-filtered (docs/guide/programbench.md, "Grading").
• One run per task and condition (seed 0), 30 balanced tasks chosen by scripts/programbench_subset.py (the paper's Balanced-30 ids are not published).

4.4 Current state (2026-11-02)

• Manifest: 60 entries; 54 graded (28 solo, 26 team). Not graded: team php-src (fetched; its first grade job failed without grade.json; a second grade job 6ae8b9ac... was launched 2026-11-02 16:26, see 4.1), team rumdl (fetched, never graded), solo and team skeema, solo and team typst (done: their run dirs exist only in the bucket under runs/<tag>/).
• Grading was paused by the operator after 2026-10-23 16:51; the 2026-11-02 grade job above is the only activity since.
• results/summary.md|json (2026-10-23 07:53) and results/scores.png (10:51) are stale: they cover 21 graded runs. Re-run analyze.py and plot_scores.py once grading is done. From the manifest alone (not the analysis): mean reward solo 0.222 (28 runs), team 0.266 (26 runs); 26 paired tasks, team better on 13, solo on 6; nothing resolved.
• First-run findings: no delegation in any team run (no post tool call in the 28 fetched team runs' events.jsonl); 29 of the 56 fetched runs contain a ContextExceeded event (16 solo, 13 team).