cmpatino's picture
|
download
raw
20.7 kB
# Runbook: Flint x ProgramBench (solo PIV vs team PIV)
How to reproduce this experiment from a fresh machine, and what actually happened when it ran
(2026-10-22/23). The design is in `plan/cohere-sec4-piv-plan.md`; the layout is in `README.md`.
All times are UTC. "Mirror" means this directory, which is a copy of the bucket
`hf://buckets/cmpatino/flint-programbench`.
## 1. Prerequisites
- **Accounts.** A Hugging Face account that can run HF Jobs on flavor `h200` (the serving job),
`cpu-upgrade` (task and grade jobs) and `cpu-xl` (three big tasks). Write access to a bucket
(this run used `hf://buckets/cmpatino/flint-programbench`).
- **`HF_TOKEN`** in the shell, with write access to that bucket. It is passed to every job by
name (`--secrets HF_TOKEN`), is the API key for the served model (jobs proxy), and is what
`hf buckets sync` uses. The hidden tests (`hf://datasets/programbench/ProgramBench-Tests`) are
mounted only in grade jobs.
- **Python 3.12 and uv.** swarmlab needs Python >= 3.12. On this machine uv's default managed
Python location has no exec bit, so `UV_PYTHON_INSTALL_DIR=$AM_LOCAL/uv-python` must be set,
or the env and the wheel that `swarmlab job run` builds fail to install.
- **Code.** GitHub `huggingface/swarmlab`, branch `exp/flint-programbench` at
`699846d21a8d499ba5fa2a194c805e4e49348de1` (main + PR #32 self-served models, + PR #33
`guidance` with the solo template and `guidance: file:`, + the `ContextExceeded` commit).
The first serving job and the first 4 task jobs (step 2, see section 4) ran at `3a77725`,
which lacks only the `ContextExceeded` commit.
- **Model.** `RecursiveSI/flint` (public, Apache 2.0), revision
`2a247be0c0aec895ccb0ac7ca92229ec6df37288`. The repo has exactly one commit ("Initial
release", 2026-10-21T14:24:03Z) and `lastModified` is still 2026-10-21, so this is the
revision every serving job downloaded on 2026-10-22/23 (the serving jobs did not pass
`--revision`; the ledger rows have `"revision": null`). It is served with the SGLang kit
shipped in the model repo, image `lmsysorg/sglang:v0.5.16-cu130`, on one H200
(the kit needs >= 141 GB per GPU; the profile refuses other flavors). Context 262,144 tokens
(`probe/probe2-*.log`: `context_len=262144`).
- **Money.** The agreed ceiling was $100 (plan, "Decisions 2026-10-22 (later)"). The actual
bill was about $60 (section 4.2).
## 2. Step by step
Variables used below (adapt them):
```bash
export AM_LOCAL=/home/node/local # any local, executable disk
export EXP=/data/workspaces/swarmlab/experiments/flint-programbench # the mirror
export ENV=$AM_LOCAL/envs/flint-exp
export UV_PYTHON_INSTALL_DIR=$AM_LOCAL/uv-python
export SWARMLAB_CMD=$ENV/bin/swarmlab # what sweep.py calls
BUCKET=hf://buckets/cmpatino/flint-programbench # default in scripts/sweep.py (BUCKET)
```
### (a) Rebuild the code env
```bash
uv python install 3.12
git clone https://github.com/huggingface/swarmlab $AM_LOCAL/git/wt-flint-exp
git -C $AM_LOCAL/git/wt-flint-exp checkout 699846d21a8d499ba5fa2a194c805e4e49348de1
cd $AM_LOCAL/git/wt-flint-exp
UV_PROJECT_ENVIRONMENT=$ENV uv sync --python 3.12 --extra dev --extra hub --extra anthropic --extra programbench
$ENV/bin/swarmlab --help # sanity check
```
The env used for the run has `programbench 1.2.5`, `huggingface_hub 2.1.1`, `anthropic 1.11.0`.
`scripts/plot_scores.py` also needs matplotlib, which is not in the env: run it with
`uv run --with matplotlib` (step f). Docs to read in the checkout: `docs/guide/real-models.md`
("Self-served models in HF Jobs"), `docs/guide/programbench.md` ("Async team", "Running in HF
Jobs", "Grading", "Documented deviations"), `docs/guide/cli.md`.
### (b) Get the mirror and adapt its paths
```bash
hf buckets sync $BUCKET $EXP --exclude "jobs/*" # the bucket also holds every job's stage dir under jobs/
```
The specs reference the prompt files by **absolute path**
(`file:/data/workspaces/swarmlab/experiments/flint-programbench/prompts/...`), in `specs/*.yaml`
and in all 60 `sweep/specs/*.yaml`. On another machine, rewrite them:
```bash
OLD=/data/workspaces/swarmlab/experiments/flint-programbench
grep -rl "$OLD" $EXP/specs $EXP/sweep/specs | xargs sed -i "s#$OLD#$EXP#g"
```
or regenerate `sweep/specs/` from the condition specs with `sweep.py plan` (below). Note the
difference: the condition specs now carry `timeout_s: 900` on the `served` provider, but 54 of
the 60 materialised specs that actually ran do not (only the 6 retried ones do; section 2e).
To reproduce what ran, keep `sweep/specs/` and only rewrite the paths; to run the corrected
design, regenerate. If the bucket is not `cmpatino/flint-programbench`, change `BUCKET` in
`scripts/sweep.py` (or the `bucket` field of a new manifest) and pass `--bucket` to every
manual `swarmlab job ...` command.
`sweep.py` takes `--exp-dir` (default: the directory above `scripts/`) and runs every command
with the experiment dir as cwd, so the launch ledger is `$EXP/runs/jobs.jsonl`.
### (c) Serve Flint
```bash
cd $EXP
$ENV/bin/python -I scripts/sweep.py serve # prints the command and the GPU ceiling (12 h x $5/h = $60)
$ENV/bin/python -I scripts/sweep.py serve --launch # submits, records the job in sweep/manifest.json "serve"
$ENV/bin/python -I scripts/sweep.py endpoint # repeat until "ready": true (about 3-4 min)
```
`sweep.py` imports swarmlab for `serve` and the analysis scripts import it too, so run them with
the env's python (`$ENV/bin/python`). It runs `swarmlab job serve RecursiveSI/flint --engine sglang
--profile flint --flavor h200 --timeout 12h --bucket $BUCKET`. To pin the model revision, launch
by hand instead (`sweep.py serve` has no revision flag) and record the job id in the manifest:
```bash
$ENV/bin/swarmlab job serve RecursiveSI/flint --engine sglang --profile flint --flavor h200 \
--timeout 12h --revision 2a247be0c0aec895ccb0ac7ca92229ec6df37288 --bucket $BUCKET --launch --json
# then set manifest["serve"] = {"job_id": "<id>", "launched_at": "<UTC>", "timeout": "12h"}
```
`sweep.py serve --launch` refuses when the manifest already has a serve job. To start a new
one, move the old `serve` object into the `serve_history` list (add a `stopped` note) and set
`"serve": {}`; this is how serve job 1 was retired.
The serving job bills $5/h from start to stop, busy or idle. It ends only at its `--timeout`
or with `swarmlab job stop <id>`.
### (d) Plan and run the sweep under the budget
```bash
$ENV/bin/python -I scripts/sweep.py plan # 60 specs in sweep/specs/, manifest entries "planned"
$ENV/bin/python -I scripts/sweep.py run --dry-run # prints the job commands only
$ENV/bin/python -I scripts/sweep.py run --max-concurrent 6 --budget-usd 100 --poll 120
$ENV/bin/python -I scripts/sweep.py status --refresh # budget line, counts, serve cost
```
The main sweep ran exactly `sweep.py run --max-concurrent 6 --budget-usd 100 --poll 120`.
Each task job is `swarmlab job run sweep/specs/<condition>__<task>.yaml --endpoint <serve job>
--bucket $BUCKET --arm <condition> --seeds 0 --launch --json` on `cpu-upgrade` (7 h timeout).
`plan` keeps the status of existing entries; `run` resumes from the manifest.
Manifest semantics (`scripts/sweep.py` docstring): statuses go `planned -> running ->
done|failed -> fetched -> graded`. `--stop-at N` bounds the total number of entries ever
launched (any status but `planned`), across invocations; `--max-concurrent` is the cap on jobs
running at once. `--retry-failed` moves a failed, ungraded entry's job fields into its
`attempts` list and sets it back to `planned`.
Budget guard: `run`, `fetch` and `serve --launch` refuse a launch when spent + worst case of
running jobs + the next job's worst case > `--budget-usd`. **Pitfall:** `hf jobs inspect` gives
no running time for a CANCELED job, so the manifest caches every stopped serve job with
`secs: 0` (see `sweep/manifest.json` `jobs`). After a serve job is stopped, its GPU cost
drops out of "spent", and the guard undercounts by that amount. Keep the GPU total by hand
(section 4.2) and lower `--budget-usd` accordingly.
**Never run `plan`, `run`, `fetch` (or `status --refresh`) concurrently**: they all rewrite
`sweep/manifest.json`, and the last writer wins.
### (e) Known failure modes, handled as they were
1. **Provider request timeout (4 solo jobs: dsq, code-minimap, rust-sloth, zstd).** The
`served` provider's default request timeout is 90 s; Flint's long reasoning answers on a
shared server exceeded it and the job ended ERROR. Fix: add `timeout_s: 900` to the
`served` provider params of the spec, then relaunch (below).
2. **OOM on cpu-upgrade (team codesnap, team php-src; then solo rust-sloth on its first retry).**
Exit code 137, `OOMKilled`. Fix: relaunch with `--flavor cpu-xl` ($1.00/h in the ledger).
3. **Context overflow.** Before `699846d`, a context-length error from the server crashed the
job (solo shellharden at step 2). With `699846d` it ends the agent's episode as
`ContextExceeded` and the run stays gradable. No action needed.
The relaunches were done by hand, one job per entry, against a fresh serving job:
```bash
$ENV/bin/swarmlab job run $EXP/sweep/specs/solo-piv__multiprocessio__dsq.c3ae0ba.yaml \
--endpoint <serve job id> --bucket $BUCKET --arm solo-piv --seeds 0 --launch --json
$ENV/bin/swarmlab job run $EXP/sweep/specs/team-piv__php__php-src.c891263.yaml \
--endpoint <serve job id> --bucket $BUCKET --arm team-piv --seeds 0 --flavor cpu-xl --launch --json
```
and folded into the manifest by hand: the failed job's `job_id`, `job_stage`, `tag` go into the
entry's `attempts` list; the entry gets the new `job_id`, `tag`, `flavor`, `launched_at` and
status `running` (then `done` when the job completes; `sweep.py run --once` polls it). Each
relaunch is also a line in `sweep/retries.jsonl`. (`sweep.py run --retry-failed` is the
scripted equivalent for the timeout case, but it cannot pass `--flavor`.) Entries reset for
other reasons keep the old fields under `history` with `reason` and `reset_at` (team cmatrix:
re-run with the lead template variant; solo shellharden: crashed on context length before
`699846d`).
### (f) Fetch, grade, analyse, plot
```bash
$ENV/bin/python -I scripts/sweep.py fetch --budget-usd 100 # every done entry: fetch, then grade --launch --wait
$ENV/bin/python -I scripts/analyze.py # results/summary.{md,json}
UV_PROJECT_ENVIRONMENT=$ENV uv run --project $AM_LOCAL/git/wt-flint-exp --with matplotlib \
python -I scripts/plot_scores.py # results/scores.png
```
`fetch` runs `swarmlab job fetch <tag>/<run_id> --out runs --bucket $BUCKET` and then
`swarmlab programbench grade runs/<run_id> --bucket $BUCKET --launch --wait --json` one entry
at a time (a big task waits up to an hour or more). It is resumable: an interrupted `fetch`
leaves the entry `fetched` and the next `fetch` grades it. A grade job that ends without a
`grade.json` leaves `grade_error` on the entry and the status `fetched`. If a `--wait` was cut
but the grade job finished, `swarmlab programbench fetch-grade runs/<run_id>` picks up the score
(the manifest then needs `score/passed/total/grade_job/status: graded` by hand).
### (g) Stop the server and sync
```bash
$ENV/bin/swarmlab job stop <serve job id> # as soon as no task job needs it
$ENV/bin/swarmlab job status <serve job id> # CANCELED
find $EXP -name __pycache__ -o -name .ruff_cache | xargs rm -rf
hf buckets sync $EXP $BUCKET # needs HF_TOKEN with write access
```
**Do not sync the mirror to the bucket root with `--delete`.** The bucket is also the jobs'
bucket: `jobs/<tag>/` holds every job's stage dir (and each grade job's `grade.json`), and
`runs/<tag>/` holds every task job's remote run dir. The mirror has neither (its `runs/` holds
the fetched copies under the run id). A `--delete` sync would delete them, including the only
copies of runs not yet fetched. To remove a stale path from the bucket, sync an empty
directory onto that prefix only.
## 3. Pitfalls (summary)
- Absolute prompt paths in the specs (step b).
- The bucket is a default in `scripts/sweep.py` (`BUCKET`), stored in the manifest's `bucket`.
- `hf buckets sync` needs `HF_TOKEN` with write access; never `--delete` onto the bucket root.
- The serving job bills $5/h until stopped; stop it by hand (`swarmlab job stop`). Its timeout
(12 h) is only a ceiling.
- The budget guard forgets stopped serve jobs (`secs: 0`), step d.
- `--stop-at` counts every entry ever launched, across invocations; `--retry-failed` resets
only failed, ungraded entries.
- Never run `plan`/`run`/`fetch` concurrently; `serve --launch` refuses while `serve` is set.
- Retries on `cpu-xl` and manifest resets were manual; `sweep.py` alone does not reproduce them.
- The 54 specs that ran first lack `timeout_s: 900`; regenerated specs have it (step b).
## 4. What actually happened
### 4.1 Timeline (UTC)
| when | what | evidence |
|---|---|---|
| 2026-10-21 14:24 | `RecursiveSI/flint` released, commit `2a247be` (its only commit) | HF API |
| 2026-10-22 11:59, 12:03 | H200 probe jobs `6ad9fa8d...` (died on a huggingface_hub upgrade in the script) and `6ad9fb9c...` (server ready, tool calls parsed, 514 tok/s at 4 streams). Not in the ledger. | `probe/` |
| 10-22 13:37 | Serve job 1 `6ada117bc191903149a71704` (code `3a77725`), ready 13:40 | `sweep/serve-ready.log`, ledger |
| 10-22 13:46-14:06 | Step 2 (pipeline proof): solo cmatrix `6ada1390...` (graded 0.947, kept), team cmatrix `6ada13f4...` (graded 0.966, later reset), solo shellharden `6ada1834...` (ERROR: context length; reset) | manifest `history`, `commands.log` |
| 10-22 14:36-15:10 | Team cmatrix with `prompts/piv-lead-strict.txt` and the official lead template (`specs/team-piv-strict__cmatrix.yaml`), job `6ada1f81...`, grade job `6ada2819...`; not a manifest entry, run dir `runs/pb-team-piv-strict-cmatrix-3a77725-20261022-143558/` | ledger, `sweep/strict-status.log` |
| 10-22 ~15:35 | Serve job 1 stopped by hand, moved to `serve_history` | manifest `serve_history[0].stopped` |
| 10-22 16:01 | Commit `699846d` (`ContextExceeded`) | git |
| 10-22 16:05 | Serve job 2 `6ada3417925af0e27d4f4e59` (12 h timeout), ready 16:09:45 | `sweep/serve2-ready.log` |
| 10-22 16:09-22:53 | Main sweep `sweep.py run --max-concurrent 6 --budget-usd 100 --poll 120`: 59 task jobs launched (all entries but solo cmatrix), last launch 22:53 (team typst); last job done 00:12 (team rumdl) | ledger, `commands.log` |
| 10-23 ~00:13 | Serve job 2 stopped by hand: 29,212 s = 8.11 GPU h, $40.57 at the last check (00:12:52) | manifest `serve` |
| 10-23 00:15 | Serve job 3 `6adaa715925af0e27d4f8e3b` (3 h timeout) for the retries | ledger |
| 10-23 00:20-00:29 | 6 relaunches: dsq, code-minimap, rust-sloth, zstd (solo, `timeout_s: 900`, cpu-upgrade); codesnap, php-src (team, cpu-xl) | `sweep/retries.jsonl` |
| 10-23 00:34 | rust-sloth relaunch `6adaa928...` ERROR (OOM on cpu-upgrade); the other 5 completed by 01:12 | `sweep/retries-status.log` |
| 10-23 ~01:13 | Serve job 3 stopped by hand (last poll 01:13:03) | `commands.log` |
| 10-23 01:14-01:39 | Serve job 4 `6adab4d5c191903149a7886e` (2 h timeout), ready 01:17:26; rust-sloth on cpu-xl `6adab5d5...` 01:19-01:39 COMPLETED; serve job 4 stopped 01:39:17 | `sweep/retries-status.log` |
| 10-22 14:21 - 10-23 16:51 | Grading with `sweep.py fetch` (step-2 runs on 10-22, the rest from 10-23 00:14). The log pauses at 01:39-07:52, 09:34-10:50 (marked `fetch resumed 10:50`) and after 16:51 (machine restarts; resumed each time). 57 grade jobs in the ledger by then. | `commands.log`, ledger |
| 10-23 15:45-16:33 | Grade job `6adb80fc...` for team php-src ended ERROR with no `grade.json` (`grade_error` in the manifest) | manifest |
| 10-23 16:51 | Grading stopped at 54/60 graded | manifest |
| 2026-11-02 16:21 | `commands.log` says `fetch resumed 2026-11-02T16:21`; a `swarmlab programbench grade runs/pb-team-piv-php--php-src.c891263__team-piv__s0 --launch --wait` process started 16:21:28 and launched grade job `6ae8b9acc78863f9c630bae1` at 16:26 (RUNNING when this runbook was written). Its parent `sweep.py` process was no longer running, so the manifest will not record its result by itself. | `commands.log`, ledger, `ps` |
### 4.2 Jobs and cost
From `runs/jobs.jsonl` (132 rows on 2026-11-02: 4 serve, 70 task, 58 grade, the last one running) and the manifest's
`jobs` cache (running seconds x the ledger's list price). The serve jobs' hours come from the
manifest and the logs, because canceled jobs report no running time.
| item | jobs | hours | cost |
|---|---|---|---|
| Serve job 1 `6ada117b...` (13:37:02 - ~15:35) | 1 | ~1.97 | ~$9.8 at list price; the manifest note says "~$6" (not resolved) |
| Serve job 2 `6ada3417...` | 1 | 8.11 | $40.57 |
| Serve job 3 `6adaa715...` (00:15:30 - ~01:13) | 1 | ~0.96 | ~$4.8 (estimate) |
| Serve job 4 `6adab4d5...` (01:14:01 - 01:39:17) | 1 | 0.42 | ~$2.1 |
| Task jobs, cpu-upgrade ($0.03/h), incl. failed and step-2/strict | 67 | 43.5 | $1.30 |
| Task jobs, cpu-xl ($1.00/h) | 3 | 1.58 | $1.58 |
| Grade jobs, cpu-upgrade (57 by 10-23; `6ae8b9ac...` running) | 57 | 7.53 | $0.23 |
| Probe jobs (2 x h200, minutes each) | 2 | - | not in the ledger |
| **Total** | | | **~$56-60** (budget $100) |
GPU time dominates. The billing page of the HF account is the authority for the serve jobs.
### 4.3 Deviations from the official ProgramBench run
- **Solo prompt:** the official solo template plus the PIV paragraph as `guidance`
(`prompts/piv-solo.txt`; PR #33). Not the official solo run (`specs/solo-piv.yaml` comment).
- **Lead template variant:** all 30 team runs use `prompts/lead-template-team.yaml`, swarmlab's
lead template with one rule changed for the lead only: "Your response MUST include AT LEAST ONE
bash tool call" becomes "AT LEAST ONE tool call (bash, post, read_board or wait_for_workers)".
The first team cmatrix run (`6ada13f4...`, 0.966) predates the variant and was reset and re-run.
- **Team shape:** async, 1 lead + 4 workers (fixed pool), pull delivery, `rooms` topology; lead
1000 steps, workers 300, `rounds_per_agent: 120` (`specs/team-piv.yaml`). PIV texts are
swarmlab's wording (`prompts/`), not the paper's App B.
- **`ContextExceeded`** (`699846d`): a context-length error ends the episode instead of
crashing the job. Flint runs with a 262,144-token context, `max_tokens 32768`, temperature at
the server default (1.0), and its reasoning is kept in the history (`history_reasoning: true`,
which Flint requires), so long episodes fill the context.
- **Provider timeout:** 90 s default request timeout for 54 of the 60 runs; 900 s for the 6
relaunched ones (`sweep/specs/*` with `timeout_s`).
- **Resources:** `cpu-upgrade` (8 vCPU, 32 GB) vs the official 20 CPU / 60 GB; team codesnap,
team php-src and solo rust-sloth ran on `cpu-xl`.
- **Network:** DNS blackhole + pinned hosts (Hub, Xet, the serving endpoint) + sudo removed,
instead of `--network none` (`docs/guide/programbench.md`, "Documented deviations").
- **Grading:** the submitted code (git seed, `compile.sh`, `eval/run.sh`) runs as the
unprivileged `agent` user; test branches share one job instead of a fresh container each;
scores are not cheat-filtered (`docs/guide/programbench.md`, "Grading").
- **One run per task and condition** (seed 0), 30 balanced tasks chosen by
`scripts/programbench_subset.py` (the paper's Balanced-30 ids are not published).
### 4.4 Current state (2026-11-02)
- Manifest: 60 entries; **54 graded** (28 solo, 26 team). Not graded: team php-src (fetched;
its first grade job failed without `grade.json`; a second grade job `6ae8b9ac...` was launched
2026-11-02 16:26, see 4.1), team rumdl (fetched, never graded), solo and team skeema, solo
and team typst (`done`: their run dirs exist only in the bucket under `runs/<tag>/`).
- Grading was paused by the operator after 2026-10-23 16:51; the 2026-11-02 grade job above
is the only activity since.
- `results/summary.md|json` (2026-10-23 07:53) and `results/scores.png` (10:51) are stale:
they cover 21 graded runs. Re-run `analyze.py` and `plot_scores.py` once grading is done.
From the manifest alone (not the analysis): mean reward solo 0.222 (28 runs), team 0.266
(26 runs); 26 paired tasks, team better on 13, solo on 6; nothing resolved.
- First-run findings: **no delegation in any team run** (no `post` tool call in the 28 fetched
team runs' `events.jsonl`); **29 of the 56 fetched runs contain a `ContextExceeded` event**
(16 solo, 13 team).

Xet Storage Details

Size:
20.7 kB
·
Xet hash:
7db19a557c0d11cad5fbd02c5b025a0b47365bf4c862fa4eaa17ddf646b637f3

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.