Buckets:
| # Flint x ProgramBench (replicating Cohere Sec 4: solo PIV vs team PIV) | |
| Mirror of the HF bucket `cmpatino/flint-programbench`. Everything specific to this experiment lives | |
| here, not in the swarmlab repo: the plan, prompts, specs, the task subset, the sweep and analysis | |
| scripts, probe logs and run outputs. | |
| Read `TLDR.md` first: the question, the setup and the results on one page. | |
| ## Results (54 of 60 runs graded) | |
|  | |
| Per-task reward for the 30 tasks: solo PIV = filled blue circle, team PIV = hollow orange diamond, | |
| a grey connector joins a task's pair; tasks are ordered easy / medium / hard by the benchmark's | |
| label (`results/scores.png`, regenerated by `scripts/plot_scores.py`; the numbers are in | |
| `results/summary.md`). | |
| - **Flint does not collaborate.** In all 30 team runs the lead never posted an assignment and the | |
| workers never made a model call, even with instructions that forbade it from working alone. The | |
| team condition is a solo run with idle workers, so the measured difference (team minus solo | |
| +0.033 over 26 paired tasks, 95% interval [-0.036, +0.103]) is run-to-run noise. | |
| - **Context, not teamwork, is what limits Flint on ProgramBench.** It requires its reasoning to be | |
| resent with every earlier turn, so the 262k context fills after about 160 steps; 29 of 56 | |
| fetched runs ended that way, usually before a build existed, which is where the zero scores come | |
| from. | |
| - **Absolute performance is low**: mean reward 0.22 (solo) and 0.27 (team), no task fully solved, | |
| against 0.58 for the paper's GPT-5.6 solo on the full benchmark (a different task set, so not a | |
| direct comparison). Total cost about $60: one H200 serving job at $5/h plus cheap CPU jobs. | |
| ## Layout | |
| - `RUNBOOK.md`: how to reproduce the experiment from a fresh machine, and what actually happened | |
| (timeline, job ids, costs, deviations, current state). | |
| - `plan/cohere-sec4-piv-plan.md`: the decided design (2026-10-22), later decisions, the pinned | |
| model revision. | |
| - `probe/`: the H200 serving probe (`probe.sh`) and its two job logs (the first died on a | |
| huggingface_hub upgrade in the script; the second succeeded: server ready in ~106 s, tool calls | |
| parsed, 514 tok/s aggregate at 4 streams). | |
| - `prompts/`: the PIV guidance texts (solo, lead, lead-strict, worker), swarmlab's wording of the | |
| paper's regime, and `lead-template-team.yaml` (the lead template variant all team runs used). | |
| - `specs/`: the condition specs (solo-piv, team-piv) and `team-piv-strict__cmatrix.yaml` (a | |
| one-off cmatrix run with the strict lead text). | |
| - `tasks/`: the balanced 30-task subset and the metadata table it was drawn from. | |
| - `scripts/`: `programbench_subset.py` (task selection), `sweep.py` (serve, plan, run, fetch and | |
| grade, status), `analyze.py` (results/summary), `plot_scores.py` (results/scores.png). | |
| - `sweep/`: `manifest.json` (every entry: job ids, tags, statuses, scores, `attempts`/`history` | |
| for relaunches and resets; the serve jobs; the job cost cache), `commands.log` (every external | |
| command with its exit code), `retries.jsonl` (the manual relaunches), `*-status.log` and | |
| `serve*-ready.log` (polling logs), `specs/` (the 60 materialised specs that ran). | |
| - `results/`: `summary.md`, `summary.json`, `scores.png` (from analyze.py and plot_scores.py; | |
| regenerate after grading). | |
| - `runs/`: fetched run directories and grades (one per task x condition), and `jobs.jsonl`, the | |
| launch ledger of every job (serve, task and grade). | |
| The bucket also holds what the jobs wrote there (`jobs/<tag>/` stage dirs, `runs/<tag>/` remote run | |
| dirs), which this mirror does not: never sync this directory to the bucket with `--delete`. | |
Xet Storage Details
- Size:
- 3.83 kB
- Xet hash:
- 6960911bbde7798a239a74532f38a29fe9db968d97a52f1ab57d8445324db0b4
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.