README.md

Flint x ProgramBench (replicating Cohere Sec 4: solo PIV vs team PIV)

Mirror of the HF bucket cmpatino/flint-programbench. Everything specific to this experiment lives here, not in the swarmlab repo: the plan, prompts, specs, the task subset, the sweep and analysis scripts, probe logs and run outputs.

Read TLDR.md first: the question, the setup and the results on one page.

Results (54 of 60 runs graded)

Per-task reward for the 30 tasks: solo PIV = filled blue circle, team PIV = hollow orange diamond, a grey connector joins a task's pair; tasks are ordered easy / medium / hard by the benchmark's label (results/scores.png, regenerated by scripts/plot_scores.py; the numbers are in results/summary.md).

• Flint does not collaborate. In all 30 team runs the lead never posted an assignment and the workers never made a model call, even with instructions that forbade it from working alone. The team condition is a solo run with idle workers, so the measured difference (team minus solo +0.033 over 26 paired tasks, 95% interval [-0.036, +0.103]) is run-to-run noise.
• Context, not teamwork, is what limits Flint on ProgramBench. It requires its reasoning to be resent with every earlier turn, so the 262k context fills after about 160 steps; 29 of 56 fetched runs ended that way, usually before a build existed, which is where the zero scores come from.
• Absolute performance is low: mean reward 0.22 (solo) and 0.27 (team), no task fully solved, against 0.58 for the paper's GPT-5.6 solo on the full benchmark (a different task set, so not a direct comparison). Total cost about $60: one H200 serving job at $5/h plus cheap CPU jobs.

Layout

• RUNBOOK.md: how to reproduce the experiment from a fresh machine, and what actually happened (timeline, job ids, costs, deviations, current state).
• plan/cohere-sec4-piv-plan.md: the decided design (2026-10-22), later decisions, the pinned model revision.
• probe/: the H200 serving probe (probe.sh) and its two job logs (the first died on a huggingface_hub upgrade in the script; the second succeeded: server ready in ~106 s, tool calls parsed, 514 tok/s aggregate at 4 streams).
• prompts/: the PIV guidance texts (solo, lead, lead-strict, worker), swarmlab's wording of the paper's regime, and lead-template-team.yaml (the lead template variant all team runs used).
• specs/: the condition specs (solo-piv, team-piv) and team-piv-strict__cmatrix.yaml (a one-off cmatrix run with the strict lead text).
• tasks/: the balanced 30-task subset and the metadata table it was drawn from.
• scripts/: programbench_subset.py (task selection), sweep.py (serve, plan, run, fetch and grade, status), analyze.py (results/summary), plot_scores.py (results/scores.png).
• sweep/: manifest.json (every entry: job ids, tags, statuses, scores, attempts/history for relaunches and resets; the serve jobs; the job cost cache), commands.log (every external command with its exit code), retries.jsonl (the manual relaunches), *-status.log and serve*-ready.log (polling logs), specs/ (the 60 materialised specs that ran).
• results/: summary.md, summary.json, scores.png (from analyze.py and plot_scores.py; regenerate after grading).
• runs/: fetched run directories and grades (one per task x condition), and jobs.jsonl, the launch ledger of every job (serve, task and grade).

The bucket also holds what the jobs wrote there (jobs/<tag>/ stage dirs, runs/<tag>/ remote run dirs), which this mirror does not: never sync this directory to the bucket with --delete.