Buckets:
| # TL;DR: does Flint collaborate? A ProgramBench team-vs-solo experiment (October 2026) | |
| **Question.** Cohere's paper "Scaling Inference-Time Compute with Teams of Coding Agents" | |
| (arXiv 2610.17471) reports that a lead agent delegating to workers beats a solo agent on | |
| ProgramBench, at 3-7x the cost, and that whether a model delegates at all is a property of the | |
| model. We asked that question of `RecursiveSI/flint`, a 40B-parameter open-weights | |
| mixture-of-experts model (2.5B active) released on 2026-10-21: given the paper's | |
| plan-implement-verify ("PIV") instructions, does a team of Flint agents beat one Flint agent? | |
| **Benchmark.** ProgramBench (Meta, arXiv 2605.03546): 200 tasks, each a black-box reference | |
| program plus its documentation; the agent must rebuild the program from scratch and is scored by | |
| hidden behavioural tests (reward = fraction of tests passed). The official agent is mini-swe-agent | |
| with a single `bash` tool. We used 30 tasks, 10 easy / 10 medium / 10 hard by the benchmark's own | |
| labels, spread across languages (`tasks/tasks-balanced-30.txt`). | |
| **Harness.** swarmlab (github.com/huggingface/swarmlab) runs ProgramBench as a "world": the same | |
| image, limits, bash tool and official evaluator as the benchmark, with mini-swe-agent's own loop | |
| as the participant. For the team shape, agents run concurrently with no round barrier (`commit: | |
| async`): a lead and 4 workers share one workspace; the lead assigns work by posting to a worker's | |
| channel, a worker that has nothing to do waits at zero cost, and `wait_for_workers` returns the | |
| reports. Flint was served from its own SGLang kit on one H200 GPU (HF Jobs, $5/h); each task ran in | |
| its own CPU job inside the task image, calling that server. Code: branch `exp/flint-programbench` | |
| at commit `699846d`. Procedure and every job id: `RUNBOOK.md`, `sweep/manifest.json`. | |
| **Conditions.** `solo-piv`: one agent, the official prompt plus the PIV paragraph. `team-piv`: | |
| lead + 4 workers, all Flint, the PIV instructions for a lead and for workers | |
| (`prompts/`), one run per task. Budget: a hard $100 ceiling; actual spend about $60. | |
| **Results (54 of 60 runs graded; the last six hard-task grades were not finished when this was written; one is in progress).** | |
| | | solo PIV | team PIV | | |
| |---|---|---| | |
| | runs graded | 28 | 26 | | |
| | mean reward (tests passed / total) | 0.222 | 0.266 | | |
| | tasks fully solved | 0 | 0 | | |
| | runs that submitted a program | 12 of 28 | 15 of 26 | | |
| | runs that ended by context overflow | 16 of 28 | 13 of 28 | | |
| | mean wall time per run | 35 min | 46 min | | |
| | assignments posted by the lead | - | **0 in all 30 runs** | | |
| | worker model calls | - | **0** | | |
| Paired over the 26 tasks graded in both conditions: team minus solo = +0.033, 95% bootstrap | |
| interval [-0.036, +0.103], sign test p = 0.17. Per-task picture: `results/scores.png`. | |
| **What it means.** | |
| 1. **Flint does not delegate.** In 30 team runs, with instructions that told it to use its | |
| workers in every phase, and in a variant that forbade it from working alone, the lead never | |
| posted one assignment and the workers never ran. The "team" condition is a solo run with four | |
| idle spectators, so the small reward difference above is run-to-run noise (the same task, | |
| re-run, varied by up to 0.04 with identical behaviour). This matches the paper's finding that | |
| delegation is a model trait: Opus 4.7 rarely delegated, Haiku 4.5 did readily; Flint is at the | |
| far end of "never". | |
| 2. **The binding limit for Flint on this benchmark is context, not collaboration.** Flint requires | |
| its reasoning to be sent back with every earlier turn; mini-swe-agent keeps the whole | |
| transcript; so each step adds roughly 1.3k tokens and the 262k context fills after about 160 | |
| model calls, far short of the 1000-step limit. Half of all runs ended that way, usually before | |
| a build existed, which is where the zero scores come from. | |
| 3. **Absolute performance is low**: no task fully solved, mean reward about 0.25, against 0.58 | |
| for the paper's GPT-5.6 solo PIV on the full benchmark. Different task subset, so not a direct | |
| comparison. | |
| **Deviations from the official ProgramBench run** (all documented in `plan/` and the specs): the | |
| PIV paragraph appended to the official solo prompt; for the lead only, the rule "every reply must | |
| include a bash call" relaxed to "a tool call", because Flint followed it literally and could not | |
| post; a context-overflow error ends the episode and the workspace is graded as is (the official | |
| agent would crash, and the benchmark grades the workspace either way); the provider request | |
| timeout raised to 15 minutes for the 6 re-run jobs (the other 54 ran with the 90 s default); two memory-hungry tasks re-run on a larger CPU flavor; submitted | |
| code runs as an unprivileged user and with DNS blackholed during grading. | |
| **Where everything is.** Bucket `hf://buckets/cmpatino/flint-programbench` (public): this file, | |
| `RUNBOOK.md`, the plan, prompts, specs, task list, scripts, the manifest and logs, every fetched | |
| run with its transcript and grade, `results/`. Written by Carlos Patiño's swarmlab agent session, | |
| 2026-10-22 to 2026-11-02. | |
Xet Storage Details
- Size:
- 5.12 kB
- Xet hash:
- cbbd44c8bd0a403622caf23f5e1501a7c8dab0d66f10a472163b9e790155e295
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.