TLDR.md

TL;DR: does Flint collaborate? A ProgramBench team-vs-solo experiment (October 2026)

Question. Cohere's paper "Scaling Inference-Time Compute with Teams of Coding Agents" (arXiv 2610.17471) reports that a lead agent delegating to workers beats a solo agent on ProgramBench, at 3-7x the cost, and that whether a model delegates at all is a property of the model. We asked that question of RecursiveSI/flint, a 40B-parameter open-weights mixture-of-experts model (2.5B active) released on 2026-10-21: given the paper's plan-implement-verify ("PIV") instructions, does a team of Flint agents beat one Flint agent?

Benchmark. ProgramBench (Meta, arXiv 2605.03546): 200 tasks, each a black-box reference program plus its documentation; the agent must rebuild the program from scratch and is scored by hidden behavioural tests (reward = fraction of tests passed). The official agent is mini-swe-agent with a single bash tool. We used 30 tasks, 10 easy / 10 medium / 10 hard by the benchmark's own labels, spread across languages (tasks/tasks-balanced-30.txt).

Harness. swarmlab (github.com/huggingface/swarmlab) runs ProgramBench as a "world": the same image, limits, bash tool and official evaluator as the benchmark, with mini-swe-agent's own loop as the participant. For the team shape, agents run concurrently with no round barrier (commit: async): a lead and 4 workers share one workspace; the lead assigns work by posting to a worker's channel, a worker that has nothing to do waits at zero cost, and wait_for_workers returns the reports. Flint was served from its own SGLang kit on one H200 GPU (HF Jobs, $5/h); each task ran in its own CPU job inside the task image, calling that server. Code: branch exp/flint-programbench at commit 699846d. Procedure and every job id: RUNBOOK.md, sweep/manifest.json.

Conditions. solo-piv: one agent, the official prompt plus the PIV paragraph. team-piv: lead + 4 workers, all Flint, the PIV instructions for a lead and for workers (prompts/), one run per task. Budget: a hard $100 ceiling; actual spend about $60.

Results (54 of 60 runs graded; the last six hard-task grades were not finished when this was written; one is in progress).

	solo PIV	team PIV
runs graded	28	26
mean reward (tests passed / total)	0.222	0.266
tasks fully solved	0	0
runs that submitted a program	12 of 28	15 of 26
runs that ended by context overflow	16 of 28	13 of 28
mean wall time per run	35 min	46 min
assignments posted by the lead	-	0 in all 30 runs
worker model calls	-	0

Paired over the 26 tasks graded in both conditions: team minus solo = +0.033, 95% bootstrap interval [-0.036, +0.103], sign test p = 0.17. Per-task picture: results/scores.png.

What it means.

1. Flint does not delegate. In 30 team runs, with instructions that told it to use its workers in every phase, and in a variant that forbade it from working alone, the lead never posted one assignment and the workers never ran. The "team" condition is a solo run with four idle spectators, so the small reward difference above is run-to-run noise (the same task, re-run, varied by up to 0.04 with identical behaviour). This matches the paper's finding that delegation is a model trait: Opus 4.7 rarely delegated, Haiku 4.5 did readily; Flint is at the far end of "never".
2. The binding limit for Flint on this benchmark is context, not collaboration. Flint requires its reasoning to be sent back with every earlier turn; mini-swe-agent keeps the whole transcript; so each step adds roughly 1.3k tokens and the 262k context fills after about 160 model calls, far short of the 1000-step limit. Half of all runs ended that way, usually before a build existed, which is where the zero scores come from.
3. Absolute performance is low: no task fully solved, mean reward about 0.25, against 0.58 for the paper's GPT-5.6 solo PIV on the full benchmark. Different task subset, so not a direct comparison.

Deviations from the official ProgramBench run (all documented in plan/ and the specs): the PIV paragraph appended to the official solo prompt; for the lead only, the rule "every reply must include a bash call" relaxed to "a tool call", because Flint followed it literally and could not post; a context-overflow error ends the episode and the workspace is graded as is (the official agent would crash, and the benchmark grades the workspace either way); the provider request timeout raised to 15 minutes for the 6 re-run jobs (the other 54 ran with the 90 s default); two memory-hungry tasks re-run on a larger CPU flavor; submitted code runs as an unprivileged user and with DNS blackholed during grading.

Where everything is. Bucket hf://buckets/cmpatino/flint-programbench (public): this file, RUNBOOK.md, the plan, prompts, specs, task list, scripts, the manifest and logs, every fetched run with its transcript and grade, results/. Written by Carlos Patiño's swarmlab agent session, 2026-10-22 to 2026-11-02.