results/summary.md

Flint x ProgramBench: solo PIV vs team PIV

Serve job 6ada3417925af0e27d4f4e59: 8.11 GPU h ($40.57); tokens are priced $0.

Per condition

condition	planned	graded	reward	% resolved	% almost	wall min	model calls	tool calls	workers	delegations	reports	worker share	format err	CPU job h	GPU h (est.)	submitted by
solo-piv	30	28	0.222	0.0	0.0	34.6	151.0	174.5	-	-	-	-	0.04	16.35	3.49	solo 12, none 16
team-piv	30	26	0.266	0.0	0.0	46.0	158.4	403.2	0	0	0	0.00	0	21.20	4.63	lead 15, none 13

Means are over runs; team-only columns are - for solo. GPU h per condition splits the serve job's hours by summed run wall time.

Paired comparison (team - solo, per task)

Tasks with both graded: 26; mean diff 0.033; 95% bootstrap CI [-0.036, 0.103]; sign test: team better 13, solo better 6, ties 7, p = 0.167.

task	difficulty	solo	team	diff
abishekvashok__cmatrix.5c082c6	easy	0.947	0.949	0.002
anordal__shellharden.6a6ffd4	easy	0.332	0.448	0.116
arthursonzogni__json-tui.17a22b6	easy	0.426	0.531	0.105
cslarsen__jp2a.61d205f	easy	0.217	0.000	-0.217
kisielk__errcheck.dacab89	easy	0.512	0.403	-0.109
multiprocessio__dsq.c3ae0ba	easy	0.554	0.507	-0.046
rbakbashev__elfcat.52f8cc7	easy	0.000	0.000	0.000
sclevine__yj.8016400	easy	0.000	0.000	0.000
sirwart__ripsecrets.34c9e03	easy	0.000	0.439	0.439
wfxr__code-minimap.0ddeea5	easy	0.000	0.000	0.000
brocode__fblog.3b54330	medium	0.000	0.000	0.000
codesnap-rs__codesnap.f81e4f3	medium	0.468	0.235	-0.233
ecumene__rust-sloth.051c559	medium	0.121	0.000	-0.121
facebook__zstd.1168da0	medium	0.000	0.323	0.323
go-critic__go-critic.9aea378	medium	0.312	0.349	0.037
madler__pigz.fe4894f	medium	0.000	0.403	0.403
nikolassv__bartib.6b9b5ce	medium	0.000	0.000	0.000
ninja-build__ninja.cc60300	medium	0.000	0.000	0.000
peco__peco.4e58dad	medium	0.648	0.687	0.040
sharkdp__fd.40d8eb3	medium	0.400	0.000	-0.400
canop__broot.d6c798e	hard	0.608	0.628	0.020
dandavison__delta.acd758f	hard	0.000	0.000	0.000
doxygen__doxygen.966d98e	hard	0.188	0.248	0.060
jesseduffield__lazygit.1d0db51	hard	0.255	0.382	0.127
johnkerl__miller.8d85b46	hard	0.057	0.063	0.005
parcel-bundler__lightningcss.aa2ed1e	hard	0.000	0.309	0.309
php__php-src.c891263	hard	0.000	-	-
rvben__rumdl.2d75c4d	hard	0.165	-	-
skeema__skeema.6a76243	hard	-	-	-
typst__typst.88356d0	hard	-	-	-

Missing or ungraded (6)

• team-piv__php__php-src.c891263__s0: not graded (status fetched, job 6adaaa5b925af0e27d4f9009)
• team-piv__rvben__rumdl.2d75c4d__s0: not graded (status fetched, job 6ada8c38925af0e27d4f7e11)
• solo-piv__skeema__skeema.6a76243__s0: no run dir (status done, job 6ada8d9f925af0e27d4f7ea1)
• team-piv__skeema__skeema.6a76243__s0: no run dir (status done, job 6ada8e0a925af0e27d4f7ed8)
• solo-piv__typst__typst.88356d0__s0: no run dir (status done, job 6ada8e77c191903149a76c13)
• team-piv__typst__typst.88356d0__s0: no run dir (status done, job 6ada93bf925af0e27d4f8291)