Buckets:
| # Flint x ProgramBench: solo PIV vs team PIV | |
| Serve job 6ada3417925af0e27d4f4e59: 8.11 GPU h ($40.57); tokens are priced $0. | |
| ## Per condition | |
| | condition | planned | graded | reward | % resolved | % almost | wall min | model calls | tool calls | workers | delegations | reports | worker share | format err | CPU job h | GPU h (est.) | submitted by | | |
| |---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---| | |
| | solo-piv | 30 | 28 | 0.222 | 0.0 | 0.0 | 34.6 | 151.0 | 174.5 | - | - | - | - | 0.04 | 16.35 | 3.49 | solo 12, none 16 | | |
| | team-piv | 30 | 26 | 0.266 | 0.0 | 0.0 | 46.0 | 158.4 | 403.2 | 0 | 0 | 0 | 0.00 | 0 | 21.20 | 4.63 | lead 15, none 13 | | |
| Means are over runs; team-only columns are `-` for solo. GPU h per condition splits the serve job's hours by summed run wall time. | |
| ## Paired comparison (team - solo, per task) | |
| Tasks with both graded: 26; mean diff 0.033; 95% bootstrap CI [-0.036, 0.103]; sign test: team better 13, solo better 6, ties 7, p = 0.167. | |
| | task | difficulty | solo | team | diff | | |
| |---|---|---|---|---| | |
| | abishekvashok__cmatrix.5c082c6 | easy | 0.947 | 0.949 | 0.002 | | |
| | anordal__shellharden.6a6ffd4 | easy | 0.332 | 0.448 | 0.116 | | |
| | arthursonzogni__json-tui.17a22b6 | easy | 0.426 | 0.531 | 0.105 | | |
| | cslarsen__jp2a.61d205f | easy | 0.217 | 0.000 | -0.217 | | |
| | kisielk__errcheck.dacab89 | easy | 0.512 | 0.403 | -0.109 | | |
| | multiprocessio__dsq.c3ae0ba | easy | 0.554 | 0.507 | -0.046 | | |
| | rbakbashev__elfcat.52f8cc7 | easy | 0.000 | 0.000 | 0.000 | | |
| | sclevine__yj.8016400 | easy | 0.000 | 0.000 | 0.000 | | |
| | sirwart__ripsecrets.34c9e03 | easy | 0.000 | 0.439 | 0.439 | | |
| | wfxr__code-minimap.0ddeea5 | easy | 0.000 | 0.000 | 0.000 | | |
| | brocode__fblog.3b54330 | medium | 0.000 | 0.000 | 0.000 | | |
| | codesnap-rs__codesnap.f81e4f3 | medium | 0.468 | 0.235 | -0.233 | | |
| | ecumene__rust-sloth.051c559 | medium | 0.121 | 0.000 | -0.121 | | |
| | facebook__zstd.1168da0 | medium | 0.000 | 0.323 | 0.323 | | |
| | go-critic__go-critic.9aea378 | medium | 0.312 | 0.349 | 0.037 | | |
| | madler__pigz.fe4894f | medium | 0.000 | 0.403 | 0.403 | | |
| | nikolassv__bartib.6b9b5ce | medium | 0.000 | 0.000 | 0.000 | | |
| | ninja-build__ninja.cc60300 | medium | 0.000 | 0.000 | 0.000 | | |
| | peco__peco.4e58dad | medium | 0.648 | 0.687 | 0.040 | | |
| | sharkdp__fd.40d8eb3 | medium | 0.400 | 0.000 | -0.400 | | |
| | canop__broot.d6c798e | hard | 0.608 | 0.628 | 0.020 | | |
| | dandavison__delta.acd758f | hard | 0.000 | 0.000 | 0.000 | | |
| | doxygen__doxygen.966d98e | hard | 0.188 | 0.248 | 0.060 | | |
| | jesseduffield__lazygit.1d0db51 | hard | 0.255 | 0.382 | 0.127 | | |
| | johnkerl__miller.8d85b46 | hard | 0.057 | 0.063 | 0.005 | | |
| | parcel-bundler__lightningcss.aa2ed1e | hard | 0.000 | 0.309 | 0.309 | | |
| | php__php-src.c891263 | hard | 0.000 | - | - | | |
| | rvben__rumdl.2d75c4d | hard | 0.165 | - | - | | |
| | skeema__skeema.6a76243 | hard | - | - | - | | |
| | typst__typst.88356d0 | hard | - | - | - | | |
| ## Missing or ungraded (6) | |
| - team-piv__php__php-src.c891263__s0: not graded (status fetched, job 6adaaa5b925af0e27d4f9009) | |
| - team-piv__rvben__rumdl.2d75c4d__s0: not graded (status fetched, job 6ada8c38925af0e27d4f7e11) | |
| - solo-piv__skeema__skeema.6a76243__s0: no run dir (status done, job 6ada8d9f925af0e27d4f7ea1) | |
| - team-piv__skeema__skeema.6a76243__s0: no run dir (status done, job 6ada8e0a925af0e27d4f7ed8) | |
| - solo-piv__typst__typst.88356d0__s0: no run dir (status done, job 6ada8e77c191903149a76c13) | |
| - team-piv__typst__typst.88356d0__s0: no run dir (status done, job 6ada93bf925af0e27d4f8291) | |
Xet Storage Details
- Size:
- 3.46 kB
- Xet hash:
- 046c2140fdce7ef462eec33be91a283a20577e28480f0388000782c3fb1e62ba
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.