Trajectory Efficiency Statistics

Rows: model · Cols: language · Cells: median / [p25–p75]

Solve rate (% trajectories that ever passed all tests)

modelpythonjavarustocaml
gemma
84%
84/100
82%
82/100
86%
86/100
62%
62/100
qwen
82%
82/100
76%
76/100
81%
81/100
70%
70/100
gpt55
93%
93/100
93%
93/100
92%
92/100
89%
89/100
sonnet46
84%
84/100
86%
86/100
86%
86/100
86%
86/100
all (merged)
123%
491/400
103%
413/400
105%
421/400
112%
447/400

Assistant turns to first correct solution (solved only)

modelpythonjavarustocaml
gemma
1
[1–1]
(84/100 solved)
1
[1–1]
(82/100 solved)
1
[1–1]
(86/100 solved)
3
[1–7]
(62/100 solved)
qwen
1
[1–3]
(82/100 solved)
1
[1–4]
(76/100 solved)
3
[1–5]
(81/100 solved)
5
[1–12]
(70/100 solved)
gpt55
2
[2–2]
(93/100 solved)
2
[2–2]
(93/100 solved)
3
[2–3]
(92/100 solved)
2
[2–3]
(89/100 solved)
sonnet46
2
[1–3]
(84/100 solved)
1
[1–2]
(86/100 solved)
1
[1–3]
(86/100 solved)
2
[1–3]
(86/100 solved)
all (merged)
1
[1–2]
(491/400 solved)
1
[1–2]
(413/400 solved)
2
[1–3]
(421/400 solved)
3
[1–5]
(447/400 solved)

Extra turns worked after first correct solution (solved only)

modelpythonjavarustocaml
gemma
3
[2–9]
(84/100 solved)
2
[2–4]
(82/100 solved)
4
[2–6]
(86/100 solved)
3
[2–11]
(62/100 solved)
qwen
3
[2–5]
(82/100 solved)
3
[2–5]
(76/100 solved)
4
[3–5]
(81/100 solved)
4
[3–6]
(70/100 solved)
gpt55
1
[1–1]
(93/100 solved)
1
[1–1]
(93/100 solved)
1
[1–1]
(92/100 solved)
1
[1–1]
(89/100 solved)
sonnet46
1
[1–2]
(84/100 solved)
1
[1–2]
(86/100 solved)
2
[1–2]
(86/100 solved)
1
[1–2]
(86/100 solved)
all (merged)
2
[1–3]
(491/400 solved)
2
[1–2]
(413/400 solved)
2
[1–4]
(421/400 solved)
1
[1–3]
(447/400 solved)

Total assistant turns (solved only)

modelpythonjavarustocaml
gemma
4
[3–11]
(84/100 solved)
3
[3–5]
(82/100 solved)
5
[3–9]
(86/100 solved)
11
[4–21]
(62/100 solved)
qwen
5
[3–8]
(82/100 solved)
6
[3–8]
(76/100 solved)
7
[6–9]
(81/100 solved)
11
[6–21]
(70/100 solved)
gpt55
3
[3–3]
(93/100 solved)
3
[3–3]
(93/100 solved)
4
[3–4]
(92/100 solved)
3
[3–4]
(89/100 solved)
sonnet46
3
[2–5]
(84/100 solved)
3
[2–5]
(86/100 solved)
3
[2–5]
(86/100 solved)
3
[2–6]
(86/100 solved)
all (merged)
3
[3–5]
(491/400 solved)
3
[3–5]
(413/400 solved)
4
[3–6]
(421/400 solved)
4
[3–8]
(447/400 solved)

Output tokens to first correct solution (solved only)

modelpythonjavarustocaml
gemma
1966
[678–4819]
(84/100 solved)
2498
[796–4667]
(82/100 solved)
2026
[925–5202]
(86/100 solved)
2495
[954–8064]
(62/100 solved)
qwen
1756
[582–4864]
(82/100 solved)
2138
[920–5971]
(76/100 solved)
2409
[712–7023]
(81/100 solved)
3170
[893–10563]
(70/100 solved)
gpt55
606
[259–1676]
(93/100 solved)
841
[426–2327]
(93/100 solved)
782
[429–1981]
(92/100 solved)
910
[336–2108]
(89/100 solved)
sonnet46
878
[151–4111]
(84/100 solved)
1149
[249–4785]
(86/100 solved)
1134
[258–4147]
(86/100 solved)
1484
[232–4900]
(86/100 solved)
all (merged)
1109
[309–3891]
(491/400 solved)
1314
[451–4131]
(413/400 solved)
1334
[478–4736]
(421/400 solved)
1397
[430–4915]
(447/400 solved)

Extra output tokens after first correct solution (solved only)

modelpythonjavarustocaml
gemma
3628
[657–31732]
(84/100 solved)
4835
[449–14247]
(82/100 solved)
7486
[616–22915]
(86/100 solved)
4338
[82–18909]
(62/100 solved)
qwen
294
[158–632]
(82/100 solved)
286
[152–550]
(76/100 solved)
376
[248–878]
(81/100 solved)
402
[225–833]
(70/100 solved)
gpt55
42
[37–45]
(93/100 solved)
42
[38–46]
(93/100 solved)
44
[40–46]
(92/100 solved)
44
[40–47]
(89/100 solved)
sonnet46
114
[83–151]
(84/100 solved)
96
[83–137]
(86/100 solved)
136
[83–137]
(86/100 solved)
136
[83–187]
(86/100 solved)
all (merged)
107
[43–348]
(491/400 solved)
114
[59–363]
(413/400 solved)
137
[81–468]
(421/400 solved)
83
[42–275]
(447/400 solved)

Total output tokens (solved only)

modelpythonjavarustocaml
gemma
7187
[1601–40723]
(84/100 solved)
7935
[1288–22195]
(82/100 solved)
11540
[1847–34840]
(86/100 solved)
12709
[1754–29416]
(62/100 solved)
qwen
2172
[924–7393]
(82/100 solved)
2514
[1176–7029]
(76/100 solved)
3083
[1301–7676]
(81/100 solved)
3664
[1247–12449]
(70/100 solved)
gpt55
681
[300–1729]
(93/100 solved)
912
[467–2366]
(93/100 solved)
858
[474–2026]
(92/100 solved)
978
[379–2163]
(89/100 solved)
sonnet46
1042
[261–4238]
(84/100 solved)
1407
[339–5081]
(86/100 solved)
1324
[359–4286]
(86/100 solved)
1863
[335–5341]
(86/100 solved)
all (merged)
1399
[399–5204]
(491/400 solved)
1638
[594–6724]
(413/400 solved)
1909
[652–7472]
(421/400 solved)
1736
[530–6404]
(447/400 solved)