Spaces:
Sleeping
Sleeping
| # Spreadsheet Gym — Model Comparison | |
| **5 SOTA models × 2 reward modes** evaluated on 12 scenarios. | |
| ## Summary | |
| | Model | Custom Avg | OpenEnv Avg | GT Pass Rate | Avg Steps | Time | | |
| |---|:---:|:---:|:---:|:---:|:---:| | |
| | **gpt-5.4** | **0.67** | **0.65** | 10/12 (83%) | 7.8 | 161s | | |
| | claude-opus-4-20250514 | 0.39 | 0.46 | 7/12 (58%) | 33.6 | 1759s | | |
| | claude-sonnet-4-6 | 0.33 | 0.44 | 6/12 (50%) | 18.9 | 895s | | |
| | claude-opus-4-6 | -0.03 | 0.39 | 5/12 (42%) | 41.3 | 1062s | | |
| | gpt-5 | -0.44 | 0.14 | 1/12 (8%) | 8.8 | 1876s | | |
| **Best model:** gpt-5.4 — highest scores on both reward modes, fastest execution, most scenarios solved. | |
| ## Per-Scenario Breakdown (Custom Mode) | |
| | Scenario | gpt-5.4 | sonnet-4-6 | opus-4-6 | opus-0514 | gpt-5 | | |
| |---|:---:|:---:|:---:|:---:|:---:| | |
| | buggy_template_fix_01 | **0.92** | 0.94 | 0.94 | 0.85 | 0.89 | | |
| | conditional_aggregation_01 | **0.96** | 0.87 | 0.81 | -0.78 | -0.71 | | |
| | conditional_aggregation_02 | **0.91** | -0.68 | -0.66 | 0.85 | -0.68 | | |
| | cross_sheet_lookup_01 | **0.92** | -0.68 | -0.70 | -0.80 | -0.68 | | |
| | cross_sheet_lookup_02 | **0.95** | 0.23 | 0.82 | 0.16 | -0.68 | | |
| | formula_repair_01 | **0.88** | 0.88 | -0.69 | 0.86 | -0.72 | | |
| | formula_repair_02 | **0.93** | 0.89 | 0.91 | 0.91 | 0.92 | | |
| | ledger_reconciliation_01 | -0.66 | 0.28 | -0.68 | **0.83** | -0.68 | | |
| | ledger_reconciliation_02 | **0.92** | 0.22 | -0.66 | 0.13 | -0.80 | | |
| | messy_table_extraction_01 | -0.66 | **0.83** | -0.68 | ERROR | -0.75 | | |
| | range_transformation_01 | **0.98** | 0.86 | 0.91 | 0.83 | -0.68 | | |
| | schedule_grid_fill_01 | **0.96** | -0.68 | -0.68 | 0.88 | -0.69 | | |
| ## Per-Scenario Breakdown (OpenEnv Mode) | |
| | Scenario | gpt-5.4 | sonnet-4-6 | opus-4-6 | opus-0514 | gpt-5 | | |
| |---|:---:|:---:|:---:|:---:|:---:| | |
| | buggy_template_fix_01 | **0.76** | 0.76 | 0.76 | 0.74 | 0.15 | | |
| | conditional_aggregation_01 | **0.76** | 0.74 | 0.73 | 0.74 | 0.14 | | |
| | conditional_aggregation_02 | **0.76** | 0.14 | 0.14 | 0.74 | 0.14 | | |
| | cross_sheet_lookup_01 | **0.75** | 0.14 | 0.14 | 0.14 | 0.14 | | |
| | cross_sheet_lookup_02 | **0.76** | 0.74 | 0.73 | 0.74 | 0.14 | | |
| | formula_repair_01 | **0.75** | 0.14 | 0.14 | ERROR | 0.15 | | |
| | formula_repair_02 | **0.76** | 0.75 | 0.75 | ERROR | 0.15 | | |
| | ledger_reconciliation_01 | 0.14 | **0.74** | 0.14 | ERROR | 0.14 | | |
| | ledger_reconciliation_02 | **0.75** | 0.14 | 0.14 | 0.13 | 0.14 | | |
| | messy_table_extraction_01 | 0.14 | 0.13 | 0.14 | **0.75** | 0.14 | | |
| | range_transformation_01 | **0.76** | 0.76 | 0.75 | 0.74 | 0.14 | | |
| | schedule_grid_fill_01 | **0.76** | 0.14 | 0.14 | 0.75 | 0.14 | | |
| ## Step Count Comparison | |
| | Scenario | max_steps | gpt-5.4 | sonnet-4-6 | opus-4-6 | opus-0514 | gpt-5 | | |
| |---|:---:|:---:|:---:|:---:|:---:|:---:| | |
| | buggy_template_fix_01 | 50 | **10** | 9 | 10 | 26 | 15 | | |
| | conditional_aggregation_01 | 55 | **6** | 10 | 234 | 55 | 9 | | |
| | conditional_aggregation_02 | 55 | **8** | 5 | 4 | 12 | 5 | | |
| | cross_sheet_lookup_01 | 60 | **9** | 6 | 7 | 60 | 5 | | |
| | cross_sheet_lookup_02 | 60 | **7** | 84 | 198 | 60 | 6 | | |
| | formula_repair_01 | 40 | **12** | 15 | 7 | 27 | 12 | | |
| | formula_repair_02 | 40 | **8** | 15 | 13 | 13 | 11 | | |
| | ledger_reconciliation_01 | 60 | 3 | 7 | 5 | **23** | 5 | | |
| | ledger_reconciliation_02 | 60 | **9** | 18 | 5 | 55 | 21 | | |
| | messy_table_extraction_01 | 55 | 3 | **15** | 4 | 17 | 8 | | |
| | range_transformation_01 | 50 | **5** | 12 | 7 | 39 | 4 | | |
| | schedule_grid_fill_01 | 55 | **8** | 5 | 5 | 26 | 6 | | |
| Bold = model that solved the scenario (GT=1.0) in fewest steps. | |
| ## Key Observations | |
| ### 1. Step Score Compression (OpenEnv Mode) | |
| All step_score values fall between 0.31–0.41 regardless of agent quality. This is a known issue — the per-step reward magnitudes are too small (0.02–0.50) relative to the normalizer's expected range [-0.5, +1.0]. The entire ranking comes from ground truth alone. | |
| ### 2. Hallucination Penalty Dominance (Custom Mode) | |
| The -1.0 hallucination penalty fires frequently (when all tool calls succeed but ground truth fails). This causes scores like -0.66 to -0.80, making the average for models that fail a few scenarios deeply negative — even if they perform well on others. | |
| ### 3. Efficiency Score Bug (Custom Mode) | |
| The efficiency denominator uses `len(expected_tools)` (tool types, not steps). A scenario with 8 expected tool types but a realistic 20-step solution gives efficiency = 8/20 = 0.40 even for a perfect agent. | |
| ### 4. gpt-5 OpenEnv Anomaly | |
| gpt-5 scored 0.14 across ALL scenarios in OpenEnv mode (GT=0.0 for every scenario). The same model scored higher on some scenarios in custom mode. This suggests a session or environment issue during the OpenEnv batch run, not a model capability problem. | |
| ### 5. Step Count vs Quality | |
| Some models take extremely many steps (opus-4-6: 234 on conditional_aggregation_01, 198 on cross_sheet_lookup_02) but still achieve GT=1.0. Others fail in just 3-5 steps. The current scoring doesn't properly differentiate efficient correct agents from slow correct agents. | |
| ## Hardening Assessment | |
| **SOTA average (custom): 0.67** (gpt-5.4) — below the 0.7 threshold. | |
| No hardening required at this time. The scenarios are challenging enough — even the best model fails 2/12 scenarios. | |