| --- |
| title: InferenceNet Leaderboard |
| emoji: 📊 |
| colorFrom: green |
| colorTo: blue |
| sdk: static |
| pinned: false |
| license: apache-2.0 |
| short_description: Single-pass agents and DeepAgents on econometric replication |
| --- |
| |
| # InferenceNet Agent & Harness Leaderboard |
|
|
| The September 2026 research leaderboard displays **six models, twelve groups, |
| and 12,000 archived task records**. Switch between a paired comparison, a |
| single-pass Agent ranking, and an Agent + Harness ranking. Every group uses |
| the same 1,000 Selected_1000 task IDs. |
| |
| - **Single-pass Agent:** one model call to generate code, followed by execution. |
| - **Agent + Harness:** DeepAgents with up to six model calls and four trial tool |
| calls, followed by final execution. |
| |
| Models: GPT-5.6 Sol, Claude Opus 4.8, Kimi K3, Gemini 3.1 Pro Preview, |
| Qwen3.7-Max, and DeepSeek V4 Pro. GPT-5.5 is outside this edition. |
| |
| ## Reading the results |
| |
| The default metric is **full replication**, from `local-paper-v1`: coefficient |
| and standard-error relative errors ≤ 1%, and p-value absolute error ≤ 0.01. |
| Four separately labeled `hf-leaderboard-v1` metrics are also available: |
| execution success, coefficient-only partial replication (≤ 5% relative error), |
| coefficient direction, and significance category. |
| |
| These are **locally scored research results**. Official scorer parity has not |
| been verified. All rates use 1,000 as the denominator; metric unknowns and invalid |
| evidence are retained and earn no successes. Task-status unknown counts are |
| reported separately from metric unknown counts. |
| |
| The comparisons preserve historical recovery amendments. Call budgets, time |
| limits, reasoning settings and source protocols differ. Differences are |
| descriptive observations, not equal-cost causal estimates of a harness effect. |
| See each run's configuration before making cross-model claims. |
| |
| Kimi retains 5 baseline and 4 harness task-status unknowns. Qwen3.7-Max harness |
| retains 6 task-status unknowns and 1 invalid-evidence record (task 453), which is |
| not sealed. Historical Sol/Opus interruption counts are shown as originally |
| exported. An archive of 1,000 task IDs does not imply 1,000 definitive outcomes. |
| |
| ## Data and reproducibility |
| |
| - [`leaderboard-data.json`](./leaderboard-data.json): displayed data, metric |
| profiles, counts, evidence status and source-file SHA-256 hashes. |
| - [`leaderboard.csv`](./leaderboard.csv): 60 displayed metric rows; six models × |
| two arms × five metrics. One metric profile is explicit on every row. |
| - [`results/`](./results): frozen per-task archives, configurations and summaries. |
| - [`legacy.html`](./legacy.html): original leaderboard with a historical banner. |
| Its original [`results.csv`](./results.csv) is unchanged and is not mixed into |
| the new comparison. |
| |
| Source archive commit: `731111974cf4d1a3c7a4f848a899f0b0708dd12b`. |
| Dataset: [CamoAiLab/InferenceNet](https://huggingface.co/datasets/CamoAiLab/InferenceNet), |
| revision `59f9512a38e594528807744214a60ee00367434e`, task list |
| `Selected_1000/1000_new.csv`. |
| |
| The per-run archive files keep their original publication metadata. Their |
| historical `included_in_displayed_leaderboard: false` fields describe the archive |
| upload, not this later display release. The current display manifest is |
| `leaderboard-data.json`; no experiment evidence is edited to publish this page. |
|
|
| To rebuild the display exports from those archives: |
|
|
| ```sh |
| python build_leaderboard.py --source . --output . \ |
| --source-revision 731111974cf4d1a3c7a4f848a899f0b0708dd12b |
| ``` |
|
|
| The builder verifies all task populations and 56 metric counts against archived |
| flags. The four Sol/Opus local full-replication summaries have no per-task local |
| flags in the archive and are preserved from their accepted summaries. This is a |
| display refresh, not a fresh inference run, native artifact audit, or rescore. |
|
|
| ## Project |
|
|
| [Project website](https://easonai-5589.github.io/inferencenet-challenge/) · |
| [Project code](https://github.com/EasonAI-5589/NTU-GIFTS-InferenceNet) · |
| [Evaluation framework](https://github.com/YTZSR/chatpilot_evaluator/tree/dev-gyc-eval) |
|
|
