| I benchmark the GGUF quants I make on a 42-task agentic-coding suite, and I've put the results in one place: |
|
|
| **https://huggingface.co/Myric/apex-quant-benchmarks** |
|
|
| **What it is.** OpenCode drives each model through a real tool-use loop (bash/read/write/edit) against 42 tasks in four suites — 25 general, 5 hard, 9 frontier, 3 repair-in-place. The code is graded by a fresh `pytest` per task, partial credit. Timeouts and output-cap truncations are **voided, not scored as failures** (`valid = 42 − void`). |
|
|
| The table covers APEX tiers, stock K-quants, QAT and vendor GGUFs. Each row links to the actual model/quant card where one is public, and to the base/official card where the tested GGUF is local-only. |
|
|
| **A few things that fell out:** |
|
|
| - **42/42 is the entry bar, not a ranking.** The suites saturate for competent coders. The signal is the suite where a model falls off, and output tokens at equal score. |
| - **Wall-clock is not a proxy for parameter count.** A 4B Spark-X2.5 BF16 run took **14.5 h** (10 voids) while the 124B Ornith-1.5-122B finished in **1.75 h**. Output tokens and timeouts drive the clock, not size. |
| - **Voids change the story.** Spark-X2.6 is **39/39 valid** with 3 voids — not 39/42. Several others look worse than they are under the raw denominator. |
| - **Aggressive APEX tiers do cost correctness at the edge.** Mellum2.1 APEX 3.7 bpw matched 42/42; 3.1 bpw dropped to 40/42, both misses on the frontier. |
|
|
| **Caveat:** pass/fail is a gate; the harness does not measure code quality, and single-run rows are noisy. The repo includes the raw per-run JSON/CSV, the four plots, and a runbook for adding new results. |
|
|
| Repo, data and plots: **https://huggingface.co/Myric/apex-quant-benchmarks** |
|
|
