--- license: apache-2.0 tags: - benchmark - agentic-coding - quantization - apex - gguf - moe language: - en --- # APEX quant benchmarks — 42-task agentic coding suite Results for the GGUF models I quantize and benchmark, measured on a real agentic coding loop rather than a single-completion prompt. Every row below is a full **42-task** run across all four suites. This is a report, not a model. The quants themselves live in the individual `Myric/*-APEX-GGUF` repositories, and the table links each row to the **actual model tested** where a public card exists (otherwise to the base/official card — see the note under the table). ## The benchmark **42 tasks in four suites** (private benchmark harness): | suite | tasks | what it is | |---|---:|---| | `opencode_tasks` | 25 | general algorithmic ladder: data structures, graphs, DP, streams | | `opencode_tasks_hard` | 5 | fix/extend an existing codebase | | `opencode_tasks_frontier` | 9 | the hard-coder frontier (regex, btree, SQL, CRDT, rate limiters, txns) | | `opencode_tasks_tier4a` | 3 | multi-file repair in place (SWE-bench-style), not a harder rung | **How a run works.** The model is served locally through `llama-server` and driven by OpenCode's real tool-use loop (bash / read / write / edit) against each task's `PROMPT.md`. The code it writes is graded by running the task's own `pytest test_solution.py` **fresh** — the model's self-report is never trusted. Scoring is **partial credit**: the fraction of individual pytest items passed, accumulated per suite. **Fixed conditions.** `--parallel 1` (a shared KV pool with concurrent OpenCode requests is not comparable); compaction **off**; context auto-sized to the largest that fits up to 262,144; per-family output cap above worst-case reasoning; per-task wall 3,600 s; sampling at the model card's recommended temperature/top-p. **Boxes.** *GB10* = NVIDIA GB10 (DGX Spark-class), ~119 GiB unified memory. *Thor* = a 2013 dual-Xeon workstation with 128 GB DDR3 and a 16 GB RTX 4060. The box is noted per row where it matters; wall-clock is only comparable within a box. **A perfect score is the entry bar, not a finding.** These suites are designed to saturate for a competent coder. A full 42/42 says the arm *qualified for comparison*; it is not evidence that anything had no effect. The measurement lives in **which suite a model falls off at**, and — once pass/fail saturates — in **output tokens at equal score**. Code quality is not captured by this harness. **Noise.** Seven of the nine frontier tasks can flip between identical repeats, and within-arm token spread runs to double digits. The table shows the **best run** per row and the **Reps** column gives the number of runs and the total range. A single-run row is a data point, not a noise floor. Tokens compare within a model, never across models of different scale. ### Time is part of the measurement A full 42-task run is typically **~2 hours** on the GB10. Across the 74 full runs here the median is **1.8 h**, p90 **4.9 h**, and the slowest outlier is **14.5 h** (Spark-X2.5-4B BF16, 10 voids — each timeout can burn the full 3,600 s per-task wall). A **12-hour outer budget** is therefore a real bound rather than a hypothetical: most runs never approach it, but a slow or verbose model that keeps timing out can exhaust it. A model that cannot finish inside that budget is, for this box, **too slow to be useful** — the benchmark is meant to answer the deployment question, not the abstract-capability question. So a missing perfect score is not always a missing capability: - A task that hits the wall or the output cap is **voided**, not marked wrong. The model may have been on the right track and simply not finished. That is why `Valid = 42 − void` is the column to read, and why a row like Spark-X2.6-4B is 39/39 valid despite a 39/42 raw total. - But a void is also **not evidence of capability**. A model that voids the frontier tasks has not demonstrated that it can do them under the budget; a 100% valid ratio on a shrunken denominator is not the same as a clean 42/42. The clean rows are the stronger result. - A smarter-but-slower model could move up on faster hardware or with a longer budget. The table measures capability **under this GB10's time budget**, not capability in the abstract. For a local deployment, the budget is part of the requirement. This is why the frontier column and the `Void` count sit next to the score: they separate "did not finish" from "got it wrong", and neither is silently folded into the other. ## Full 42-task results `G` = general (25), `H` = hard (5), `F` = frontier (9), `T4A` = tier4a (3). Per-suite and total columns are the **best run** (most valid tasks passed); `Void` is that run's harness-limit count and `Reps` gives run count and the total-passed range. `Tok/task` is output tokens per valid task (`tokens_out / valid`); `Tok/s` is end-to-end output-token rate (`tokens_out / wall_s`), so it includes prefill and tool-loop idle time, not just decode speed. **A void is not a failure.** It is a timeout, output-cap truncation, or no-completion that the harness refuses to score (`n_valid = 42 − void`). The `Valid` column is therefore `passed / (42 − void)`, and it is the number to read. The 42-slot denominator is kept only so the void count is visible. This is not academic: **Spark-X2.6-4B is 39/39 valid (3 void), not 39/42**; Spark-X2.5 BF16 is 32/32 with 10 void; MiniCPM5-2B is 35/35 with 7 void. Reporting those as 39/42, 32/42, or 35/42 would count harness limits as model failures. | Model | Quant | G (25) | H (5) | F (9) | T4A (3) | Valid | Void | Tok/task | Tok/s | Reps | |---|---|---:|---:|---:|---:|---:|---:|---:|---:|---| | [Muse-Glimmer-30B](https://huggingface.co/unsloth/Muse-Glimmer-30B-GGUF) | UD-Q4_K_XL | 25/25 | 5/5 | 9/9 | 3/3 | **42/42** | 0 | 5,021 | 9.3 | 1 | | [K2-Horizon-MoVA-36B-A4B](https://huggingface.co/Myric/K2-Horizon-MoVA-36B-A4B-APEX-GGUF) | APEX i-quality v2 | 25/25 | 5/5 | 9/9 | 3/3 | **42/42** | 0 | 2,597 | 6.2 | 1 | | [Kolibri-1 78B-A3.5B](https://huggingface.co/Myric/Kolibri-1-APEX-GGUF) | APEX 2.4 bpw | 25/25 | 5/5 | 9/9 | 3/3 | **42/42** | 0 | 8,466 | 44.7 | 1 | | [Kolibri-1 78B-A3.5B](https://huggingface.co/Myric/Kolibri-1-APEX-GGUF) | Q4_K uniform | 25/25 | 5/5 | 9/9 | 3/3 | **42/42** | 0 | 8,932 | 41.1 | 2 (42) | | [Laguna-XS-2.1](https://huggingface.co/Myric/Laguna-XS-2.1-APEX-GGUF) | APEX i-quality v2 | 25/25 | 5/5 | 9/9 | 3/3 | **42/42** | 0 | 4,180 | 47.3 | 2 (42) | | [Laguna-XS-2.1](https://huggingface.co/Myric/Laguna-XS-2.1-APEX-GGUF) | APEX dynamic v2 | 25/25 | 5/5 | 8/9 | 3/3 | **41/42** | 0 | 7,174 | 40.0 | 2 (41) | | [Laguna-S-2.1](https://huggingface.co/Myric/Laguna-S-2.1-APEX-GGUF) | APEX i-quality v2 | 25/25 | 5/5 | 7/7 | 3/3 | **40/40** | 2 | 5,558 | 19.6 | 1 | | [Nemotron-3-Super-120B-A12B](https://huggingface.co/Myric/Nemotron-3-Super-120B-A12B-APEX-GGUF) | APEX i-quality | 25/25 | 5/5 | 9/9 | 3/3 | **42/42** | 0 | 3,943 | 13.5 | 1 | | [Nemotron-3.5-Super-VL-120B-A12B](https://huggingface.co/Myric/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-APEX-GGUF) | APEX i-quality | 25/25 | 5/5 | 9/9 | 3/3 | **42/42** | 0 | 2,856 | 10.9 | 1 | | [Ornith-1.5-122B](https://huggingface.co/ornith-ai/Ornith-1.5-122B) | APEX i-quality | 25/25 | 5/5 | 9/9 | 3/3 | **42/42** | 0 | 3,206 | 20.1 | 2 (42) | | [Ornith-1.5-122B](https://huggingface.co/ornith-ai/Ornith-1.5-122B) | APEX i-compact | 25/25 | 5/5 | 9/9 | 3/3 | **42/42** | 0 | 3,148 | 20.6 | 2 (42) | | [Ternary-Bonsai-2-27B](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf) | PQ2_0 (2.13 bpw) | 25/25 | 5/5 | 8/8 | 3/3 | **41/41** | 1 | 11,578 | 21.6 | 1 | | [Gemma-4-12B-it QAT](https://huggingface.co/google/gemma-4-12B-it-qat-q4_0-gguf) | Q4_0 (grid) | 24/25 | 5/5 | 9/9 | 3/3 | **41/42** | 0 | 5,931 | 13.5 | 1 | | [Gemma-4-12B-it QAT](https://huggingface.co/google/gemma-4-12B-it-qat-q4_0-gguf) | Q4_0 (Google) | 24/25 | 5/5 | 8/8 | 3/3 | **40/41** | 1 | 7,621 | 12.5 | 2 (39–40) | | [Mellum2.1-12B-A2.5B-Thinking](https://huggingface.co/JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF) | Q8_0 | 25/25 | 5/5 | 8/8 | 3/3 | **41/41** | 1 | 5,029 | 51.8 | 2 (40–41) | | [Mellum2.1-12B-A2.5B-Thinking](https://huggingface.co/Myric/Mellum2.1-12B-A2.5B-Thinking-APEX-GGUF) | APEX 4.9 bpw | 25/25 | 5/5 | 8/9 | 3/3 | **41/42** | 0 | 6,782 | 71.7 | 2 (41) | | [Mellum2.1-12B-A2.5B-Thinking](https://huggingface.co/Myric/Mellum2.1-12B-A2.5B-Thinking-APEX-GGUF) | APEX 4.3 bpw | 25/25 | 5/5 | 8/9 | 3/3 | **41/42** | 0 | 6,356 | 73.8 | 2 (40–41) | | [Mellum2.1-12B-A2.5B-Thinking](https://huggingface.co/Myric/Mellum2.1-12B-A2.5B-Thinking-APEX-GGUF) | APEX 3.7 bpw | 25/25 | 5/5 | 9/9 | 3/3 | **42/42** | 0 | 5,992 | 83.0 | 2 (41–42) | | [Mellum2.1-12B-A2.5B-Thinking](https://huggingface.co/Myric/Mellum2.1-12B-A2.5B-Thinking-APEX-GGUF) | APEX 3.1 bpw | 25/25 | 5/5 | 7/9 | 3/3 | **40/42** | 0 | 10,729 | 93.8 | 2 (36–40) | | [Mellum2.1-12B-A2.5B-Thinking](https://huggingface.co/JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF) | Q4_K_M stock | 25/25 | 5/5 | 8/9 | 3/3 | **41/42** | 0 | 5,920 | 68.9 | 2 (40–41) | | [Mellum2.1-12B-A2.5B-Thinking](https://huggingface.co/bartowski/Mellum2.1-12B-A2.5B-Thinking-GGUF) | Q6_K_L bartowski | 25/25 | 5/5 | 8/9 | 3/3 | **41/42** | 0 | 5,397 | 57.6 | 2 (41) | | [GPT-OSS-20B](https://huggingface.co/ggml-org/gpt-oss-20b-GGUF) | MXFP4 (Thor) | 25/25 | 5/5 | 8/9 | 3/3 | **41/42** | 0 | 7,653 | 50.3 | 2 (40–41) | | [Nemotron-3.5-Lightning-30B-A3B](https://huggingface.co/Myric/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-APEX-GGUF) | APEX compact (Thor) | 25/25 | 5/5 | 9/9 | 3/3 | **42/42** | 0 | 13,211 | 34.9 | 2 (40–42) | | [Gemma-4-26B-A4B](https://huggingface.co/google/gemma-4-26B-A4B-it-qat-q4_0-gguf) | Q4_0 QAT (Thor) | 25/25 | 5/5 | 9/9 | 3/3 | **42/42** | 0 | 6,000 | 30.3 | 2 (41–42) | | [Kolibri-1 78B-A3.5B](https://huggingface.co/Myric/Kolibri-1-APEX-GGUF) | APEX i-quality high (Thor) | 25/25 | 5/5 | 9/9 | 3/3 | **42/42** | 0 | 8,269 | 15.8 | 1 | | [Kolibri-1 78B-A3.5B](https://huggingface.co/Myric/Kolibri-1-APEX-GGUF) | APEX i-quality low (Thor) | 25/25 | 5/5 | 9/9 | 3/3 | **42/42** | 0 | 5,600 | 15.7 | 1 | | [LFM2.5-24B-A2B](https://huggingface.co/LiquidAI/LFM2.5-24B-A2B-GGUF) | QAD Q4_0 | 25/25 | 5/5 | 7/9 | 3/3 | **40/42** | 0 | 12,442 | 73.9 | 4 (37–40) | | [LFM2.5-24B-A2B](https://huggingface.co/LiquidAI/LFM2.5-24B-A2B-GGUF) | Q8_0 | 25/25 | 3/3 | 8/8 | 3/3 | **39/39** | 3 | 5,875 | 38.7 | 1 | | [LFM2.5-24B-A2B](https://huggingface.co/LiquidAI/LFM2.5-24B-A2B) | IQ3_XXS (BF16 src) | 25/25 | 4/4 | 7/9 | 3/3 | **39/41** | 1 | 11,937 | 78.4 | 2 (36–39) | | [Nemotron-3.5-Bolt-74B-A9B](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF) | 24 GB quant | 24/24 | 5/5 | 7/7 | 3/3 | **39/39** | 3 | 4,214 | 16.7 | 1 | | [Spark-X2.6-4B](https://huggingface.co/Myric/Spark-X2.6-4B-GGUF) | Q6_K | 25/25 | 4/4 | 7/7 | 3/3 | **39/39** | 3 | 5,760 | 25.0 | 1 | | [Spark-X2.5-4B](https://huggingface.co/Myric/Spark-X2.5-4B-GGUF) | BF16 | 22/22 | 3/3 | 4/4 | 3/3 | **32/32** | 10 | 17,953 | 11.0 | 1 | | [Spark-X2.5-4B](https://huggingface.co/Myric/Spark-X2.5-4B-GGUF) | Q6_K | 22/22 | 3/3 | 2/2 | 3/3 | **30/30** | 12 | 12,916 | 16.2 | 1 | | [MiniCPM5-2B](https://huggingface.co/openbmb/MiniCPM5-2B-GGUF) | Q8_0 | 23/23 | 3/3 | 6/6 | 3/3 | **35/35** | 7 | 11,148 | 57.1 | 1 | | [LFM3-1.7B](https://huggingface.co/LiquidAI/LFM3-1.7B-GGUF) | Q4_0 | 23/24 | 3/4 | 4/7 | 2/3 | **32/38** | 4 | 18,431 | 85.3 | 1 | | [Flint](https://huggingface.co/Myric/Flint-APEX-GGUF) | APEX 5bpw | 25/25 | 5/5 | 9/9 | 3/3 | **42/42** | 0 | 4,397 | 40.1 | 1 | | [Qwen3.8-Flash-Next](https://huggingface.co/Myric/Qwen3.8-Flash-Next-APEX-GGUF) | APEX MIDDLE | 24/24 | 5/5 | 9/9 | 3/3 | **41/41** | 1 | 5,970 | 16.1 | 1 | | [Ornith-1.5-35B-A3B](https://huggingface.co/Myric/Ornith-1.5-35B-A3B-APEX-GGUF) | APEX i-quality | 25/25 | 5/5 | 9/9 | 3/3 | **42/42** | 0 | 5,028 | 50.8 | 1 | | [GLM-4.7-Flash](https://huggingface.co/Myric/GLM-4.7-Flash-APEX-GGUF) | APEX i-quality | 24/25 | 3/5 | 6/9 | 3/3 | **36/42** | 0 | 10,226 | 28.2 | 1 | | [Muse-Glimmer-30B-heretic](https://huggingface.co/darkc0de/Muse-Glimmer-30B-heretic) | Q4_K_M | 25/25 | 5/5 | 9/9 | 3/3 | **42/42** | 0 | 4,895 | 8.8 | 1 | | [Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) | Q4_K_M | 25/25 | 5/5 | 9/9 | 3/3 | **42/42** | 0 | 2,236 | 9.6 | 1 | Every row links to the public card for the exact quant where one exists; where the tested GGUF is local-only (the Gemma grid variant, Ornith's APEX tiers before publication, LFM's BF16-source IQ3_XXS), the link goes to the base/official card and the Quant column names the variant. ### What the fall-off says With voids excluded, the picture is cleaner — and different from the raw denominators: - **Clean 42/42, no voids:** Muse-Glimmer-30B, K2-Horizon v2, Kolibri-1 (APEX and uniform), Laguna-XS i-quality v2, Nemotron-3-Super-120B, Nemotron-3.5-Super-VL-120B-A12B, Ornith-1.5-122B (i-quality and i-compact). Their comparison is tokens-at-equal-score, not correctness. - **Perfect on every valid task, via voids:** Spark-X2.6-4B (39/39, 3 void), Spark-X2.5 at BF16 and Q6_K (32/32 and 30/30; 10 and 12 void), MiniCPM5-2B (35/35, 7 void), LFM2.5 Q8_0 (39/39, 3 void), Nemotron-3.5-Bolt-74B (39/39, 3 void), Laguna-S (40/40, 2 void), and **Bonsai 2 27B (41/41, 1 void)**. The voids are the output cap or the wall on the longest tasks; the write-ups note no degeneration. These are not capability losses. Bonsai's one void is the "smart but too slow" case: **56.8k output tokens and the full 3600 s wall, still unfinished** — not a wrong answer. - **Real frontier misses (1–2 valid tasks):** Laguna-XS dynamic v2 41/42, Mellum2.1 APEX 4.9 / 4.3 / stock Q4_K_M / bartowski Q6_K_L all 41/42, Mellum2.1 APEX 3.1 40/42, LFM2.5 QAD Q4_0 40/42, LFM2.5 IQ3_XXS 39/41. This is the scale doing its job: the frontier tier is where the quant or the smaller model gives up first. - **General-suite miss:** Gemma-4-12B QAT grid 41/42 (one general task). The Google QAT arm is 40/41 on its best-passed rep (one general miss, one void); its second rep is 39/39 with 3 void. The general-suite miss is the one real failure; the void is a harness limit. - **Mellum2.1 APEX 3.7 bpw reaches 42/42** on its best rep (41/42 on the other), while 3.1 bpw falls to 40/42 — the aggressive tier is where the quant starts to cost correctness, and it shows up in the frontier column. ## Plots Measured wall-clock from the full 42-task runs. Time plots use **GB10 runs only** — Thor wall-clock is not comparable to the GB10. Each point is the best valid run for that model/quant. Parameter counts are nominal from the model cards/configs; Ornith-1.5-122B's active count is derived from its 256-expert top-8 config (~12 B). **Wall clock vs active parameters.** Active count alone explains little: the 4 B Spark pair spans 2.5–14.5 h while the 12 B-active 120–124 B MoEs finish in 1.8–3.4 h. ![Wall clock vs active parameters](plots/wall_vs_active.png) **Wall clock vs active parameters — perfect-score runs, by model origin.** Filtered to runs that passed every valid task (`passed == valid`); voids are allowed, and their wall time is still included because this is elapsed wall clock, not valid-task time. **Fill = pretrained-base origin; edge = post-training/compression origin.** Blue = US, red = China, amber = Germany, purple = Czechia, green = UAE. The origin plots omit the Spark-X2.5 14.5 h BF16 outlier to keep the vertical scale readable. Bonsai 27B and Ornith-1.5-122B show red fill / blue edge: Chinese pretrained bases, US post-processing. `Muse-Glimmer-30B-heretic` has a US/Meta base (blue fill) and a gray edge labelled **Abliterated**, because its Heretic abliteration publisher's origin is undocumented. ![Wall clock vs active parameters, perfect-score runs by country](plots/wall_vs_active_perfect_country.png) **Wall clock vs total parameters.** Total size is not the driver either: a 124 B model can finish in 1.75 h while a 4 B model takes 14.5 h. ![Wall clock vs total parameters](plots/wall_vs_total.png) **Wall clock vs total parameters — perfect-score runs, by model origin.** Same filter, fill/edge encoding, and color key as the active-parameter version (Spark-X2.5 omitted); wall time still includes voided tasks. ![Wall clock vs total parameters, perfect-score runs by country](plots/wall_vs_total_perfect_country.png) **Wall clock vs output tokens.** The clearest single driver: the runs that spend more output tokens take longer. The Spark outliers are the high-token runs, not the large models. ![Wall clock vs output tokens](plots/wall_vs_tokens.png) **Verbosity vs active parameters.** Output tokens per valid task; circles are GB10, squares Thor. Larger active models are not uniformly more verbose, and the 4 B Spark BF16 run is the outlier at ~18 k tokens/task. ![Verbosity vs active parameters](plots/tokens_per_task_vs_active.png) **Performance per byte.** Capability vs quant file size (left; top-left is best) and end-to-end output tokens/s vs size (right; GB10 only — throughput is box-specific). The ternary Bonsai 27B PQ2_0 is the efficiency outlier: **41/41 valid at 7.21 GB**, with one void (`btree_insert_delete` — 3600 s / 32k cap). Full capability on every valid task, at the cost of raw decode speed from the custom ternary kernels. ![Performance per byte](plots/perf_per_byte.png) ## Also measured (subset runs) Not comparable to the 42-task table above — these ran one suite, or fewer tasks. Kept here so the frontier signal is not lost. | Model | Quant | Suite | Score | Note | |---|---|---|---:|---| | Qwen3.8-Flash-Next | APEX (several arms) | frontier | 9/9 | one arm 8/9 | | K2-Horizon-MoVA-36B-A4B | APEX compact / mini | frontier | 9/9 | both tiers | | K2-Horizon-MoVA-36B-A4B | APEX i-quality | frontier | 8/9 GB10, 9/9 Thor | box difference | | Ornith-1.5-35B-A3B | APEX i-quality | frontier | 9/9 | — | | Nemotron-3.5-Lightning-30B-A3B | APEX i-quality | frontier | 7/9 | — | | Spark-X2.5-4B | Q5 (1M-context) | general | 10/13 | long-context variant | | Spark-X2.5-4B | Q4 | hard | 3/5 | — | | Solar-Open2-148B-A15B REAP | Q4_K_M | tier4a | 2/3 (25/27, 91.7%) | 131k ctx, low reasoning, Solar proxy parser | ## Quantization note APEX tiers are **per-tensor allocations**, not fixed recipes: the edge/near/middle bands, the shared-expert and latent-bottleneck protection, and the F32 state/norm/router pins all change per architecture. `i-quality` is a shelf label for the tier I reach for by default, not a claim of identity with another project's recipe. Sizes and bpw live in each model repo. ## Files | path | contents | |---|---| | `results.json` | every full-42 and subset run, per-suite passed/valid/partial, model path, box, void/wall | | `results_full42.csv` | flat table of the 74 full-42 runs, one row per run | | `plots/` | wall-clock vs parameters/tokens, verbosity, performance-per-byte, and country-colored perfect-score plots | ## Caveats - **Pass/fail is a gate.** This harness saturates by design; a perfect score is not a ranking. - **Code quality is unmeasured.** Tests-passed and tokens are what is recorded; clarity, structure, and whether the model understood the problem are not. - **Tokens are within-model.** Cross-model token comparisons confound capability with verbosity and are not reported. - **Single runs are noisy.** Where only one run exists, treat the score as a data point.