| # k448 ("Tessera", Qwen3.8-Flash-Next, 448 experts) - capability lab and speed report, 2026-10-25 |
|
|
| All quality numbers: Colab L4 VM, llama.cpp, 120-task hard suite (HumanEval+, MATH-500, MMLU-Pro; 40 each; seed 1). |
| Wording rule used throughout: "beats Q4" only if the paired sign test is p < 0.05 in our favour. |
|
|
| ## 1. Verdict against Q4 (unsloth UD-IQ4_XS, 87 GiB) |
| - Hard test, no-think greedy: Q4 103/120, k448 101/120 (code 39/40 both, math 35/40 both, reasoning 29 vs 27). |
| - Paired sign test p = 0.754. **k448 MATCHES Q4 within noise. It does not beat it.** |
| - Speed with llama.cpp on the VM: 11.7 vs 8.2 tokens/s (+43%). Memory while running: 41.9 GiB resident, of which 6.3 GiB is real working memory and 35.5 GiB is the mapped model file; the 26.8 GiB n-gram table is lazy. |
| - Not compared: Strata's own ISTA-DASLab packs (IQ2_XS 39.2 GB, IQ3_XXS 47 GB) are the real size peers; k448 keeps about 47 GB resident. |
| |
| ## 2. What raised capability (weights unchanged, run-time settings only) |
| | Setting | Result | |
| |---|---| |
| | Thinking, reasoning budget 4096, temp 0.6 / top_p 0.95 / top_k 20 | rescued 8 of 12 tasks the greedy run failed (math 5/5, reasoning 3/7); 9/15 on the full redo. About 3.8k tokens per task. | |
| | Resampling control (temp 0.6, no thinking) | 2/12 | |
| | 3-vote majority (temp 0.7) | 2/12, the same two tasks as resampling; no gain from voting | |
| | Thinking vs resampling, paired | 6 vs 0, exact sign test p = 0.031 (small n = 12, tasks pre-selected as failures) | |
| | 14 active experts instead of 10 | math 37 vs 35, reasoning 27 vs 27, p = 0.688: no significant gain; perplexity gap to Q4 shrinks but KLD to Q4 does not | |
| | MTP draft (llama.cpp) | x1.34 speed with identical answers | |
| |
| Recommended profile: everyday use no-think greedy, 10 experts, MTP on. For hard math or reasoning turn thinking on with a 4096 budget (about twice the tokens). Do not use voting or 14 experts. |
| |
| ## 3. Speed with Strata's engine (github.com/Niko1221/Strata, MIT, engine 0.1.51) |
| Same Colab VM (L4, 12 vCPU with AVX2 only, 54 GB RAM), k448 packed with Strata's own tool, no MTP head loaded: |
| | Run | Decode tokens/s | |
| |---|---| |
| | llama.cpp, no MTP | 11.7 | |
| | llama.cpp with MTP | about 16 | |
| | Strata, 256 tokens, `--spec 4` | 15.4 | |
| | Strata, 256 tokens, `--spec 2` | 18.4 | |
| | Strata, 1024 tokens, `--spec 2` | **27.2** | |
| - The 27.2 figure includes lookup drafting (37.5% of drafted tokens accepted) on a repetitive code answer; prose will gain less. |
| - Output of the 256-token run was coherent, correct Python. Quality of the Strata path has NOT been run through the hard suite. |
| - NOT measured on the RTX 4060 laptop. The VM's slow disk (12 MB/s random reads) and small RAM held Strata back here; the PC may differ either way. |
| |
| Steps that worked on the VM (a source build; no installer scripts were run): |
| 1. `cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89 && cmake --build build --target strata -j4` |
| 2. `STRATA_GGUF_PY=build/_deps/strata_llamacpp-src/gguf-py python3 tools/iq_pack.py --gguf <k448.gguf> --out packs/k448 --compat-bf16` (under a minute; 303 tensors served natively, 195 small tensors rounded to BF16, max error 0.0144) |
| 3. `python3 tools/make_profile.py --no-base --n-expert 448 --out k448-profile.bin` (the shipped profile is for 512 experts) |
| 4. `./build/strata --pack packs/k448 --tokens-file prompt.txt --native <k448.gguf> --ple-gguf <k448.gguf> --ple-io mmap --max-new 1024 --max-context 4096 --expert-profile k448-profile.bin --expert-cache auto --kv int8 --spec 2 --prefill 512` |
| (the binary takes no subcommand; native packs require `--native`, `--spec T>=2` and `--prefill`; on a fast NVMe drop `--ple-io mmap`) |
|
|
| ## 4. Open items |
| - Strata's MTP draft layer: needs about 5-6 GB fetched from the public Qwen/Qwen3.8-Flash-Next repo (tools/mtp_fetch.py). Not downloaded; awaiting the user's yes. Our llama.cpp-format MTP file cannot be used by Strata. |
| - Measure on the user's PC (the user downloads the k448 file themselves; no token is ever entered by Claude). |
| - Run the hard suite through Strata's server to confirm quality parity. |
| - Compare k448 against Strata's ISTA IQ3_XXS / IQ2_XS on the same tasks. |
| - Autonomous (agent-style) coding was never tested; only function-level tasks. |
| |
| ## 5. The big test suite (`09_big_suite.py`, built 2026-10-25) |
| Why: 120 tasks only detect differences of about 10 points. This set has 4 domains x 300 tasks = 1200, all public benchmarks, every answer checked by code. |
| |
| | Domain | 300 tasks = | Check | |
| |---|---|---| |
| | code | 120 HumanEval+ (all but HumanEval/32, whose own reference fails) + 180 MBPP sanitized | unit tests on the model's function | |
| | math | 225 MATH-500 integer answers (levels 2-5) + 75 AIME (AI-MO/aimo-validation-aime) | final line `#### <integer>` | |
| | reasoning | 200 MMLU-Pro (14 subjects round-robin, 10 options) + 100 BIG-Bench Hard (8 tasks) | final line `Answer: <x>` | |
| | English | 100 RACE-high reading comprehension + 100 CoLA grammar (half acceptable) + 100 writing prompts with machine-checked rules (15 rule types: word counts, paragraphs, bullets, case, keywords, start/end text, quotes, title, commas, P.S., letter count) | `Answer:` line, or the rules in code | |
| |
| - **Nested**: the first 60 tasks of every domain are a prefix of the 300 (`--hard-n 60` = quick set, 240 tasks; `--hard-n 300` = full set). Results of the two can be pooled. |
| - **Checked**: selftest passes locally and on the VM (references pass, garbage fails, each rule checker passes its own good/bad example); all 420 code reference solutions run; the VM file has the same sha256 prefix as the local one (`5700b5f8682b8978`). |
| - **Run it**: `python3 07x_domain_eval.py run --suite big --hard-n 60 --seed 1 --no-think --name <n> --out results_<n>.json` plus the usual `--bin/--model/--ot/--extra` flags; compare with `07x_domain_eval.py compare a.json b.json` (paired sign test per domain). |
| - **Power** (paired exact sign test, 5%): about 5 points per domain at 300 tasks, about 10 points at 60, about 2.5 points when the four domains are pooled (1200). Treat the English rule-following prompts as a relative measure only: they are our own set, not IFEval, so they cannot be compared with published numbers. |
| - **Smoke run** (3 tasks per domain, llama.cpp + MTP, no-think, Colab L4): 12/12 pass, 4781 tokens in 471 s = about 400 tokens and 39 s per task, 10.2 tokens/s overall. That puts the quick set (240 tasks) at about 2.6 hours and the full set (1200 tasks) at about 13 hours for one setting and one model; thinking mode costs roughly 5-10x more tokens per task. A 12-task sample, so treat the times as rough; Strata at about 27 tokens/s on this machine would cut them by more than half, but it has not been run through this suite. |
| - **Known limits**: code is already near the ceiling (39/40 on HumanEval+), so code gains will be small; there is no agent-style coding test. |
|
|
| ## 6. Quick-set baseline (run 2026-10-25) |
| k448 + MTP draft, llama.cpp, no-think greedy, Colab L4, big suite `--hard-n 60` (60 tasks per domain = 240, seed 1). 137,690 tokens in 9,799 s (14.1 tokens/s overall, about 570 tokens and 41 s per task). |
|
|
| | Domain | Pass | Parts | |
| |---|---|---| |
| | code | 55/60 (91.7%) | HumanEval+ 22/24, MBPP 33/36 | |
| | math | 39/60 (65.0%) | MATH-500 39/45, AIME 0/15 | |
| | reasoning | 51/60 (85.0%) | MMLU-Pro 31/40, BBH 20/20 | |
| | English | 52/60 (86.7%) | RACE 18/20, CoLA 17/20, rule-following 17/20 | |
| | **All** | **197/240 (82.1%)** | | |
|
|
| - **AIME 0/15 is a token-cap result, not a measure of ability.** The no-think runs on this suite are capped at 2048 tokens (the harness raises `--max-tokens` to at least 2048), and the AIME answers used 2048 tokens on average, so every one ran to the cap. MATH-500 averaged 1,024 tokens, so some of its misses are probably cut off too. Math is the domain where thinking mode (budget 4096) should move the number most. |
| - Strongest domain is code (HumanEval+ and MBPP are near the ceiling, as noted in section 5); BBH is a clean 20/20. |
| - With 60 tasks per domain only differences of about 10 points are detectable (section 5), so this is a reference point for later changes, not a ranking. It has not been run on Q4 yet, so it says nothing about k448 vs Q4 on this suite. |
| - Raw per-task results: `results_big60_k448.json` on the L4 VM (`/content/q38/eval/`); compare later runs with `07x_domain_eval.py compare`. |
|
|
| ## 7. The harder coding suite (`10_code_hard.py`, built 2026-10-25) |
| Why: HumanEval+ and MBPP (92% for k448) are small textbook functions that top models already pass. These two are the harder, widely used replacements that we can run and score by code without a token. |
|
|
| | Benchmark | What it tests | In the suite | |
| |---|---|---| |
| | LiveCodeBench (code_generation_lite, release 6) | contest programming: the model writes a whole program (AtCoder, stdin/stdout) or a LeetCode class; every one of about 40 hidden tests must pass | 175 problems from 2025-01 to 2025-04 (newer than most training data); ids carry the difficulty (easy 43, medium 52, hard 80) | |
| | CRUXEval | code understanding: predict the output of a function for an input (CRUXEval-O), or find an input that gives a given output (CRUXEval-I) | 800 functions, half O and half I per task list | |
|
|
| - **Run it**: `python3 07x_domain_eval.py run --suite codehard --hard-n 60 --max-tokens 4096 --ctx 8192 --no-think --seed 1 ...` (60 = quick set: 60 LiveCodeBench + 60 CRUXEval; nested prefixes, so `--hard-n 100` extends it). New categories `lcb` and `crux` in the compare table. |
| - **Checked**: selftest passes locally and on the VM (lookup-table "solutions" pass every test of 4 stdin and 4 LeetCode-style problems and a wrong solution fails; CRUXEval references pass, garbage fails); all 800 CRUXEval references run; all 175 LiveCodeBench test sets decode (the private tests are a pickled string, loaded with a loader that refuses every global, so a dataset cannot run code). The VM file has the same sha256 prefix as the local one (`2c9c82f1effce90a`). |
| - **Expect low numbers**: contest problems are hard. Without thinking, most hard AtCoder problems will fail; thinking mode will matter far more here than on the old sets. The release is recent, but the model's training cutoff is unknown, so some problems may still be familiar. |
| - **Time**: LiveCodeBench answers are long (about 1-2k tokens). Roughly 2 hours for the 60+60 quick set and 3.5 hours for 100+100 at llama.cpp speeds; thinking mode several times that. |
| - **Not included, and why**: SWE-bench Verified and the Aider polyglot benchmark need a Docker or multi-language toolchain and an agent loop that edits real repositories, which the Colab VM cannot host; BigCodeBench-Hard (148 library-heavy tasks) needs dozens of heavy libraries installed first and is the next candidate if we want one more. |
|
|
| ## 8. Hard coding results (run 2026-10-26, Colab A100, llama.cpp + MTP draft, seed 1) |
| 60 LiveCodeBench problems + 60 CRUXEval tasks, the same 120 tasks in both runs. Thinking: budget 4,096 tokens, temperature 0.6, top-p 0.95, top-k 20. No-think: greedy. Both: 8,192-token answer cap, all experts on the GPU. Per-task tables: `results/k448_code_think_compact.csv`, `results/k448_code_nothink_a100_compact.csv`. |
|
|
| | | Thinking (4k budget) | No-think (greedy) | |
| |---|---|---| |
| | **All 120** | **99/120 (82.5%)** | **96/120 (80.0%)** | |
| | LiveCodeBench | 41/60 (68.3%) | 38/60 (63.3%) | |
| | - easy / medium / hard | 12/12, 15/20, 14/28 | 12/12, 15/20, 11/28 | |
| | CRUXEval | 58/60 (96.7%) | 58/60 (96.7%) | |
| | - CRUX-O / CRUX-I | 28/30, 30/30 | 28/30, 30/30 | |
| | Tokens (per task) | 277,274 (2,310) | 262,258 (2,185) | |
| | Time (per task), A100 | 3,689 s (31 s) | 2,888 s (24 s) | |
|
|
| - **Thinking does not beat no-think here.** Paired exact sign test: only thinking passes 6 tasks (CRUX-O 155; LCB hard 3777, abc388_g, arc192_d; LCB medium 3722, 3779), only no-think passes 3 (CRUX-O 445; LCB medium 3805, abc400_d), p = 0.508. Both runs are inside each other's noise; the earlier thinking gain (section 2) was on math and reasoning tasks that greedy had already failed, not on coding. |
| - **CRUXEval is at its ceiling** (96.7% both ways), so the room for improvement is in LiveCodeBench, mostly the hard problems (about half of the 60 problems, 46% of the pool). |
| - **Both modes lose their points by running out of room.** Thinking: all 19 LiveCodeBench failures used the whole 4,096-token budget (4,268 tokens or more); problems that finished thinking earlier went 18 for 18, budget-hit problems 23 for 42. No-think: 17 of its 22 LiveCodeBench failures ran to the 8,192-token answer cap (13 of the 28 hard problems and 5 of the 20 medium ones hit the cap). The budget is correlated with difficulty, so this shows where the failures sit, not by itself that more room would fix them; the 16k-budget rerun (below) tests that. |
| - **Why the 19 thinking-run LiveCodeBench failures failed** (re-running the problem's visible sample tests on the model's last answer): 12 give a wrong answer on a sample, 4 crash at run time, 2 pass the samples but fail hidden tests, 1 has no code block (it ran to the 8,192 cap). So 16 of 19 fail the problem's own examples, which the model never checked; this is the case a run-the-tests-and-repair loop (the Claude Code harness) is built for. The two that pass the samples get no feedback from `--retry`. |
| - **Speed.** Overall 75 tokens/s thinking and 91 tokens/s no-think on the A100 with all experts on the GPU; the L4 managed about 14 tokens/s overall and your PC has not been measured, so a 16k budget could cost many minutes per hard problem. |
| - **Caveat.** 60 problems per benchmark: the 95% interval on a LiveCodeBench rate is about +/-12 points, and six tasks of difference between two runs is within sampling noise. |
| |
| ### 8.1 Rerun of the failed tasks with a 16,384-token thinking budget (run 2026-10-26, A100) |
| The 21 tasks the 4k-budget thinking run failed (19 LiveCodeBench + 2 CRUXEval) were run again with `--reasoning-budget 16384` (context 24,576, answer cap 20,480, same temperature and sampling). Table: `results/k448_code_think16k_redo_compact.csv`. |
| |
| | | Rescued | Still failing | |
| |---|---|---| |
| | LiveCodeBench medium (5 failed before) | **5 of 5** | 0 | |
| | LiveCodeBench hard (14 failed before) | **4 of 14** | 10 | |
| | CRUXEval (2 failed before) | 0 of 2 | 2 (both finished early; the budget was not their problem) | |
| | **Total** | **9 of 21** | 12 | |
| |
| - **Estimate, not a measurement of all 120:** adding the 9 rescues to the 99 earlier passes gives **108/120 (90%)**, with LiveCodeBench at **50/60 (83.3%, 95% interval about 72-91%)**, so 105/120 is reached on this basis. The 99 tasks that passed before were not rerun; some of the 23 that passed on the very edge of the 4k budget could flip either way under a different sample. A clean figure needs all 120 tasks rerun at 16k (about 2 hours of A100 time; not run). |
| - **Mostly the budget, not luck.** A plain resample rescued 2 of 12 failures in an earlier control (about 17%), which would predict about 3 of 19 here; 9 of 19 is well above that. Some of the 9 are still likely luck. |
| - **The budget gets used up either way.** All 19 LiveCodeBench tasks used 16,400 tokens or more (average 17,300), including the 9 that passed and one that hit the 20,480 cap. So 16k is not a "finish early" budget for hard problems; a larger budget (32k) may rescue more of the remaining 10, which has not been tested. |
| - **Cost.** 333,605 tokens in 4,516 s for the 21 tasks (238 s per LiveCodeBench task at about 74 tokens/s on the A100), about 4 times the tokens per task of the 4k run. On the L4 earlier (14 tokens/s) this would be about 20 minutes per hard problem; your PC has not been measured. |
| |
| ### 8.2 Retry on top of the 16k budget (run 2026-10-26, A100) |
| The 10 LiveCodeBench problems still unsolved after the 16,384-token rerun were given up to 2 repair turns (`--retry 2`). After a failed answer the model is shown its first failing VISIBLE sample test (the problem's own examples; hidden tests are used only for the final score) and asked for a corrected solution, again with a 16,384-token thinking budget. The first attempt was the stored 16k answer, so any rescue comes from the repair turns. Table: `results/k448_code_think16k_retry_compact.csv`. |
| |
| | Result | Tasks | |
| |---|---| |
| | Rescued by 1 repair turn | arc194_d, 3674 | |
| | Rescued by 2 repair turns | arc192_e, arc196_b | |
| | Still failing after 2 turns | arc196_d, abc398_g, arc190_c, arc193_b | |
| | No feedback possible (the code already passes the visible samples, so the loop has nothing to show) | abc392_f, abc387_f | |
|
|
| - **4 of 10 rescued (all hard problems).** Combined estimate: 99 (4k run) + 9 (16k budget) + 4 (repair turns) = **112/120 (93.3%)**, LiveCodeBench **54/60 (90.0%)**, CRUXEval 58/60. That is above the 105/120 target and, on this sample, close to the published LiveCodeBench v6 figures in section 9 (88.8 to 91.9). |
| - **Caveats, stated plainly.** (1) This is a chain of estimates: the 99 tasks that passed in the first run were never rerun under the 16k or retry settings. A clean number needs all 120 tasks rerun (run 5, below). (2) The repair turns are an agent-style loop with test feedback; the published single-shot figures are not, so the comparison favours k448. (3) Selection effect: only previously failed tasks were retried, and plain resampling alone rescues some (2 of 12 in an earlier control). (4) 60 problems: the 95% interval on 54/60 is about 80-95%. |
| - **The two failure kinds the loop cannot help** are visible in the data: 2 of 10 tasks produce code that passes the examples and fails hidden tests, and 4 of 10 never get right even with feedback. |
| - **Cost.** 240,734 tokens in 3,411 s for the 10 tasks (341 s per task on the A100), all of it in repair turns because the first answers were reused. |
|
|
| ### 8.3 Clean rerun of all 120 tasks with a 16,384-token thinking budget (run 2026-10-26, A100) |
| The same 120 tasks as section 8, all of them rerun with `--reasoning-budget 16384` (context 24,576, answer cap 20,480, temperature 0.6 / top-p 0.95 / top-k 20, single attempt, no repair turns). This replaces the chained estimate of 8.1. Table: `results/k448_code_think16k_full_compact.csv`. |
|
|
| | | 4k budget (section 8) | **16k budget, clean rerun** | No-think | |
| |---|---|---|---| |
| | **All 120** | 99 (82.5%) | **107 (89.2%)** | 96 (80.0%) | |
| | LiveCodeBench (60) | 41 (68.3%) | **49 (81.7%, interval about 70-90%)** | 38 (63.3%) | |
| | - easy / medium / hard | 12/12, 15/20, 14/28 | 12/12, 19/20, 18/28 | 12/12, 15/20, 11/28 | |
| | CRUXEval (60) | 58 | 58 | 58 | |
| | Tokens (per task) | 277,274 (2,310) | 723,767 (6,031) | 262,258 (2,185) | |
| | Time (per task), A100 | 3,689 s (31 s) | 9,992 s (83 s) | 2,888 s (24 s) | |
|
|
| - **A longer thinking budget helps, and the gain is statistically solid.** Paired exact sign test, 16k vs 4k on the same 120 tasks: only the 16k run passes 9 tasks, only the 4k run passes 1, **p = 0.021**. That is the first coding result in this report that clears p < 0.05. The earlier chained estimate (99 + 9 rescues = 108, section 8.1) held: the clean run gave 107. |
| - **The gain is all in LiveCodeBench, and mostly medium and hard problems** (medium 15 -> 19 of 20, hard 14 -> 18 of 28). CRUXEval did not move (58/60 at every setting), as expected: those answers finish early. |
| - **It costs 2.6 times the tokens**: about 6,000 tokens and 83 s per task on the A100 (about 16,000 tokens per medium or hard LiveCodeBench problem; easy problems and CRUXEval stay short). On the 14 tokens/s measured on the L4 earlier that is about 20 minutes per hard problem; your PC has not been measured. |
| - **11 LiveCodeBench problems still fail**: arc192_e, arc196_d, arc194_d, abc392_f, abc398_g, arc190_c, abc387_f, abc393_e, medium arc192_a, arc193_b, 3674. Three of them (arc192_e, arc194_d, 3674) were rescued by repair turns in the earlier retry run (section 8.2, a different sample of answers), so repair turns on top of 16k would probably add a few more tasks (roughly 110/120), but that combination was not rerun cleanly. |
| - **Noise.** Single run per setting; 60 problems per benchmark. Run-to-run sampling noise on LiveCodeBench is a few problems, which is why the paired test (p = 0.021) matters more than the raw 8-task gap. |
|
|
| ### 8.4 A 32,768-token budget on the problems that still fail at 16k (run 2026-10-26, A100) |
| The 11 LiveCodeBench problems that failed in the clean 16k rerun (section 8.3) were run again with `--reasoning-budget 32768` (context 40,960, answer cap 36,864, single attempt). Table: `results/k448_code_think32k_redo_compact.csv`. |
|
|
| - **2 of 11 pass** (`abc393_e` and the medium problem `arc192_a`). Both had passed in other runs of the same model at a smaller budget (`abc393_e` at 4k, `arc192_a` at 16k in the 21-task rerun), so they are better read as sampling luck on borderline problems than as a gain from the larger budget. The **nine hard problems that failed at 16k still fail at 32k**: arc192_e, arc196_d, arc194_d, abc392_f, abc398_g, arc190_c, abc387_f, arc193_b, 3674. |
| - **Every one of the 11 used the whole 32k budget** (33,000 to 35,300 tokens). The model keeps thinking until it is stopped rather than converging on an answer for these problems. |
| - **Cost:** 373,556 tokens in 5,438 s (about 34,000 tokens and 494 s per problem). Doubling the budget again cost twice as much per problem for no measurable gain. |
| - **Conclusion:** the budget gain stops between 16k and 32k on this set. 16k is the sensible setting; for the remaining hard problems the repair-turn loop of section 8.2 (feedback from the problem's own sample tests) rescued more (4 of 10) than a bigger budget did (0 of 9 hard ones). |
|
|
| ## 9. Comparison with strong models (published numbers, and where k448 sits) |
| Source: the Qwen3.8-Flash-Next model card on Hugging Face (August 2026). All numbers are vendor-reported, not independently verified; the card does not say whether a score is thinking or non-thinking (the model thinks by default) and does not state the thinking budget. |
|
|
| | Benchmark | Flash-Next (parent, full precision) | Qwen3.8-27B | Qwen3.7-Plus | DeepSeek-V4-Flash-0731 | Claude-Opus-4.6 (Max) | |
| |---|---|---|---|---|---| |
| | LiveCodeBench v6 | 91.9 | 90.3 | 89.6 | 90.6 | 88.8 | |
| | k448, thinking 4k budget (ours, 60-problem sample) | 68.3 (95% interval 56-79) | | | | | |
| | k448, no-think (ours, same 60 problems) | 63.3 (95% interval 51-74) | | | | | |
| | k448, 16k budget, clean rerun of all 60 problems (section 8.3) | **81.7 (interval about 70-90)** | | | | | |
| | k448, 16k budget on the failed tasks (estimate, section 8.1) | 83.3 (interval about 72-91) | | | | | |
| | k448, 16k budget + test-feedback repair turns (estimate, section 8.2; an agent-style loop, not single-shot) | 90.0 (interval about 80-95) | | | | | |
|
|
| - **Only LiveCodeBench lines up.** CRUXEval, HumanEval, MBPP, MATH-500, MMLU-Pro and BBH are not in the card, so no published figure for those is quoted here. |
| - **Not like for like.** Our LiveCodeBench is 60 problems sampled from a 175-problem slice (2025-01 to 2025-04) of the release 6 data, answered with a 4,096-token thinking budget and an 8,192-token answer cap; the vendor figure is the full v6 set, presumably with a far longer budget. With 60 problems one run has a 95% interval of roughly +/-12 points. Expect our number to sit below the published one; the gap mixes quantization, budget and sampling, and cannot be split without running the full-precision model (or Q4) through the same harness. |
| - **Where k448 sits.** With a 4,096-token thinking budget k448 scores 68.3% on LiveCodeBench against the published 91.9% for the full-precision parent and 88.8% for Claude-Opus-4.6 (Max). With a 16,384-token budget, rerun cleanly on all 60 problems (section 8.3), it scores **81.7% (interval about 70-90%)**: a significant improvement over 4k (paired p = 0.021), and the published 91.9 now sits just above the top of our interval, while the other published figures (88.8 to 90.6) sit inside it. So with a long enough thinking budget k448 is plausibly within the range of the published models on this benchmark, but 60 problems cannot separate it from them. Adding up to two repair turns with the problems' own sample tests (section 8.2) would add a few more problems, but that is an agent-style loop with feedback, which the published single-shot numbers do not use, and it was measured on a different sample of answers. What remains could be quantization, a still longer budget (32k test: section 8.4, which found no gain over 16k), or sampling noise; separating them needs a Q4 or full-precision run on the same harness. |
| - **We did not run the other models ourselves** (no API access from this setup), so every competitor number above is as published. |
|
|
| ## 10. Reasoning: does a bigger thinking budget help? (run 2026-10-26, A100) |
| Quick reasoning set from the big suite (`--hard-n 60 --only reason`): 40 MMLU-Pro questions (10 answer options, 14 subjects) and 20 BIG-Bench Hard tasks, seed 1, same tasks every run. The no-think run is the baseline; the 10 questions it failed were then redone with thinking at two budgets (the 50 it passed were not rerun). Tables: `results/k448_reason_nothink_compact.csv`, `..._think4k_compact.csv`, `..._think16k_compact.csv`. |
|
|
| | Setting | Result | Tokens (per task) | |
| |---|---|---| |
| | No-think, 2,048-token cap (60 tasks) | **50/60 (83.3%)**: BBH 20/20, MMLU-Pro 30/40 | 40,632 (677) in 468 s | |
| | Thinking, 4,096 budget, the 10 failures | **5 of 10 rescued** | 31,492 (3,149) in 393 s | |
| | Thinking, 16,384 budget, the 10 failures | **3 of 10 rescued** | 106,193 (10,619) in 1,315 s | |
|
|
| - **Every failure was an MMLU-Pro question.** BBH (logic puzzles, tracking, date and fallacy questions) was 20/20 without thinking; there is nothing to gain there. 6 of the 10 failures ran into the 2,048-token cap without producing an answer, so part of the no-think failure is just running out of room. |
| - **Thinking at 4k rescued 5 of the 10** (mmlu11759, 4613, 12187, 11789, 11236). Applied to the whole set that would be about 55/60 (91.7%), but only the failures were redone, so that is an estimate. The earlier lab on the L4 found the same thing (thinking at 4k rescued 3 of 7 reasoning failures). |
| - **A 16k budget did not help more; it rescued fewer (3 of 10)** at 3.4 times the tokens. Rescued at 16k: mmlu11759, 12187, 11236 (all three also rescued at 4k). On MMLU-Pro, which mostly tests knowledge and short multi-step reasoning, thinking longer than a few thousand tokens just does not buy accuracy; this is the opposite of the coding result (section 8.3), where the extra budget mattered. |
| - **How much is luck?** Resampling alone rescued 2 of 12 failures in an earlier control (about 17%). Against that rate, 5 of 10 is unlikely by chance (rough one-sided binomial p about 0.016) and 3 of 10 is not (p about 0.22). The 10 questions and the 17% control rate are both small, so treat this as suggestive, not proven. A proper test would rerun all 60 under each setting. |
| - **Two failures never flipped** (mmlu527, mmlu4987): short wrong answers in every setting, probably knowledge gaps or ambiguous questions rather than a budget problem. |
| - **Recommendation for reasoning:** thinking with a 4,096-token budget, not more. Use 16k only for coding (section 8.3). |
|
|
| ## 12. Agentic coding on SWE-bench Verified Mini through Claude Code (run 2026-10-26, A100) |
| The real Claude Code CLI (2.1.197, headless `claude -p`, run as an unprivileged user) was pointed at k448 (llama-server, `/v1/messages`, thinking off, 65,536-token context, MTP draft) and given 20 instances of SWE-bench Verified Mini: 10 django and 10 sphinx whose own gold patch reproduces without Docker here (37 of the 50 instances do: 25 django + 12 sphinx; the 13 old sphinx 3.x instances fail even with the gold patch and are excluded). Grading uses swebench 4.0.4's own log parsers on FAIL_TO_PASS and PASS_TO_PASS. Runner `13_swe_native.py`, proxy `14_msg_proxy.py`. Tables: `results/k448_swe_compact.csv` (50-turn pass) and `results/k448_swe_t100_compact.csv` (100-turn rerun of the 8 misses). |
|
|
| | | Result | |
| |---|---| |
| | As run, 50-turn cap | **12/20 (60%)**: django 8/10, sphinx 4/10 | |
| | The 8 misses rerun with a 100-turn cap | +3 (django-11790, sphinx-9229, sphinx-9230) | |
| | As run, combined (a second attempt for the misses) | **15/20 (75%)**, 95% interval about 53-89% | |
| | **Resolved with no lookup of the upstream project on the internet** | **7/20 (35%)**, interval about 18-57%: 6 at 50 turns (django 11880, 11951, 11999, 12039, 12050; sphinx 9367) plus django-11790 in the rerun | |
|
|
| ### The headline number is not clean: the agent went to the internet |
| The sandbox had open internet and Claude Code had its normal tools (Bash, WebFetch, WebSearch). The transcripts show it used them to look for the upstream fix: |
| - In the first django-11790 run the agent searched GitHub for the upstream pull request and went to fetch it (it wrote that it had found the upstream PR and would fetch the actual fix). |
| - **8 of the 15 resolved instances consulted the internet**: django-11848 (downloaded raw files from GitHub), 11885 (listed and cloned upstream django), 11964 (fetched django 3.1.3's `enums.py`, which already contains the fix), sphinx-10323 (listed and cloned upstream sphinx, one WebFetch), 10466 (WebFetch plus a GitHub issue lookup), 9320 (GitHub issue and timeline lookups), 9229 (fetched sphinx 4.1.0 source and searched issues), 9230 (nine WebFetch calls and an issue search). The other 7 resolved without any such lookup (django-11790's rerun only downloaded dependency packages). |
| - Not checked: whether the local git clone still holds future commits. Several transcripts ran `git log --all` or `git show <sha>`. |
| - So **7/20 is a floor** (a lookup does not prove it was needed) and **15/20 is a ceiling**; the true no-help rate is somewhere between. The 15 is also a best of two attempts for the 8 reruns. |
| - Official SWE-bench style runs do not give the agent the web, so the as-run number cannot be compared with leaderboard figures. |
|
|
| ### Numbers |
| - **Tool calls:** 912 in the 50-turn pass, 62 errors (6.8%); 694 in the rerun, 36 errors (5.2%). |
| - **Turns and time:** 45.8 turns and 343 s per instance on average at the 50-turn cap; 87.1 turns and 668 s in the rerun. The 28 agent runs took 12,191 s (3.4 h) of A100 time, about 23 compute units. |
| - **When the agent stopped on its own it was usually right:** 9 of 20 runs ended with the agent declaring it was done, and 8 of those 9 were resolved. The 11 runs cut off by the turn cap resolved 4. |
|
|
| ### Failure modes (per-instance results plus the transcripts I read) |
| 1. **Turn cap with the work unfinished:** 11 of 20 in the first pass, 7 of those unresolved. |
| 2. **Context overflow in long runs:** 5 of the 8 reruns (sphinx-10435, 10673, 11510, 8265, 9229) died with HTTP 400, "request (65,5xx to 66,3xx tokens) exceeds the available context size (65,536 tokens)". Claude Code does not know the server window is 65k, so it never compacts. The GPU holds 49.7 of 80 GB at 65k, so a 128k context fits. |
| 3. **A fix that passes the agent's own checks but not the target tests:** django-11815 (1 of 2 target tests, the same at 50 and 100 turns), sphinx-10673 (0/1, patches of 164 and 175 lines), 11510 (0/2), 8265 (0/1), 9230 at 50 turns (0/1). |
| 4. **Empty patch after long exploration:** django-11790 and sphinx-8265 and 9229 at 50 turns, sphinx-10435 in the rerun. |
| 5. **A malformed tool call:** one WebFetch call in sphinx-10466 had template markup (`</function></tool_call><tool_call><function=WebFetch>`) inside its URL argument; one example seen, not counted across all runs. |
| 6. **Environment friction:** in the first django-11790 transcript the agent spent turns finding a Python that works for the unprivileged user (one interpreter path returned permission denied) and imported Django through the system Python with a path hack. |
| 7. **An oversized patch:** the resolved sphinx-9229 patch is 1,657 lines (the other resolved ones are 12 to 215); it passed with no regressions but would not survive code review. Contents not inspected. |
|
|
| ### Setup caveats |
| - Thinking off (`enable_thinking` false), 65,536 context, sampling left to Claude Code and the server defaults. |
| - **Claude Code needs the rewriting proxy:** Qwen3.8's chat template raises "System message must be at the beginning" for the system-role entry Claude Code sends after the first user turn. The proxy folds it into the last user message; 1,594 requests went through, all 200 except the 5 context overflows. |
| - Docker-free environments, not the official images: uv venvs, `setuptools<70`, `Pygments==2.18.0`, Python 3.8 for django (the images use older versions), 37 of 50 instances reproduce. |
| - 20 instances: the interval is about +/-18-20 points, so 60% and 75% are not different from each other statistically. |
| - **Section 9's published agentic table (SWE-bench Pro, Multilingual, DeepSWE, NL2Repo, Toolathlon) is a different, harder set of benchmarks and is not comparable.** There is no published number for this exact setup. |
|
|
| ### Verdict |
| k448 under Claude Code can do small, localized bug fixes: the 7 clean resolves are short patches (12 to 215 lines) at about 6 minutes each on the A100, and a "done" from the agent was right 8 times out of 9. It is not shown to be a daily coding agent. The no-help rate is between 35% and 75% on 20 instances, hard tasks hit the turn cap or the 65k context, and the agent does look up upstream fixes when it can, so its own skill cannot be separated from the lookups in this run. Speed on your PC is not measured (14 tokens/s on the L4 earlier would make 6 minutes into about 30). |
|
|
| **Next experiment, proposed and not run:** the same 20 instances offline (network blocked for the agent user, WebFetch and WebSearch disallowed, future git objects removed), a 128k context, the same turn caps, and a paired comparison with this table. About 2.5 to 3 hours of A100, roughly 17 to 20 units. |
|
|
| ## 13. Overnight summary (2026-10-26, A100) |
| A100 time is from each run's own clock; units are time x 6.77 per hour, a rough figure that leaves out loading and idle time. |
|
|
| | # | Experiment | Result | Test | A100 time (about units) | |
| |---|---|---|---|---| |
| | 1 | Coding, 4k thinking budget (120 tasks) | 99/120 (LCB 41/60, CRUX 58/60) | baseline | 3,689 s (6.9) | |
| | 2 | Coding, no-think | 96/120 (LCB 38/60) | vs 4k: p = 0.508 | 2,888 s (5.4) | |
| | 3 | Redo of the 21 failures at 16k | 9 of 21 rescued | luck baseline about 17% | 4,516 s (8.5) | |
| | 4 | Two repair turns on the 10 still failing | 4 of 10 rescued (chained estimate 112/120) | estimate only | 3,411 s (6.4) | |
| | 5 | **Coding, clean rerun of all 120 at 16k** | **107/120 (89.2%), LCB 49/60** | vs 4k: p = 0.021 | 9,992 s (18.8) | |
| | 6 | 32k on the 11 still failing | 2 of 11 (likely luck) | none | 5,438 s (10.2) | |
| | 7 | Reasoning, no-think (60 tasks) | 50/60 | baseline | 468 s | |
| | 8 | Reasoning, 4k thinking on the 10 failures | 5 of 10 rescued | p about 0.016 vs luck baseline | 393 s | |
| | 9 | Reasoning, 16k thinking on the 10 failures | 3 of 10 rescued | p about 0.22 vs luck baseline | 1,315 s (rows 7 to 9: 4.1) | |
| | 10 | SWE-bench Verified Mini through Claude Code, 50 turns | 12/20 as run; 6/20 with no web lookup | none | 6,850 s (12.9) | |
| | 11 | Rerun of the 8 misses, 100 turns | +3 as run (+1 with no lookup) | none | 5,341 s (10.0) | |
|
|
| - **Coding:** a 16k thinking budget is the setting to use (107/120, significantly better than 4k); 32k adds nothing; repair turns on top rescue a few hard problems. |
| - **Reasoning:** thinking at 4k, not more; a bigger budget did not help. |
| - **Agentic:** 7 to 15 of 20, depending on whether web lookups count; not established until the offline rerun. |
| - **Units:** 84.10 at 06:20, 58.12 at 10:13, 51.37 at 11:13, **45.17 at 12:05**, burning 6.77 per hour while the runtime stays connected, idle or not. |
| - **A100 runtime:** disconnected and deleted at 12:10 on 2026-10-26 at your request, after the units reading of 45.17 at 12:05 (the balance cannot be read without a runtime, so the final figure is a little lower). The k448 and MTP files on that VM came from Hugging Face (download cache with the same commit hash), so they are re-downloadable; the raw SWE-bench transcripts, results JSONs and logs on the VM are gone (the compact CSVs and the report are saved locally). llama.cpp PR 30594 was not touched. |
|
|
