Add benchmark tables, scripts and report

benchmarks/REPORT.md

@@ -0,0 +1,272 @@
+# k448 ("Tessera", Qwen3.8-Flash-Next, 448 experts) - capability lab and speed report, 2026-10-25
+
+All quality numbers: Colab L4 VM, llama.cpp, 120-task hard suite (HumanEval+, MATH-500, MMLU-Pro; 40 each; seed 1).
+Wording rule used throughout: "beats Q4" only if the paired sign test is p < 0.05 in our favour.
+
+## 1. Verdict against Q4 (unsloth UD-IQ4_XS, 87 GiB)
+- Hard test, no-think greedy: Q4 103/120, k448 101/120 (code 39/40 both, math 35/40 both, reasoning 29 vs 27).
+- Paired sign test p = 0.754. **k448 MATCHES Q4 within noise. It does not beat it.**
+- Speed with llama.cpp on the VM: 11.7 vs 8.2 tokens/s (+43%). Memory while running: 41.9 GiB resident, of which 6.3 GiB is real working memory and 35.5 GiB is the mapped model file; the 26.8 GiB n-gram table is lazy.
+- Not compared: Strata's own ISTA-DASLab packs (IQ2_XS 39.2 GB, IQ3_XXS 47 GB) are the real size peers; k448 keeps about 47 GB resident.
+
+## 2. What raised capability (weights unchanged, run-time settings only)
+| Setting | Result |
+|---|---|
+| Thinking, reasoning budget 4096, temp 0.6 / top_p 0.95 / top_k 20 | rescued 8 of 12 tasks the greedy run failed (math 5/5, reasoning 3/7); 9/15 on the full redo. About 3.8k tokens per task. |
+| Resampling control (temp 0.6, no thinking) | 2/12 |
+| 3-vote majority (temp 0.7) | 2/12, the same two tasks as resampling; no gain from voting |
+| Thinking vs resampling, paired | 6 vs 0, exact sign test p = 0.031 (small n = 12, tasks pre-selected as failures) |
+| 14 active experts instead of 10 | math 37 vs 35, reasoning 27 vs 27, p = 0.688: no significant gain; perplexity gap to Q4 shrinks but KLD to Q4 does not |
+| MTP draft (llama.cpp) | x1.34 speed with identical answers |
+
+Recommended profile: everyday use no-think greedy, 10 experts, MTP on. For hard math or reasoning turn thinking on with a 4096 budget (about twice the tokens). Do not use voting or 14 experts.
+
+## 3. Speed with Strata's engine (github.com/Niko1221/Strata, MIT, engine 0.1.51)
+Same Colab VM (L4, 12 vCPU with AVX2 only, 54 GB RAM), k448 packed with Strata's own tool, no MTP head loaded:
+| Run | Decode tokens/s |
+|---|---|
+| llama.cpp, no MTP | 11.7 |
+| llama.cpp with MTP | about 16 |
+| Strata, 256 tokens, `--spec 4` | 15.4 |
+| Strata, 256 tokens, `--spec 2` | 18.4 |
+| Strata, 1024 tokens, `--spec 2` | **27.2** |
+- The 27.2 figure includes lookup drafting (37.5% of drafted tokens accepted) on a repetitive code answer; prose will gain less.
+- Output of the 256-token run was coherent, correct Python. Quality of the Strata path has NOT been run through the hard suite.
+- NOT measured on the RTX 4060 laptop. The VM's slow disk (12 MB/s random reads) and small RAM held Strata back here; the PC may differ either way.
+
+Steps that worked on the VM (a source build; no installer scripts were run):
+1. `cmake -S . -B build -DCMAKE_BUILD_TYPE=Release -DSTRATA_ENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89 && cmake --build build --target strata -j4`
+2. `STRATA_GGUF_PY=build/_deps/strata_llamacpp-src/gguf-py python3 tools/iq_pack.py --gguf <k448.gguf> --out packs/k448 --compat-bf16` (under a minute; 303 tensors served natively, 195 small tensors rounded to BF16, max error 0.0144)
+3. `python3 tools/make_profile.py --no-base --n-expert 448 --out k448-profile.bin` (the shipped profile is for 512 experts)
+4. `./build/strata --pack packs/k448 --tokens-file prompt.txt --native <k448.gguf> --ple-gguf <k448.gguf> --ple-io mmap --max-new 1024 --max-context 4096 --expert-profile k448-profile.bin --expert-cache auto --kv int8 --spec 2 --prefill 512`
+   (the binary takes no subcommand; native packs require `--native`, `--spec T>=2` and `--prefill`; on a fast NVMe drop `--ple-io mmap`)
+
+## 4. Open items
+- Strata's MTP draft layer: needs about 5-6 GB fetched from the public Qwen/Qwen3.8-Flash-Next repo (tools/mtp_fetch.py). Not downloaded; awaiting the user's yes. Our llama.cpp-format MTP file cannot be used by Strata.
+- Measure on the user's PC (the user downloads the k448 file themselves; no token is ever entered by Claude).
+- Run the hard suite through Strata's server to confirm quality parity.
+- Compare k448 against Strata's ISTA IQ3_XXS / IQ2_XS on the same tasks.
+- Autonomous (agent-style) coding was never tested; only function-level tasks.
+
+## 5. The big test suite (`09_big_suite.py`, built 2026-10-25)
+Why: 120 tasks only detect differences of about 10 points. This set has 4 domains x 300 tasks = 1200, all public benchmarks, every answer checked by code.
+
+| Domain | 300 tasks = | Check |
+|---|---|---|
+| code | 120 HumanEval+ (all but HumanEval/32, whose own reference fails) + 180 MBPP sanitized | unit tests on the model's function |
+| math | 225 MATH-500 integer answers (levels 2-5) + 75 AIME (AI-MO/aimo-validation-aime) | final line `#### <integer>` |
+| reasoning | 200 MMLU-Pro (14 subjects round-robin, 10 options) + 100 BIG-Bench Hard (8 tasks) | final line `Answer: <x>` |
+| English | 100 RACE-high reading comprehension + 100 CoLA grammar (half acceptable) + 100 writing prompts with machine-checked rules (15 rule types: word counts, paragraphs, bullets, case, keywords, start/end text, quotes, title, commas, P.S., letter count) | `Answer:` line, or the rules in code |
+
+- **Nested**: the first 60 tasks of every domain are a prefix of the 300 (`--hard-n 60` = quick set, 240 tasks; `--hard-n 300` = full set). Results of the two can be pooled.
+- **Checked**: selftest passes locally and on the VM (references pass, garbage fails, each rule checker passes its own good/bad example); all 420 code reference solutions run; the VM file has the same sha256 prefix as the local one (`5700b5f8682b8978`).
+- **Run it**: `python3 07x_domain_eval.py run --suite big --hard-n 60 --seed 1 --no-think --name <n> --out results_<n>.json` plus the usual `--bin/--model/--ot/--extra` flags; compare with `07x_domain_eval.py compare a.json b.json` (paired sign test per domain).
+- **Power** (paired exact sign test, 5%): about 5 points per domain at 300 tasks, about 10 points at 60, about 2.5 points when the four domains are pooled (1200). Treat the English rule-following prompts as a relative measure only: they are our own set, not IFEval, so they cannot be compared with published numbers.
+- **Smoke run** (3 tasks per domain, llama.cpp + MTP, no-think, Colab L4): 12/12 pass, 4781 tokens in 471 s = about 400 tokens and 39 s per task, 10.2 tokens/s overall. That puts the quick set (240 tasks) at about 2.6 hours and the full set (1200 tasks) at about 13 hours for one setting and one model; thinking mode costs roughly 5-10x more tokens per task. A 12-task sample, so treat the times as rough; Strata at about 27 tokens/s on this machine would cut them by more than half, but it has not been run through this suite.
+- **Known limits**: code is already near the ceiling (39/40 on HumanEval+), so code gains will be small; there is no agent-style coding test.
+
+## 6. Quick-set baseline (run 2026-10-25)
+k448 + MTP draft, llama.cpp, no-think greedy, Colab L4, big suite `--hard-n 60` (60 tasks per domain = 240, seed 1). 137,690 tokens in 9,799 s (14.1 tokens/s overall, about 570 tokens and 41 s per task).
+
+| Domain | Pass | Parts |
+|---|---|---|
+| code | 55/60 (91.7%) | HumanEval+ 22/24, MBPP 33/36 |
+| math | 39/60 (65.0%) | MATH-500 39/45, AIME 0/15 |
+| reasoning | 51/60 (85.0%) | MMLU-Pro 31/40, BBH 20/20 |
+| English | 52/60 (86.7%) | RACE 18/20, CoLA 17/20, rule-following 17/20 |
+| **All** | **197/240 (82.1%)** | |
+
+- **AIME 0/15 is a token-cap result, not a measure of ability.** The no-think runs on this suite are capped at 2048 tokens (the harness raises `--max-tokens` to at least 2048), and the AIME answers used 2048 tokens on average, so every one ran to the cap. MATH-500 averaged 1,024 tokens, so some of its misses are probably cut off too. Math is the domain where thinking mode (budget 4096) should move the number most.
+- Strongest domain is code (HumanEval+ and MBPP are near the ceiling, as noted in section 5); BBH is a clean 20/20.
+- With 60 tasks per domain only differences of about 10 points are detectable (section 5), so this is a reference point for later changes, not a ranking. It has not been run on Q4 yet, so it says nothing about k448 vs Q4 on this suite.
+- Raw per-task results: `results_big60_k448.json` on the L4 VM (`/content/q38/eval/`); compare later runs with `07x_domain_eval.py compare`.
+
+## 7. The harder coding suite (`10_code_hard.py`, built 2026-10-25)
+Why: HumanEval+ and MBPP (92% for k448) are small textbook functions that top models already pass. These two are the harder, widely used replacements that we can run and score by code without a token.
+
+| Benchmark | What it tests | In the suite |
+|---|---|---|
+| LiveCodeBench (code_generation_lite, release 6) | contest programming: the model writes a whole program (AtCoder, stdin/stdout) or a LeetCode class; every one of about 40 hidden tests must pass | 175 problems from 2025-01 to 2025-04 (newer than most training data); ids carry the difficulty (easy 43, medium 52, hard 80) |
+| CRUXEval | code understanding: predict the output of a function for an input (CRUXEval-O), or find an input that gives a given output (CRUXEval-I) | 800 functions, half O and half I per task list |
+
+- **Run it**: `python3 07x_domain_eval.py run --suite codehard --hard-n 60 --max-tokens 4096 --ctx 8192 --no-think --seed 1 ...` (60 = quick set: 60 LiveCodeBench + 60 CRUXEval; nested prefixes, so `--hard-n 100` extends it). New categories `lcb` and `crux` in the compare table.
+- **Checked**: selftest passes locally and on the VM (lookup-table "solutions" pass every test of 4 stdin and 4 LeetCode-style problems and a wrong solution fails; CRUXEval references pass, garbage fails); all 800 CRUXEval references run; all 175 LiveCodeBench test sets decode (the private tests are a pickled string, loaded with a loader that refuses every global, so a dataset cannot run code). The VM file has the same sha256 prefix as the local one (`2c9c82f1effce90a`).
+- **Expect low numbers**: contest problems are hard. Without thinking, most hard AtCoder problems will fail; thinking mode will matter far more here than on the old sets. The release is recent, but the model's training cutoff is unknown, so some problems may still be familiar.
+- **Time**: LiveCodeBench answers are long (about 1-2k tokens). Roughly 2 hours for the 60+60 quick set and 3.5 hours for 100+100 at llama.cpp speeds; thinking mode several times that.
+- **Not included, and why**: SWE-bench Verified and the Aider polyglot benchmark need a Docker or multi-language toolchain and an agent loop that edits real repositories, which the Colab VM cannot host; BigCodeBench-Hard (148 library-heavy tasks) needs dozens of heavy libraries installed first and is the next candidate if we want one more.
+
+## 8. Hard coding results (run 2026-10-26, Colab A100, llama.cpp + MTP draft, seed 1)
+60 LiveCodeBench problems + 60 CRUXEval tasks, the same 120 tasks in both runs. Thinking: budget 4,096 tokens, temperature 0.6, top-p 0.95, top-k 20. No-think: greedy. Both: 8,192-token answer cap, all experts on the GPU. Per-task tables: `results/k448_code_think_compact.csv`, `results/k448_code_nothink_a100_compact.csv`.
+
+| | Thinking (4k budget) | No-think (greedy) |
+|---|---|---|
+| **All 120** | **99/120 (82.5%)** | **96/120 (80.0%)** |
+| LiveCodeBench | 41/60 (68.3%) | 38/60 (63.3%) |
+| - easy / medium / hard | 12/12, 15/20, 14/28 | 12/12, 15/20, 11/28 |
+| CRUXEval | 58/60 (96.7%) | 58/60 (96.7%) |
+| - CRUX-O / CRUX-I | 28/30, 30/30 | 28/30, 30/30 |
+| Tokens (per task) | 277,274 (2,310) | 262,258 (2,185) |
+| Time (per task), A100 | 3,689 s (31 s) | 2,888 s (24 s) |
+
+- **Thinking does not beat no-think here.** Paired exact sign test: only thinking passes 6 tasks (CRUX-O 155; LCB hard 3777, abc388_g, arc192_d; LCB medium 3722, 3779), only no-think passes 3 (CRUX-O 445; LCB medium 3805, abc400_d), p = 0.508. Both runs are inside each other's noise; the earlier thinking gain (section 2) was on math and reasoning tasks that greedy had already failed, not on coding.
+- **CRUXEval is at its ceiling** (96.7% both ways), so the room for improvement is in LiveCodeBench, mostly the hard problems (about half of the 60 problems, 46% of the pool).
+- **Both modes lose their points by running out of room.** Thinking: all 19 LiveCodeBench failures used the whole 4,096-token budget (4,268 tokens or more); problems that finished thinking earlier went 18 for 18, budget-hit problems 23 for 42. No-think: 17 of its 22 LiveCodeBench failures ran to the 8,192-token answer cap (13 of the 28 hard problems and 5 of the 20 medium ones hit the cap). The budget is correlated with difficulty, so this shows where the failures sit, not by itself that more room would fix them; the 16k-budget rerun (below) tests that.
+- **Why the 19 thinking-run LiveCodeBench failures failed** (re-running the problem's visible sample tests on the model's last answer): 12 give a wrong answer on a sample, 4 crash at run time, 2 pass the samples but fail hidden tests, 1 has no code block (it ran to the 8,192 cap). So 16 of 19 fail the problem's own examples, which the model never checked; this is the case a run-the-tests-and-repair loop (the Claude Code harness) is built for. The two that pass the samples get no feedback from `--retry`.
+- **Speed.** Overall 75 tokens/s thinking and 91 tokens/s no-think on the A100 with all experts on the GPU; the L4 managed about 14 tokens/s overall and your PC has not been measured, so a 16k budget could cost many minutes per hard problem.
+- **Caveat.** 60 problems per benchmark: the 95% interval on a LiveCodeBench rate is about +/-12 points, and six tasks of difference between two runs is within sampling noise.
+
+### 8.1 Rerun of the failed tasks with a 16,384-token thinking budget (run 2026-10-26, A100)
+The 21 tasks the 4k-budget thinking run failed (19 LiveCodeBench + 2 CRUXEval) were run again with `--reasoning-budget 16384` (context 24,576, answer cap 20,480, same temperature and sampling). Table: `results/k448_code_think16k_redo_compact.csv`.
+
+| | Rescued | Still failing |
+|---|---|---|
+| LiveCodeBench medium (5 failed before) | **5 of 5** | 0 |
+| LiveCodeBench hard (14 failed before) | **4 of 14** | 10 |
+| CRUXEval (2 failed before) | 0 of 2 | 2 (both finished early; the budget was not their problem) |
+| **Total** | **9 of 21** | 12 |
+
+- **Estimate, not a measurement of all 120:** adding the 9 rescues to the 99 earlier passes gives **108/120 (90%)**, with LiveCodeBench at **50/60 (83.3%, 95% interval about 72-91%)**, so 105/120 is reached on this basis. The 99 tasks that passed before were not rerun; some of the 23 that passed on the very edge of the 4k budget could flip either way under a different sample. A clean figure needs all 120 tasks rerun at 16k (about 2 hours of A100 time; not run).
+- **Mostly the budget, not luck.** A plain resample rescued 2 of 12 failures in an earlier control (about 17%), which would predict about 3 of 19 here; 9 of 19 is well above that. Some of the 9 are still likely luck.
+- **The budget gets used up either way.** All 19 LiveCodeBench tasks used 16,400 tokens or more (average 17,300), including the 9 that passed and one that hit the 20,480 cap. So 16k is not a "finish early" budget for hard problems; a larger budget (32k) may rescue more of the remaining 10, which has not been tested.
+- **Cost.** 333,605 tokens in 4,516 s for the 21 tasks (238 s per LiveCodeBench task at about 74 tokens/s on the A100), about 4 times the tokens per task of the 4k run. On the L4 earlier (14 tokens/s) this would be about 20 minutes per hard problem; your PC has not been measured.
+
+### 8.2 Retry on top of the 16k budget (run 2026-10-26, A100)
+The 10 LiveCodeBench problems still unsolved after the 16,384-token rerun were given up to 2 repair turns (`--retry 2`). After a failed answer the model is shown its first failing VISIBLE sample test (the problem's own examples; hidden tests are used only for the final score) and asked for a corrected solution, again with a 16,384-token thinking budget. The first attempt was the stored 16k answer, so any rescue comes from the repair turns. Table: `results/k448_code_think16k_retry_compact.csv`.
+
+| Result | Tasks |
+|---|---|
+| Rescued by 1 repair turn | arc194_d, 3674 |
+| Rescued by 2 repair turns | arc192_e, arc196_b |
+| Still failing after 2 turns | arc196_d, abc398_g, arc190_c, arc193_b |
+| No feedback possible (the code already passes the visible samples, so the loop has nothing to show) | abc392_f, abc387_f |
+
+- **4 of 10 rescued (all hard problems).** Combined estimate: 99 (4k run) + 9 (16k budget) + 4 (repair turns) = **112/120 (93.3%)**, LiveCodeBench **54/60 (90.0%)**, CRUXEval 58/60. That is above the 105/120 target and, on this sample, close to the published LiveCodeBench v6 figures in section 9 (88.8 to 91.9).
+- **Caveats, stated plainly.** (1) This is a chain of estimates: the 99 tasks that passed in the first run were never rerun under the 16k or retry settings. A clean number needs all 120 tasks rerun (run 5, below). (2) The repair turns are an agent-style loop with test feedback; the published single-shot figures are not, so the comparison favours k448. (3) Selection effect: only previously failed tasks were retried, and plain resampling alone rescues some (2 of 12 in an earlier control). (4) 60 problems: the 95% interval on 54/60 is about 80-95%.
+- **The two failure kinds the loop cannot help** are visible in the data: 2 of 10 tasks produce code that passes the examples and fails hidden tests, and 4 of 10 never get right even with feedback.
+- **Cost.** 240,734 tokens in 3,411 s for the 10 tasks (341 s per task on the A100), all of it in repair turns because the first answers were reused.
+
+### 8.3 Clean rerun of all 120 tasks with a 16,384-token thinking budget (run 2026-10-26, A100)
+The same 120 tasks as section 8, all of them rerun with `--reasoning-budget 16384` (context 24,576, answer cap 20,480, temperature 0.6 / top-p 0.95 / top-k 20, single attempt, no repair turns). This replaces the chained estimate of 8.1. Table: `results/k448_code_think16k_full_compact.csv`.
+
+| | 4k budget (section 8) | **16k budget, clean rerun** | No-think |
+|---|---|---|---|
+| **All 120** | 99 (82.5%) | **107 (89.2%)** | 96 (80.0%) |
+| LiveCodeBench (60) | 41 (68.3%) | **49 (81.7%, interval about 70-90%)** | 38 (63.3%) |
+| - easy / medium / hard | 12/12, 15/20, 14/28 | 12/12, 19/20, 18/28 | 12/12, 15/20, 11/28 |
+| CRUXEval (60) | 58 | 58 | 58 |
+| Tokens (per task) | 277,274 (2,310) | 723,767 (6,031) | 262,258 (2,185) |
+| Time (per task), A100 | 3,689 s (31 s) | 9,992 s (83 s) | 2,888 s (24 s) |
+
+- **A longer thinking budget helps, and the gain is statistically solid.** Paired exact sign test, 16k vs 4k on the same 120 tasks: only the 16k run passes 9 tasks, only the 4k run passes 1, **p = 0.021**. That is the first coding result in this report that clears p < 0.05. The earlier chained estimate (99 + 9 rescues = 108, section 8.1) held: the clean run gave 107.
+- **The gain is all in LiveCodeBench, and mostly medium and hard problems** (medium 15 -> 19 of 20, hard 14 -> 18 of 28). CRUXEval did not move (58/60 at every setting), as expected: those answers finish early.
+- **It costs 2.6 times the tokens**: about 6,000 tokens and 83 s per task on the A100 (about 16,000 tokens per medium or hard LiveCodeBench problem; easy problems and CRUXEval stay short). On the 14 tokens/s measured on the L4 earlier that is about 20 minutes per hard problem; your PC has not been measured.
+- **11 LiveCodeBench problems still fail**: arc192_e, arc196_d, arc194_d, abc392_f, abc398_g, arc190_c, abc387_f, abc393_e, medium arc192_a, arc193_b, 3674. Three of them (arc192_e, arc194_d, 3674) were rescued by repair turns in the earlier retry run (section 8.2, a different sample of answers), so repair turns on top of 16k would probably add a few more tasks (roughly 110/120), but that combination was not rerun cleanly.
+- **Noise.** Single run per setting; 60 problems per benchmark. Run-to-run sampling noise on LiveCodeBench is a few problems, which is why the paired test (p = 0.021) matters more than the raw 8-task gap.
+
+### 8.4 A 32,768-token budget on the problems that still fail at 16k (run 2026-10-26, A100)
+The 11 LiveCodeBench problems that failed in the clean 16k rerun (section 8.3) were run again with `--reasoning-budget 32768` (context 40,960, answer cap 36,864, single attempt). Table: `results/k448_code_think32k_redo_compact.csv`.
+
+- **2 of 11 pass** (`abc393_e` and the medium problem `arc192_a`). Both had passed in other runs of the same model at a smaller budget (`abc393_e` at 4k, `arc192_a` at 16k in the 21-task rerun), so they are better read as sampling luck on borderline problems than as a gain from the larger budget. The **nine hard problems that failed at 16k still fail at 32k**: arc192_e, arc196_d, arc194_d, abc392_f, abc398_g, arc190_c, abc387_f, arc193_b, 3674.
+- **Every one of the 11 used the whole 32k budget** (33,000 to 35,300 tokens). The model keeps thinking until it is stopped rather than converging on an answer for these problems.
+- **Cost:** 373,556 tokens in 5,438 s (about 34,000 tokens and 494 s per problem). Doubling the budget again cost twice as much per problem for no measurable gain.
+- **Conclusion:** the budget gain stops between 16k and 32k on this set. 16k is the sensible setting; for the remaining hard problems the repair-turn loop of section 8.2 (feedback from the problem's own sample tests) rescued more (4 of 10) than a bigger budget did (0 of 9 hard ones).
+
+## 9. Comparison with strong models (published numbers, and where k448 sits)
+Source: the Qwen3.8-Flash-Next model card on Hugging Face (August 2026). All numbers are vendor-reported, not independently verified; the card does not say whether a score is thinking or non-thinking (the model thinks by default) and does not state the thinking budget.
+
+| Benchmark | Flash-Next (parent, full precision) | Qwen3.8-27B | Qwen3.7-Plus | DeepSeek-V4-Flash-0731 | Claude-Opus-4.6 (Max) |
+|---|---|---|---|---|---|
+| LiveCodeBench v6 | 91.9 | 90.3 | 89.6 | 90.6 | 88.8 |
+| k448, thinking 4k budget (ours, 60-problem sample) | 68.3 (95% interval 56-79) | | | | |
+| k448, no-think (ours, same 60 problems) | 63.3 (95% interval 51-74) | | | | |
+| k448, 16k budget, clean rerun of all 60 problems (section 8.3) | **81.7 (interval about 70-90)** | | | | |
+| k448, 16k budget on the failed tasks (estimate, section 8.1) | 83.3 (interval about 72-91) | | | | |
+| k448, 16k budget + test-feedback repair turns (estimate, section 8.2; an agent-style loop, not single-shot) | 90.0 (interval about 80-95) | | | | |
+
+- **Only LiveCodeBench lines up.** CRUXEval, HumanEval, MBPP, MATH-500, MMLU-Pro and BBH are not in the card, so no published figure for those is quoted here.
+- **Not like for like.** Our LiveCodeBench is 60 problems sampled from a 175-problem slice (2025-01 to 2025-04) of the release 6 data, answered with a 4,096-token thinking budget and an 8,192-token answer cap; the vendor figure is the full v6 set, presumably with a far longer budget. With 60 problems one run has a 95% interval of roughly +/-12 points. Expect our number to sit below the published one; the gap mixes quantization, budget and sampling, and cannot be split without running the full-precision model (or Q4) through the same harness.
+- **Where k448 sits.** With a 4,096-token thinking budget k448 scores 68.3% on LiveCodeBench against the published 91.9% for the full-precision parent and 88.8% for Claude-Opus-4.6 (Max). With a 16,384-token budget, rerun cleanly on all 60 problems (section 8.3), it scores **81.7% (interval about 70-90%)**: a significant improvement over 4k (paired p = 0.021), and the published 91.9 now sits just above the top of our interval, while the other published figures (88.8 to 90.6) sit inside it. So with a long enough thinking budget k448 is plausibly within the range of the published models on this benchmark, but 60 problems cannot separate it from them. Adding up to two repair turns with the problems' own sample tests (section 8.2) would add a few more problems, but that is an agent-style loop with feedback, which the published single-shot numbers do not use, and it was measured on a different sample of answers. What remains could be quantization, a still longer budget (32k test: section 8.4, which found no gain over 16k), or sampling noise; separating them needs a Q4 or full-precision run on the same harness.
+- **We did not run the other models ourselves** (no API access from this setup), so every competitor number above is as published.
+
+## 10. Reasoning: does a bigger thinking budget help? (run 2026-10-26, A100)
+Quick reasoning set from the big suite (`--hard-n 60 --only reason`): 40 MMLU-Pro questions (10 answer options, 14 subjects) and 20 BIG-Bench Hard tasks, seed 1, same tasks every run. The no-think run is the baseline; the 10 questions it failed were then redone with thinking at two budgets (the 50 it passed were not rerun). Tables: `results/k448_reason_nothink_compact.csv`, `..._think4k_compact.csv`, `..._think16k_compact.csv`.
+
+| Setting | Result | Tokens (per task) |
+|---|---|---|
+| No-think, 2,048-token cap (60 tasks) | **50/60 (83.3%)**: BBH 20/20, MMLU-Pro 30/40 | 40,632 (677) in 468 s |
+| Thinking, 4,096 budget, the 10 failures | **5 of 10 rescued** | 31,492 (3,149) in 393 s |
+| Thinking, 16,384 budget, the 10 failures | **3 of 10 rescued** | 106,193 (10,619) in 1,315 s |
+
+- **Every failure was an MMLU-Pro question.** BBH (logic puzzles, tracking, date and fallacy questions) was 20/20 without thinking; there is nothing to gain there. 6 of the 10 failures ran into the 2,048-token cap without producing an answer, so part of the no-think failure is just running out of room.
+- **Thinking at 4k rescued 5 of the 10** (mmlu11759, 4613, 12187, 11789, 11236). Applied to the whole set that would be about 55/60 (91.7%), but only the failures were redone, so that is an estimate. The earlier lab on the L4 found the same thing (thinking at 4k rescued 3 of 7 reasoning failures).
+- **A 16k budget did not help more; it rescued fewer (3 of 10)** at 3.4 times the tokens. Rescued at 16k: mmlu11759, 12187, 11236 (all three also rescued at 4k). On MMLU-Pro, which mostly tests knowledge and short multi-step reasoning, thinking longer than a few thousand tokens just does not buy accuracy; this is the opposite of the coding result (section 8.3), where the extra budget mattered.
+- **How much is luck?** Resampling alone rescued 2 of 12 failures in an earlier control (about 17%). Against that rate, 5 of 10 is unlikely by chance (rough one-sided binomial p about 0.016) and 3 of 10 is not (p about 0.22). The 10 questions and the 17% control rate are both small, so treat this as suggestive, not proven. A proper test would rerun all 60 under each setting.
+- **Two failures never flipped** (mmlu527, mmlu4987): short wrong answers in every setting, probably knowledge gaps or ambiguous questions rather than a budget problem.
+- **Recommendation for reasoning:** thinking with a 4,096-token budget, not more. Use 16k only for coding (section 8.3).
+
+## 12. Agentic coding on SWE-bench Verified Mini through Claude Code (run 2026-10-26, A100)
+The real Claude Code CLI (2.1.197, headless `claude -p`, run as an unprivileged user) was pointed at k448 (llama-server, `/v1/messages`, thinking off, 65,536-token context, MTP draft) and given 20 instances of SWE-bench Verified Mini: 10 django and 10 sphinx whose own gold patch reproduces without Docker here (37 of the 50 instances do: 25 django + 12 sphinx; the 13 old sphinx 3.x instances fail even with the gold patch and are excluded). Grading uses swebench 4.0.4's own log parsers on FAIL_TO_PASS and PASS_TO_PASS. Runner `13_swe_native.py`, proxy `14_msg_proxy.py`. Tables: `results/k448_swe_compact.csv` (50-turn pass) and `results/k448_swe_t100_compact.csv` (100-turn rerun of the 8 misses).
+
+| | Result |
+|---|---|
+| As run, 50-turn cap | **12/20 (60%)**: django 8/10, sphinx 4/10 |
+| The 8 misses rerun with a 100-turn cap | +3 (django-11790, sphinx-9229, sphinx-9230) |
+| As run, combined (a second attempt for the misses) | **15/20 (75%)**, 95% interval about 53-89% |
+| **Resolved with no lookup of the upstream project on the internet** | **7/20 (35%)**, interval about 18-57%: 6 at 50 turns (django 11880, 11951, 11999, 12039, 12050; sphinx 9367) plus django-11790 in the rerun |
+
+### The headline number is not clean: the agent went to the internet
+The sandbox had open internet and Claude Code had its normal tools (Bash, WebFetch, WebSearch). The transcripts show it used them to look for the upstream fix:
+- In the first django-11790 run the agent searched GitHub for the upstream pull request and went to fetch it (it wrote that it had found the upstream PR and would fetch the actual fix).
+- **8 of the 15 resolved instances consulted the internet**: django-11848 (downloaded raw files from GitHub), 11885 (listed and cloned upstream django), 11964 (fetched django 3.1.3's `enums.py`, which already contains the fix), sphinx-10323 (listed and cloned upstream sphinx, one WebFetch), 10466 (WebFetch plus a GitHub issue lookup), 9320 (GitHub issue and timeline lookups), 9229 (fetched sphinx 4.1.0 source and searched issues), 9230 (nine WebFetch calls and an issue search). The other 7 resolved without any such lookup (django-11790's rerun only downloaded dependency packages).
+- Not checked: whether the local git clone still holds future commits. Several transcripts ran `git log --all` or `git show <sha>`.
+- So **7/20 is a floor** (a lookup does not prove it was needed) and **15/20 is a ceiling**; the true no-help rate is somewhere between. The 15 is also a best of two attempts for the 8 reruns.
+- Official SWE-bench style runs do not give the agent the web, so the as-run number cannot be compared with leaderboard figures.
+
+### Numbers
+- **Tool calls:** 912 in the 50-turn pass, 62 errors (6.8%); 694 in the rerun, 36 errors (5.2%).
+- **Turns and time:** 45.8 turns and 343 s per instance on average at the 50-turn cap; 87.1 turns and 668 s in the rerun. The 28 agent runs took 12,191 s (3.4 h) of A100 time, about 23 compute units.
+- **When the agent stopped on its own it was usually right:** 9 of 20 runs ended with the agent declaring it was done, and 8 of those 9 were resolved. The 11 runs cut off by the turn cap resolved 4.
+
+### Failure modes (per-instance results plus the transcripts I read)
+1. **Turn cap with the work unfinished:** 11 of 20 in the first pass, 7 of those unresolved.
+2. **Context overflow in long runs:** 5 of the 8 reruns (sphinx-10435, 10673, 11510, 8265, 9229) died with HTTP 400, "request (65,5xx to 66,3xx tokens) exceeds the available context size (65,536 tokens)". Claude Code does not know the server window is 65k, so it never compacts. The GPU holds 49.7 of 80 GB at 65k, so a 128k context fits.
+3. **A fix that passes the agent's own checks but not the target tests:** django-11815 (1 of 2 target tests, the same at 50 and 100 turns), sphinx-10673 (0/1, patches of 164 and 175 lines), 11510 (0/2), 8265 (0/1), 9230 at 50 turns (0/1).
+4. **Empty patch after long exploration:** django-11790 and sphinx-8265 and 9229 at 50 turns, sphinx-10435 in the rerun.
+5. **A malformed tool call:** one WebFetch call in sphinx-10466 had template markup (`</function></tool_call><tool_call><function=WebFetch>`) inside its URL argument; one example seen, not counted across all runs.
+6. **Environment friction:** in the first django-11790 transcript the agent spent turns finding a Python that works for the unprivileged user (one interpreter path returned permission denied) and imported Django through the system Python with a path hack.
+7. **An oversized patch:** the resolved sphinx-9229 patch is 1,657 lines (the other resolved ones are 12 to 215); it passed with no regressions but would not survive code review. Contents not inspected.
+
+### Setup caveats
+- Thinking off (`enable_thinking` false), 65,536 context, sampling left to Claude Code and the server defaults.
+- **Claude Code needs the rewriting proxy:** Qwen3.8's chat template raises "System message must be at the beginning" for the system-role entry Claude Code sends after the first user turn. The proxy folds it into the last user message; 1,594 requests went through, all 200 except the 5 context overflows.
+- Docker-free environments, not the official images: uv venvs, `setuptools<70`, `Pygments==2.18.0`, Python 3.8 for django (the images use older versions), 37 of 50 instances reproduce.
+- 20 instances: the interval is about +/-18-20 points, so 60% and 75% are not different from each other statistically.
+- **Section 9's published agentic table (SWE-bench Pro, Multilingual, DeepSWE, NL2Repo, Toolathlon) is a different, harder set of benchmarks and is not comparable.** There is no published number for this exact setup.
+
+### Verdict
+k448 under Claude Code can do small, localized bug fixes: the 7 clean resolves are short patches (12 to 215 lines) at about 6 minutes each on the A100, and a "done" from the agent was right 8 times out of 9. It is not shown to be a daily coding agent. The no-help rate is between 35% and 75% on 20 instances, hard tasks hit the turn cap or the 65k context, and the agent does look up upstream fixes when it can, so its own skill cannot be separated from the lookups in this run. Speed on your PC is not measured (14 tokens/s on the L4 earlier would make 6 minutes into about 30).
+
+**Next experiment, proposed and not run:** the same 20 instances offline (network blocked for the agent user, WebFetch and WebSearch disallowed, future git objects removed), a 128k context, the same turn caps, and a paired comparison with this table. About 2.5 to 3 hours of A100, roughly 17 to 20 units.
+
+## 13. Overnight summary (2026-10-26, A100)
+A100 time is from each run's own clock; units are time x 6.77 per hour, a rough figure that leaves out loading and idle time.
+
+| # | Experiment | Result | Test | A100 time (about units) |
+|---|---|---|---|---|
+| 1 | Coding, 4k thinking budget (120 tasks) | 99/120 (LCB 41/60, CRUX 58/60) | baseline | 3,689 s (6.9) |
+| 2 | Coding, no-think | 96/120 (LCB 38/60) | vs 4k: p = 0.508 | 2,888 s (5.4) |
+| 3 | Redo of the 21 failures at 16k | 9 of 21 rescued | luck baseline about 17% | 4,516 s (8.5) |
+| 4 | Two repair turns on the 10 still failing | 4 of 10 rescued (chained estimate 112/120) | estimate only | 3,411 s (6.4) |
+| 5 | **Coding, clean rerun of all 120 at 16k** | **107/120 (89.2%), LCB 49/60** | vs 4k: p = 0.021 | 9,992 s (18.8) |
+| 6 | 32k on the 11 still failing | 2 of 11 (likely luck) | none | 5,438 s (10.2) |
+| 7 | Reasoning, no-think (60 tasks) | 50/60 | baseline | 468 s |
+| 8 | Reasoning, 4k thinking on the 10 failures | 5 of 10 rescued | p about 0.016 vs luck baseline | 393 s |
+| 9 | Reasoning, 16k thinking on the 10 failures | 3 of 10 rescued | p about 0.22 vs luck baseline | 1,315 s (rows 7 to 9: 4.1) |
+| 10 | SWE-bench Verified Mini through Claude Code, 50 turns | 12/20 as run; 6/20 with no web lookup | none | 6,850 s (12.9) |
+| 11 | Rerun of the 8 misses, 100 turns | +3 as run (+1 with no lookup) | none | 5,341 s (10.0) |
+
+- **Coding:** a 16k thinking budget is the setting to use (107/120, significantly better than 4k); 32k adds nothing; repair turns on top rescue a few hard problems.
+- **Reasoning:** thinking at 4k, not more; a bigger budget did not help.
+- **Agentic:** 7 to 15 of 20, depending on whether web lookups count; not established until the offline rerun.
+- **Units:** 84.10 at 06:20, 58.12 at 10:13, 51.37 at 11:13, **45.17 at 12:05**, burning 6.77 per hour while the runtime stays connected, idle or not.
+- **A100 runtime:** disconnected and deleted at 12:10 on 2026-10-26 at your request, after the units reading of 45.17 at 12:05 (the balance cannot be read without a runtime, so the final figure is a little lower). The k448 and MTP files on that VM came from Hugging Face (download cache with the same commit hash), so they are re-downloadable; the raw SWE-bench transcripts, results JSONs and logs on the VM are gone (the compact CSVs and the report are saved locally). llama.cpp PR 30594 was not touched.

benchmarks/SPEED_PC.md

@@ -0,0 +1,27 @@
+# One-token routed-expert matvec on this PC (2026-10-26)
+
+Tool: `bench.c` (build with `build.bat`, run with `run.bat <threads>`), CPU backend of `C:\Projects\llama.cpp-strata\build-clang`
+(AVX2 + AVX-VNNI, no AVX-512), i7-13620H, DDR5-5200, 10 threads. 10 of 128 experts per op, ids rotated so the weights stream from RAM.
+Shapes are k448-like guesses: gate/up k=2048 m=1024, down k=1024 m=2048. A 78 GB download was running in the background (light CPU use).
+
+| type | gate/up Gweights/s (GB/s) | down Gweights/s (GB/s) | note |
+|---|---|---|---|
+| **iq2_s** (k448 gate/up) | 96.7 (31.0) | 90.4 (29.0) | compute-bound: about half of what RAM allows |
+| **iq4_nl** (k448 down) | 93.4 (52.5) | 101.2 (57.0) | close to RAM speed |
+| iq3_xxs | 61.4 (23.5) | 70.3 (26.9) | the slow one (Strata's model type) |
+| q2_K | 183.1 (60.1) | 174.7 (57.3) | what a RAM-speed 2-bit kernel looks like |
+| q4_K | 115.9 (65.2) | 118.8 (66.8) | RAM ceiling is about 65 GB/s |
+| q4_0 | 112.7 (63.4) | 114.6 (64.5) | |
+| q6_K | 71.7 (58.8) | 63.0 (51.7) | |
+
+Reading: with k448's types the CPU expert work is about 95 Gweights/s. An IQ2_S kernel at RAM speed would be about 190 Gweights/s.
+Estimate (not measured on the model): 2.36 B routed weights per token / 95 G = about 25 ms per token if every expert ran on the CPU.
+
+## Server baseline on the model (Minecraft closed, Overwolf on)
+no MTP 17.5 tok/s; MTP 1 draft 20.3-20.7; MTP 2 drafts 21-22 (acceptance 93%); GPU cache 645 MiB, hit rate about 14%. With Minecraft running: 11.7 no-MTP. Bigger cache + MTP draft fails to load (bug).
+
+## All-VRAM test
+no MTP, cache 3200 MiB, margin 250, ubatch 512: 20.1-22.5 tok/s, hit 52%, VRAM 7947/8188. MTP2 + cache 1800 (margin 700): 21.4-22.4, hit 29%. Cache 2400+ with MTP: draft fails to load (VRAM). They do not stack.
+
+## Cache/draft, long prompts, kernel (2026-10-26)
+Draft on CPU + cache 3000: 21.6 tok/s (no gain). Prefill: 5,198 tokens at 164 tok/s; repeat prompt 0.2 s. Decode on prose explanation 14-16 tok/s (draft acceptance 65% vs 93% for code). IQ2_S kernel already AVX2 with table gathers; est. upside under 10%, not written.

benchmarks/results/k448_code_nothink_a100_compact.csv

@@ -0,0 +1,121 @@
+id,pass,tokens
+lcb_hard_3744,1,2060
+lcb_easy_3778,1,85
+lcb_easy_abc392_a,1,177
+lcb_medium_arc195_a,1,3119
+lcb_hard_abc397_g,0,8192
+lcb_hard_arc192_e,0,7505
+lcb_hard_arc196_d,0,8192
+lcb_hard_abc400_e,1,1019
+lcb_hard_arc194_d,0,8192
+lcb_medium_3779,0,8192
+lcb_hard_abc392_f,0,818
+lcb_easy_abc395_a,1,155
+lcb_medium_3786,1,1317
+lcb_hard_arc195_e,0,8192
+lcb_hard_abc398_d,1,3789
+lcb_hard_abc398_g,0,8192
+lcb_medium_abc392_d,1,512
+lcb_easy_abc393_a,1,518
+lcb_hard_abc391_g,1,1557
+lcb_hard_abc394_g,0,8192
+lcb_medium_3805,1,8192
+lcb_hard_abc396_g,1,754
+lcb_easy_3817,1,100
+lcb_easy_3832,1,1697
+lcb_hard_3777,0,8192
+lcb_hard_abc393_d,1,423
+lcb_hard_abc388_g,0,2049
+lcb_hard_abc394_f,0,6907
+lcb_hard_abc389_f,1,3545
+lcb_medium_3677,1,692
+lcb_medium_3722,0,2127
+lcb_hard_arc190_c,0,8165
+lcb_hard_abc387_f,0,8192
+lcb_hard_3783,1,5241
+lcb_hard_arc192_d,0,8192
+lcb_hard_abc393_e,1,748
+lcb_medium_abc388_c,1,863
+lcb_easy_3753,1,152
+lcb_hard_3697,1,3650
+lcb_hard_arc196_b,0,8192
+lcb_easy_3736,1,109
+lcb_hard_3789,1,4919
+lcb_medium_arc192_a,0,8192
+lcb_medium_abc392_c,1,978
+lcb_medium_3785,1,1206
+lcb_medium_3737,1,1899
+lcb_medium_3748,1,1066
+lcb_easy_3731,1,93
+lcb_medium_abc398_b,1,676
+lcb_easy_abc400_b,1,803
+lcb_hard_arc193_b,0,8192
+lcb_medium_abc399_d,0,8192
+lcb_medium_abc400_d,1,3457
+lcb_medium_abc399_c,1,520
+lcb_medium_3760,1,1475
+lcb_medium_3776,0,8192
+lcb_medium_3743,1,2328
+lcb_easy_abc396_b,1,177
+lcb_easy_3799,1,298
+lcb_hard_3674,0,8192
+crux_o_sample_595,1,203
+crux_i_sample_123,1,3696
+crux_o_sample_32,1,209
+crux_i_sample_198,1,1311
+crux_o_sample_630,1,284
+crux_i_sample_423,1,6083
+crux_o_sample_388,1,217
+crux_i_sample_379,1,663
+crux_o_sample_64,1,555
+crux_i_sample_506,1,593
+crux_o_sample_560,1,142
+crux_i_sample_204,1,624
+crux_o_sample_760,1,547
+crux_i_sample_308,1,594
+crux_o_sample_379,1,224
+crux_i_sample_223,1,656
+crux_o_sample_184,1,237
+crux_i_sample_374,1,804
+crux_o_sample_385,1,509
+crux_i_sample_11,1,431
+crux_o_sample_155,0,1004
+crux_i_sample_39,1,339
+crux_o_sample_387,1,221
+crux_i_sample_544,1,760
+crux_o_sample_445,1,460
+crux_i_sample_257,1,950
+crux_o_sample_144,1,137
+crux_i_sample_610,1,2444
+crux_o_sample_728,1,176
+crux_i_sample_490,1,421
+crux_o_sample_670,1,345
+crux_i_sample_20,1,385
+crux_o_sample_694,1,252
+crux_i_sample_551,1,772
+crux_o_sample_747,1,315
+crux_i_sample_260,1,910
+crux_o_sample_286,1,397
+crux_i_sample_549,1,880
+crux_o_sample_650,1,166
+crux_i_sample_783,1,860
+crux_o_sample_114,1,1400
+crux_i_sample_457,1,942
+crux_o_sample_611,1,202
+crux_i_sample_515,1,617
+crux_o_sample_405,1,314
+crux_i_sample_336,1,515
+crux_o_sample_349,1,301
+crux_i_sample_663,1,751
+crux_o_sample_599,1,505
+crux_i_sample_149,1,1256
+crux_o_sample_669,1,291
+crux_i_sample_703,1,1390
+crux_o_sample_136,0,590
+crux_i_sample_608,1,563
+crux_o_sample_210,1,198
+crux_i_sample_673,1,568
+crux_o_sample_682,1,220
+crux_i_sample_291,1,1010
+crux_o_sample_561,1,202
+crux_i_sample_557,1,635

benchmarks/results/k448_code_think16k_full_compact.csv

@@ -0,0 +1,121 @@
+id,pass,tokens
+lcb_hard_3744,1,13560
+lcb_easy_3778,1,458
+lcb_easy_abc392_a,1,919
+lcb_medium_arc195_a,1,4474
+lcb_hard_abc397_g,1,17532
+lcb_hard_arc192_e,0,17527
+lcb_hard_arc196_d,0,16399
+lcb_hard_abc400_e,1,16663
+lcb_hard_arc194_d,0,16934
+lcb_medium_3779,1,16498
+lcb_hard_abc392_f,0,16839
+lcb_easy_abc395_a,1,370
+lcb_medium_3786,1,16865
+lcb_hard_arc195_e,1,17009
+lcb_hard_abc398_d,1,9100
+lcb_hard_abc398_g,0,16718
+lcb_medium_abc392_d,1,16074
+lcb_easy_abc393_a,1,779
+lcb_hard_abc391_g,1,14693
+lcb_hard_abc394_g,1,17640
+lcb_medium_3805,1,16656
+lcb_hard_abc396_g,1,17045
+lcb_easy_3817,1,360
+lcb_easy_3832,1,7308
+lcb_hard_3777,1,17628
+lcb_hard_abc393_d,1,4230
+lcb_hard_abc388_g,1,17384
+lcb_hard_abc394_f,1,16954
+lcb_hard_abc389_f,1,17488
+lcb_medium_3677,1,16896
+lcb_medium_3722,1,12331
+lcb_hard_arc190_c,0,17422
+lcb_hard_abc387_f,0,17261
+lcb_hard_3783,1,11649
+lcb_hard_arc192_d,1,17481
+lcb_hard_abc393_e,0,16979
+lcb_medium_abc388_c,1,3173
+lcb_easy_3753,1,448
+lcb_hard_3697,1,16831
+lcb_hard_arc196_b,1,17371
+lcb_easy_3736,1,772
+lcb_hard_3789,1,16774
+lcb_medium_arc192_a,0,16900
+lcb_medium_abc392_c,1,822
+lcb_medium_3785,1,1293
+lcb_medium_3737,1,6858
+lcb_medium_3748,1,1953
+lcb_easy_3731,1,1419
+lcb_medium_abc398_b,1,562
+lcb_easy_abc400_b,1,1476
+lcb_hard_arc193_b,0,20480
+lcb_medium_abc399_d,1,16745
+lcb_medium_abc400_d,1,16900
+lcb_medium_abc399_c,1,1324
+lcb_medium_3760,1,6367
+lcb_medium_3776,1,16989
+lcb_medium_3743,1,16140
+lcb_easy_abc396_b,1,510
+lcb_easy_3799,1,3001
+lcb_hard_3674,0,17132
+crux_o_sample_595,1,564
+crux_i_sample_123,1,2206
+crux_o_sample_32,1,402
+crux_i_sample_198,1,982
+crux_o_sample_630,1,1180
+crux_i_sample_423,1,4725
+crux_o_sample_388,1,428
+crux_i_sample_379,1,467
+crux_o_sample_64,1,838
+crux_i_sample_506,1,542
+crux_o_sample_560,1,425
+crux_i_sample_204,1,541
+crux_o_sample_760,1,319
+crux_i_sample_308,1,753
+crux_o_sample_379,1,421
+crux_i_sample_223,1,386
+crux_o_sample_184,1,322
+crux_i_sample_374,1,654
+crux_o_sample_385,1,726
+crux_i_sample_11,1,404
+crux_o_sample_155,1,1422
+crux_i_sample_39,1,346
+crux_o_sample_387,1,195
+crux_i_sample_544,1,2334
+crux_o_sample_445,0,1312
+crux_i_sample_257,1,999
+crux_o_sample_144,1,164
+crux_i_sample_610,1,2645
+crux_o_sample_728,1,274
+crux_i_sample_490,1,513
+crux_o_sample_670,1,694
+crux_i_sample_20,1,295
+crux_o_sample_694,1,623
+crux_i_sample_551,1,940
+crux_o_sample_747,1,394
+crux_i_sample_260,1,698
+crux_o_sample_286,1,795
+crux_i_sample_549,1,468
+crux_o_sample_650,1,230
+crux_i_sample_783,1,673
+crux_o_sample_114,1,536
+crux_i_sample_457,1,790
+crux_o_sample_611,1,287
+crux_i_sample_515,1,326
+crux_o_sample_405,1,355
+crux_i_sample_336,1,1078
+crux_o_sample_349,1,867
+crux_i_sample_663,1,302
+crux_o_sample_599,1,999
+crux_i_sample_149,1,1678
+crux_o_sample_669,1,647
+crux_i_sample_703,1,787
+crux_o_sample_136,0,2757
+crux_i_sample_608,1,633
+crux_o_sample_210,1,379
+crux_i_sample_673,1,792
+crux_o_sample_682,1,505
+crux_i_sample_291,1,1122
+crux_o_sample_561,1,282
+crux_i_sample_557,1,983

benchmarks/results/k448_code_think16k_redo_compact.csv

@@ -0,0 +1,22 @@
+id,pass,tokens
+lcb_hard_abc397_g,1,17629
+lcb_hard_arc192_e,0,17677
+lcb_hard_arc196_d,0,16399
+lcb_hard_arc194_d,0,16936
+lcb_hard_abc392_f,0,16895
+lcb_hard_arc195_e,1,16935
+lcb_hard_abc398_g,0,16691
+lcb_hard_abc394_g,1,17923
+lcb_medium_3805,1,16656
+lcb_hard_abc394_f,1,16982
+lcb_hard_arc190_c,0,17264
+lcb_hard_abc387_f,0,17305
+lcb_hard_arc196_b,0,17359
+lcb_medium_arc192_a,1,17108
+lcb_hard_arc193_b,0,20480
+lcb_medium_abc399_d,1,16745
+lcb_medium_abc400_d,1,16900
+lcb_medium_3776,1,16989
+lcb_hard_3674,0,18361
+crux_o_sample_445,0,1312
+crux_o_sample_136,0,3059

benchmarks/results/k448_code_think16k_retry_compact.csv

@@ -0,0 +1,11 @@
+id,pass,tokens,retries
+lcb_hard_arc192_e,1,35351,2
+lcb_hard_arc196_d,0,32807,2
+lcb_hard_arc194_d,1,16997,1
+lcb_hard_abc392_f,0,0,0
+lcb_hard_abc398_g,0,33498,2
+lcb_hard_arc190_c,0,34698,2
+lcb_hard_abc387_f,0,0,0
+lcb_hard_arc196_b,1,34387,2
+lcb_hard_arc193_b,0,35784,2
+lcb_hard_3674,1,17212,1

benchmarks/results/k448_code_think32k_redo_compact.csv

@@ -0,0 +1,12 @@
+id,pass,tokens
+lcb_hard_arc192_e,0,34464
+lcb_hard_arc196_d,0,34805
+lcb_hard_arc194_d,0,33330
+lcb_hard_abc392_f,0,33330
+lcb_hard_abc398_g,0,33039
+lcb_hard_arc190_c,0,35309
+lcb_hard_abc387_f,0,34266
+lcb_hard_abc393_e,1,33970
+lcb_medium_arc192_a,1,33293
+lcb_hard_arc193_b,0,33647
+lcb_hard_3674,0,34103

benchmarks/results/k448_code_think_compact.csv

@@ -0,0 +1,121 @@
+id,pass,tokens
+lcb_hard_3744,1,4410
+lcb_easy_3778,1,458
+lcb_easy_abc392_a,1,919
+lcb_medium_arc195_a,1,4448
+lcb_hard_abc397_g,0,6412
+lcb_hard_arc192_e,0,6590
+lcb_hard_arc196_d,0,4268
+lcb_hard_abc400_e,1,4389
+lcb_hard_arc194_d,0,4900
+lcb_medium_3779,1,4274
+lcb_hard_abc392_f,0,4437
+lcb_easy_abc395_a,1,370
+lcb_medium_3786,1,4515
+lcb_hard_arc195_e,0,4552
+lcb_hard_abc398_d,1,4364
+lcb_hard_abc398_g,0,4413
+lcb_medium_abc392_d,1,4493
+lcb_easy_abc393_a,1,779
+lcb_hard_abc391_g,1,4675
+lcb_hard_abc394_g,0,4923
+lcb_medium_3805,0,4727
+lcb_hard_abc396_g,1,4706
+lcb_easy_3817,1,360
+lcb_easy_3832,1,4169
+lcb_hard_3777,1,4719
+lcb_hard_abc393_d,1,4230
+lcb_hard_abc388_g,1,4779
+lcb_hard_abc394_f,0,4531
+lcb_hard_abc389_f,1,5117
+lcb_medium_3677,1,4521
+lcb_medium_3722,1,4362
+lcb_hard_arc190_c,0,8192
+lcb_hard_abc387_f,0,5298
+lcb_hard_3783,1,4697
+lcb_hard_arc192_d,1,5087
+lcb_hard_abc393_e,1,4479
+lcb_medium_abc388_c,1,3173
+lcb_easy_3753,1,448
+lcb_hard_3697,1,4452
+lcb_hard_arc196_b,0,5168
+lcb_easy_3736,1,772
+lcb_hard_3789,1,4593
+lcb_medium_arc192_a,0,5127
+lcb_medium_abc392_c,1,822
+lcb_medium_3785,1,1293
+lcb_medium_3737,1,4344
+lcb_medium_3748,1,1953
+lcb_easy_3731,1,1419
+lcb_medium_abc398_b,1,562
+lcb_easy_abc400_b,1,1476
+lcb_hard_arc193_b,0,4545
+lcb_medium_abc399_d,0,4540
+lcb_medium_abc400_d,0,4633
+lcb_medium_abc399_c,1,1324
+lcb_medium_3760,1,4249
+lcb_medium_3776,0,5012
+lcb_medium_3743,1,4275
+lcb_easy_abc396_b,1,510
+lcb_easy_3799,1,3001
+lcb_hard_3674,0,8192
+crux_o_sample_595,1,564
+crux_i_sample_123,1,2206
+crux_o_sample_32,1,402
+crux_i_sample_198,1,982
+crux_o_sample_630,1,1180
+crux_i_sample_423,1,4149
+crux_o_sample_388,1,428
+crux_i_sample_379,1,467
+crux_o_sample_64,1,838
+crux_i_sample_506,1,542
+crux_o_sample_560,1,425
+crux_i_sample_204,1,541
+crux_o_sample_760,1,319
+crux_i_sample_308,1,753
+crux_o_sample_379,1,421
+crux_i_sample_223,1,386
+crux_o_sample_184,1,322
+crux_i_sample_374,1,654
+crux_o_sample_385,1,726
+crux_i_sample_11,1,404
+crux_o_sample_155,1,1422
+crux_i_sample_39,1,346
+crux_o_sample_387,1,195
+crux_i_sample_544,1,2334
+crux_o_sample_445,0,1312
+crux_i_sample_257,1,999
+crux_o_sample_144,1,164
+crux_i_sample_610,1,2645
+crux_o_sample_728,1,274
+crux_i_sample_490,1,513
+crux_o_sample_670,1,694
+crux_i_sample_20,1,295
+crux_o_sample_694,1,623
+crux_i_sample_551,1,940
+crux_o_sample_747,1,394
+crux_i_sample_260,1,698
+crux_o_sample_286,1,795
+crux_i_sample_549,1,468
+crux_o_sample_650,1,230
+crux_i_sample_783,1,673
+crux_o_sample_114,1,536
+crux_i_sample_457,1,790
+crux_o_sample_611,1,287
+crux_i_sample_515,1,326
+crux_o_sample_405,1,355
+crux_i_sample_336,1,1078
+crux_o_sample_349,1,867
+crux_i_sample_663,1,302
+crux_o_sample_599,1,999
+crux_i_sample_149,1,1678
+crux_o_sample_669,1,647
+crux_i_sample_703,1,787
+crux_o_sample_136,0,2757
+crux_i_sample_608,1,633
+crux_o_sample_210,1,379
+crux_i_sample_673,1,792
+crux_o_sample_682,1,505
+crux_i_sample_291,1,1122
+crux_o_sample_561,1,282
+crux_i_sample_557,1,983

benchmarks/results/k448_reason_nothink_compact.csv

@@ -0,0 +1,61 @@
+id,pass,tokens
+reason_mmlu3028,1,719
+reason_mmlu10879,1,271
+reason_bbh_logical_deduction_five_objects_0,1,630
+reason_mmlu11759,0,2048
+reason_mmlu8693,1,302
+reason_bbh_tracking_shuffled_objects_five_objects_0,1,494
+reason_mmlu1760,1,742
+reason_mmlu6359,1,603
+reason_bbh_date_understanding_0,1,432
+reason_mmlu4584,0,2048
+reason_mmlu7301,1,685
+reason_bbh_web_of_lies_0,1,263
+reason_mmlu5893,1,259
+reason_mmlu9408,1,739
+reason_bbh_boolean_expressions_0,1,228
+reason_mmlu4770,1,491
+reason_mmlu2310,1,543
+reason_bbh_navigate_0,1,128
+reason_mmlu656,1,557
+reason_mmlu10734,1,933
+reason_bbh_formal_fallacies_0,1,1333
+reason_mmlu4613,0,2048
+reason_mmlu4984,1,733
+reason_bbh_object_counting_0,1,191
+reason_mmlu6184,0,2048
+reason_mmlu3337,1,676
+reason_bbh_logical_deduction_five_objects_1,1,512
+reason_mmlu10806,1,168
+reason_mmlu12187,0,2048
+reason_bbh_tracking_shuffled_objects_five_objects_1,1,551
+reason_mmlu2156,1,573
+reason_mmlu527,0,282
+reason_bbh_date_understanding_1,1,289
+reason_mmlu10323,1,527
+reason_mmlu5812,1,216
+reason_bbh_web_of_lies_1,1,212
+reason_mmlu962,1,714
+reason_mmlu8463,1,445
+reason_bbh_boolean_expressions_1,1,285
+reason_mmlu10422,1,659
+reason_mmlu7515,1,586
+reason_bbh_navigate_1,1,158
+reason_mmlu2640,1,446
+reason_mmlu4987,0,567
+reason_bbh_formal_fallacies_1,1,1616
+reason_mmlu3166,1,552
+reason_mmlu1366,0,830
+reason_bbh_object_counting_1,1,263
+reason_mmlu11789,0,2048
+reason_mmlu5632,1,398
+reason_bbh_logical_deduction_five_objects_2,1,673
+reason_mmlu200,1,680
+reason_mmlu8011,1,990
+reason_bbh_tracking_shuffled_objects_five_objects_2,1,461
+reason_mmlu6177,1,600
+reason_mmlu10443,1,482
+reason_bbh_date_understanding_2,1,244
+reason_mmlu11236,0,454
+reason_mmlu7172,1,680
+reason_bbh_web_of_lies_2,1,279

benchmarks/results/k448_reason_think16k_compact.csv

@@ -0,0 +1,11 @@
+id,pass,tokens
+reason_mmlu11759,1,16721
+reason_mmlu4584,0,12966
+reason_mmlu4613,0,16697
+reason_mmlu6184,0,16457
+reason_mmlu12187,1,16565
+reason_mmlu527,0,454
+reason_mmlu4987,0,902
+reason_mmlu1366,0,4612
+reason_mmlu11789,0,20480
+reason_mmlu11236,1,339

benchmarks/results/k448_reason_think4k_compact.csv

@@ -0,0 +1,11 @@
+id,pass,tokens
+reason_mmlu11759,1,4447
+reason_mmlu4584,0,4199
+reason_mmlu4613,1,4278
+reason_mmlu6184,0,4159
+reason_mmlu12187,1,4206
+reason_mmlu527,0,454
+reason_mmlu4987,0,902
+reason_mmlu1366,0,4268
+reason_mmlu11789,1,4240
+reason_mmlu11236,1,339

benchmarks/results/k448_swe_compact.csv

@@ -0,0 +1,21 @@
+id,resolved,turns,tool_calls,tool_errors,patch_lines,secs
+django__django-11790,0,51,50,4,0,316
+django__django-11815,0,51,50,9,18,360
+django__django-11848,1,22,21,2,64,177
+django__django-11880,1,39,38,4,12,257
+django__django-11885,1,51,50,1,215,340
+django__django-11951,1,49,48,1,14,334
+django__django-11964,1,58,57,3,19,379
+django__django-11999,1,47,46,3,17,293
+django__django-12039,1,51,50,1,32,306
+django__django-12050,1,42,41,9,16,266
+sphinx-doc__sphinx-10323,1,32,31,1,29,230
+sphinx-doc__sphinx-10435,0,51,52,1,27,380
+sphinx-doc__sphinx-10466,1,51,50,2,27,366
+sphinx-doc__sphinx-10673,0,51,57,1,164,370
+sphinx-doc__sphinx-11510,0,51,54,3,75,325
+sphinx-doc__sphinx-8265,0,51,50,2,0,920
+sphinx-doc__sphinx-9229,0,51,51,5,0,314
+sphinx-doc__sphinx-9230,0,51,53,4,41,365
+sphinx-doc__sphinx-9320,1,51,50,4,57,448
+sphinx-doc__sphinx-9367,1,14,13,2,15,104

benchmarks/results/k448_swe_t100_compact.csv

@@ -0,0 +1,9 @@
+id,resolved,turns,tool_calls,tool_errors,patch_lines,secs
+django__django-11790,1,51,50,5,16,319
+django__django-11815,0,101,100,4,26,641
+sphinx-doc__sphinx-10435,0,94,93,9,0,800
+sphinx-doc__sphinx-10673,0,80,79,2,175,557
+sphinx-doc__sphinx-11510,0,96,95,5,162,640
+sphinx-doc__sphinx-8265,0,98,97,2,39,794
+sphinx-doc__sphinx-9229,1,76,75,4,1657,561
+sphinx-doc__sphinx-9230,1,101,105,5,13,1029

benchmarks/scripts/07_domain_eval.py

@@ -0,0 +1,630 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+"""
+07_domain_eval.py - does a quantized Tessera keep what matters (Turkish, English, code, math)?  Everything is scored against a
+reference model (the Q4), because the original checkpoint cannot be run on the test machines.
+
+  run      56 checkable prompts sent to a llama-server (OpenAI API): 11 Python functions with hidden unit tests (3 asked in Turkish),
+           16 math word problems (8 English + the same 8 in Turkish), 14 Turkish and 15 English language tasks (reading comprehension,
+           Turkish alphabet order, i/I casing, vowel-harmony plurals, instruction following, extraction, cross-language answers).
+           Problems with numbers are generated from a seed, so the answers are computed, not typed.  Greedy decoding.
+  kld      llama-perplexity per domain over the held-out text of 06_build_eval_set.py: mean KL divergence and same-top-token rate
+           against the reference model's saved logits (--save-base stores them for the reference)
+  compare  table of result files; every task category as a % of the first file (the reference)
+  selftest the checkers accept the reference answers and reject garbage; no model needed
+
+  python 07_domain_eval.py run --bin <llama.cpp>/build/bin --model q4.gguf --ot '<-ot regex>' --name q4 --no-think     (starts llama-server)
+  python 07_domain_eval.py run --bin <llama.cpp>/build/bin --model ours.gguf --ot '<-ot regex>' --name ours --no-think
+  python 07_domain_eval.py run --endpoint http://127.0.0.1:8080 --name x --no-think          (a llama-server that is already running)
+  python 07_domain_eval.py compare results_q4.json results_ours.json
+  python 07_domain_eval.py kld --bin <llama.cpp>/build/bin --model q4.gguf --ot '<-ot regex>' --name q4 --save-base
+  python 07_domain_eval.py kld --bin <llama.cpp>/build/bin --model ours.gguf --ot '<-ot regex>' --name ours
+
+`run` EXECUTES THE PYTHON THE MODEL WRITES (10 s limit, 2 GB, empty temp directory, nothing else isolated): use Colab or a throw-away
+machine.  56 tasks are a smoke test: a difference of a few tasks is noise (one task is about 9 points of the 11 code tasks).
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import math
+import os
+import random
+import re
+import shlex
+import subprocess
+import sys
+import tempfile
+import time
+import unicodedata
+import urllib.request
+from fractions import Fraction as Fr
+from pathlib import Path
+
+CATS = ("code", "math", "tr", "en", "reason", "lcb", "crux")
+TR_ALPHA = "abcçdefgğhıijklmnoöprsştuüvyz"
+
+
+# ---------------------------------------------------------------------------------------------------------------- checkers
+def strip_think(r: str) -> str:
+    return re.sub(r"<think>.*", "", re.sub(r"<think>.*?</think>", "", r, flags=re.S), flags=re.S).strip()
+
+
+def numbers(text: str) -> list:
+    out = []
+    for tok in re.findall(r"-?\d[\d.,]*", text):
+        tok = tok.rstrip(".,")
+        tok = re.sub(r"[.,]", "", tok) if re.fullmatch(r"-?\d{1,3}([.,]\d{3})+", tok) else tok.replace(",", ".")
+        try:
+            out.append(Fr(tok))
+        except ValueError:
+            pass
+    return out
+
+
+def num(ans):
+    def f(r):
+        if "####" in r:
+            n = numbers(r.rsplit("####", 1)[1])
+            return bool(n) and n[0] == ans
+        n = numbers(r)
+        return bool(n) and n[-1] == ans
+    return f
+
+
+def has(*words):               # every word present (case-insensitive)
+    return lambda r: all(w.lower() in r.lower() for w in words)
+
+
+def ordered(words):            # the words appear in this order
+    def f(r):
+        pos = 0
+        for w in words:
+            i = r.lower().find(w.lower(), pos)
+            if i < 0:
+                return False
+            pos = i + len(w)
+        return True
+    return f
+
+
+def word(*ok):                 # the whole answer is one of these words
+    return lambda r: re.sub(r"[^\w]", "", r.casefold()) in ok
+
+
+def bullets(n):
+    def f(r):
+        lines = [l for l in r.splitlines() if l.strip()]
+        return len(lines) == n and all(l.startswith("- ") for l in lines)
+    return f
+
+
+def deaccent(s: str) -> str:
+    return "".join(c for c in unicodedata.normalize("NFD", s) if unicodedata.category(c) != "Mn").lower()
+
+
+def as_json(want: dict):
+    def f(r):
+        m = re.search(r"\{.*\}", r, flags=re.S)
+        try:
+            d = json.loads(m.group(0))
+        except Exception:
+            return False
+        return all(deaccent(str(d.get(k, ""))) == deaccent(v) for k, v in want.items())
+    return f
+
+
+def english_only(*words):      # contains the words and no Turkish-specific letter
+    return lambda r: has(*words)(r) and not any(c in r for c in "çğıöşüÇĞİÖŞÜ")
+
+
+def code_of(r: str) -> str:
+    blocks = [b for b in re.findall(r"```(?:python|py)?[ \t]*\n(.*?)```", r, flags=re.S) if re.search(r"^\s*(def|class) ", b, re.M)]
+    return blocks[0] if blocks else r
+
+
+def run_code(code: str, tests: str, timeout: int = 10) -> bool:
+    pre = None
+    if os.name == "posix":
+        import resource
+        pre = lambda: resource.setrlimit(resource.RLIMIT_AS, (2 << 30, 2 << 30))
+    with tempfile.TemporaryDirectory() as d:
+        script = Path(d) / "task.py"      # a file, not -c: the HumanEval+ tests are bigger than the 128 KiB limit of one argument
+        script.write_text(code + "\n\n" + tests, encoding="utf-8")
+        try:
+            return subprocess.run([sys.executable, "-I", str(script)], cwd=d, capture_output=True, timeout=timeout, preexec_fn=pre,
+                                  env={**os.environ, "OPENBLAS_NUM_THREADS": "1", "OMP_NUM_THREADS": "1"}).returncode == 0
+        except subprocess.TimeoutExpired:
+            return False
+
+
+# ---------------------------------------------------------------------------------------------------------------- tasks
+ONE_BLOCK_EN = " Reply with one Python code block only."
+ONE_BLOCK_TR = " Yalnızca tek bir Python kod bloğu ile cevap ver."
+CODE = [  # (id, prompt, tests, reference solution)
+    ("roman", "Write a Python function `roman_to_int(s)` that converts a Roman numeral (I V X L C D M, with subtractive pairs such as "
+     "IV and IX) to an integer." + ONE_BLOCK_EN,
+     "assert roman_to_int('III')==3 and roman_to_int('IV')==4 and roman_to_int('XLII')==42 and roman_to_int('MCMXCIV')==1994",
+     "def roman_to_int(s):\n    v = {'I': 1, 'V': 5, 'X': 10, 'L': 50, 'C': 100, 'D': 500, 'M': 1000}\n    return sum(-v[c] if i + 1 < len(s) and "
+     "v[c] < v[s[i + 1]] else v[c] for i, c in enumerate(s))"),
+    ("merge_tr", "Python'da `merge_intervals(intervals)` fonksiyonunu yaz: [başlangıç, bitiş] listelerinden oluşan bir listeyi alır, "
+     "kesişen veya uç uca değen aralıkları birleştirir ve başlangıca göre sıralı bir liste (iç içe listeler olarak) döndürür." + ONE_BLOCK_TR,
+     "assert merge_intervals([[1,3],[2,6],[8,10],[15,18]])==[[1,6],[8,10],[15,18]]\nassert merge_intervals([[1,4],[4,5]])==[[1,5]]\n"
+     "assert merge_intervals([])==[]\nassert merge_intervals([[5,6],[1,2]])==[[1,2],[5,6]]",
+     "def merge_intervals(iv):\n    out = []\n    for a, b in sorted(iv):\n        if out and a <= out[-1][1]:\n"
+     "            out[-1][1] = max(out[-1][1], b)\n        else:\n            out.append([a, b])\n    return out"),
+    ("balanced_tr", "Python'da `is_balanced(s)` fonksiyonunu yaz: s içindeki (), [] ve {} parantezleri doğru sırada kapatılmışsa True, "
+     "değilse False döndürsün (s yalnızca bu altı karakterden oluşur, boş dize True'dur)." + ONE_BLOCK_TR,
+     "assert is_balanced('([]{})') and not is_balanced('(]') and is_balanced('') and not is_balanced('((') and not is_balanced('())')"
+     " and not is_balanced('([)]')",
+     "def is_balanced(s):\n    st, pair = [], {')': '(', ']': '[', '}': '{'}\n    for c in s:\n        if c in pair:\n"
+     "            if not st or st.pop() != pair[c]:\n                return False\n        else:\n            st.append(c)\n    return not st"),
+    ("flatten", "Write a Python function `flatten(x)` that turns an arbitrarily nested list into a flat list, keeping the order. "
+     "Strings are single items, not lists." + ONE_BLOCK_EN,
+     "assert flatten([1,[2,[3,[4]]],5])==[1,2,3,4,5] and flatten([])==[] and flatten([[],[[]]])==[] and flatten(['ab',[['c']]])==['ab','c']",
+     "def flatten(x):\n    out = []\n    for i in x:\n        out += flatten(i) if isinstance(i, list) else [i]\n    return out"),
+    ("word_freq", "Write a Python function `word_freq(text)` that returns a dict mapping each lowercase word to its count; words are the "
+     "maximal runs of ASCII letters after lowercasing, everything else is a separator." + ONE_BLOCK_EN,
+     "assert word_freq('Hello, hello world!')=={'hello':2,'world':1} and word_freq('')=={} and word_freq(\"It's it\")=={'it':2,'s':1}",
+     "import re\nfrom collections import Counter\ndef word_freq(t):\n    return dict(Counter(re.findall('[a-z]+', t.lower())))"),
+    ("rle", "Write a Python function `rle_decode(s)` that decodes run-length encoding: each run is a decimal count (one or more digits) "
+     "followed by one non-digit character, e.g. '3a2b1c' -> 'aaabbc' and '12x' -> 'xxxxxxxxxxxx'." + ONE_BLOCK_EN,
+     "assert rle_decode('3a2b1c')=='aaabbc' and rle_decode('')=='' and rle_decode('12x')=='x'*12 and rle_decode('10a1b')=='a'*10+'b'",
+     "import re\ndef rle_decode(s):\n    return ''.join(c * int(n) for n, c in re.findall(r'(\\d+)(\\D)', s))"),
+    ("moving_avg", "This function should return the averages of every window of k consecutive items of xs (an empty list when k > len(xs)), "
+     "but it has a bug. Return the corrected function.\n\n```python\ndef moving_average(xs, k):\n    out = []\n    for i in range(len(xs) - k):\n"
+     "        out.append(sum(xs[i:i + k]) / k)\n    return out\n```" + ONE_BLOCK_EN,
+     "assert moving_average([1,2,3,4,5],2)==[1.5,2.5,3.5,4.5] and moving_average([1,2,3],3)==[2.0] and moving_average([1,2],3)==[]",
+     "def moving_average(xs, k):\n    return [sum(xs[i:i + k]) / k for i in range(len(xs) - k + 1)]"),
+    ("second_tr", "Python'da `ikinci_buyuk(xs)` fonksiyonunu yaz: sayı listesindeki en büyük ikinci FARKLI değeri döndürsün; böyle bir değer "
+     "yoksa None döndürsün." + ONE_BLOCK_TR,
+     "assert ikinci_buyuk([3,1,4,1,5])==4 and ikinci_buyuk([2,2]) is None and ikinci_buyuk([7]) is None and ikinci_buyuk([5,5,3])==3 "
+     "and ikinci_buyuk([]) is None",
+     "def ikinci_buyuk(xs):\n    s = sorted(set(xs))\n    return s[-2] if len(s) > 1 else None"),
+    ("two_sum", "Write a Python function `two_sum(nums, target)` that returns a list [i, j] with i < j and nums[i] + nums[j] == target "
+     "(exactly one such pair exists)." + ONE_BLOCK_EN,
+     "assert two_sum([2,7,11,15],9)==[0,1] and two_sum([3,2,4],6)==[1,2] and two_sum([3,3],6)==[0,1]",
+     "def two_sum(nums, target):\n    seen = {}\n    for j, x in enumerate(nums):\n        if target - x in seen:\n"
+     "            return [seen[target - x], j]\n        seen[x] = j"),
+    ("ipv4", "Write a Python function `valid_ipv4(s)` that returns True only for a dotted-quad IPv4 address: exactly four decimal parts "
+     "separated by dots, digits only, each 0-255, no leading zeros (except the single digit '0')." + ONE_BLOCK_EN,
+     "assert valid_ipv4('192.168.1.1') and valid_ipv4('0.0.0.0') and not valid_ipv4('256.1.1.1') and not valid_ipv4('1.2.3') "
+     "and not valid_ipv4('01.2.3.4') and not valid_ipv4('1.2.3.4.5') and not valid_ipv4('a.b.c.d') and not valid_ipv4('1..2.3')",
+     "def valid_ipv4(s):\n    p = s.split('.')\n    return len(p) == 4 and all(x.isascii() and x.isdigit() and (x == '0' or x[0] != '0') "
+     "and int(x) <= 255 for x in p)"),
+    ("lru", "Write a Python class `LRUCache(capacity)` with `get(key)` (returns the value, or -1 if absent) and `put(key, value)`. When "
+     "a put would exceed the capacity, evict the least recently used key; both get and put count as a use." + ONE_BLOCK_EN,
+     "c = LRUCache(2); c.put(1,1); c.put(2,2); assert c.get(1)==1; c.put(3,3); assert c.get(2)==-1; c.put(4,4); assert c.get(1)==-1\n"
+     "assert c.get(3)==3 and c.get(4)==4\nc = LRUCache(1); c.put(1,1); c.put(1,5); assert c.get(1)==5; c.put(2,2); assert c.get(1)==-1",
+     "from collections import OrderedDict\nclass LRUCache:\n    def __init__(self, capacity):\n        self.c, self.n = OrderedDict(), capacity\n"
+     "    def get(self, k):\n        if k not in self.c:\n            return -1\n        self.c.move_to_end(k)\n        return self.c[k]\n"
+     "    def put(self, k, v):\n        self.c[k] = v\n        self.c.move_to_end(k)\n        if len(self.c) > self.n:\n"
+     "            self.c.popitem(last=False)"),
+]
+
+
+# math: each template returns (English, Turkish, answer); the numbers are drawn so that the answers are whole and sensible
+def m_shop(r):
+    p, n, a, b = r.randint(2, 9), r.randint(5, 15), r.randint(2, 9), r.randint(2, 6)
+    spend = a * p + b * n
+    m = (spend // 10 + 1) * 10 + r.choice([0, 10, 20])
+    return (f"A shop sells pencils for {p} dollars each and notebooks for {n} dollars each. Maria buys {a} pencils and {b} notebooks "
+            f"and pays with a {m}-dollar bill. How many dollars of change does she get?",
+            f"Bir kırtasiyede kalem tanesi {p} lira, defter tanesi {n} liradır. Maria {a} kalem ve {b} defter alıp {m} liralık "
+            f"banknotla ödeme yapıyor. Kaç lira para üstü alır?", m - spend)
+
+
+def m_train(r):
+    v1, t1, v2, t2 = r.randint(40, 90), r.randint(2, 5), r.randint(60, 120), r.randint(1, 4)
+    return (f"A train travels at {v1} km/h for {t1} hours and then at {v2} km/h for {t2} hours. How many kilometers does it cover in total?",
+            f"Bir tren saatte {v1} km hızla {t1} saat, ardından saatte {v2} km hızla {t2} saat gidiyor. Toplam kaç km yol alır?",
+            v1 * t1 + v2 * t2)
+
+
+def m_discount(r):
+    x, d, c = 20 * r.randint(5, 20), r.choice([10, 15, 20, 25, 30, 40]), r.randint(3, 15)
+    return (f"A jacket costs {x} dollars. It is discounted by {d}% and then a coupon takes another {c} dollars off the discounted price. "
+            f"What is the final price in dollars?",
+            f"Bir montun fiyatı {x} liradır. Önce %{d} indirim yapılıyor, sonra indirimli fiyattan {c} lira daha düşülüyor. "
+            f"Son fiyat kaç liradır?", Fr(x * (100 - d), 100) - c)
+
+
+def m_age(r):
+    a, x, n = r.randint(8, 30), r.randint(2, 9), r.randint(2, 8)
+    s = 2 * a + x + 2 * n
+    return (f"Tom is {x} years older than Ann. In {n} years the sum of their ages will be {s}. How old is Ann now?",
+            f"Ali, Ayşe'den {x} yaş büyüktür. {n} yıl sonra ikisinin yaşları toplamı {s} olacaktır. Ayşe şimdi kaç yaşındadır?", a)
+
+
+def m_work(r):
+    while True:
+        a, b = r.choice([2, 3, 4, 5, 6, 10, 12, 15]), r.choice([2, 3, 4, 5, 6, 10, 12, 15])
+        mins = Fr(60 * a * b, a + b)
+        if a != b and mins.denominator == 1:
+            return (f"Pipe A can fill a tank in {a} hours and pipe B can fill it in {b} hours. Working together, how many minutes do "
+                    f"they need to fill the empty tank?",
+                    f"A musluğu bir depoyu {a} saatte, B musluğu {b} saatte doldurabiliyor. İkisi birlikte çalışırsa boş depoyu kaç "
+                    f"dakikada doldurur?", mins)
+
+
+def m_rect(r):
+    w = r.randint(3, 20)
+    l, p = 2 * w + 3, 2 * (3 * w + 3)
+    return (f"The length of a rectangle is 3 more than twice its width. Its perimeter is {p} cm. What is its area in square cm?",
+            f"Bir dikdörtgenin uzun kenarı, kısa kenarının iki katından 3 cm fazladır. Çevresi {p} cm olduğuna göre alanı kaç cm²'dir?",
+            w * l)
+
+
+def m_ratio(r):
+    a, b, c, k = r.randint(1, 4), r.randint(2, 6), r.randint(3, 8), r.randint(4, 15)
+    s = k * (a + b + c)
+    return (f"{s} dollars are shared among three friends in the ratio {a}:{b}:{c}. How many dollars does the friend with the largest "
+            f"share get?",
+            f"{s} lira, üç arkadaş arasında {a}:{b}:{c} oranında paylaştırılıyor. En büyük payı alan kaç lira alır?", k * max(a, b, c))
+
+
+def m_codes(r):
+    n = r.randint(5, 9)
+    return (f"How many different 3-letter codes can be made from {n} distinct letters if no letter may be used twice in a code?",
+            f"{n} farklı harften, hiçbir harf bir şifrede iki kez kullanılmamak koşuluyla, kaç farklı 3 harfli şifre oluşturulabilir?",
+            n * (n - 1) * (n - 2))
+
+
+MATH = [m_shop, m_train, m_discount, m_age, m_work, m_rect, m_ratio, m_codes]
+HOW_EN = "\n\nThink step by step, then give the final answer on the last line as: #### <number>"
+HOW_TR = "\n\nAdım adım düşün, sonra son satırda nihai cevabı şu biçimde yaz: #### <sayı>"
+TR_PLURAL = [("kitap", "kitaplar"), ("göz", "gözler"), ("okul", "okullar"), ("kalem", "kalemler"), ("çiçek", "çiçekler"),
+             ("araba", "arabalar"), ("kedi", "kediler"), ("ağaç", "ağaçlar"), ("öğretmen", "öğretmenler"), ("dağ", "dağlar"),
+             ("deniz", "denizler"), ("masa", "masalar"), ("kuş", "kuşlar"), ("gül", "güller")]
+TR_WORDS = ["cam", "çanta", "dağ", "ışık", "iyi", "orman", "ödev", "sandalye", "şeker", "uzay", "ümit", "zeytin", "gül", "hava", "kalem",
+            "lale", "yol", "yüz", "üzüm", "masa", "pencere", "çiçek", "şehir", "ılık", "ince", "öğle", "göl", "fırın", "tarla"]
+MONTHS_EN = ["January", "March", "May", "July", "September", "November"]
+MONTHS_TR = ["Ocak", "Mart", "Mayıs", "Temmuz", "Eylül", "Kasım"]
+CITIES = ["Izmir", "Bursa", "Konya", "Trabzon", "Antalya", "Adana"]
+
+
+def tr_key(w: str):
+    return [TR_ALPHA.index(c) for c in w]
+
+
+def build(seed: int = 1) -> list:
+    """[(id, category, prompt, check(response) -> bool, reference response)]"""
+    T = []
+    for tid, prompt, tests, ref in CODE:
+        T.append((f"code_{tid}", "code", prompt, (lambda t: lambda r: run_code(code_of(r), t))(tests), f"```python\n{ref}\n```"))
+    for i, f in enumerate(MATH):
+        en, tr, ans = f(random.Random(f"{seed}m{i}"))
+        T.append((f"math_{f.__name__[2:]}_en", "math", en + HOW_EN, num(ans), f"#### {ans}"))
+        T.append((f"math_{f.__name__[2:]}_tr", "math", tr + HOW_TR, num(ans), f"#### {ans}"))
+    # Turkish
+    for i in range(3):
+        r = random.Random(f"{seed}ta{i}")
+        a, b, c = r.randint(2, 7), r.randint(2, 6), r.randint(15, 40)
+        T.append((f"tr_age{i}", "tr", f"Zeynep, Ahmet'ten {a} yaş büyüktür. Ahmet, Deniz'den {b} yaş küçüktür. Deniz {c} yaşındadır. "
+                  f"Zeynep kaç yaşındadır? Yalnızca sayıyı yaz.", num(c - b + a), str(c - b + a)))
+    for i in range(2):
+        ws = random.Random(f"{seed}ts{i}").sample(TR_WORDS, 6)
+        want = sorted(ws, key=tr_key)
+        T.append((f"tr_sort{i}", "tr", "Şu kelimeleri Türk alfabesine göre alfabetik sıraya diz ve aralarına virgül koyarak tek satırda yaz: "
+                  + ", ".join(ws), ordered(want), ", ".join(want)))
+    ph = random.Random(f"{seed}tu").choice(["istanbul ışığı", "ilkbahar ışıkları", "iğne ve ipek", "ılık süt içtim", "çiçekli bir bahçe"])
+    up = ph.replace("i", "İ").replace("ı", "I").upper()
+    T.append(("tr_upper", "tr", f"'{ph}' ifadesini Türkçe yazım kurallarına göre tamamen BÜYÜK harfe çevir. Yalnızca sonucu yaz.",
+              lambda r, up=up: up in r, up))
+    for i in range(2):
+        pairs = random.Random(f"{seed}tp{i}").sample(TR_PLURAL, 5)
+        T.append((f"tr_plural{i}", "tr", "Şu kelimelerin çoğul hâlini yaz (her biri için yalnızca çoğul biçimi, virgülle ayırarak): "
+                  + ", ".join(w for w, _ in pairs), has(*[p for _, p in pairs]), ", ".join(p for _, p in pairs)))
+    T.append(("tr_bullets", "tr", "Ankara hakkında tam olarak 3 madde yaz. Her madde satır başında '- ' ile başlasın, başka hiçbir şey yazma.",
+              bullets(3), "- Başkenttir.\n- İç Anadolu'dadır.\n- Kalabalıktır."))
+    T.append(("tr_yesno", "tr", "Yalnızca 'evet' veya 'hayır' yaz: Türkiye'nin başkenti İstanbul mu?", word("hayır", "hayir"), "hayır"))
+    T.append(("tr_translate", "tr", "Şu cümleyi İngilizceye çevir, yalnızca çeviriyi yaz: 'Yarın sabah erken kalkıp işe gideceğim.'",
+              lambda r: sum(bool(re.search(p, r.lower())) for p in (r"tomorrow", r"morning", r"early", r"\bwork|\bjob|office",
+                                                                    r"wake|get up|rise|getting up")) >= 4,
+              "Tomorrow morning I will get up early and go to work."))
+    for i in range(2):
+        r = random.Random(f"{seed}te{i}")
+        day, mon, hh, mm, city = r.randint(2, 28), r.choice(MONTHS_TR), r.randint(9, 17), r.choice(["00", "15", "30", "45"]), r.choice(CITIES)
+        txt = f"Toplantı {day} {mon} 2025 tarihinde, saat {hh}:{mm} olarak, {city} şubesinde yapılacaktır."
+        if i == 0:
+            T.append((f"tr_extract{i}", "tr", f"Metin: {txt}\nSoru: Toplantı saat kaçta? Yalnızca saati yaz.",
+                      lambda r, a=f"{hh}:{mm}", b=f"{hh}.{mm}": a in r or b in r, f"{hh}:{mm}"))
+        else:
+            T.append((f"tr_extract{i}", "tr", f"Metin: {txt}\nSoru: Toplantı hangi şehirde yapılacak? Yalnızca şehir adını yaz.",
+                      has(city), city))
+    T.append(("tr_cross", "tr", "İngilizce olarak cevap ver, tek cümle: Fransa'nın başkenti neresidir?", english_only("paris"),
+              "The capital of France is Paris."))
+    # English
+    for i in range(3):
+        r = random.Random(f"{seed}ea{i}")
+        a, b, c = r.randint(2, 7), r.randint(2, 6), r.randint(15, 40)
+        T.append((f"en_age{i}", "en", f"Emma is {a} years older than Noah. Noah is {b} years younger than Liam. Liam is {c} years old. "
+                  f"How old is Emma? Answer with just the number.", num(c - b + a), str(c - b + a)))
+    T.append(("en_bullets", "en", "Write exactly 3 bullet points about Paris. Start each line with '- ' and write nothing else.",
+              bullets(3), "- Capital of France.\n- On the Seine.\n- Home of the Louvre."))
+    T.append(("en_yesno1", "en", "Answer with only 'yes' or 'no': Is the Pacific Ocean the largest ocean on Earth?", word("yes"), "yes"))
+    T.append(("en_yesno2", "en", "Answer with only 'yes' or 'no': Is 91 a prime number?", word("no"), "no"))
+    T.append(("en_reverse", "en", "Reverse the order of the words in this sentence and output only the result: 'the quick brown fox jumps'",
+              lambda r: "jumps fox brown quick the" in re.sub(r"[^\w ]", "", r.lower()), "jumps fox brown quick the"))
+    for i in range(2):
+        r = random.Random(f"{seed}ee{i}")
+        day, mon, hh, mm, city = r.randint(2, 28), r.choice(MONTHS_EN), r.randint(9, 17), r.choice(["00", "15", "30", "45"]), r.choice(CITIES)
+        txt = f"The meeting will take place on {mon} {day}, 2025 at {hh}:{mm} in the {city} office."
+        if i == 0:
+            T.append((f"en_extract{i}", "en", f"Text: {txt}\nQuestion: At what time is the meeting? Answer with the time only.",
+                      has(f"{hh}:{mm}"), f"{hh}:{mm}"))
+        else:
+            T.append((f"en_extract{i}", "en", f"Text: {txt}\nQuestion: In which city is the meeting? Answer with the city only.", has(city), city))
+    T.append(("en_logic1", "en", "All bloops are razzies. All razzies are lazzies. Is every bloop a lazzie? Answer only yes or no.",
+              word("yes"), "yes"))
+    T.append(("en_logic2", "en", "All birds have feathers. Penguins have feathers. Does it follow that penguins are birds? Answer only yes or no.",
+              word("no"), "no"))
+    T.append(("en_json1", "en", "Give the capital of Japan as a JSON object with keys \"city\" and \"country\". Output only the JSON.",
+              as_json({"city": "Tokyo", "country": "Japan"}), '{"city": "Tokyo", "country": "Japan"}'))
+    T.append(("en_json2", "en", "Give the capital of Canada as a JSON object with keys \"city\" and \"country\". Output only the JSON.",
+              as_json({"city": "Ottawa", "country": "Canada"}), '{"city": "Ottawa", "country": "Canada"}'))
+    T.append(("en_cross", "en", "Answer in Turkish, in one sentence: What is the capital of Turkey?",
+              lambda r: "ankara" in r.lower() and ("başkent" in r.lower() or "baskent" in r.lower()), "Türkiye'nin başkenti Ankara'dır."))
+    T.append(("en_words", "en", "Summarize in at most 12 words: 'The city council voted on Tuesday to build a new bridge over the river, "
+              "after years of debate about the cost.'", lambda r: len(r.split()) <= 12 and "bridge" in r.lower(),
+              "Council votes to build new river bridge after years of debate."))
+    return T
+
+
+# ---------------------------------------------------------------------------------------------------------------- commands
+def table(rows, head):
+    w = [max(len(str(x)) for x in col) for col in zip(head, *rows)]
+    for r in [head] + rows:
+        print("  ".join(str(x).ljust(n) for x, n in zip(r, w)))
+
+
+def serve(a):
+    """starts llama-server on the model and returns once /health says ok"""
+    cmd = [str(Path(a.bin) / "llama-server"), "-m", a.model, "-ngl", "99", "-fa", "on", "-lzm", "on", "-t", str(a.threads), "-c", str(a.ctx),
+           "-np", "1", "--host", "127.0.0.1", "--port", str(a.port)] + (["-ot", a.ot] if a.ot else []) + shlex.split(a.extra)
+    log = open(f"server_{a.name}.log", "w")
+    p = subprocess.Popen(cmd, stdout=log, stderr=subprocess.STDOUT)
+    t0 = time.time()
+    while time.time() - t0 < a.load_timeout:
+        if p.poll() is not None:
+            sys.exit(f"llama-server exited early, see {log.name}")
+        try:
+            if urllib.request.urlopen(f"http://127.0.0.1:{a.port}/health", timeout=5).status == 200:
+                print(f"llama-server ready after {time.time() - t0:.0f} s", flush=True)
+                return p
+        except Exception:
+            time.sleep(5)
+    p.terminate()
+    sys.exit(f"llama-server not ready after {a.load_timeout} s, see {log.name}")
+
+
+def cmd_run(a):
+    srv = serve(a) if a.model else None
+    if srv:
+        a.endpoint = f"http://127.0.0.1:{a.port}"
+    try:
+        run_tasks(a)
+    finally:
+        if srv:
+            srv.terminate()
+            srv.wait(60)
+
+
+def run_tasks(a):
+    if a.suite != "basic":     # public benchmarks from 08_hard_tasks.py / 09_big_suite.py / 10_code_hard.py; long answers need room
+        sys.path.insert(0, str(Path(__file__).resolve().parent))
+        import importlib
+        mod, fn = {"hard": ("08_hard_tasks", "build_hard"), "big": ("09_big_suite", "build_big"), "codehard": ("10_code_hard", "build_codehard")}[a.suite]
+        tasks, a.max_tokens = getattr(importlib.import_module(mod), fn)(a.seed, a.hard_n, run_code, code_of, num), max(a.max_tokens, 2048)
+    else:
+        tasks = build(a.seed)
+    tasks = [t for t in tasks if t[1] in a.only.split(",")]
+    if a.redo:        # only the tasks that file failed (math and reasoning first): does a setting rescue them?
+        prev = json.loads(Path(a.redo).read_text(encoding="utf-8"))["results"]
+        order = {"math": 0, "reason": 1, "code": 2}
+        tasks = sorted([t for t in tasks if t[0] in prev and not prev[t[0]]["pass"]], key=lambda t: order.get(t[1], 3))[:a.redo_max]
+    res, t0, toks = {}, time.time(), 0
+    out = a.out or f"results_{a.name}.json"
+
+    def save():       # after every task, so a dropped session keeps what is done
+        Path(out).write_text(json.dumps({"kind": "tasks", "name": a.name, "seed": a.seed, "tokens": toks, "seconds": round(time.time() - t0),
+                                         "results": res}, indent=1, ensure_ascii=False), encoding="utf-8")
+    def ask(messages, v):      # one chat request -> (reply text, completion tokens)
+        body = {"model": a.name, "messages": messages, "max_tokens": a.max_tokens,
+                "temperature": a.temperature if a.temperature is not None else (a.temp if a.votes > 1 else 0), "seed": a.seed * 1000 + v}
+        for key, val in (("top_p", a.top_p), ("top_k", a.top_k)):
+            if val is not None:
+                body[key] = val
+        if a.no_think:
+            body["chat_template_kwargs"] = {"enable_thinking": False}
+        elif a.think:
+            body["chat_template_kwargs"] = {"enable_thinking": True}
+        req = urllib.request.Request(a.endpoint.rstrip("/") + "/v1/chat/completions", json.dumps(body).encode(),
+                                     {"Content-Type": "application/json"})
+        try:
+            with urllib.request.urlopen(req, timeout=a.timeout) as f:
+                d = json.load(f)
+            return d["choices"][0]["message"].get("content") or "", d.get("usage", {}).get("completion_tokens", 0)
+        except Exception as e:
+            return f"[request failed: {e}]", 0
+
+    first = {k: v["resp"] for k, v in json.loads(Path(a.first_from).read_text(encoding="utf-8"))["results"].items()} if a.first_from else {}
+    for k, (tid, cat, prompt, check, _) in enumerate(tasks, 1):
+        resps, tt, tries = [], 0, 0
+        for v in range(a.votes):
+            text, n = (first[tid], 0) if tid in first else ask([{"role": "user", "content": prompt}], v)     # --first-from: reuse the stored first answer
+            resps.append(text)
+            tt += n
+        oks = [bool(check(strip_think(r))) for r in resps]
+        if a.retry and a.votes == 1 and not oks[0] and hasattr(check, "feedback"):   # repair loop: only visible sample tests are shown to the model
+            msgs = [{"role": "user", "content": prompt}]
+            for _try in range(a.retry):
+                fb = check.feedback(strip_think(resps[0]))
+                if fb is None:      # the visible tests pass: nothing more to tell it
+                    break
+                msgs += [{"role": "assistant", "content": strip_think(resps[0])}, {"role": "user", "content": fb}]
+                resps[0], n = ask(msgs, 0)
+                tt += n
+                tries += 1
+                oks = [bool(check(strip_think(resps[0])))]
+                if oks[0]:
+                    break
+        if a.votes > 1 and cat == "math":     # self-consistency: the most common final number wins, ties go to the first sample
+            ans = [(numbers(strip_think(r)) or [None])[-1] for r in resps]
+            best = max(ans, key=lambda x: (ans.count(x), -ans.index(x)))
+            ok = oks[ans.index(best)]
+        else:                                 # no canonical answer to vote on: the majority of the samples must pass
+            ok = sum(oks) * 2 > len(oks)
+        toks += tt
+        res[tid] = {"cat": cat, "pass": ok, "frac": sum(oks) / len(oks), "toks": tt, "resp": resps[0], **({"retries": tries} if a.retry else {})}
+        print(f"{k:3d}/{len(tasks)} {tid:22s} {'PASS' if ok else 'fail'}  {sum(oks)}/{len(oks)}  {time.time() - t0:5.0f} s" + (f"  retries {tries}" if a.retry else ""), flush=True)
+        save()
+    print(f"{toks} tokens in {time.time() - t0:.0f} s -> {out}")
+    cmd_compare(argparse.Namespace(files=[out]))
+
+
+def cmd_compare(a):
+    docs = [json.loads(Path(f).read_text(encoding="utf-8")) for f in a.files]
+    tasks = [d for d in docs if d["kind"] == "tasks"]
+    if tasks:
+        def rate(d, c):
+            r = [v["pass"] for v in d["results"].values() if c is None or v["cat"] == c]
+            return sum(r), len(r)
+        ref = tasks[0]
+        rows = []
+        for c in list(CATS) + [None]:
+            if c and not rate(ref, c)[1]:
+                continue
+            row = [c or "ALL"]
+            for d in tasks:
+                k, n = rate(d, c)
+                rk, rn = rate(ref, c)
+                pct = f"{100 * (k / n) / (rk / rn):.0f}%" if n and rk else "-"
+                row.append(f"{k}/{n}" + ("" if d is ref else f" ({pct} of ref)"))
+            rows.append(row)
+        rows.append(["time / tokens"] + [f"{d.get('seconds', 0)} s / {d.get('tokens', 0)} tok ({d.get('tokens', 0) / max(d.get('seconds', 1), 1):.1f} t/s)" for d in tasks])
+        table(rows, ["tasks"] + [d["name"] + (" (ref)" if d is ref else "") for d in tasks])
+        for d in tasks[1:]:       # paired sign test on the tasks only one of the two passes ("beats" needs p < 0.05 in its favour)
+            ids = [i for i in ref["results"] if i in d["results"]]
+            b = sum(ref["results"][i]["pass"] and not d["results"][i]["pass"] for i in ids)
+            c = sum(d["results"][i]["pass"] and not ref["results"][i]["pass"] for i in ids)
+            p = min(1.0, 2 * sum(math.comb(b + c, i) for i in range(min(b, c) + 1)) / 2 ** (b + c)) if b + c else 1.0
+            print(f"paired {d['name']} vs {ref['name']}: only {ref['name']} passes {b}, only {d['name']} passes {c}, sign test p = {p:.3f}")
+    klds = [d for d in docs if d["kind"] == "kld"]
+    if klds:
+        print()
+        rows = [[dom] + [f"KLD {d['domains'][dom]['kld']:.3f}  top {d['domains'][dom]['top']:.1f}%  pplx {d['domains'][dom]['ppl_ratio']:.3f}"
+                         if dom in d["domains"] else "-" for d in klds] for dom in ("tr", "en", "code", "math")]
+        table(rows, ["domain"] + [d["name"] for d in klds])
+
+
+def parse_kld(text: str) -> dict:
+    g = lambda p: float(re.search(p, text).group(1))
+    return {"kld": g(r"Mean\s+KLD:\s+([\d.]+)"), "top": g(r"Same top p:\s+([\d.]+)"),
+            "ppl_ratio": g(r"Mean PPL\(Q\)/PPL\(base\)\s*:\s+([\d.]+)"), "ppl": g(r"Mean PPL\(Q\)\s*:\s+([\d.]+)")}
+
+
+def cmd_kld(a):
+    ev = Path(a.eval_dir)
+    res = {}
+    for dom in ("tr", "en", "code", "math"):
+        base = ev / f"base_{dom}.kld"
+        cmd = [str(Path(a.bin) / "llama-perplexity"), "-m", a.model, "-f", str(ev / f"eval_{dom}.txt"), "-c", "512", "-b", "512", "-ub", "512",
+               "--chunks", str(a.chunks), "-ngl", "99", "-fa", "on", "-lzm", "on", "-t", str(a.threads)]
+        cmd += (["-ot", a.ot] if a.ot else []) + shlex.split(a.extra) + ["--kl-divergence-base", str(base)]
+        cmd += [] if a.save_base else ["--kl-divergence"]
+        t0 = time.time()
+        p = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True)
+        if p.returncode != 0:
+            sys.exit(f"{dom}: llama-perplexity failed ({p.returncode}):\n{p.stdout[-1500:]}")
+        if a.save_base:
+            m = re.search(r"Final estimate: PPL = ([\d.]+)", p.stdout)
+            res[dom] = {"ppl": float(m.group(1)) if m else None}
+            print(f"{dom:5s} base saved {base} ({time.time() - t0:.0f} s)  PPL {res[dom]['ppl']}", flush=True)
+            continue
+        try:
+            res[dom] = parse_kld(p.stdout)
+        except AttributeError:
+            sys.exit(f"{dom}: no statistics in the output:\n{p.stdout[-1500:]}")
+        r = res[dom]
+        print(f"{dom:5s} KLD {r['kld']:.3f}  same top {r['top']:.1f}%  PPL ratio {r['ppl_ratio']:.3f}  ({time.time() - t0:.0f} s)", flush=True)
+    if not a.save_base:
+        out = a.out or f"kld_{a.name}.json"
+        Path(out).write_text(json.dumps({"kind": "kld", "name": a.name, "domains": res}, indent=1), encoding="utf-8")
+        print("->", out)
+
+
+def cmd_selftest(a):
+    T = build(a.seed)
+    bad = [t[0] for t in T if not t[3](t[4])]
+    junk = [t[0] for t in T if t[3]("zzz 0")]
+    code_wrong = run_code("def roman_to_int(s):\n    return 0", CODE[0][2])
+    sample = "Mean PPL(Q)                   :   3.059708 ±   0.061630\nMean PPL(Q)/PPL(base)         :   1.131630 ±   0.011859\n" \
+             "Mean    KLD:   0.350462 ±   0.007497\nSame top p: 82.255 ± 0.378 %\n"
+    k = parse_kld(sample)
+    ids = [t[0] for t in T]
+    assert len(set(ids)) == len(ids), "duplicate task ids"
+    assert numbers("1.250 ve 3,5 ve 12") == [1250, Fr(7, 2), 12] and strip_think("<think>x</think>ok") == "ok" and strip_think("<think>x") == ""
+    assert not bad and not junk and not code_wrong, (bad, junk, code_wrong)
+    assert k == {"kld": 0.350462, "top": 82.255, "ppl_ratio": 1.13163, "ppl": 3.059708}, k
+    n = {c: sum(t[1] == c for t in T) for c in CATS}
+    print(f"selftest OK: {len(T)} tasks {n}; reference answers pass, garbage fails, kld parser reads the sample")
+
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    sub = ap.add_subparsers(dest="cmd", required=True)
+    r = sub.add_parser("run")
+    r.add_argument("--endpoint", default="http://127.0.0.1:8080")
+    r.add_argument("--name", required=True)
+    r.add_argument("--out")
+    r.add_argument("--seed", type=int, default=1)
+    r.add_argument("--only", default=",".join(CATS), help="categories, comma separated")
+    r.add_argument("--suite", choices=("basic", "hard", "big", "codehard"), default="basic", help="hard: HumanEval+, MATH-500, MMLU-Pro (08_hard_tasks.py); big: four domains, nested pools (09_big_suite.py); codehard: LiveCodeBench + CRUXEval (10_code_hard.py, use --max-tokens 4096 --ctx 8192); use --out")
+    r.add_argument("--hard-n", type=int, default=100, help="tasks per category in the hard / big suite (big: 60 = quick prefix, 300 = full)")
+    r.add_argument("--max-tokens", type=int, default=768)
+    r.add_argument("--think", action="store_true", help="send chat_template_kwargs enable_thinking=true")
+    r.add_argument("--redo", help="results file: run only the tasks it failed (same --suite/--seed/--hard-n)")
+    r.add_argument("--redo-max", type=int, default=1000)
+    r.add_argument("--first-from", help="with --retry: take each task's first answer from this results file (its 'resp') instead of generating it again")
+    r.add_argument("--retry", type=int, default=0, help="lcb tasks: after a failed answer, show the model its first failing VISIBLE sample test and ask again, up to N times (needs --ctx big enough for the extra turns)")
+    r.add_argument("--temperature", type=float, help="sampling temperature (default: 0, or --temp when --votes > 1)")
+    r.add_argument("--top-p", type=float)
+    r.add_argument("--top-k", type=int)
+    r.add_argument("--timeout", type=int, default=1200)
+    r.add_argument("--no-think", action="store_true", help="send chat_template_kwargs enable_thinking=false")
+    r.add_argument("--votes", type=int, default=1, help="samples per task; math: the most common final number wins, other categories: majority must pass")
+    r.add_argument("--temp", type=float, default=0.7, help="sampling temperature when --votes > 1 ")
+    r.add_argument("--model", help="start llama-server on this GGUF (needs --bin) instead of using --endpoint")
+    r.add_argument("--bin", help="directory with llama-server")
+    r.add_argument("--ot", default="", help="-ot overrides for the server")
+    r.add_argument("--extra", default="", help="more llama-server flags, one string: --extra='--flag value'")
+    r.add_argument("--threads", type=int, default=8)
+    r.add_argument("--ctx", type=int, default=4096)
+    r.add_argument("--port", type=int, default=18080)   # 8080 is taken on Colab
+    r.add_argument("--load-timeout", type=int, default=1800)
+    r.set_defaults(f=cmd_run)
+    c = sub.add_parser("compare")
+    c.add_argument("files", nargs="+", help="result files; the first task file is the reference")
+    c.set_defaults(f=cmd_compare)
+    k = sub.add_parser("kld")
+    k.add_argument("--bin", required=True, help="directory with llama-perplexity")
+    k.add_argument("--model", required=True)
+    k.add_argument("--name", required=True)
+    k.add_argument("--eval-dir", default="evalset")
+    k.add_argument("--chunks", type=int, default=16)
+    k.add_argument("--ot", default="", help="-ot overrides (which tensors stay on the CPU)")
+    k.add_argument("--extra", default="", help="more llama-perplexity flags, one string: --extra='--flag value'")
+    k.add_argument("--threads", type=int, default=8)
+    k.add_argument("--save-base", action="store_true", help="this is the reference: store its logits")
+    k.add_argument("--out")
+    k.set_defaults(f=cmd_kld)
+    s = sub.add_parser("selftest")
+    s.add_argument("--seed", type=int, default=1)
+    s.set_defaults(f=cmd_selftest)
+    a = ap.parse_args()
+    a.f(a)
+
+
+if __name__ == "__main__":
+    main()

benchmarks/scripts/09_big_suite.py

@@ -0,0 +1,309 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+"""
+09_big_suite.py - the large four-domain task set behind `07_domain_eval.py run --suite big --hard-n N` (N tasks per domain, up to the pool size).
+Public benchmarks, English only, every answer checked by code; the tasks of a smaller N are the FIRST N of a bigger one (nested), so a quick run
+(--hard-n 60) is a prefix of the full run (--hard-n 300) and the two can be pooled:
+  code    HumanEval+ (164) and MBPP sanitized test (257), 2:3 mix; the model writes the function, the unit tests run on it
+  math    MATH-500 integer answers of level 2-5 (278) and AIME (90, AI-MO/aimo-validation-aime), 3:1 mix; final line "#### <integer>"
+  reason  MMLU-Pro (240, 14 subjects round-robin, 10 options) and BIG-Bench Hard (120, 8 tasks), 2:1 mix; final line "Answer: <x>"
+  en      RACE-high reading comprehension (120), CoLA grammar acceptability (120, half acceptable), and 120 writing prompts with machine-checked
+          constraints (word counts, paragraphs, bullets, case, keywords, start/end text, quotes, title, commas, P.S.), 1:1:1 mix
+Needs `pip install datasets` and network access; the model-written code is EXECUTED (see 07).  Selftest (same needs):
+  python 09_big_suite.py selftest          reference answers pass, garbage fails, every constraint checker passes its own good/bad example
+  python 09_big_suite.py validate          runs EVERY reference solution (421 code tasks) and prints the ids to put in BAD_CODE
+"""
+import random
+import re
+import sys
+from fractions import Fraction as Fr
+
+HOW = "\n\nThink step by step, then give the final answer on the last line as: #### <integer>"
+LETTERS = "ABCDEFGHIJ"
+BAD_CODE = {"code_he32"}      # code task ids whose own reference solution fails its tests (`validate`: HumanEval/32 find_zero is float-sensitive)
+
+
+def mix(pools, pattern):
+    """The pools interleaved by the digits of `pattern` ("00101" = two from pool 0, one from pool 1, ...) until every pool is used up."""
+    pos, out, i = [0] * len(pools), [], 0
+    while any(pos[k] < len(pools[k]) for k in range(len(pools))) and i < 10 ** 6:
+        k = int(pattern[i % len(pattern)])
+        i += 1
+        if pos[k] < len(pools[k]):
+            out.append(pools[k][pos[k]])
+            pos[k] += 1
+    return out
+
+
+def letter(L, k):
+    """check: the last 'Answer: X' of the response is the letter L (X among the first k letters)."""
+    pat = rf"Answer:[\s*(]*([{LETTERS[:k]}])\b"
+    return lambda resp: (re.findall(pat, resp) or [None])[-1] == L
+
+
+def token(target):
+    """check: the last 'Answer: X' of the response has the single word/number X equal to the target (case-insensitive, '(A)' = 'a')."""
+    want = re.sub(r"[^a-z0-9\-]", "", target.casefold())
+    return lambda resp: bool((got := re.findall(r"Answer:[\s*(\[]*([A-Za-z0-9\-]+)", resp))) and got[-1].casefold() == want
+
+
+# ---------------------------------------------------------------------------------------------------------------------------------- code
+def pool_code(rnd, run_code, code_of):
+    from datasets import load_dataset
+    he, mb = [], []
+    for r in load_dataset("evalplus/humanevalplus", split="test"):
+        tests = r["test"] + f"\n\ncheck({r['entry_point']})"
+        he.append((f"code_he{r['task_id'].split('/')[-1]}", "code",
+                   "Complete this Python function. Reply with one Python code block containing the whole function (and any imports it needs).\n\n"
+                   f"```python\n{r['prompt']}```",
+                   (lambda t: lambda resp: run_code(code_of(resp), t, 20))(tests), f"```python\n{r['prompt']}{r['canonical_solution']}\n```"))
+    for r in load_dataset("google-research-datasets/mbpp", "sanitized", split="test"):
+        tests = "\n".join(r["test_imports"]) + "\n" + "\n".join(r["test_list"])
+        mb.append((f"code_mbpp{r['task_id']}", "code",
+                   "Write a Python function for the task below. Reply with one Python code block containing the whole function (and any imports it "
+                   f"needs).\n\nTask: {r['prompt']}\n\nYour function must pass these tests (they fix its name and signature):\n" + "\n".join(r["test_list"]),
+                   (lambda t: lambda resp: run_code(code_of(resp), t, 20))(tests), f"```python\n{r['code']}\n```"))
+    he, mb = [t for t in he if t[0] not in BAD_CODE], [t for t in mb if t[0] not in BAD_CODE]
+    rnd.shuffle(he)
+    rnd.shuffle(mb)
+    return mix([he, mb], "00111")       # 2 HumanEval+ : 3 MBPP per cycle of five
+
+
+# ---------------------------------------------------------------------------------------------------------------------------------- math
+def pool_math(rnd, num):
+    from datasets import load_dataset
+    ma, ai = [], []
+    for r in load_dataset("HuggingFaceH4/MATH-500", split="test"):
+        if re.fullmatch(r"-?\d+", str(r["answer"]).strip()) and int(r["level"]) >= 2:
+            ans = Fr(str(r["answer"]).strip())
+            ma.append((f"math_{r['unique_id'].replace('/', '_').removesuffix('.json')}", "math", r["problem"] + HOW, num(ans), f"#### {ans}"))
+    for r in load_dataset("AI-MO/aimo-validation-aime", split="train"):
+        if re.fullmatch(r"-?\d+", str(r["answer"]).strip()):
+            ans = Fr(str(r["answer"]).strip())
+            ai.append((f"math_aime{r['id']}", "math", r["problem"] + HOW, num(ans), f"#### {ans}"))
+    rnd.shuffle(ma)
+    rnd.shuffle(ai)
+    return mix([ma, ai], "0001")                  # 3 MATH-500 : 1 AIME
+
+
+# ---------------------------------------------------------------------------------------------------------------------------------- reasoning
+BBH = {       # task -> the closing instruction that names the answer format
+    "logical_deduction_five_objects": "Answer: <option letter>", "tracking_shuffled_objects_five_objects": "Answer: <option letter>",
+    "date_understanding": "Answer: <option letter>", "web_of_lies": "Answer: Yes or Answer: No", "boolean_expressions": "Answer: True or Answer: False",
+    "navigate": "Answer: Yes or Answer: No", "formal_fallacies": "Answer: valid or Answer: invalid", "object_counting": "Answer: <integer>"}
+
+
+def pool_reason(rnd):
+    from datasets import load_dataset
+    by = {}
+    for r in load_dataset("TIGER-Lab/MMLU-Pro", split="test"):
+        by.setdefault(r["category"], []).append(r)
+    cats = sorted(by)
+    for c in cats:
+        rnd.shuffle(by[c])
+    mm = []
+    for i in range(18):                                       # round-robin over the 14 subjects, the subject order shuffled each round
+        order = cats[:]
+        rnd.shuffle(order)
+        for c in order:
+            r = by[c][i]
+            opts = "\n".join(f"{LETTERS[j]}. {o}" for j, o in enumerate(r["options"]))
+            mm.append((f"reason_mmlu{r['question_id']}", "reason", f"{r['question']}\n\n{opts}\n\nThink step by step, then end with exactly: Answer: <letter>",
+                       letter(r["answer"], 10), f"Answer: {r['answer']}"))
+    mm = mm[:240]
+    bb = []
+    for task, fmt in BBH.items():
+        rows = list(load_dataset("lukaemon/bbh", task, split="test"))
+        rnd.shuffle(rows)
+        bb.append([(f"reason_bbh_{task}_{k}", "reason", f"{r['input']}\n\nThink step by step, then end with exactly: {fmt}", token(r["target"]),
+                    f"Answer: {r['target'].strip('()')}") for k, r in enumerate(rows[:15])])
+    return mix([mm, mix(bb, "01234567")], "001")             # 2 MMLU-Pro : 1 BBH
+
+
+# ---------------------------------------------------------------------------------------------------------------------------------- English
+def _words(r):
+    return len(re.findall(r"\w+", r))
+
+
+C = {}      # constraint name -> (tag, make); make(rnd) -> (rule sentence, check, a response that passes, a response that fails)
+CLASH = [("lowercase", "end_phrase"), ("lowercase", "start_word"), ("lowercase", "postscript"), ("uppercase", "end_phrase"),
+         ("uppercase", "start_word"), ("wrap_quotes", "end_phrase"), ("wrap_quotes", "start_word"), ("wrap_quotes", "postscript"),
+         ("end_phrase", "postscript"), ("title", "start_word")]
+KEYS = ("river", "lantern", "courage", "market", "silver", "window", "winter", "bridge")
+BANNED = ("very", "really", "good", "nice", "thing")
+ENDS = ("Is there anything else I can help with?", "That is all for now.", "Thank you for reading.")
+STARTS = ("Today", "Imagine", "Remember", "Consider")
+
+
+def cons(tag):
+    def deco(f):
+        C[f.__name__] = (tag, f)
+        return f
+    return deco
+
+
+@cons("len")
+def words_min(rnd):
+    n = rnd.choice((80, 120, 150, 200))
+    return f"Your response must contain at least {n} words.", lambda r: _words(r) >= n, "word " * n, "word " * (n - 1)
+
+
+@cons("len")
+def words_max(rnd):
+    n = rnd.choice((60, 80, 100, 120))
+    return f"Your response must contain fewer than {n} words.", lambda r: _words(r) < n, "word " * (n - 1), "word " * n
+
+
+@cons("struct")
+def paragraphs(rnd):
+    n = rnd.choice((2, 3, 4))
+    return (f"Write exactly {n} paragraphs, separated from each other by a line containing only ***.",
+            lambda r: len(p := re.split(r"(?m)^[ \t]*\*\*\*[ \t]*$", r.strip())) == n and all(x.strip() for x in p),
+            "\n***\n".join(["para"] * n), "\n***\n".join(["para"] * (n + 1)))
+
+
+@cons("struct")
+def bullets(rnd):
+    n = rnd.choice((3, 4, 5))
+    return (f"Use exactly {n} bullet points in markdown format, each line starting with \"* \", and no other bullets.",
+            lambda r: sum(1 for l in r.splitlines() if re.match(r"\s*\* ", l)) == n, "\n".join(["* item"] * n), "\n".join(["* item"] * (n + 1)))
+
+
+@cons("kw")
+def keywords(rnd):
+    a, b = rnd.sample(KEYS, 2)
+    return (f"Include the words \"{a}\" and \"{b}\".", lambda r: all(re.search(rf"\b{k}\b", r, re.I) for k in (a, b)), f"{a} and {b}", f"only {a}")
+
+
+@cons("forbid")
+def forbid(rnd):
+    w = rnd.choice(BANNED)
+    return f"Do not use the word \"{w}\" anywhere.", lambda r: not re.search(rf"\b{w}\b", r, re.I), "fine text", f"a {w} text"
+
+
+@cons("case")
+def lowercase(rnd):
+    return "Your entire response must be in lowercase letters, with no capital letters at all.", lambda r: r == r.lower(), "all lower", "Not lower"
+
+
+@cons("case")
+def uppercase(rnd):
+    return "Your entire response must be in ALL CAPITAL LETTERS.", lambda r: r == r.upper(), "ALL UPPER", "Not upper"
+
+
+@cons("end")
+def end_phrase(rnd):
+    p = rnd.choice(ENDS)
+    return f"Finish your response with the exact phrase \"{p}\" and add nothing after it.", lambda r: r.strip().endswith(p), f"text {p}", f"{p} text"
+
+
+@cons("start")
+def start_word(rnd):
+    w = rnd.choice(STARTS)
+    return f"Begin your response with the word \"{w}\".", lambda r: bool(re.match(rf"\W*{w}\b", r.strip())), f"{w} is a day", f"Then {w}"
+
+
+@cons("wrap")
+def wrap_quotes(rnd):
+    return ("Wrap your entire response in double quotation marks.", lambda r: len(r.strip()) > 1 and r.strip()[0] == '"' and r.strip()[-1] == '"',
+            '"text"', "text")
+
+
+@cons("title")
+def title(rnd):
+    return "Include a title wrapped in double angular brackets, like <<title>>.", lambda r: bool(re.search(r"<<[^<>\n]+>>", r)), "<<A title>> body", "A title body"
+
+
+@cons("punct")
+def no_commas(rnd):
+    return "Do not use any commas in your response.", lambda r: "," not in r, "no comma", "a, b"
+
+
+@cons("ps")
+def postscript(rnd):
+    return "At the end, add a postscript that starts with P.S.", lambda r: bool(re.search(r"(?m)^\s*P\.S\.", r)), "text\nP.S. more", "text"
+
+
+@cons("letter")
+def letter_z(rnd):
+    n = rnd.choice((3, 5))
+    return f"The letter \"z\" must appear at least {n} times.", lambda r: r.lower().count("z") >= n, "z" * n, "z" * (n - 1)
+
+
+TOPICS = ("a rainy morning in a small town", "why sleep matters", "the history of tea", "a lost umbrella", "learning to swim as an adult", "a city without cars",
+          "the first day at a new job", "a village market", "the smell of fresh bread", "an old lighthouse", "a long train journey", "keeping a diary",
+          "a robot that learns to paint", "the quiet of a library", "planting a vegetable garden", "a broken clock", "moving to another country",
+          "the last bus home", "a mountain hike", "a neighbour who plays the violin", "saving money as a student", "a power cut at night",
+          "the invention of the bicycle", "a street musician", "cooking for a big family", "an unexpected letter", "the sound of the sea",
+          "a school sports day", "a very small apartment", "the first snow of winter")
+FORMS = ("short story", "product description", "email to a colleague", "blog post", "poem", "short speech", "review", "letter to a friend",
+         "news summary", "explanation for a ten-year-old")
+
+
+def pool_en(rnd):
+    from datasets import load_dataset
+    race = list(load_dataset("ehovy/race", "high", split="test"))
+    rnd.shuffle(race)
+    ra = []
+    for k, r in enumerate(race[:120]):          # several questions share one passage id, so the position is part of the task id
+        opts = "\n".join(f"{LETTERS[j]}. {o}" for j, o in enumerate(r["options"]))
+        ra.append((f"en_race_{k:03d}_{r['example_id'].replace('/', '_').replace('.txt', '')}", "en",
+                   f"Read the passage, then answer the question.\n\nPassage:\n{r['article']}\n\nQuestion: {r['question']}\n\n{opts}\n\n"
+                   "Think briefly, then end with exactly: Answer: <letter>", letter(r["answer"], 4), f"Answer: {r['answer']}"))
+    cola = list(load_dataset("nyu-mll/glue", "cola", split="validation"))
+    yes, no = [r for r in cola if r["label"] == 1], [r for r in cola if r["label"] == 0]
+    rnd.shuffle(yes)
+    rnd.shuffle(no)
+    co = []
+    for r in mix([yes[:60], no[:60]], "01"):
+        co.append((f"en_cola_{r['idx']}", "en", "Is this sentence grammatically acceptable English, as a careful native speaker would judge it?\n\n"
+                   f"Sentence: {r['sentence']}\n\nGive at most two sentences of reasoning, then end with exactly: Answer: Yes or Answer: No",
+                   token("Yes" if r["label"] == 1 else "No"), f"Answer: {'Yes' if r['label'] == 1 else 'No'}"))
+    wr, names = [], sorted(C)
+    for i in range(120):
+        for _ in range(200):
+            pick = rnd.sample(names, 2 if rnd.random() < 0.7 else 3)
+            if len({C[n][0] for n in pick}) == len(pick) and not any(a in pick and b in pick for a, b in CLASH):
+                break
+        parts = [C[n][1](rnd) for n in pick]
+        wr.append((f"en_if{i:03d}", "en", f"Write a {rnd.choice(FORMS)} about {rnd.choice(TOPICS)}. Follow every rule below.\n" + "\n".join(f"- {p[0]}" for p in parts),
+                   (lambda cs: lambda resp: all(c(resp) for c in cs))([p[1] for p in parts]), None))
+    return mix([ra, co, wr], "012")
+
+
+# ---------------------------------------------------------------------------------------------------------------------------------- the suite
+def build_big(seed, n, run_code, code_of, num):
+    """[(id, category, prompt, check(response) -> bool, reference response or None)]: the first n tasks of each pool (nested for any n)."""
+    T = []
+    for name, pool in (("code", lambda r: pool_code(r, run_code, code_of)), ("math", lambda r: pool_math(r, num)), ("reason", pool_reason), ("en", pool_en)):
+        T += pool(random.Random(f"{seed}big{name}"))[:n]
+    assert len({t[0] for t in T}) == len(T), "duplicate task ids"
+    return T
+
+
+def constraint_unit_tests():
+    bad = []
+    for name, (tag, make) in C.items():
+        for s in range(8):
+            _, chk, good, worse = make(random.Random(s))
+            if not chk(good) or chk(worse):
+                bad.append((name, s))
+    return bad
+
+
+if __name__ == "__main__" and sys.argv[1:] in (["selftest"], ["validate"]):
+    import importlib
+    sys.path.insert(0, __import__("pathlib").Path(__file__).resolve().parent.as_posix())
+    m = importlib.import_module("07_domain_eval")
+    if sys.argv[1] == "selftest":
+        assert not constraint_unit_tests(), constraint_unit_tests()
+        T = build_big(1, 12, m.run_code, m.code_of, m.num)
+        ref = [t for t in T if t[4] is not None]
+        bad = [t[0] for t in ref if not t[3](t[4])]
+        junk = [t[0] for t in T if t[4] is not None and t[3]("zzz 987654321")]      # the constraint tasks have no reference and garbage may satisfy a rule
+        per = {c: sum(t[1] == c for t in T) for c in ("code", "math", "reason", "en")}
+        assert not bad and not junk and per == {c: 12 for c in per} and len({t[0] for t in T}) == len(T), (bad, junk, per)
+        print(f"selftest OK: {len(T)} tasks {per}, {len(ref)} references pass, garbage fails, {len(C)} constraint checkers pass their own examples")
+    else:
+        bad = [t[0] for t in pool_code(random.Random("v"), m.run_code, m.code_of) if not t[3](t[4])]
+        print(f"{len(bad)} reference solutions fail: BAD_CODE = {set(bad)!r}")

benchmarks/scripts/10_code_hard.py

@@ -0,0 +1,321 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+"""
+10_code_hard.py - harder coding benchmarks behind `07_domain_eval.py run --suite codehard --hard-n N` (N tasks per benchmark, nested prefixes):
+  lcb    LiveCodeBench code_generation_lite, release 6: 175 AtCoder and LeetCode contest problems from 2025-01 to 2025-04 (newer than most training
+         data), about 40 hidden tests each; the model writes a whole program (stdin/stdout) or a LeetCode class, every test must pass.
+         The task id carries the difficulty: lcb_easy_..., lcb_medium_..., lcb_hard_...
+  crux   CRUXEval (800 short Python functions), half output prediction (CRUXEval-O) and half input prediction (CRUXEval-I): code understanding,
+         checked by running `assert f(...) == ...` on the model's answer.
+Public data, no token.  Needs `pip install datasets` and network (the LiveCodeBench file is about 1 GB).  THE MODEL-WRITTEN CODE IS EXECUTED (see 07).
+Use `--max-tokens 4096 --ctx 8192` for the lcb tasks (long answers).  Checks:
+  python 10_code_hard.py selftest     lookup-table solutions pass every test of a few lcb problems, wrong ones fail; crux references pass
+  python 10_code_hard.py validate     runs all 800 crux references and prints the ids to put in BAD_CRUX
+"""
+import base64
+import io
+import json
+import os
+import pickle
+import random
+import re
+import subprocess
+import sys
+import tempfile
+import time
+import zlib
+from pathlib import Path
+
+LCB_FILES = ("test6",)          # release 6 (2025-01 .. 2025-04); add "test5" (2024-09 .. 2025-01) for a bigger pool
+BAD_CRUX = set()                # crux ids whose own reference fails (filled by `validate`)
+HEADER = ("from string import *\nfrom re import *\nfrom datetime import *\nfrom collections import *\nfrom heapq import *\nfrom bisect import *\n"
+          "from copy import deepcopy\nfrom math import *\nfrom typing import *\nfrom functools import *\nfrom itertools import *\nfrom operator import *\n"
+          "from random import *\nfrom statistics import *\nimport sys, heapq, bisect, math, collections, itertools, functools, random, string, re, json, copy\n")
+FUNC_RUNNER = '''import json, sys
+ns = {}
+exec(compile(open("sol.py", encoding="utf-8").read(), "sol.py", "exec"), ns)
+f = getattr(ns["Solution"](), sys.argv[1])
+for inp, out in json.load(open("tests.json", encoding="utf-8")):
+    got = f(*[json.loads(l) for l in inp.split("\\n") if l.strip()])
+    if json.loads(json.dumps(got)) != json.loads(out):
+        print("FAIL")
+        sys.exit(1)
+print("OK")
+'''
+
+
+# ---------------------------------------------------------------------------------------------------------------------------------- LiveCodeBench
+class _Plain(pickle.Unpickler):
+    """The private tests are a pickled JSON string; a string needs no globals, so refuse every one (a dataset must not run code while loading)."""
+    def find_class(self, module, name):
+        raise pickle.UnpicklingError(f"global {module}.{name} refused")
+
+
+def decode_private(s):
+    return json.loads(_Plain(io.BytesIO(zlib.decompress(base64.b64decode(s)))).load())
+
+
+def lcb_tests(row):
+    return json.loads(row["public_test_cases"]) + decode_private(row["private_test_cases"])
+
+
+def _same(got, want):
+    g, w = [l.rstrip() for l in got.strip().splitlines()], [l.rstrip() for l in want.strip().splitlines()]
+    if len(g) != len(w):
+        return False
+    for a, b in zip(g, w):
+        if a == b:
+            continue
+        ta, tb = a.split(), b.split()
+        if len(ta) != len(tb):
+            return False
+        for x, y in zip(ta, tb):
+            if x == y:
+                continue
+            if re.fullmatch(r"-?\d+", x) and re.fullmatch(r"-?\d+", y):      # integers are exact
+                return False
+            try:
+                if abs(float(x) - float(y)) > 1e-6 * max(1.0, abs(float(y))):
+                    return False
+            except ValueError:
+                return False
+    return True
+
+
+def _limits():
+    if os.name != "posix":
+        return None
+    import resource
+    return lambda: resource.setrlimit(resource.RLIMIT_AS, (2 << 30, 2 << 30))
+
+
+def run_lcb(code, tests, func, total=90, per=8):
+    """True when `code` passes every test: func None = a stdin/stdout program (one process per test), else a LeetCode Solution method."""
+    env = {**os.environ, "OPENBLAS_NUM_THREADS": "1", "OMP_NUM_THREADS": "1"}
+    with tempfile.TemporaryDirectory() as d:
+        d = Path(d)
+        (d / "sol.py").write_text(HEADER + code, encoding="utf-8")
+        t0 = time.time()
+        if func is None:
+            for t in tests:
+                if time.time() - t0 > total:
+                    return False
+                try:
+                    r = subprocess.run([sys.executable, "-I", "sol.py"], input=t["input"], capture_output=True, text=True, timeout=per, cwd=d, env=env,
+                                       preexec_fn=_limits())
+                except (subprocess.TimeoutExpired, UnicodeDecodeError):
+                    return False
+                if r.returncode != 0 or not _same(r.stdout, t["output"]):
+                    return False
+            return True
+        (d / "tests.json").write_text(json.dumps([[t["input"], t["output"]] for t in tests]), encoding="utf-8")
+        (d / "run.py").write_text(FUNC_RUNNER, encoding="utf-8")
+        try:
+            r = subprocess.run([sys.executable, "-I", "run.py", func], capture_output=True, text=True, timeout=total, cwd=d, env=env, preexec_fn=_limits())
+        except (subprocess.TimeoutExpired, UnicodeDecodeError):
+            return False
+        return r.returncode == 0 and r.stdout.strip().endswith("OK")
+
+
+FUNC_DEBUG = '''import json, sys, traceback
+ns = {}
+exec(compile(open("sol.py", encoding="utf-8").read(), "sol.py", "exec"), ns)
+f = getattr(ns["Solution"](), sys.argv[1])
+for inp, out in json.load(open("tests.json", encoding="utf-8")):
+    try:
+        got = f(*[json.loads(l) for l in inp.split("\\n") if l.strip()])
+    except Exception:
+        print("Input:\\n" + inp.strip()[:300] + "\\nRuntime error:\\n" + traceback.format_exc()[-600:])
+        sys.exit(1)
+    if json.loads(json.dumps(got)) != json.loads(out):
+        print("Input:\\n" + inp.strip()[:300] + "\\nExpected output:\\n" + out.strip()[:300] + "\\nYour output:\\n" + json.dumps(got)[:300])
+        sys.exit(1)
+print("OK")
+'''
+
+
+def first_failure(code, tests, func, per=8):
+    """None when `code` passes every test in `tests` (the VISIBLE sample tests), else a short text on the first failure, for a repair prompt."""
+    env = {**os.environ, "OPENBLAS_NUM_THREADS": "1", "OMP_NUM_THREADS": "1"}
+    with tempfile.TemporaryDirectory() as d:
+        d = Path(d)
+        (d / "sol.py").write_text(HEADER + code, encoding="utf-8")
+        if func is None:
+            for t in tests:
+                shown = "Input:\n" + t["input"].strip()[:300]
+                try:
+                    r = subprocess.run([sys.executable, "-I", "sol.py"], input=t["input"], capture_output=True, text=True, timeout=per, cwd=d, env=env,
+                                       preexec_fn=_limits())
+                except subprocess.TimeoutExpired:
+                    return f"{shown}\nThe program did not finish within {per} seconds."
+                except UnicodeDecodeError:
+                    return f"{shown}\nThe program printed bytes that are not valid text."
+                if r.returncode != 0:
+                    return f"{shown}\nRuntime error:\n{r.stderr[-600:]}"
+                if not _same(r.stdout, t["output"]):
+                    return f"{shown}\nExpected output:\n{t['output'].strip()[:300]}\nYour output:\n{r.stdout.strip()[:300]}"
+            return None
+        (d / "tests.json").write_text(json.dumps([[t["input"], t["output"]] for t in tests]), encoding="utf-8")
+        (d / "run.py").write_text(FUNC_DEBUG, encoding="utf-8")
+        try:
+            r = subprocess.run([sys.executable, "-I", "run.py", func], capture_output=True, text=True, timeout=per * 4, cwd=d, env=env, preexec_fn=_limits())
+        except subprocess.TimeoutExpired:
+            return "The code did not finish in time on the sample tests."
+        except UnicodeDecodeError:
+            return "The code printed bytes that are not valid text."
+        if r.returncode == 0 and r.stdout.strip().endswith("OK"):
+            return None
+        return (r.stdout or r.stderr).strip()[-800:] or "The code crashed."
+
+
+def lcb_feedback(resp, row, func):
+    """What to tell the model after a failed attempt: only the visible sample tests are used (never the hidden ones); None = no new information."""
+    code = last_block(resp)
+    if not code.strip():
+        return "I could not find a ```python code block in your answer. Give the complete solution in one ```python block."
+    bad = first_failure(code, json.loads(row["public_test_cases"]), func)
+    if bad is None:
+        return None
+    return "Your code fails this sample test.\n\n" + bad + "\n\nFix the code. Reply with the complete corrected solution in one ```python block."
+
+
+def last_block(resp):
+    blocks = re.findall(r"```(?:python|py|Python)?[ \t]*\n(.*?)```", resp, flags=re.S)
+    return blocks[-1] if blocks else ""
+
+
+def lcb_prompt(row):
+    if row["starter_code"]:
+        fmt = "You will use the following starter code to write the solution to the problem and enclose your code within delimiters."
+        block = row["starter_code"]
+    else:
+        fmt = ("Read the inputs from stdin solve the problem and write the answer to stdout (do not directly test on the sample inputs). Enclose your "
+               "code within delimiters as follows. Ensure that when the python program runs, it reads the inputs, runs the algorithm and writes output "
+               "to STDOUT.")
+        block = "# YOUR CODE HERE"
+    return f"### Question:\n{row['question_content']}\n\n### Format: {fmt}\n```python\n{block}\n```\n\n### Answer: (use the provided format with backticks)\n"
+
+
+def lcb_rows():
+    from datasets import load_dataset
+    rows = []
+    for f in LCB_FILES:
+        rows += list(load_dataset("json", data_files=f"hf://datasets/livecodebench/code_generation_lite/{f}.jsonl", split="train"))
+    return rows
+
+
+def pool_lcb(rnd):
+    rows = lcb_rows()
+    rnd.shuffle(rows)
+    T = []
+    for r in rows:
+        func = json.loads(r["metadata"] or "{}").get("func_name") if r["starter_code"] else None
+
+        def check(resp, r=r, func=func, cache={}):
+            code = last_block(resp)
+            if not code.strip():
+                return False
+            if r["question_id"] not in cache:
+                cache.clear()
+                cache[r["question_id"]] = lcb_tests(r)
+            return run_lcb(code, cache[r["question_id"]], func)
+        check.feedback = lambda resp, r=r, func=func: lcb_feedback(resp, r, func)     # used by `07 run --retry N`
+        T.append((f"lcb_{r['difficulty']}_{r['question_id']}", "lcb", lcb_prompt(r), check, None))
+    return T
+
+
+# ---------------------------------------------------------------------------------------------------------------------------------- CRUXEval
+def pool_crux(rnd, run_code):
+    from datasets import load_dataset
+    rows = [r for r in load_dataset("cruxeval-org/cruxeval", split="test")]
+    ro, ri = rows[:], rows[:]
+    rnd.shuffle(ro)
+    rnd.shuffle(ri)
+    O, I = [], []
+    for r in ro:
+        if f"crux_o_{r['id']}" in BAD_CRUX:
+            continue
+        pro = ("You are given a Python function and an assertion containing an input to the function. Complete the assertion with a literal (no "
+               "unsimplified expressions, no function calls) containing the output when executing the provided code on the given input, even if the "
+               "function is incorrect or incomplete. Think step by step, then end with exactly one line in this form: [ANSWER] assert f(...) == <literal> "
+               f"[/ANSWER]\n\n```python\n{r['code']}\nassert f({r['input']}) == ??\n```")
+
+        def check_o(resp, r=r):
+            m = re.findall(r"\[ANSWER\]\s*assert\s+f\(.*?\)\s*==\s*(.*?)\s*\[/ANSWER\]", resp, flags=re.S)
+            return bool(m) and run_code(r["code"], f"assert f({r['input']}) == ({m[-1]})", 10)
+        O.append((f"crux_o_{r['id']}", "crux", pro, check_o, f"[ANSWER] assert f({r['input']}) == {r['output']} [/ANSWER]"))
+    for r in ri:
+        if f"crux_i_{r['id']}" in BAD_CRUX:
+            continue
+        pri = ("You are given a Python function and an assertion containing an output of the function. Find any input such that executing the function "
+               "on the input leads to the given output. There may be several answers, give only one. Think step by step, then end with exactly one line "
+               f"in this form: [ANSWER] assert f(<your input>) == {r['output']} [/ANSWER]\n\n```python\n{r['code']}\nassert f(??) == {r['output']}\n```")
+
+        def check_i(resp, r=r):
+            m = re.findall(r"\[ANSWER\]\s*assert\s+f\((.*)\)\s*==", resp, flags=re.S)
+            return bool(m) and run_code(r["code"], f"assert f({m[-1]}) == {r['output']}", 10)
+        I.append((f"crux_i_{r['id']}", "crux", pri, check_i, f"[ANSWER] assert f({r['input']}) == {r['output']} [/ANSWER]"))
+    mixed, i = [], 0
+    while i < min(len(O), len(I)):
+        mixed += [O[i], I[i]]
+        i += 1
+    return mixed
+
+
+# ---------------------------------------------------------------------------------------------------------------------------------- the suite
+def build_codehard(seed, n, run_code, code_of, num):
+    """[(id, category, prompt, check(response) -> bool, reference response or None)]: the first n lcb problems and the first n crux tasks."""
+    T = pool_lcb(random.Random(f"{seed}lcb"))[:n] + pool_crux(random.Random(f"{seed}crux"), run_code)[:n]
+    assert len({t[0] for t in T}) == len(T), "duplicate task ids"
+    return T
+
+
+def _lookup_solution(row, tests):
+    """A 'solution' that only looks the answers up: it passes iff the runner, the decoding and the comparison are right."""
+    if not row["starter_code"]:
+        return "import sys\nD = " + repr({t["input"]: t["output"] for t in tests}) + "\nprint(D[sys.stdin.read()], end='')\n", None
+    func = json.loads(row["metadata"])["func_name"]
+    D = {json.dumps([json.loads(l) for l in t["input"].split("\n") if l.strip()]): t["output"] for t in tests}
+    return f"import json\nD = {D!r}\nclass Solution:\n    def {func}(self, *a):\n        return json.loads(D[json.dumps(list(a))])\n", func
+
+
+if __name__ == "__main__" and sys.argv[1:] in (["selftest"], ["validate"]):
+    import importlib
+    sys.path.insert(0, Path(__file__).resolve().parent.as_posix())
+    m = importlib.import_module("07_domain_eval")
+    if sys.argv[1] == "selftest":
+        rows = lcb_rows()
+        stdin, func = [r for r in rows if not r["starter_code"]][:4], [r for r in rows if r["starter_code"]][:4]
+        for r in stdin + func:
+            tests = lcb_tests(r)
+            code, fn = _lookup_solution(r, tests)
+            assert run_lcb(code, tests, fn), ("lookup solution failed", r["question_id"])
+            wrong = "print(0)\n" if fn is None else f"class Solution:\n    def {fn}(self, *a):\n        return None\n"
+            assert not run_lcb(wrong, tests, fn), ("wrong solution passed", r["question_id"])
+        assert last_block("text\n```python\nA\n```\nmore\n```python\nB\n```") == "B\n"
+        for r in stdin + func:          # the repair feedback: a correct answer needs none, wrong / crashing / empty answers get a message
+            tests = json.loads(r["public_test_cases"])
+            good, fn = _lookup_solution(r, tests)
+            assert lcb_feedback("```python\n" + good + "```", r, fn) is None, ("feedback for a correct solution", r["question_id"])
+            bad_code = "print(0)\n" if fn is None else f"class Solution:\n    def {fn}(self, *a):\n        return None\n"
+            crash = "raise ValueError('boom')\n" if fn is None else f"class Solution:\n    def {fn}(self, *a):\n        raise ValueError('boom')\n"
+            assert "sample test" in (lcb_feedback("```python\n" + bad_code + "```", r, fn) or ""), ("no feedback for a wrong answer", r["question_id"])
+            assert "boom" in (lcb_feedback("```python\n" + crash + "```", r, fn) or ""), ("no error text for a crash", r["question_id"])
+            assert "could not find" in lcb_feedback("no code here", r, fn)
+        T = build_codehard(1, 10, m.run_code, m.code_of, m.num)
+        crux = [t for t in T if t[1] == "crux"]
+        bad = [t[0] for t in crux if not t[3](t[4])]
+        junk = [t[0] for t in T if t[1] == "crux" and t[3]("zzz 987654321")] + [t[0] for t in T if t[1] == "lcb" and t[3]("zzz 987654321")]
+        per = {c: sum(t[1] == c for t in T) for c in ("lcb", "crux")}
+        assert not bad and not junk and per == {"lcb": 10, "crux": 10}, (bad, junk, per)
+        print(f"selftest OK: {len(T)} tasks {per}; {len(stdin)} stdin + {len(func)} functional lcb problems pass their lookup solution and fail a wrong one; "
+              f"crux references pass, garbage fails")
+    else:
+        from datasets import load_dataset
+        bad = []
+        for r in load_dataset("cruxeval-org/cruxeval", split="test"):
+            if not m.run_code(r["code"], f"assert f({r['input']}) == ({r['output']})", 10):
+                bad += [f"crux_o_{r['id']}", f"crux_i_{r['id']}"]
+        print(f"{len(bad) // 2} crux references fail: BAD_CRUX = {set(bad)!r}")
+        n_ok = sum(1 for r in lcb_rows() if lcb_tests(r))
+        print(f"lcb: {n_ok} problems decode their tests")

benchmarks/scripts/13_swe_native.py

@@ -0,0 +1,288 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+"""
+13_swe_native.py - SWE-bench Verified (Mini) WITHOUT Docker or conda, on a plain Linux VM, driving the real Claude Code CLI.
+
+It reuses swebench 4.0.4's own per-repo specs, setup/eval scripts and log parsers (pip install --no-deps --target /content/sb4 swebench==4.0.4),
+translating the conda lines to `uv venv` environments, one /testbed checkout at a time.
+
+  python 13_swe_native.py list                                  the instances of SWE-bench Verified Mini with their repo / version
+  python 13_swe_native.py gold  INSTANCE_ID                     build the environment and check that the GOLD patch resolves the instance
+  python 13_swe_native.py agent --ids a,b,c --out results_swe_k448.json --endpoint http://127.0.0.1:18080 --model k448 --user agent
+        [--max-turns 60] [--timeout 2700]                      Claude Code solves each instance, the patch is graded by the official parsers
+
+A resolved instance = every FAIL_TO_PASS test passes and no PASS_TO_PASS test breaks (the official SWE-bench definition).  THE MODEL'S COMMANDS
+AND THE REPOSITORIES' TESTS ARE EXECUTED: use a disposable VM; the agent runs as a separate unix user.
+"""
+import argparse
+import json
+import os
+import re
+import shlex
+import shutil
+import subprocess
+import sys
+import time
+from pathlib import Path
+
+sys.path.insert(0, "/content/sb4")
+ENVS, TESTBED, WORK = Path("/content/sb_envs"), Path("/testbed"), Path("/content/sb_work")
+CONSTRAINTS = Path("/content/sb_constraints.txt")
+PY_MAP = {"3.5": "3.8", "3.6": "3.8", "3.7": "3.8"}          # uv offers CPython 3.8+ ; these old pins run fine on 3.8
+
+
+def rows():
+    from datasets import load_dataset
+    return [dict(r) for r in load_dataset("MariusHobbhahn/swe-bench-verified-mini", split="test")]
+
+
+def get(iid):
+    for r in rows():
+        if r["instance_id"] == iid:
+            return r
+    sys.exit(f"unknown instance {iid}")
+
+
+def test_spec(inst):
+    from swebench.harness.test_spec.test_spec import make_test_spec
+    return make_test_spec(inst)
+
+
+def env_dir(inst):
+    return ENVS / f"{inst['repo'].replace('/', '__')}__{inst['version']}__v4"
+
+
+def adapt(script, venv, py):
+    """conda lines -> uv venv lines (the scripts are plain bash otherwise)."""
+    s = script.replace("source /opt/miniconda3/bin/activate\n", "")
+    s = re.sub(r"conda create -n testbed python=([\d.]+)\s+-y", lambda m: f"uv venv --seed --python {PY_MAP.get(m.group(1), m.group(1))} {venv} && {venv}/bin/python -m pip install 'setuptools<70'", s)
+    s = s.replace("conda activate testbed", f"source {venv}/bin/activate")
+    s = s.replace('echo "Current environment: $CONDA_DEFAULT_ENV"', "true")
+    # sphinx: tox would build its own interpreter without pytest here; run what its tox.ini runs, directly in the venv
+    s = re.sub(r"tox --current-env -e\S+ -v -- ", "PYTHONWARNINGS=all python -X dev -m pytest --durations 25 -rA ", s)
+    return s
+
+
+def bash(script, log, timeout, env=None):
+    Path(log).parent.mkdir(parents=True, exist_ok=True)
+    f = Path(log).with_suffix(".sh")
+    f.write_text(script, encoding="utf-8")
+    try:
+        r = subprocess.run(["bash", str(f)], stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, timeout=timeout, errors="replace",
+                           env={**os.environ, "PIP_CONSTRAINT": str(CONSTRAINTS), **(env or {})})      # one stream, in order: the grader reads the output between the START/END markers
+        out = r.stdout
+        code = r.returncode
+    except subprocess.TimeoutExpired as e:
+        out = ((e.stdout or b"").decode("utf-8", "replace") if isinstance(e.stdout, bytes) else (e.stdout or "")) + "\n[timeout]"
+        code = 124
+    Path(log).write_text(out, encoding="utf-8")
+    return code, out
+
+
+def prepare(inst):
+    """Environment (cached per repo+version) and a fresh /testbed at the base commit with the repo installed."""
+    ts = test_spec(inst)
+    venv = env_dir(inst)
+    py = re.search(r"python=([\d.]+)", ts.setup_env_script)
+    py = PY_MAP.get(py.group(1), py.group(1)) if py else "3.9"
+    if not (venv / ".done").exists():
+        shutil.rmtree(venv, ignore_errors=True)
+        ENVS.mkdir(parents=True, exist_ok=True)
+        code, out = bash(adapt(ts.setup_env_script, venv, py), WORK / "logs" / f"{venv.name}_env.log", 3000)
+        if code != 0 or not (venv / "bin" / "python").exists():
+            raise RuntimeError(f"environment build failed (exit {code}): {out[-600:]}")
+        (venv / ".done").write_text("ok")
+    shutil.rmtree(TESTBED, ignore_errors=True)
+    code, out = bash(adapt(ts.install_repo_script, venv, py), WORK / "logs" / f"{inst['instance_id']}_repo.log", 2400)
+    if code != 0 or not (TESTBED / ".git").exists():
+        raise RuntimeError(f"repo install failed (exit {code}): {out[-600:]}")
+    return ts, venv
+
+
+def alias_docstring_lines(out):
+    """Python 3.8 prints `test_x (mod.Class)` and the docstring line separately; add the one-line form that the dataset's test names use."""
+    res, prev = [], None
+    for line in out.split("\n"):
+        res.append(line)
+        t = line.strip()
+        m = re.match(r"^(test\S* \([\w.]+\))$", t)
+        if m:
+            prev = m.group(1)
+        elif prev and " ... " in t:
+            res.append(f"{prev} ... {t.rsplit(' ... ', 1)[1]}")
+            prev = None
+        else:
+            prev = None
+    return "\n".join(res)
+
+
+def evaluate(inst, ts, venv, patch):
+    """Fresh checkout, apply the model patch, run the official eval script, grade with the official parser. -> (resolved, note)"""
+    from swebench.harness.grading import get_eval_report
+    py = re.search(r"python=([\d.]+)", ts.setup_env_script)
+    py = PY_MAP.get(py.group(1), py.group(1)) if py else "3.9"
+    subprocess.run("git reset --hard -q && git clean -fdq", shell=True, cwd=TESTBED)
+    if not patch.strip():
+        return False, "empty patch"
+    pf = WORK / f"{inst['instance_id']}.patch"
+    pf.write_text(patch if patch.endswith("\n") else patch + "\n", encoding="utf-8")
+    r = subprocess.run(["git", "apply", "-v", str(pf)], cwd=TESTBED, capture_output=True, text=True)
+    if r.returncode != 0:
+        return False, "patch does not apply: " + (r.stdout + r.stderr)[-200:]
+    code, out = bash(adapt(ts.eval_script, venv, py), WORK / "logs" / f"{inst['instance_id']}_eval.log", 2700)
+    logp = WORK / "logs" / f"{inst['instance_id']}_report.log"
+    logp.write_text(">>>>> Applied Patch\n" + alias_docstring_lines(out), encoding="utf-8")
+    try:
+        rep = get_eval_report(test_spec=ts, prediction={"instance_id": inst["instance_id"], "model_name_or_path": "k448", "model_patch": patch},
+                              test_log_path=str(logp), include_tests_status=True)[inst["instance_id"]]
+    except Exception as e:
+        return False, f"grading error: {e!r}"
+    ts_ = rep.get("tests_status", {})
+    f2p, p2p = ts_.get("FAIL_TO_PASS", {}), ts_.get("PASS_TO_PASS", {})
+    note = f"F2P {len(f2p.get('success', []))}/{len(f2p.get('success', [])) + len(f2p.get('failure', []))}, P2P ok {len(p2p.get('success', []))} broken {len(p2p.get('failure', []))}"
+    return bool(rep.get("resolved")), note
+
+
+def cmd_goldall(a):
+    """Gold-patch check of every instance: the ones that resolve are the runnable benchmark."""
+    out = Path(a.out)
+    res = json.loads(out.read_text(encoding="utf-8")) if out.exists() else {}
+    for inst in rows():
+        iid = inst["instance_id"]
+        if iid in res:
+            continue
+        t0 = time.time()
+        try:
+            ts, venv = prepare(inst)
+            ok, note = evaluate(inst, ts, venv, inst["patch"])
+        except Exception as e:
+            ok, note = False, f"error: {e!r}"[:300]
+        res[iid] = {"ok": ok, "note": note, "secs": round(time.time() - t0)}
+        out.write_text(json.dumps(res, indent=1), encoding="utf-8")
+        print(f"{iid:36s} gold_resolved={ok}  {note[:100]}  ({time.time() - t0:.0f} s)", flush=True)
+    print(f"runnable {sum(v['ok'] for v in res.values())}/{len(res)}")
+
+
+def cmd_list(a):
+    for r in rows():
+        print(f"{r['instance_id']:36s} {r['repo']:20s} {r['version']:5s} F2P={len(json.loads(r['FAIL_TO_PASS']) if isinstance(r['FAIL_TO_PASS'], str) else r['FAIL_TO_PASS'])}")
+
+
+def cmd_gold(a):
+    inst = get(a.instance)
+    t0 = time.time()
+    ts, venv = prepare(inst)
+    print(f"prepared in {time.time() - t0:.0f} s", flush=True)
+    ok, note = evaluate(inst, ts, venv, inst["patch"])
+    print(f"GOLD {inst['instance_id']}: resolved={ok}  {note}  ({time.time() - t0:.0f} s)")
+    return 0 if ok else 1
+
+
+def agent_prompt(inst):
+    return (f"<uploaded_files>\n/testbed\n</uploaded_files>\nI've uploaded a python code repository in the directory /testbed. Consider the following issue "
+            f"description:\n\n<issue>\n{inst['problem_statement']}\n</issue>\n\nCan you help me implement the necessary changes to the repository so that the "
+            "requirements specified in the issue are met? I've already taken care of all changes to any of the test files described in the issue, so you do NOT "
+            "need to modify the testing logic or any of the tests in any way. Make the minimal changes to non-test files in /testbed so that the issue is resolved. "
+            "You can run the existing tests to check your work. When you are done, stop.")
+
+
+def parse_stream(text):
+    calls = errs = 0
+    last, final = "", {}
+    for line in text.splitlines():
+        if not line.startswith("{"):
+            continue
+        try:
+            ev = json.loads(line)
+        except ValueError:
+            continue
+        t = ev.get("type")
+        if t == "assistant":
+            for b in (ev.get("message", {}).get("content") or []):
+                if b.get("type") == "tool_use":
+                    calls += 1
+                elif b.get("type") == "text" and b.get("text", "").strip():
+                    last = b["text"].strip()
+        elif t == "user":
+            c = ev.get("message", {}).get("content")
+            if isinstance(c, list):
+                errs += sum(1 for b in c if isinstance(b, dict) and b.get("type") == "tool_result" and b.get("is_error"))
+        elif t == "result":
+            final = ev
+    u = final.get("usage") or {}
+    return {"turns": final.get("num_turns", 0), "tool_calls": calls, "tool_errors": errs, "out_tok": u.get("output_tokens", 0),
+            "in_tok": u.get("input_tokens", 0) + u.get("cache_read_input_tokens", 0) + u.get("cache_creation_input_tokens", 0),
+            "stop": final.get("subtype") or ("no_result" if not final else "?"), "last": last[-300:]}
+
+
+def cmd_agent(a):
+    ids = a.ids.split(",") if a.ids else [r["instance_id"] for r in rows()]
+    out = Path(a.out)
+    res = json.loads(out.read_text(encoding="utf-8")).get("results", {}) if out.exists() else {}
+    logs = Path(str(out.with_suffix("")) + "_logs")
+    logs.mkdir(parents=True, exist_ok=True)
+    home = WORK / "_home"
+    home.mkdir(parents=True, exist_ok=True)
+    t0 = time.time()
+    for k, iid in enumerate(ids, 1):
+        if iid in res:
+            continue
+        inst = get(iid)
+        s0 = time.time()
+        info = {"repo": inst["repo"], "version": inst["version"]}
+        try:
+            ts, venv = prepare(inst)
+            subprocess.run(["chown", "-R", f"{a.user}:{a.user}", str(TESTBED), str(home)], check=True)
+            env = {"PATH": f"{venv}/bin:" + os.environ.get("PATH", ""), "VIRTUAL_ENV": str(venv), "HOME": str(home), "LANG": "C.UTF-8",
+                   "ANTHROPIC_BASE_URL": a.endpoint, "ANTHROPIC_AUTH_TOKEN": "local", "ANTHROPIC_API_KEY": "local", "ANTHROPIC_MODEL": a.model,
+                   "ANTHROPIC_SMALL_FAST_MODEL": a.model, "ANTHROPIC_DEFAULT_HAIKU_MODEL": a.model, "ANTHROPIC_DEFAULT_SONNET_MODEL": a.model,
+                   "ANTHROPIC_DEFAULT_OPUS_MODEL": a.model, "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC": "1", "DISABLE_TELEMETRY": "1",
+                   "DISABLE_AUTOUPDATER": "1", "DISABLE_ERROR_REPORTING": "1", "IS_SANDBOX": "1"}
+            cmd = ["runuser", "-u", a.user, "--", "env"] + [f"{kk}={vv}" for kk, vv in env.items()] + shlex.split(a.claude) + [
+                "-p", agent_prompt(inst), "--output-format", "stream-json", "--verbose", "--dangerously-skip-permissions",
+                "--max-turns", str(a.max_turns), "--model", a.model]
+            try:
+                r = subprocess.run(cmd, cwd=TESTBED, capture_output=True, text=True, timeout=a.timeout, errors="replace", stdin=subprocess.DEVNULL)
+                text, err, timed_out = r.stdout, r.stderr, False
+            except subprocess.TimeoutExpired as e:
+                text = ((e.stdout or b"").decode("utf-8", "replace") if isinstance(e.stdout, bytes) else (e.stdout or ""))
+                err, timed_out = "timeout", True
+            (logs / f"{iid}.jsonl").write_text(text + ("\n[stderr]\n" + err[-2000:] if err else ""), encoding="utf-8")
+            info.update(parse_stream(text))
+            info["timed_out"] = timed_out
+            subprocess.run("git add -A && git diff --cached HEAD > /tmp/model.patch", shell=True, cwd=TESTBED)
+            patch = Path("/tmp/model.patch").read_text(encoding="utf-8", errors="replace")
+            info["patch_lines"] = patch.count("\n")
+            info["pass"], info["note"] = evaluate(inst, ts, venv, patch)
+        except Exception as e:
+            info.update({"pass": False, "note": f"harness error: {e!r}"[:300]})
+        info["secs"] = round(time.time() - s0)
+        res[iid] = info
+        out.write_text(json.dumps({"kind": "swe", "name": a.model, "seconds": round(time.time() - t0), "results": res}, indent=1), encoding="utf-8")
+        print(f"{k:3d}/{len(ids)} {iid:34s} {'RESOLVED' if info['pass'] else 'no':8s} turns={info.get('turns', 0):3d} calls={info.get('tool_calls', 0):3d} "
+              f"tool_errors={info.get('tool_errors', 0):2d} patch={info.get('patch_lines', 0):3d} lines {info['secs']:5d} s  {info.get('note', '')[:80]}", flush=True)
+    n = len(res)
+    print(f"\nresolved {sum(v['pass'] for v in res.values())}/{n}")
+
+
+if __name__ == "__main__":
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    sub = ap.add_subparsers(dest="cmd", required=True)
+    sub.add_parser("list")
+    ga = sub.add_parser("goldall")
+    ga.add_argument("--out", default="/content/gold_all.json")
+    g = sub.add_parser("gold")
+    g.add_argument("instance")
+    ag = sub.add_parser("agent")
+    ag.add_argument("--ids")
+    ag.add_argument("--out", default="results_swe.json")
+    ag.add_argument("--endpoint", default="http://127.0.0.1:18080")
+    ag.add_argument("--model", default="k448")
+    ag.add_argument("--user", default="agent")
+    ag.add_argument("--claude", default="claude")
+    ag.add_argument("--max-turns", type=int, default=60)
+    ag.add_argument("--timeout", type=int, default=2700)
+    a = ap.parse_args()
+    WORK.mkdir(parents=True, exist_ok=True)
+    CONSTRAINTS.write_text("Pygments==2.18.0\n", encoding="utf-8")      # the version current when the official images were built
+    {"list": cmd_list, "gold": cmd_gold, "goldall": cmd_goldall, "agent": cmd_agent}[a.cmd](a)

benchmarks/scripts/14_msg_proxy.py

@@ -0,0 +1,87 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+"""
+14_msg_proxy.py - a small logging / rewriting proxy between Claude Code and llama-server's /v1/messages endpoint.
+
+  python 14_msg_proxy.py            listens on 127.0.0.1:18082, forwards to 127.0.0.1:18080, streams the answer back, and appends every
+                                    /v1/messages request body to /content/proxy_log.jsonl.  fix() is the hook for rewriting a request
+                                    (the chat template of Qwen3.8 rejects some shapes that Claude Code sends).
+"""
+import http.client
+import http.server
+import json
+import time
+
+UP = ("127.0.0.1", 18080)
+CRLF = bytes([13, 10])
+
+
+def fix(d):
+    """Qwen3.8's chat template raises 'System message must be at the beginning' for a system-role entry in `messages`, which Claude Code sends
+    after the user turn.  Fold every such entry into the last user message as an extra text block (nothing is dropped)."""
+    msgs = d.get("messages") or []
+    keep, extra = [], []
+    for m in msgs:
+        if m.get("role") == "system":
+            c = m.get("content")
+            text = c if isinstance(c, str) else chr(10).join(b.get("text", "") for b in c if isinstance(b, dict) and b.get("type") == "text")
+            if text.strip():
+                extra.append(text)
+        else:
+            keep.append(m)
+    if len(keep) != len(msgs):
+        block = dict(type="text", text=(chr(10) * 2).join(extra) if extra else "(no content)")
+        for m in reversed(keep):
+            if m.get("role") == "user":
+                c = m.get("content")
+                m["content"] = ([dict(type="text", text=c)] if isinstance(c, str) else list(c)) + [block]
+                break
+        else:
+            keep.append(dict(role="user", content=[block]))
+        d["messages"] = keep
+    return d
+
+
+class H(http.server.BaseHTTPRequestHandler):
+    protocol_version = "HTTP/1.1"
+
+    def log_message(self, *a):
+        pass
+
+    def handle_any(self):
+        n = int(self.headers.get("Content-Length") or 0)
+        body = self.rfile.read(n) if n else b""
+        if self.path.startswith("/v1/messages") and body:
+            try:
+                d = json.loads(body)
+                with open("/content/proxy_log.jsonl", "a") as f:
+                    f.write(json.dumps(dict(t=time.time(), path=self.path, req=d))[:600000] + "\n")
+                body = json.dumps(fix(d)).encode()
+            except Exception as e:
+                open("/content/proxy_err.log", "a").write(repr(e) + "\n")
+        c = http.client.HTTPConnection(*UP, timeout=7200)
+        hdr = dict((k, v) for k, v in self.headers.items() if k.lower() not in ("host", "content-length", "connection", "accept-encoding", "transfer-encoding"))
+        c.request(self.command, self.path, body, hdr)
+        r = c.getresponse()
+        with open("/content/proxy_status.log", "a") as f:
+            f.write(f"{time.time():.0f} {self.command} {self.path} {r.status}\n")
+        self.send_response(r.status)
+        for k, v in r.getheaders():
+            if k.lower() not in ("transfer-encoding", "connection", "content-length"):
+                self.send_header(k, v)
+        self.send_header("Transfer-Encoding", "chunked")
+        self.end_headers()
+        while True:
+            chunk = r.read1(65536)
+            if not chunk:
+                break
+            self.wfile.write(b"%x" % len(chunk) + CRLF + chunk + CRLF)
+            self.wfile.flush()
+        self.wfile.write(b"0" + CRLF + CRLF)
+        self.wfile.flush()
+
+    do_GET = do_POST = do_PUT = do_DELETE = handle_any
+
+
+if __name__ == "__main__":
+    http.server.ThreadingHTTPServer(("127.0.0.1", 18082), H).serve_forever()