Release unmodified quantization with complete saved evaluation metrics and historical reference
1816fff verified | # VeriLoop-E2 paired quantized evaluation | |
| Same frozen questions, seed 20260816, thinking enabled, publisher raw-content protocol, 32768-token output budget. Legacy custom scoring, including partial IFEval rules. Format comparisons use different engines/KV types; compare original versus ablated within each format. | |
| Runtime: dual RTX 3090. W4A16 uses vLLM 0.29.0 with a local FA2/FP8-KV plugin, tensor parallel 2, concurrency 16, 65,536-token context, FP8 E5M2 KV. GGUF uses llama.cpp layer split 1:1, concurrency 8, 40,960-token context per slot, Q8_0 key/value cache. Both use temperature 0 and top_p 1. See evaluation-runtime-versions.json for exact runtime identifiers. | |
| ## w4a16 | |
| | Task | n | Original | Ablated | Delta (pp) | | |
| |---|---:|---:|---:|---:| | |
| | mmlu | 150 | 84.00% | 76.00% | -8.00 | | |
| | cmmlu | 150 | 84.67% | 75.33% | -9.33 | | |
| | mmlu_pro | 150 | 78.67% | 70.67% | -8.00 | | |
| | ceval | 150 | 84.00% | 76.00% | -8.00 | | |
| | arc | 150 | 80.67% | 85.33% | +4.67 | | |
| | truthfulqa | 150 | 58.67% | 71.33% | +12.67 | | |
| | gsm8k | 100 | 95.00% | 88.00% | -7.00 | | |
| | math500 | 100 | 75.00% | 73.00% | -2.00 | | |
| | bbh | 150 | 75.33% | 65.33% | -10.00 | | |
| | humaneval | 164 | 89.63% | 89.63% | +0.00 | | |
| | ifeval | 100 | 76.00% | 69.00% | -7.00 | | |
| Overall: 1212/1514 (80.05%) → 1157/1514 (76.42%). Delta -3.63 pp, paired approximate 95% CI [-5.894, -1.371] pp. Lost/gained: 181/126. | |
| Empty: [9, 8]; explicitly delimited reasoning: [1186, 1007]; length-limited: [9, 8]. Empty-rate validity gate (≤2%): False. | |
| Refusal head markers (100 prompts, thinking off): [98, 0]; validity: [True, True]; 2048-token budget hits: [1, 9]. See comparison.json for all marker counts. | |
| Empty-rate gate exception: original-w4a16 / humaneval: 5/164 (3.05%), above 2%. These outputs remain failed in the denominator. | |
| ### Refusal marker diagnostics | |
| | Metric | Original | Ablated | | |
| |---|---:|---:| | |
| | substring_any | 98 | 2 | | |
| | substring_head | 98 | 0 | | |
| | heretic33_any | 98 | 69 | | |
| | ega26_any | 98 | 55 | | |
| | short | 0 | 0 | | |
| | empty | 0 | 0 | | |
| | empty_but_reasoning | 0 | 0 | | |
| | reasoning_leaked | 0 | 0 | | |
| | errors | 0 | 0 | | |
| Selection-independent 84-item subset: opening markers [82, 0]; anywhere markers [82, 2]. These are heuristic counts, not a semantic compliance score. | |
| ## gguf | |
| | Task | n | Original | Ablated | Delta (pp) | | |
| |---|---:|---:|---:|---:| | |
| | mmlu | 150 | 68.00% | 68.00% | +0.00 | | |
| | cmmlu | 150 | 61.33% | 64.67% | +3.33 | | |
| | mmlu_pro | 150 | 64.00% | 68.67% | +4.67 | | |
| | ceval | 150 | 60.00% | 70.00% | +10.00 | | |
| | arc | 150 | 90.00% | 91.33% | +1.33 | | |
| | truthfulqa | 150 | 60.67% | 60.67% | +0.00 | | |
| | gsm8k | 100 | 37.00% | 47.00% | +10.00 | | |
| | math500 | 100 | 68.00% | 67.00% | -1.00 | | |
| | bbh | 150 | 48.00% | 54.67% | +6.67 | | |
| | humaneval | 164 | 90.85% | 90.24% | -0.61 | | |
| | ifeval | 100 | 63.00% | 60.00% | -3.00 | | |
| Overall: 995/1514 (65.72%) → 1039/1514 (68.63%). Delta +2.91 pp, paired approximate 95% CI [0.956, 4.856] pp. Lost/gained: 92/136. | |
| Empty: [4, 7]; explicitly delimited reasoning: [741, 782]; length-limited: [4, 7]. Empty-rate validity gate (≤2%): True. | |
| Refusal head markers (100 prompts, thinking off): [99, 0]; validity: [True, True]; 2048-token budget hits: [0, 7]. See comparison.json for all marker counts. | |
| ### Refusal marker diagnostics | |
| | Metric | Original | Ablated | | |
| |---|---:|---:| | |
| | substring_any | 99 | 0 | | |
| | substring_head | 99 | 0 | | |
| | heretic33_any | 99 | 69 | | |
| | ega26_any | 99 | 54 | | |
| | short | 0 | 0 | | |
| | empty | 0 | 0 | | |
| | empty_but_reasoning | 0 | 0 | | |
| | reasoning_leaked | 0 | 0 | | |
| | errors | 0 | 0 | | |
| Selection-independent 84-item subset: opening markers [83, 0]; anywhere markers [83, 0]. These are heuristic counts, not a semantic compliance score. | |
| ## Limits | |
| An initial original-W4A16 preflight incorrectly enabled a Qwen3 reasoning parser that the publisher does not prescribe. It classified 33/150 direct answers as hidden reasoning. Removing the incompatible parser recovered all 33, with zero empty/length-limited outputs; streaming and nonstreaming probes passed. All reported runs keep thinking enabled and the original template, use publisher raw-content semantics, and extract final content after a closing think tag only when present. Unclosed length-limited output is not scored as an answer. | |
| These are local sampled benchmarks with the historical custom scorers, not official leaderboard scores. Truncated or empty outputs remain in the primary denominator. Marker absence does not prove semantic compliance. Refusal probes overlap the earlier 16-item selection set for the first 16 items; the remaining 84 are not used for candidate selection. No new ablation tuning was performed after these evaluations. No equivalence claim follows solely from a nonsignificant difference. | |
| GGUF protocol verification: the supplied standalone source template overrides an older embedded identity block. All 1,514 prompts and token sequences match the source tokenizer. llama.cpp echoes the opening think generation-prefix tag in content-only mode; the protocol adapter removes that framing before legacy scoring. Two streaming/nonstreaming probes passed. The partial GGUF first pass is excluded. Requests allow 7,200 seconds for the slower engine, while output budgets and scorers remain unchanged. | |
| The fixed-protocol GGUF run was paused to investigate low GSM8K accuracy. Two failed cases remained wrong with prompt caching disabled and with CPU-only F16 KV. One also failed under the original Hugging Face BF16 model in Transformers. Sampled BF16 GGUF tensors matched the source exactly. These checks did not establish a GGUF-only pipeline fault; they do not establish full BF16 benchmark accuracy or explain the entire cross-format gap. The existing run resumed without changing questions, scoring, weights, or engine. Wall-clock durations include the diagnostic pause and are not throughput measurements. An isolated newer llama.cpp CPU build was not used for the reported evaluation. | |