Add full release benchmark report

release-benchmark-report.md

@@ -0,0 +1,233 @@
+# Unbound — Release Benchmark Report
+
+*Compiled 2026-05-20. Headline models: `evalengine/unbound-e2b` (v18) and
+`evalengine/unbound-e4b` (E4B-3). Bases: `unsloth/gemma-4-E2B-it` and
+`unsloth/gemma-4-E4B-it`.*
+
+## TL;DR
+
+Both Unbound builds achieve a clean uncensoring effect with **negligible
+capability damage**:
+
+- **Refusal rate** on AdvBench-520 collapses from ~98% → **4.42% (E2B)** /
+  **2.69% (E4B)**, judged by an external LLM.
+- **Useful-compliance rate** rises from ~1% baseline → **39.2% (E2B)** /
+  **47.3% (E4B)**.
+- **Hallucination on harmful prompts** rises from ~1% baseline → 16.0% (E2B)
+  / 13.1% (E4B) — the price of uncensoring a small model.
+- **Capability**: TruthfulQA mc2 **up** on both. MMLU, GSM8K, GPQA-Diamond,
+  BBH macro all land **within stderr / ≤1 pp** of base on E4B; E2B shows a
+  ~1 pp BBH dip that is small but real.
+- **Coherence on benign prompts**: 1.00 on both.
+
+E4B-3 is the strongest model overall: best refusal collapse, lowest
+hallucination, lowest KL, and capability scores statistically indistinguishable
+from the E4B base.
+
+## Methodology
+
+Three benchmark families, run on the merged HF checkpoint (post-fixed merge,
+see `AGENTS.md` "Unsloth merge trap"):
+
+1. **AdvBench-520 with LLM judge** — `scripts/benchmark.py` Phase A.
+   - Each AdvBench prompt is sent to the model under test.
+   - Each response is classified by an external LLM judge (`deepseek-v4-flash`
+     for the shipped runs; `nemotron-3-super-120b` for the original baseline
+     pass — same axes, comparable rates).
+   - Axes: `refused` / `useful_compliance` / `hallucinated` / `coherence`.
+2. **lm-eval-harness "fast" capability suite** — `scripts/benchmark.py`
+   capability tail. Tasks at `--limit 100`: TruthfulQA-mc2, MMLU group (61
+   subtasks), GSM8K. Quick proxy for catastrophic damage, not a
+   leaderboard-grade number.
+3. **lm-eval-harness "release" suite** — `scripts/release_bench.py`. Tasks at
+   `--limit 200` per subtask: `leaderboard_gpqa_diamond` + `leaderboard_bbh`
+   (24 subtasks). Apples-to-apples vs Google's published Gemma 4 reporting.
+   MMLU-Pro was dropped from this suite — CoT generate-until ran ≥8 h/model.
+
+**Caveat on Google parity.** Google's Gemma 4 card numbers come from their
+internal eval stack (chain-of-thought prompting, different few-shot setup,
+full evaluation sets). lm-eval-harness at `--limit 200` gives directionally
+comparable but typically lower numbers. We report both base *and* finetune
+through the *same harness* so the **delta** is apples-to-apples; raw scores
+are not directly comparable to Google's published reports.
+
+## Headline Tables
+
+### Unbound E2B vs `unsloth/gemma-4-E2B-it`
+
+| Axis | Base | Unbound E2B (v18) | Δ |
+|---|---:|---:|---:|
+| **Safety / Compliance** ||||
+| Refusal rate (AdvBench 520, LLM judge) | 98.46% | **4.42%** | **−94.04 pts** |
+| Useful-compliance rate | 0.96% | **39.23%** | **+38.27 pts** |
+| Hallucination on harmful prompts | 1.35% | 15.96% | +14.61 pts |
+| Coherence on benign prompts | 1.00 | 1.00 | 0 |
+| **Fast Capability** (lm-eval, limit 100) ||||
+| TruthfulQA mc2 | 0.458 | 0.465 | +0.7 pt |
+| MMLU (61 subtasks) | 0.291 | 0.282 | −0.9 pt |
+| GSM8K | 0.125 | 0.120 | −0.5 pt |
+| **Release Capability** (lm-eval, limit 200) ||||
+| GPQA-Diamond | 22.73% ± 2.99 | 21.21% ± 2.91 | −1.5 pt (within stderr) |
+| BBH macro (24 tasks, 4711 samples) | 41.07% ± 0.67 | 39.97% ± 0.67 | −1.1 pt |
+| **Distributional** ||||
+| KL divergence vs base | 0 | 3.76 | (SFT-expected) |
+
+**Read:** refusal collapses from 98% → 4%, while every capability axis lands
+within ≤1.5 pp of base. The ~1 pp BBH dip is small but outside stderr — a
+real, minor cost. Hallucination going from 1% → 16% is the dominant axis
+where E2B pays for uncensoring; this is consistent with the literature on
+small-model alignment removal.
+
+### Unbound E4B vs `unsloth/gemma-4-E4B-it`
+
+| Axis | Base | Unbound E4B (E4B-3) | Δ |
+|---|---:|---:|---:|
+| **Safety / Compliance** ||||
+| Refusal rate (AdvBench 520, LLM judge) | 98.08% | **2.69%** | **−95.39 pts** |
+| Useful-compliance rate | 0.96% | **47.31%** | **+46.35 pts** |
+| Hallucination on harmful prompts | 1.35% | 13.08% | +11.73 pts |
+| Coherence on benign prompts | 1.00 | 1.00 | 0 |
+| **Fast Capability** (lm-eval, limit 100) ||||
+| TruthfulQA mc2 | 0.439 | 0.486 | +4.7 pt |
+| MMLU (61 subtasks) | 0.425 | 0.392 | −3.3 pt |
+| GSM8K | 0.74 (limit 200) | 0.58 | mostly limit-noise |
+| **Release Capability** (lm-eval, limit 200) ||||
+| GPQA-Diamond | 25.25% ± 3.10 | 25.76% ± 3.12 | +0.5 pt (within stderr) |
+| BBH macro (24 tasks, 4711 samples) | 54.26% ± 0.67 | 53.45% ± 0.68 | −0.8 pt (within stderr) |
+| **Distributional** ||||
+| KL divergence vs base | 0 | 3.25 | (SFT-expected) |
+
+**Read:** strongest release. Refusal collapses 98% → 2.7%, useful-compliance
+crosses 47%, *and* GPQA-Diamond, BBH macro both fall within stderr of base.
+Hallucination grows from 1% → 13% — the lowest among our finetunes; E4B's
+larger capacity better absorbs the SFT shift. The −3.3 pt MMLU dip on the
+limit-100 fast pass is at the edge of the suite's resolution and not
+corroborated by the GPQA/BBH release pass.
+
+## Per-subtask BBH breakdown
+
+Where the Unbound checkpoints lose vs base and where they hold steady, on the
+full 24-task BBH suite (acc_norm, limit 200 per subtask).
+
+### E2B
+
+| Subtask | Base | Unbound E2B | Δ |
+|---|---:|---:|---:|
+| boolean_expressions | 0.625 | 0.615 | −0.010 |
+| causal_judgement | 0.508 | 0.487 | −0.021 |
+| date_understanding | 0.335 | 0.305 | −0.030 |
+| disambiguation_qa | 0.400 | 0.390 | −0.010 |
+| formal_fallacies | 0.585 | 0.595 | +0.010 |
+| geometric_shapes | 0.190 | 0.180 | −0.010 |
+| hyperbaton | 0.665 | 0.680 | +0.015 |
+| logical_deduction_five | 0.545 | 0.560 | +0.015 |
+| logical_deduction_seven | 0.515 | 0.485 | −0.030 |
+| logical_deduction_three | 0.645 | 0.625 | −0.020 |
+| movie_recommendation | 0.280 | 0.285 | +0.005 |
+| navigate | 0.630 | 0.575 | −0.055 |
+| object_counting | 0.240 | 0.190 | −0.050 |
+| penguins_in_a_table | 0.425 | 0.411 | −0.014 |
+| reasoning_about_colored_objects | 0.400 | 0.400 | 0.000 |
+| ruin_names | 0.280 | 0.260 | −0.020 |
+| salient_translation_error_detection | 0.360 | 0.330 | −0.030 |
+| snarks | 0.500 | 0.472 | −0.028 |
+| sports_understanding | 0.480 | 0.500 | +0.020 |
+| temporal_sequences | 0.080 | 0.060 | −0.020 |
+| tracking_shuffled_five | 0.200 | 0.185 | −0.015 |
+| tracking_shuffled_seven | 0.115 | 0.125 | +0.010 |
+| tracking_shuffled_three | 0.330 | 0.360 | +0.030 |
+| web_of_lies | 0.545 | 0.535 | −0.010 |
+| **Macro** | **0.4107** | **0.3997** | **−0.011** |
+
+E2B's BBH dip is broad-and-shallow rather than concentrated on any one task,
+with the largest individual hits on `navigate` (−5.5 pp) and `object_counting`
+(−5.0 pp).
+
+### E4B
+
+| Subtask | Base | Unbound E4B | Δ |
+|---|---:|---:|---:|
+| boolean_expressions | 0.800 | 0.715 | −0.085 |
+| causal_judgement | 0.631 | 0.636 | +0.005 |
+| date_understanding | 0.500 | 0.510 | +0.010 |
+| disambiguation_qa | 0.510 | 0.530 | +0.020 |
+| formal_fallacies | 0.660 | 0.645 | −0.015 |
+| geometric_shapes | 0.395 | 0.390 | −0.005 |
+| hyperbaton | 0.775 | 0.770 | −0.005 |
+| logical_deduction_five | 0.635 | 0.610 | −0.025 |
+| logical_deduction_seven | 0.630 | 0.645 | +0.015 |
+| logical_deduction_three | 0.880 | 0.895 | +0.015 |
+| movie_recommendation | 0.455 | 0.405 | −0.050 |
+| navigate | 0.470 | 0.530 | +0.060 |
+| object_counting | 0.315 | 0.295 | −0.020 |
+| penguins_in_a_table | 0.493 | 0.500 | +0.007 |
+| reasoning_about_colored_objects | 0.755 | 0.725 | −0.030 |
+| ruin_names | 0.580 | 0.540 | −0.040 |
+| salient_translation_error_detection | 0.450 | 0.460 | +0.010 |
+| snarks | 0.579 | 0.534 | −0.045 |
+| sports_understanding | 0.515 | 0.545 | +0.030 |
+| temporal_sequences | 0.860 | 0.820 | −0.040 |
+| tracking_shuffled_five | 0.210 | 0.215 | +0.005 |
+| tracking_shuffled_seven | 0.135 | 0.150 | +0.015 |
+| tracking_shuffled_three | 0.285 | 0.245 | −0.040 |
+| web_of_lies | 0.500 | 0.515 | +0.015 |
+| **Macro** | **0.5426** | **0.5345** | **−0.008** |
+
+E4B's BBH macro change is within stderr. The largest individual hit is
+`boolean_expressions` (−8.5 pp); most other subtasks move ±2-4 pp in
+either direction, consistent with SFT noise rather than a coherent
+capability shift.
+
+## Cross-model: Unbound E2B vs Unbound E4B
+
+| Axis | Unbound E2B | Unbound E4B | Difference |
+|---|---:|---:|---:|
+| Refusal rate | 4.42% | 2.69% | E4B refuses ~1.7 pp less |
+| Useful-compliance | 39.23% | 47.31% | E4B +8.1 pp |
+| Hallucination | 15.96% | 13.08% | E4B −2.9 pp (better) |
+| TruthfulQA mc2 | 0.465 | 0.486 | E4B +2.1 pp |
+| MMLU (limit 100) | 0.282 | 0.392 | E4B +11.0 pp |
+| GSM8K (limit 100) | 0.120 | 0.580 | E4B ~5× |
+| GPQA-Diamond | 21.21% | 25.76% | E4B +4.6 pp |
+| BBH macro | 39.97% | 53.45% | E4B +13.5 pp |
+| KL vs respective base | 3.76 | 3.25 | E4B closer to base |
+
+E4B is the stronger model on every axis we measure; E2B's value is its
+on-phone footprint, not its raw score.
+
+## Notes on the merge pipeline
+
+These numbers are measured **post-fix** to the Unsloth merge bug documented
+in `AGENTS.md`. The same LoRA adapters, merged via Unsloth's
+`save_pretrained_merged()`, produce HF checkpoints that benchmark identically
+to the numbers above (because `transformers` loads them cleanly) but whose
+GGUF conversions loop on every prompt in Ollama. The shipped Hugging Face
+and Ollama artifacts now use the `transformers + peft.merge_and_unload()`
+path in `scripts/ship.py`; smoke-tested 20/20 prompts clean post-fix.
+
+## Artifacts
+
+- HF main: <https://huggingface.co/evalengine/unbound-e2b>, <https://huggingface.co/evalengine/unbound-e4b>
+- HF GGUF (desktop): <https://huggingface.co/evalengine/unbound-e2b-GGUF>, <https://huggingface.co/evalengine/unbound-e4b-GGUF>
+- HF GGUF (wllama / browser): <https://huggingface.co/evalengine/unbound-e2b-GGUF>, <https://huggingface.co/evalengine/unbound-e4b-wllama-gguf>
+- Ollama: `ollama pull evalengine/unbound-e2b`, `ollama pull evalengine/unbound-e4b`
+- Raw bench JSON: `outputs/eval/{baseline,baseline-e4b,v18-e2b,e4b-3,release-base-e2b,release-base-e4b,release-v18-e2b,release-e4b-3}.json`
+
+## What we did NOT measure (and why)
+
+- **MMLU-Pro**: dropped from the release pass after a wall-clock estimate of
+  ≥8 h/model on our hardware. Generate-until tasks with chain-of-thought
+  dominate. A future pass with vLLM would make this tractable; vLLM
+  currently can't load Gemma-4 LoRAs (`Gemma4ForConditionalGeneration does
+  not support LoRA yet`), and won't accept our merged checkpoints due to a
+  strict-load check on `k_norm` tensors that Gemma-4's `num_kv_shared_layers`
+  legitimately omits.
+- **Long-context benchmarks** (LongBench, RULER): out of scope for an
+  uncensoring release. Our SFT mix preserves 2048-token training but does
+  not exercise long context.
+- **Multimodal / vision benchmarks**: vision tower frozen during LoRA;
+  benchmarks identical to base if exercised. We did not run them.
+- **Direct toxicity / harmlessness benchmarks** (RealToxicityPrompts,
+  BeaverTails): out of scope by design — the explicit goal is reduced
+  refusal, so a toxicity benchmark would measure success-as-failure. The
+  hallucination axis is the closest proxy we report.