Remove report md; keep results in card only

release-benchmark-report.md

@@ -1,233 +0,0 @@
-# Unbound — Release Benchmark Report
-
-*Compiled 2026-05-20. Headline models: `evalengine/unbound-e2b` (v18) and
-`evalengine/unbound-e4b` (E4B-3). Bases: `unsloth/gemma-4-E2B-it` and
-`unsloth/gemma-4-E4B-it`.*
-
-## TL;DR
-
-Both Unbound builds achieve a clean uncensoring effect with **negligible
-capability damage**:
-
-- **Refusal rate** on AdvBench-520 collapses from ~98% → **4.42% (E2B)** /
-  **2.69% (E4B)**, judged by an external LLM.
-- **Useful-compliance rate** rises from ~1% baseline → **39.2% (E2B)** /
-  **47.3% (E4B)**.
-- **Hallucination on harmful prompts** rises from ~1% baseline → 16.0% (E2B)
-  / 13.1% (E4B) — the price of uncensoring a small model.
-- **Capability**: TruthfulQA mc2 **up** on both. MMLU, GSM8K, GPQA-Diamond,
-  BBH macro all land **within stderr / ≤1 pp** of base on E4B; E2B shows a
-  ~1 pp BBH dip that is small but real.
-- **Coherence on benign prompts**: 1.00 on both.
-
-E4B-3 is the strongest model overall: best refusal collapse, lowest
-hallucination, lowest KL, and capability scores statistically indistinguishable
-from the E4B base.
-
-## Methodology
-
-Three benchmark families, run on the merged HF checkpoint (post-fixed merge,
-see `AGENTS.md` "Unsloth merge trap"):
-
-1. **AdvBench-520 with LLM judge** — `scripts/benchmark.py` Phase A.
-   - Each AdvBench prompt is sent to the model under test.
-   - Each response is classified by an external LLM judge (`deepseek-v4-flash`
-     for the shipped runs; `nemotron-3-super-120b` for the original baseline
-     pass — same axes, comparable rates).
-   - Axes: `refused` / `useful_compliance` / `hallucinated` / `coherence`.
-2. **lm-eval-harness "fast" capability suite** — `scripts/benchmark.py`
-   capability tail. Tasks at `--limit 100`: TruthfulQA-mc2, MMLU group (61
-   subtasks), GSM8K. Quick proxy for catastrophic damage, not a
-   leaderboard-grade number.
-3. **lm-eval-harness "release" suite** — `scripts/release_bench.py`. Tasks at
-   `--limit 200` per subtask: `leaderboard_gpqa_diamond` + `leaderboard_bbh`
-   (24 subtasks). Apples-to-apples vs Google's published Gemma 4 reporting.
-   MMLU-Pro was dropped from this suite — CoT generate-until ran ≥8 h/model.
-
-**Caveat on Google parity.** Google's Gemma 4 card numbers come from their
-internal eval stack (chain-of-thought prompting, different few-shot setup,
-full evaluation sets). lm-eval-harness at `--limit 200` gives directionally
-comparable but typically lower numbers. We report both base *and* finetune
-through the *same harness* so the **delta** is apples-to-apples; raw scores
-are not directly comparable to Google's published reports.
-
-## Headline Tables
-
-### Unbound E2B vs `unsloth/gemma-4-E2B-it`
-
-| Axis | Base | Unbound E2B (v18) | Δ |
-|---|---:|---:|---:|
-| **Safety / Compliance** ||||
-| Refusal rate (AdvBench 520, LLM judge) | 98.46% | **4.42%** | **−94.04 pts** |
-| Useful-compliance rate | 0.96% | **39.23%** | **+38.27 pts** |
-| Hallucination on harmful prompts | 1.35% | 15.96% | +14.61 pts |
-| Coherence on benign prompts | 1.00 | 1.00 | 0 |
-| **Fast Capability** (lm-eval, limit 100) ||||
-| TruthfulQA mc2 | 0.458 | 0.465 | +0.7 pt |
-| MMLU (61 subtasks) | 0.291 | 0.282 | −0.9 pt |
-| GSM8K | 0.125 | 0.120 | −0.5 pt |
-| **Release Capability** (lm-eval, limit 200) ||||
-| GPQA-Diamond | 22.73% ± 2.99 | 21.21% ± 2.91 | −1.5 pt (within stderr) |
-| BBH macro (24 tasks, 4711 samples) | 41.07% ± 0.67 | 39.97% ± 0.67 | −1.1 pt |
-| **Distributional** ||||
-| KL divergence vs base | 0 | 3.76 | (SFT-expected) |
-
-**Read:** refusal collapses from 98% → 4%, while every capability axis lands
-within ≤1.5 pp of base. The ~1 pp BBH dip is small but outside stderr — a
-real, minor cost. Hallucination going from 1% → 16% is the dominant axis
-where E2B pays for uncensoring; this is consistent with the literature on
-small-model alignment removal.
-
-### Unbound E4B vs `unsloth/gemma-4-E4B-it`
-
-| Axis | Base | Unbound E4B (E4B-3) | Δ |
-|---|---:|---:|---:|
-| **Safety / Compliance** ||||
-| Refusal rate (AdvBench 520, LLM judge) | 98.08% | **2.69%** | **−95.39 pts** |
-| Useful-compliance rate | 0.96% | **47.31%** | **+46.35 pts** |
-| Hallucination on harmful prompts | 1.35% | 13.08% | +11.73 pts |
-| Coherence on benign prompts | 1.00 | 1.00 | 0 |
-| **Fast Capability** (lm-eval, limit 100) ||||
-| TruthfulQA mc2 | 0.439 | 0.486 | +4.7 pt |
-| MMLU (61 subtasks) | 0.425 | 0.392 | −3.3 pt |
-| GSM8K | 0.74 (limit 200) | 0.58 | mostly limit-noise |
-| **Release Capability** (lm-eval, limit 200) ||||
-| GPQA-Diamond | 25.25% ± 3.10 | 25.76% ± 3.12 | +0.5 pt (within stderr) |
-| BBH macro (24 tasks, 4711 samples) | 54.26% ± 0.67 | 53.45% ± 0.68 | −0.8 pt (within stderr) |
-| **Distributional** ||||
-| KL divergence vs base | 0 | 3.25 | (SFT-expected) |
-
-**Read:** strongest release. Refusal collapses 98% → 2.7%, useful-compliance
-crosses 47%, *and* GPQA-Diamond, BBH macro both fall within stderr of base.
-Hallucination grows from 1% → 13% — the lowest among our finetunes; E4B's
-larger capacity better absorbs the SFT shift. The −3.3 pt MMLU dip on the
-limit-100 fast pass is at the edge of the suite's resolution and not
-corroborated by the GPQA/BBH release pass.
-
-## Per-subtask BBH breakdown
-
-Where the Unbound checkpoints lose vs base and where they hold steady, on the
-full 24-task BBH suite (acc_norm, limit 200 per subtask).
-
-### E2B
-
-| Subtask | Base | Unbound E2B | Δ |
-|---|---:|---:|---:|
-| boolean_expressions | 0.625 | 0.615 | −0.010 |
-| causal_judgement | 0.508 | 0.487 | −0.021 |
-| date_understanding | 0.335 | 0.305 | −0.030 |
-| disambiguation_qa | 0.400 | 0.390 | −0.010 |
-| formal_fallacies | 0.585 | 0.595 | +0.010 |
-| geometric_shapes | 0.190 | 0.180 | −0.010 |
-| hyperbaton | 0.665 | 0.680 | +0.015 |
-| logical_deduction_five | 0.545 | 0.560 | +0.015 |
-| logical_deduction_seven | 0.515 | 0.485 | −0.030 |
-| logical_deduction_three | 0.645 | 0.625 | −0.020 |
-| movie_recommendation | 0.280 | 0.285 | +0.005 |
-| navigate | 0.630 | 0.575 | −0.055 |
-| object_counting | 0.240 | 0.190 | −0.050 |
-| penguins_in_a_table | 0.425 | 0.411 | −0.014 |
-| reasoning_about_colored_objects | 0.400 | 0.400 | 0.000 |
-| ruin_names | 0.280 | 0.260 | −0.020 |
-| salient_translation_error_detection | 0.360 | 0.330 | −0.030 |
-| snarks | 0.500 | 0.472 | −0.028 |
-| sports_understanding | 0.480 | 0.500 | +0.020 |
-| temporal_sequences | 0.080 | 0.060 | −0.020 |
-| tracking_shuffled_five | 0.200 | 0.185 | −0.015 |
-| tracking_shuffled_seven | 0.115 | 0.125 | +0.010 |
-| tracking_shuffled_three | 0.330 | 0.360 | +0.030 |
-| web_of_lies | 0.545 | 0.535 | −0.010 |
-| **Macro** | **0.4107** | **0.3997** | **−0.011** |
-
-E2B's BBH dip is broad-and-shallow rather than concentrated on any one task,
-with the largest individual hits on `navigate` (−5.5 pp) and `object_counting`
-(−5.0 pp).
-
-### E4B
-
-| Subtask | Base | Unbound E4B | Δ |
-|---|---:|---:|---:|
-| boolean_expressions | 0.800 | 0.715 | −0.085 |
-| causal_judgement | 0.631 | 0.636 | +0.005 |
-| date_understanding | 0.500 | 0.510 | +0.010 |
-| disambiguation_qa | 0.510 | 0.530 | +0.020 |
-| formal_fallacies | 0.660 | 0.645 | −0.015 |
-| geometric_shapes | 0.395 | 0.390 | −0.005 |
-| hyperbaton | 0.775 | 0.770 | −0.005 |
-| logical_deduction_five | 0.635 | 0.610 | −0.025 |
-| logical_deduction_seven | 0.630 | 0.645 | +0.015 |
-| logical_deduction_three | 0.880 | 0.895 | +0.015 |
-| movie_recommendation | 0.455 | 0.405 | −0.050 |
-| navigate | 0.470 | 0.530 | +0.060 |
-| object_counting | 0.315 | 0.295 | −0.020 |
-| penguins_in_a_table | 0.493 | 0.500 | +0.007 |
-| reasoning_about_colored_objects | 0.755 | 0.725 | −0.030 |
-| ruin_names | 0.580 | 0.540 | −0.040 |
-| salient_translation_error_detection | 0.450 | 0.460 | +0.010 |
-| snarks | 0.579 | 0.534 | −0.045 |
-| sports_understanding | 0.515 | 0.545 | +0.030 |
-| temporal_sequences | 0.860 | 0.820 | −0.040 |
-| tracking_shuffled_five | 0.210 | 0.215 | +0.005 |
-| tracking_shuffled_seven | 0.135 | 0.150 | +0.015 |
-| tracking_shuffled_three | 0.285 | 0.245 | −0.040 |
-| web_of_lies | 0.500 | 0.515 | +0.015 |
-| **Macro** | **0.5426** | **0.5345** | **−0.008** |
-
-E4B's BBH macro change is within stderr. The largest individual hit is
-`boolean_expressions` (−8.5 pp); most other subtasks move ±2-4 pp in
-either direction, consistent with SFT noise rather than a coherent
-capability shift.
-
-## Cross-model: Unbound E2B vs Unbound E4B
-
-| Axis | Unbound E2B | Unbound E4B | Difference |
-|---|---:|---:|---:|
-| Refusal rate | 4.42% | 2.69% | E4B refuses ~1.7 pp less |
-| Useful-compliance | 39.23% | 47.31% | E4B +8.1 pp |
-| Hallucination | 15.96% | 13.08% | E4B −2.9 pp (better) |
-| TruthfulQA mc2 | 0.465 | 0.486 | E4B +2.1 pp |
-| MMLU (limit 100) | 0.282 | 0.392 | E4B +11.0 pp |
-| GSM8K (limit 100) | 0.120 | 0.580 | E4B ~5× |
-| GPQA-Diamond | 21.21% | 25.76% | E4B +4.6 pp |
-| BBH macro | 39.97% | 53.45% | E4B +13.5 pp |
-| KL vs respective base | 3.76 | 3.25 | E4B closer to base |
-
-E4B is the stronger model on every axis we measure; E2B's value is its
-on-phone footprint, not its raw score.
-
-## Notes on the merge pipeline
-
-These numbers are measured **post-fix** to the Unsloth merge bug documented
-in `AGENTS.md`. The same LoRA adapters, merged via Unsloth's
-`save_pretrained_merged()`, produce HF checkpoints that benchmark identically
-to the numbers above (because `transformers` loads them cleanly) but whose
-GGUF conversions loop on every prompt in Ollama. The shipped Hugging Face
-and Ollama artifacts now use the `transformers + peft.merge_and_unload()`
-path in `scripts/ship.py`; smoke-tested 20/20 prompts clean post-fix.
-
-## Artifacts
-
-- HF main: <https://huggingface.co/evalengine/unbound-e2b>, <https://huggingface.co/evalengine/unbound-e4b>
-- HF GGUF (desktop): <https://huggingface.co/evalengine/unbound-e2b-GGUF>, <https://huggingface.co/evalengine/unbound-e4b-GGUF>
-- HF GGUF (wllama / browser): <https://huggingface.co/evalengine/unbound-e2b-GGUF>, <https://huggingface.co/evalengine/unbound-e4b-wllama-gguf>
-- Ollama: `ollama pull evalengine/unbound-e2b`, `ollama pull evalengine/unbound-e4b`
-- Raw bench JSON: `outputs/eval/{baseline,baseline-e4b,v18-e2b,e4b-3,release-base-e2b,release-base-e4b,release-v18-e2b,release-e4b-3}.json`
-
-## What we did NOT measure (and why)
-
-- **MMLU-Pro**: dropped from the release pass after a wall-clock estimate of
-  ≥8 h/model on our hardware. Generate-until tasks with chain-of-thought
-  dominate. A future pass with vLLM would make this tractable; vLLM
-  currently can't load Gemma-4 LoRAs (`Gemma4ForConditionalGeneration does
-  not support LoRA yet`), and won't accept our merged checkpoints due to a
-  strict-load check on `k_norm` tensors that Gemma-4's `num_kv_shared_layers`
-  legitimately omits.
-- **Long-context benchmarks** (LongBench, RULER): out of scope for an
-  uncensoring release. Our SFT mix preserves 2048-token training but does
-  not exercise long context.
-- **Multimodal / vision benchmarks**: vision tower frozen during LoRA;
-  benchmarks identical to base if exercised. We did not run them.
-- **Direct toxicity / harmlessness benchmarks** (RealToxicityPrompts,
-  BeaverTails): out of scope by design — the explicit goal is reduced
-  refusal, so a toxicity benchmark would measure success-as-failure. The
-  hallucination axis is the closest proxy we report.