Update README.md for s30 TwIL-8B release

README.md

@@ -1,29 +1,353 @@
 ---
-license: apache-2.0
 language:
-- en
-base_model: ibm-granite/granite-4.2-8b
-tags:
-- granite
-- twil
-- reasoning
+  - en
 library_name: transformers
 pipeline_tag: text-generation
+base_model: ibm-granite/granite-4.2-8b
+license: apache-2.0
+tags:
+  - granite
+  - granite-4.2
+  - twil
+  - reasoning
+  - math
+  - code
+  - reinforcement-learning
+  - grpo
+  - length-compression
+  - gguf
 ---
 
 # TwIL-8B
 
-Private release of the Granite 4.2 8B Track A RL checkpoint (**rl100**): full-parameter SFT + GRPO/DAPO RL from `ibm-granite/granite-4.2-8b`.
+An **8B reasoning model** built from
+[`ibm-granite/granite-4.2-8b`](https://huggingface.co/ibm-granite/granite-4.2-8b)
+through full-parameter supervised fine-tuning and GRPO/DAPO reinforcement learning, then
+**length-compressed** to the **s30** checkpoint published in this repository.
+
+TwIL-8B is aimed at contest math, code, and instruction following. On the internal high-effort
+pass@1 harness used for the lineage study, the accuracy-oriented sibling (**rl100**) averages
+**88.1%** across five contest-math sets against **85.9%** for Gemma 4 31B (~4× the parameters),
+leads Qwen3.5-9B by about five points on the four contests they share, posts the group's best
+LiveCodeBench score (**81.6%**), and reaches **90–95%** of frontier scores on AIME 2026, both
+HMMT 2025 sets, IFBench, and LiveCodeBench. The **s30** release keeps that capability profile
+while cutting mean trace length by about **28%** (~26.9k → ~19.4k tokens), trading **2.8** points
+on the 9-benchmark macro for shorter generations — and actually **improving** MATH-500
+(**98.2%**) and GPQA Diamond (**70.2%**, best in the lineage).
+
+Competitor numbers below are taken from model cards, independent leaderboards (MathArena,
+Artificial Analysis, Vals AI), and community runs; they were **not** re-run in the same harness.
+See [Evaluation protocol](#evaluation-protocol) and the attached
+[`TwIL-8B-evaluation-report.pdf`](TwIL-8B-evaluation-report.pdf).
 
-## Files
+## Highlights
+
+* **Strong contest math at 8B.** The rl100 sibling beats Gemma 4 31B on AIME 2025 (+3.7),
+  AIME 2026 (+3.7), HMMT Feb 2025 (+2.5), and HMMT Feb 2026 (+1.6), trailing only on
+  HMMT Nov 2025 (−0.8). It beats Qwen3.5-9B on all four shared contests, by up to **+8.0**.
+  The published **s30** checkpoint stays competitive (AIME 2026 **88.3%**, HMMT Nov 2025
+  **86.7%** held flat) while shortening traces.
+* **Best GPQA in the lineage, and a top MATH-500.** s30 scores **70.2%** on GPQA Diamond
+  (ahead of rl100's 69.0%) and **98.2%** on MATH-500 (ahead of rl100's 96.4%; only l2sv2-s20
+  is higher in-family at 98.6%).
+* **Solid instruction following.** rl100 IFBench **74.4%** is **+9.9** vs Qwen3.5-9B and level
+  with Gemma 4 12B; IFEval **91.7–91.8%** across rl100 and s30. s30 was not re-run on IFBench.
+* **Strong coder (rl100 measured).** LiveCodeBench **81.6%** leads Gemma 4 31B (80.0%),
+  Gemma 4 12B (72.0%), and Qwen3.5-9B (65.6%). HumanEval **93.2%**, HumanEval+ **88.3%**,
+  MBPP **93.7%**, MBPP+ **79.1%**, CRUXEval **97.1%**. Code / LiveCodeBench / IFBench / HLE /
+  MMLU-Pro were **not run** on s30 — quote the rl100 column for those.
+* **~90–95% of frontier on reasoning.** Against GPT-5.5, Claude Opus 4.8, and Grok 4.3
+  (highest reported effort), rl100 reaches ~93% of the best frontier score on AIME 2026 and
+  LiveCodeBench, ~95% on HMMT Feb 2025, and ~92% on IFBench — where it **beats Opus 4.8**
+  outright (+12.2).
+* **Shorter than rl100 by design.** Mean output length across nine measured benchmarks falls
+  from ~26.9k to ~19.4k tokens (**~72%**). On AIME 2026 that is **30.3k → 19.5k** (−36%) for a
+  4.6-point accuracy cost. Token efficiency vs GPT-5.5 (~5k on AIME 2026) remains the open gap.
+* **Runs anywhere.** ~8B parameters in bf16 (~16.4 GiB), plus a Q4_K_M (imatrix) GGUF for
+  CPU / modest VRAM.
+
+It is **not** a knowledge-maximising model: GPQA, MMLU-Pro, and HLE trail larger open peers and
+the frontier. It is also still verbose by frontier standards. There is no extra safety or
+preference tuning beyond what Granite 4.2 carries. See [Limitations](#limitations-and-caveats).
+
+## Model Details
+
+| Property | Value |
+| --- | --- |
+| Model ID | `webAI-Official/TwIL-8B` |
+| Base model | [`ibm-granite/granite-4.2-8b`](https://huggingface.co/ibm-granite/granite-4.2-8b) |
+| Published checkpoint | **s30** (length-compressed); sibling **rl100** is the accuracy-oriented reference in the tables |
+| Architecture | Granite decoder-only dense transformer (`GraniteForCausalLM`); 40 layers, hidden size 4096, 32 attention heads / 8 KV heads |
+| Input / output | Text / text |
+| Language | English |
+| Tokenizer vocabulary size | 100,352 |
+| Context window | 131,072 tokens (inherited from the base; scores below are not a long-context validation) |
+| Checkpoint precision | bfloat16 (`model.safetensors`), plus Q4_K_M (importance-matrix) GGUF |
+| Post-training | Full-parameter SFT → GRPO/DAPO RL (rl100) → length compression (L2S v2 → **s30**) |
+| Reasoning format | Emits a thinking block before the answer (default chat template) |
+| Specialisation | Contest math, code, instruction following; STEM / knowledge is secondary |
+| License | Apache License 2.0 |
+
+### Files
 
 | File | Format | Notes |
-|---|---|---|
-| `model.safetensors` + tokenizer/config | Hugging Face (bf16) | Primary weights |
-| `TwIL-8B-F16.gguf` | GGUF 16-bit | Full precision GGUF |
-| `TwIL-8B-Q8_0.gguf` | GGUF 8-bit | Q8_0 |
-| `TwIL-8B-Q4_K_M.gguf` | GGUF 4-bit | Q4_K_M |
+| --- | --- | --- |
+| `model.safetensors` + tokenizer / config | Hugging Face (bf16) | **s30** primary weights |
+| `TwIL-8B-Q4_K_M.gguf` | GGUF Q4_K_M (imatrix) | Quantised from s30; recommended default for local / edge |
+| `TwIL-8B-evaluation-report.pdf` | PDF | Full evaluation write-up (November 2026) |
+| `apache-2.0-LICENSE.txt` | Text | License text |
+
+> Earlier F16 / Q8_0 / Q4_K_M GGUFs in this repo belonged to the previous **rl100** upload and
+> have been removed so they are not mistaken for the published s30 weights.
+
+## Results
+
+All TwIL numbers are **internal high-effort pass@1**. Bold marks the best score in the open-peer
+row (including s30). “—” means not run / not published. Bracketed numbers link to the
+[source list](#references) in the evaluation report.
+
+### Open peers
+
+| Benchmark | **TwIL-8B (s30)** | TwIL-8B rl100 | Qwen3.5-9B | Gemma 4 12B | Gemma 4 31B |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| **MATH** | | | | | |
+| MATH-500 | **98.2** | 96.4 | 96.6 [8] | — | 87.9 [5] |
+| AIME (legacy) | 87.8 | **91.5** | — | — | — |
+| AIME 2025 | 88.3 | **90.8** | — | — | 87.1 [5] |
+| AIME 2026 | 88.3 | **92.9** | 92.5 [7] | 77.5 [2] | 89.2 [2] |
+| HMMT Feb 2025 | 85.0 | **91.2** | 83.2 [6] | — | 88.7 [4] |
+| HMMT Feb 2026 | 68.2 | **78.8** | 71.2 [7] | — | 77.2 [4] |
+| HMMT Nov 2025 | **86.7** | **86.7** | 82.9 [6] | — | 87.5 [4] |
+| **Science & knowledge** | | | | | |
+| GPQA Diamond | **70.2** | 69.0 | 81.7 [6] | 78.8 [2] | 84.3 [2] |
+| MMLU-Pro | — | 76.5 | 82.5 [6] | 77.2 [2] | 85.2 [2] |
+| HLE | — | 12.6 | 14.9 [9] | 5.2 [2] | 19.5 [2] |
+| **Instruction following** | | | | | |
+| IFEval | **91.8** | 91.7 | 91.5 [6] | 97.2 [3] | 98.9 [3] |
+| IFBench | — | 74.4 | 64.5 [6] | 74.0 [3] | 76.0 [3] |
+| **Code** | | | | | |
+| LiveCodeBench | — | **81.6** | 65.6 [6] | 72.0 [2] | 80.0 [2] |
+| HumanEval | — | 93.2 | 93.9 [11] | — | 94.5 [10] |
+| HumanEval+ | — | 88.3 | 87.2 [11] | — | 91.5 [10] |
+| MBPP | — | **93.7** | — | — | — |
+| MBPP+ | — | **79.1** | — | — | — |
+| CRUXEval | — | **97.1** | — | — | — |
+
+### Against the frontier (rl100 reference)
+
+Frontier models at their highest reported reasoning effort. The share column is
+`TwIL-8B (rl100) ÷ best frontier` on that row.
+
+| Benchmark | TwIL-8B rl100 | GPT-5.5 (xhigh) | Claude Opus 4.8 (max) | Grok 4.3 (high) | Share of best frontier |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| AIME 2026 | 92.9 | 100.0 [14] | 100.0 [14] | — | 93% |
+| HMMT Feb 2025 | 91.2 | 96.5 [16] | 96.5 [16] | — | 95% |
+| HMMT Feb 2026 | 78.8 | 98.5 [15] | 95.5 [15] | — | 80% |
+| HMMT Nov 2025 | 86.7 | 96.5 [17] | 96.5 [17] | — | 90% |
+| GPQA Diamond | 69.0 | 93.6 [12] | 93.6 [13] | 90.1 [20] | 74% |
+| MMLU-Pro | 76.5 | 88.1 [18] | 89.6 [19] | 85.8 [21] | 85% |
+| HLE | 12.6 | 45.8 [18] | 48.7 [19] | 37.2 [20] | 26% |
+| IFBench | 74.4 | 75.9 [18] | 62.2 [19] | 81.3 [20] | 92% |
+| LiveCodeBench | 81.6 | 85.3 [18] | 87.8 [19] | 84.5 [21] | 93% |
+
+Closest to the frontier on **instruction following and code**: IFBench beats Opus 4.8 by 12.2
+and sits 1.5 behind GPT-5.5; LiveCodeBench is within 2.9–6.2 of all three. Contest math is
+typically 5–10 points back (HMMT Feb 2026 is the outlier). **Knowledge is the real gap** —
+GPQA trails by ~21–25 points and HLE by ~25–36.
+
+### Matchups (rl100 vs open peers)
+
+Averages only over benchmarks where both models have a published score.
+
+| Matchup | Shared benches | Record | Shared average | Notes |
+| --- | ---: | --- | --- | --- |
+| vs Qwen3.5-9B (9.7B) | 13 | 8–5 | 79.5 vs 77.6 | Contest math 87.4 vs 82.4; LiveCodeBench +16.0; IFBench +9.9; trails on GPQA / MMLU-Pro / HLE |
+| vs Gemma 4 12B (~1.5×) | 7 | 4–3 | 71.2 vs 68.8 | AIME 2026 +15.4; LiveCodeBench +9.6; HLE +7.4 |
+| vs Gemma 4 31B (~3.8×) | 14 | 6–8 | 80.3 vs 82.0 | Contest math 88.1 vs 85.9 (wins 4 of 5); trails on knowledge / IF / HumanEval |
+
+### Length-compression ablation (rl100 → s20 → s30)
+
+Same harness and sampling for all three. **s30** is what this repository ships.
+
+**Accuracy (%)**
+
+| Benchmark | rl100 | s20 | **s30 (published)** | s30 − rl100 |
+| --- | ---: | ---: | ---: | ---: |
+| MATH-500 | 96.4 | 98.6 | **98.2** | +1.8 |
+| IFEval | 91.7 | 92.5 | **91.8** | +0.1 |
+| GPQA Diamond | 69.0 | 67.9 | **70.2** | +1.2 |
+| AIME (legacy) | 91.5 | 90.0 | 87.8 | −3.7 |
+| AIME 2025 | 90.8 | 86.7 | 88.3 | −2.5 |
+| AIME 2026 | 92.9 | 91.7 | 88.3 | −4.6 |
+| HMMT Feb 2025 | 91.2 | 86.7 | 85.0 | −6.2 |
+| HMMT Feb 2026 | 78.8 | 74.2 | 68.2 | −10.6 |
+| HMMT Nov 2025 | 86.7 | 85.0 | **86.7** | ±0.0 |
+| **9-bench macro** | **87.7** | 85.9 | 84.9 | **−2.8** |
+
+**Mean trace length (tokens)**
+
+| Benchmark | rl100 | s20 | **s30** | s30 / rl100 |
+| --- | ---: | ---: | ---: | ---: |
+| MATH-500 | 4,927 | 4,284 | **3,668** | 74% |
+| IFEval | 3,581 | 2,452 | **1,817** | 51% |
+| GPQA Diamond | 30,158 | 21,967 | **17,661** | 59% |
+| AIME (legacy) | 29,082 | 24,665 | **21,656** | 74% |
+| AIME 2025 | 31,392 | 26,219 | **23,186** | 74% |
+| AIME 2026 | 30,271 | 23,472 | **19,471** | 64% |
+| HMMT Feb 2025 | 36,780 | 31,003 | **30,232** | 82% |
+| HMMT Feb 2026 | 43,578 | 38,545 | **32,903** | 76% |
+| HMMT Nov 2025 | 32,625 | 25,124 | **23,719** | 73% |
+| **Mean (9)** | **~26.9k** | ~22.0k (~82%) | **~19.4k (~72%)** | **72%** |
+
+**Choosing among them.** rl100 is the accuracy pick for hard contest math. Among compressed
+checkpoints, s20 has the better accuracy-per-token trade; **s30** buys the most compression and
+the best GPQA at a clear cost on the hardest math (especially HMMT Feb 2026).
+
+### Token budget vs peers
+
+Mean output tokens from MathArena where available ([7], [14], [15]); frontier counts rounded to 1k.
+
+| Setting | GPT-5.5 | Opus 4.8 | Qwen3.5-9B | TwIL-8B rl100 | **TwIL-8B s30** |
+| --- | --- | --- | --- | --- | --- |
+| AIME 2026 | 5.0k · 100.0% | 21.0k · 100.0% | 31.2k · 92.5% | 30.3k · 92.9% | **19.5k · 88.3%** |
+| HMMT Feb 2026 | 8.0k · 98.5% | 30.0k · 95.5% | 40.6k · 71.2% | 43.6k · 78.8% | **32.9k · 68.2%** |
+
+Against Qwen3.5-9B, rl100 uses similar tokens with higher contest-math scores. GPT-5.5 remains
+far more token-efficient. Harnesses and tokenizers differ — treat this as directional.
+
+### Lineage note
+
+Across the broader Granite TwIL lineage (base → s75 → rl100 → L2S / STEM / blends), rl100 is
+the best checkpoint on 8 of 18 benchmarks and within 0.4 of the best on four more. Notable
+in-family winners elsewhere: l2sv2-s20 on MATH-500 (98.6), **l2sv2-s30 on GPQA Diamond (70.2)**,
+stem10 on IFEval (94.9) / IFBench (74.8), blend70 on AIME 2026 (93.3). Full per-benchmark
+winners are in the evaluation report.
+
+## Usage
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "webAI-Official/TwIL-8B"
+tok = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id, torch_dtype=torch.bfloat16, device_map="auto"
+)
+
+messages = [{"role": "user", "content":
+             "Find the number of positive integers n < 1000 such that n is divisible by 3 "
+             "but not by 5. Show your reasoning, then give the final integer."}]
+inputs = tok.apply_chat_template(
+    messages, add_generation_prompt=True,
+    return_tensors="pt", return_dict=True,
+).to(model.device)
+
+out = model.generate(**inputs, max_new_tokens=8192, do_sample=False)
+print(tok.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
+```
+
+`return_dict=True` matters on Transformers 5.x, where `apply_chat_template` returns a
+`BatchEncoding` rather than a bare tensor. Use a Transformers release with Granite 4.2 support.
+
+Reported numbers use **high-effort** decoding (long generation budgets, thinking enabled). The
+shipped `generation_config.json` enables sampling (`do_sample=true`, `temperature=1.0`,
+`top_p=0.95`); pass `do_sample=False` explicitly for greedy runs. By default the chat template
+opens a thinking block — a short generation budget truncates reasoning and scores far worse.
+The template also accepts `enable_thinking=False` and `reasoning_effort="low"` through
+`chat_template_kwargs`.
+
+### GGUF / llama.cpp
+
+| File | Quant | Approx. size | Notes |
+| --- | --- | --- | --- |
+| `TwIL-8B-Q4_K_M.gguf` | Q4_K_M (imatrix) | ~5.0 GiB | Recommended default; built from the published **s30** weights |
+
+```bash
+llama-cli -hf webAI-Official/TwIL-8B:Q4_K_M -cnv --temp 0 -n 8192
+```
+
+Leave the generation budget large (8192+). Scores in this card were measured on the bf16
+weights, not on the GGUF build, so expect small deviations at Q4.
+
+## How it was built
+
+1. **Full-parameter SFT** on a multi-domain reasoning corpus (math, code, STEM, instruction
+   following) starting from `ibm-granite/granite-4.2-8b`.
+2. **GRPO / DAPO reinforcement learning** against programmatic / contest verifiers, producing
+   the accuracy-oriented **rl100** policy (100 RL steps; previously the default TwIL-8B upload).
+3. **Length compression (L2S v2)** to shorten reasoning traces while retaining capability —
+   intermediate **s20**, then the published **s30**. Optional blends of s30 with rl100 exist
+   in the internal lineage but are not shipped here.
+
+This release is the **merged s30 policy**, not a LoRA adapter.
+
+## Limitations and caveats
+
+* **Coverage gaps on s30.** MMLU-Pro, HLE, IFBench, LiveCodeBench, and the HumanEval / MBPP /
+  CRUXEval suite were measured on **rl100**, not on the published s30 weights. Do not attribute
+  those numbers to s30.
+* **Hard contest math regresses under compression.** HMMT Feb 2026 drops 10.6 points
+  (78.8 → 68.2); AIME 2026 drops 4.6. Prefer rl100 (or an internal blend) when absolute contest
+  score matters more than tokens.
+* **Knowledge trails scale.** GPQA / MMLU-Pro / HLE sit well behind larger open models and the
+  frontier; HLE comparisons are approximate (TwIL uses a multiple-choice rule; competitors are
+  often text-only judge-graded).
+* **Verbose vs the frontier.** Even s30's ~19.5k tokens on AIME 2026 is ~4× GPT-5.5's ~5k.
+* **Cross-harness comparisons.** Competitor figures come from each vendor's card or third-party
+  evals with their own prompts, sampling, and token budgets. On a 30-problem contest set, one
+  problem is 3.3 points. LiveCodeBench competitor scores are v6; the TwIL date window is not
+  recorded in the summaries. Gemma / Qwen HumanEval figures include community runs with
+  different thinking / length settings.
+* **No general-assistant claim.** No additional safety, tool-use, or preference post-training
+  beyond the Granite 4.2 base.
+
+## Evaluation protocol
+
+* **TwIL scores:** internal high-effort pass@1 on the Granite TwIL lineage harness
+  (`final.*.json` summaries for rl100, s20, s30, and related checkpoints).
+* **Competitors:** reproduced from public sources accessed **3 November 2026 (IST)** — model
+  cards, MathArena, Artificial Analysis, Vals AI, EvalPlus, Red Hat lighteval, and community
+  cards as cited. **No competitor benchmarks were re-run.**
+* **Naming:** historically “TwIL-8B” referred to **rl100**; this repository now publishes
+  **s30** as TwIL-8B. “GPQA” means GPQA Diamond; “AIME” without a year is the legacy set.
+  Scores are percentages (0.964 → 96.4%).
+
+Full methodology, caveats, and the numbered source list are in
+[`TwIL-8B-evaluation-report.pdf`](TwIL-8B-evaluation-report.pdf).
+
+## Relationship to TwIL-LM
+
+TwIL-8B sits in the same webAI TwIL family as
+[`TwIL-LM3`](https://huggingface.co/webAI-Official/TwIL-LM3) and
+[`TwIL-LM3-Pro`](https://huggingface.co/webAI-Official/TwIL-LM3-Pro), but on a different base
+and objective mix: Granite 4.2 **8B** with a contest-math / code / IF emphasis, rather than the
+3B formal-logic Track A / Track B suite. The LM3 cards remain the reference for in-domain
+formal-logic gates; this card is the reference for the 8B reasoning evals above.
+
+## License and attribution
+
+Released under the **Apache License 2.0** — see `apache-2.0-LICENSE.txt`.
+
+The base model, `ibm-granite/granite-4.2-8b`, is Copyright IBM Corporation and is also
+distributed under Apache 2.0; all credit for the base model goes to IBM.
+
+## References
 
-## Source checkpoint
+Primary write-up: [`TwIL-8B-evaluation-report.pdf`](TwIL-8B-evaluation-report.pdf)
+(November 2026). Selected sources:
 
-`ckpt/track_a/hf_100` (merged FSDP actor after 100 RL steps).
+1. TwIL internal eval, high-effort pass@1 (`final.*.json` summaries) — rl100, s20, s30, lineage
+2. [Gemma 4 model card](https://ai.google.dev/gemma/docs) — Google
+3. Gemma 4 Technical Report, Table 5 (arXiv 2607.02770)
+4. Qwen3.6-27B model card — Gemma4-31B baseline column (HMMT)
+5. RedHatAI gemma-4-31B lighteval BF16 baseline
+6. [Qwen3.5-9B model card](https://huggingface.co/Qwen/Qwen3.5-9B)
+7. [MathArena](https://matharena.ai/) — Qwen3.5-9B / frontier contest math + tokens
+8. SOTA2 Research — MATH-500 leaderboard
+9. Artificial Analysis via OpenRouter — Qwen3.5 9B (Reasoning) HLE
+10. EvalPlus submission #300 — Gemma 4 31B HumanEval / HumanEval+
+11. Community Qwen3.5-9B thinking-on HumanEval / HumanEval+ baseline
+12–21. Frontier cards and aggregators (OpenAI GPT-5.5, Anthropic Opus 4.8, xAI Grok 4.3,
+     MathArena / BenchLeader, Artificial Analysis, Vals AI, Z.ai GLM-5.2, benchgap.net) —
+     see the PDF for the full numbered list