Update TwIL-LM3: weights, tokenizer and model card

README.md

@@ -37,41 +37,53 @@ why it is the recommended release of the pair.
 
 ### Track A — in-domain formal logic
 
-
 All arms below were run through the same harness, prompts and decoding settings described under
 [Evaluation protocol](#evaluation-protocol). Throughput rows are reported because in-domain score
 alone is misleading for a 3B model: `ans/s` is defined throughout as `tok/s ÷ mean generation
 length`, so it measures completed answers rather than raw decode rate.
 
-#### Track A — in-domain formal logic
-
-| lane / metric | TwIL-LM3 | TwIL-LM3* | SmolLM3-3B base | Llama-3.2-3B | LFM2-2.6B | LFM2.5-8B-A1B |
-|---|---:|---:|---:|---:|---:|---:|
-| lean_formalize token_f1 | 0.5869 | **0.6456** | 0.4347 | 0.3690 | 0.1321 | 0.4655 |
-| rule_induction derivation | 0.3192 | **0.9644** | 0.1029 | 0.0825 | 0.0615 | 0.1936 |
-| entailment_label accuracy | 0.5750 | **0.6867** | 0.3750 | 0.3300 | 0.4700 | 0.5400 |
-| mcq_answer accuracy | 0.1100 | **0.5200** | 0.0000 | 0.0000 | 0.0150 | 0.0750 |
-| semantic_parse token_f1 | 0.4416 | **0.8762** | 0.4149 | 0.3102 | 0.3665 | 0.3778 |
-| lean_critic accuracy | **0.6600** | 0.5200 | 0.6500 | 0.5300 | 0.5900 | 0.5500 |
-| lm_corpus perplexity ↓ | 2.8972 | 3.1284 | 3.1818 | **2.8478** | 4.3815 | 4.9472 |
-| math_corpus perplexity ↓ | 3.8229 | **3.5245** | 4.0685 | 4.7531 | 6.7472 | 8.3323 |
-| **macro gate** | 0.4218 | **0.5896** | 0.3466 † | 0.2925 | 0.3473 | 0.3757 |
-| **strict-7** | 0.1971 | **0.3290** | 0.1493 | 0.1229 | 0.1579 | 0.1714 |
-| macro_primary | 0.4475 | **0.4958** | 0.4075 | 0.3450 | 0.4188 | 0.4213 |
-| tok/s | 15880 | 15840 | 15564 | 16160 | 25230 | 22480 |
-| mean gen length | **564** | 572 | 999 | 696 | 2296 | 1830 |
-| **ans/s** | **28.1** | 27.7 | 15.6 | 23.2 | 10.9 | 12.0 |
+| lane / metric | TwIL-LM3 | TwIL-LM3* | SmolLM3-3B base | Llama-3.2-3B | LFM2-2.6B | LFM2.5-8B-A1B | Qwen3-8B | gpt-oss-120b ‡ |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|
+| lean_formalize token_f1 | 0.5869 | **0.6456** | 0.4347 | 0.3690 | 0.1321 | 0.4655 | 0.4022 | 0.6306 |
+| rule_induction derivation | 0.3192 | **0.9644** | 0.1029 | 0.0825 | 0.0615 | 0.1936 | 0.3680 | 0.6518 |
+| entailment_label accuracy | 0.5750 | 0.6867 | 0.3750 | 0.3300 | 0.4700 | 0.5400 | 0.5800 | **0.7750** |
+| mcq_answer accuracy | 0.1100 | **0.5200** | 0.0000 | 0.0000 | 0.0150 | 0.0750 | 0.0000 | 0.0700 |
+| semantic_parse token_f1 | 0.4416 | **0.8762** | 0.4149 | 0.3102 | 0.3665 | 0.3778 | 0.4257 | 0.4331 |
+| lean_critic accuracy | 0.6600 | 0.5200 | 0.6500 | 0.5300 | 0.5900 | 0.5500 | **0.7950** | 0.5550 |
+| lm_corpus perplexity ↓ | 2.8972 | 3.1284 | 3.1818 | 2.8478 | 4.3815 | 4.9472 | **2.5440** | 912.23 § |
+| math_corpus perplexity ↓ | 3.8229 | **3.5245** | 4.0685 | 4.7531 | 6.7472 | 8.3323 | 4.0083 | 1045.63 § |
+| average, 6 lanes | 0.4488 | **0.7021** | 0.3296 | 0.2703 | 0.2725 | 0.3670 | 0.4285 | 0.5192 |
+| **macro gate** | 0.4218 | **0.5896** | 0.3466 † | 0.2925 | 0.3473 | 0.3757 | 0.5336 | — |
+| **strict-7** | 0.1971 | **0.3290** | 0.1493 | 0.1229 | 0.1579 | 0.1714 | 0.2093 | — |
+| macro_primary | 0.4475 | 0.4958 | 0.4075 | 0.3450 | 0.4188 | 0.4213 | **0.5750** | — |
+| tok/s | 15880 | 15840 | 15564 | 16160 | **25230** | 22480 | 9420 | 3374 |
+| mean gen length | **564** | 572 | 999 | 696 | 2296 | 1830 | 2094 | 1005 |
+| **ans/s** | **28.1** | 27.7 | 15.6 | 23.2 | 10.9 | 12.0 | 4.5 | 3.4 |
 
 \* **TwIL-LM3\*** is our latest version of TwIL-LM3. **The weights will be released soon** — the
 files in this repository are the current TwIL-LM3 release, not this one. Lanes marked — are not
 yet reported for it.
 
-† The base column here comes from the external-comparison run rather than the paired run used
-for the Δ table above, hence 0.3466 against 0.3356 — run-to-run variation of the same
-checkpoint. The paired run is the correct basis for the improvement claim.
+‡ **gpt-oss-120b** runs MXFP4 weights at tensor-parallel 2 — quantized and multi-GPU, so its
+throughput rows are not directly comparable to the single-GPU BF16 arms. Its `procedural` lane
+and the loose-match scorings were not collected, so the three summary rows below the six-lane
+average cannot be computed for it; that is what the — cells mean, not a zero.
+
+§ The 120B's perplexities are three orders of magnitude off every other arm because its harmony
+response format and tokenizer make the corpus lanes score a different quantity. The number is
+reported for completeness but is not a comparable measurement.
+
+† The base column here comes from the external-comparison run rather than the paired base-vs-TwIL
+run, hence 0.3466 against the 0.3356 quoted in the summary at the top of this card — run-to-run
+variation of the same checkpoint. The paired run is the correct basis for the improvement claim.
+
+**`average, 6 lanes`** is the plain mean of the six objective rows above it, each at whatever
+scoring that row reports. It is a coarser summary than the three that follow — it mixes token-F1
+with accuracy — but it is the only summary row every arm here can be compared on, including the
+120B.
 
-The three summary rows aggregate the objective lanes in different ways. None of them include the
-perplexity lanes or the token-F1 scorings, which are not on a common 0–1 accuracy scale.
+The next three rows aggregate more carefully. None of them include the perplexity lanes or the
+token-F1 scorings, which are not on a common 0–1 accuracy scale.
 
 **`macro gate`** is the headline metric and the one the training pipeline gates on. It is the
 equal-weight mean of five objectives: the four bounded classification lanes (`entailment_label`,
@@ -92,13 +104,25 @@ earlier reports; the gate is the metric to read for overall in-domain capability
 harsh — exact match on generative lanes is near zero for every arm — so it is useful for ranking
 models against each other but not as an absolute capability measure.
 
-Among the released models, TwIL-LM3 leads both headline metrics and wins every lane the pipeline
-targets directly: `lean_formalize` token-F1 0.5869 against 0.4655 for the nearest arm,
-`rule_induction` 0.3192 against 0.1936, and strict MCQ accuracy 0.1100, the only non-trivial
-value in that row. Its closest competitor on the macro gate is LFM2.5-8B-A1B at 0.3757, roughly
-2.8x its size. The one lane shown here that it gives up is `lm_corpus` perplexity, where
-Llama-3.2-3B is marginally lower. It also trails on `procedural` and loose MCQ, which are folded
-into the macro gate and strict-7 but not listed separately above.
+TwIL-LM3 beats every arm up to and including LFM2.5-8B-A1B, and does so on all six objective
+lanes and all four summary rows, not on average alone. Against the strongest of them it is
+0.4218 to 0.3757 on the gate at roughly a third of the total parameters, with the margin coming
+from the lanes the pipeline targets directly: `lean_formalize` token-F1 0.5869 against 0.4655,
+`rule_induction` 0.3192 against 0.1936, `semantic_parse` 0.4416 against 0.3778.
+
+It does not beat the two largest arms. Qwen3-8B leads it on the gate 0.5336 to 0.4218 and
+gpt-oss-120b leads the six-lane average 0.5192 to 0.4488. That gap is worth reading carefully in
+Qwen's case: almost all of it is loose-match credit. Qwen answers MCQ correctly but never in the
+requested format — strict accuracy 0.0000 against TwIL-LM3's 0.1100, while its loose match is
+0.745 — and the macro rows credit `max(exact_match, loose_match)`. On `strict-7`, which gives no
+loose-match credit anywhere, the two are 0.2093 to 0.1971, a gap of 0.012 rather than 0.11. Qwen
+also wins `lean_critic` outright at 0.7950 and has the lowest `lm_corpus` perplexity at 2.5440.
+The 120B leads three lanes outright and is genuinely stronger at entailment (0.7750) and rule
+induction (0.6518).
+
+The size and speed context matters for both. Qwen3-8B is 2.6x the parameters and produces 4.5
+answers/sec against TwIL-LM3's 28.1; the 120B is 40x the parameters and produces 3.4. TwIL-LM3
+is the strongest arm here at its own scale and the most efficient arm at any scale.
 
 The unreleased TwIL-LM3\* moves the gate to 0.5896 and strict-7 to 0.3290, roughly +0.17 and
 +0.13 over the current release. The gains are concentrated in the two lanes where TwIL-LM3 is
@@ -112,7 +136,7 @@ generations averaging 564 tokens where every other arm except Llama runs past 69
 models decode faster in raw tokens per second, 25230 and 22480 against 15880, but their length
 more than cancels it.
 
-#### Track B — held-out benchmarks
+### Track B — held-out benchmarks
 
 | dataset | TwIL-LM3 | SmolLM3-3B base | Llama-3.2-3B | LFM2-2.6B | LFM2.5-8B-A1B | Qwen3-8B | gpt-oss-120b ‡ |
 |---|---:|---:|---:|---:|---:|---:|---:|
@@ -127,7 +151,7 @@ more than cancels it.
 | musr | 0.4957 | 0.4997 | 0.4932 | 0.4867 | 0.5703 | 0.6301 | **0.6852** |
 | mmlu_redux | 0.6667 | 0.6633 | 0.6000 | 0.7133 | 0.8367 | 0.8500 | **0.9467** |
 | ifeval | 0.6433 | 0.6767 | 0.7167 | 0.7300 | **0.8900** | 0.8400 | 0.7900 |
-| rudas_ood | 0.0365 | 0.0209 | **0.0733** | 0.0017 | 0.0061 | 0.0468 | 0.0000 § |
+| rudas_ood | 0.0365 | 0.0209 | **0.0733** | 0.0017 | 0.0061 | 0.0468 | 0.0000 ¶ |
 | bbh_logic | 0.6633 | 0.6667 | 0.5333 | 0.5713 | 0.7700 | 0.6367 | **0.9980** |
 | math500 | 0.6900 | 0.7000 | 0.4233 | 0.7133 | 0.7800 | 0.6100 | **0.8433** |
 | **macro (10 CoT datasets)** | 0.7339 | 0.7193 | 0.6997 | 0.7523 | 0.7884 | 0.8493 | **0.8689** |
@@ -137,7 +161,7 @@ more than cancels it.
 | **ans/s** | **32.9** | 24.9 | 31.7 | ≈31.7 | ≈16.9 | 4.9 | 4.2 |
 
 ‡ MXFP4 weights, tensor-parallel 2 — quantized and multi-GPU, so not directly comparable to the
-single-GPU BF16 rows. § 74% of its `rudas_ood` generations hit the length cap, so that cell is a
+single-GPU BF16 rows. ¶ 74% of its `rudas_ood` generations hit the length cap, so that cell is a
 truncation artefact rather than a measured score; excluding the row, its 13-dataset macro is
 0.8708.