| --- |
| license: apache-2.0 |
| language: |
| - vi |
| - en |
| - zh |
| base_model: |
| - Qwen/Qwen3.8-27B |
| library_name: transformers |
| tags: |
| - legal |
| - kimi-k3 |
| - long-reasoning |
| - efficient-reasoning |
| - coding |
| - math |
| - stem |
| --- |
| |
| # Kiwen-27B |
|
|
| A fine-tune of [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) trained on long chain-of-thought traces from Kimi K3. |
|
|
| This is **not a broadly stronger model**. The fine-tune primarily teaches Qwen3.8-27B to terminate its reasoning and produce an answer before exhausting its generation budget. |
|
|
| ## Model Summary |
|
|
| Qwen3.8-27B is a strong reasoning model that can sometimes spend its entire token budget deliberating without producing a final answer. |
|
|
| On GSM8K, this behavioral change produces a **+10.31 percentage-point improvement** under flexible extraction. |
|
|
| ### Key results |
|
|
| | Benchmark | Qwen3.8-27B | **Kiwen-27B** | Delta | t | |
| | ---------------------------------------- | ----------: | ---------------: | ---------: | --------: | |
| | **GSM8K** exact_match (flexible) | 0.7407 | **0.8438** | **+10.31** | **+6.58** | |
| | **GSM8K** exact_match (strict) | 0.6672 | **0.7278** | **+6.07** | **+3.40** | |
| | IFEval prompt-level strict | 0.8226 | **0.8429** | +2.03 | +0.90 | |
| | IFEval prompt-level loose | 0.8447 | **0.8669** | +2.22 | +1.04 | |
| | IFEval instruction-level strict | 0.8393 | **0.8645** | +2.52 | — | |
| | IFEval instruction-level loose | 0.8537 | **0.8801** | +2.64 | — | |
| | VMLU val (744 Vietnamese MMLU questions) | 83.47 | **86.02** | +2.55 | +1.37 | |
|
|
| Evaluated using [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) 0.4.12 with both models served on identical hardware: |
|
|
| * **Hardware:** 1× H200 |
| * **Inference:** SGLang |
| * **Model dtype:** BF16 |
| * **KV cache:** FP8 |
| * **Generation limit:** `max_gen_toks=4096` |
| * **Prompts:** identical |
| * **Decoding settings:** identical |
|
|
| The largest improvement is on GSM8K. Other benchmarks show smaller but consistently positive changes, with no measured regression. |
|
|
| ## The Mechanism: Learning When to Stop |
|
|
| GSM8K is evaluated using two extraction methods: |
|
|
| * **Strict match:** requires the canonical `#### <number>` answer format. |
| * **Flexible extraction:** extracts the final number from the generated response. |
|
|
| The improvement under flexible extraction is substantially larger: |
|
|
| > **+10.31 points flexible vs. +6.07 points strict** |
|
|
| This asymmetry suggests that much of the improvement comes from **successfully producing an answer before the generation budget is exhausted**, rather than from a fundamental improvement in mathematical reasoning. |
|
|
| ## Out-of-Domain Regression |
|
|
| To test whether the fine-tuning damaged unrelated capabilities, both models were evaluated on 100 held-out Vietnamese enterprise documents covering: |
|
|
| * Accounting |
| * CRM |
| * Internal RAG workloads |
|
|
| The metric was token-level negative log-likelihood: |
|
|
| | Model | NLL / token | |
| | ------------ | ----------: | |
| | Qwen3.8-27B | 3.3390 | |
| | Kiwen-27B | 3.3543 | |
|
|
| Paired difference: |
|
|
| * **Δ:** +0.0067 |
| * **Standard error:** 0.0041 |
| * **t:** 1.65 |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoModelForImageTextToText, AutoProcessor |
| |
| model = AutoModelForImageTextToText.from_pretrained( |
| "beyoru/Kiwen-27B", |
| dtype="bfloat16", |
| device_map="auto", |
| ) |
| |
| proc = AutoProcessor.from_pretrained( |
| "beyoru/Kiwen-27B" |
| ) |
| ``` |
|
|
| ### MTP Tensors |
|
|
| MTP tensors are absent from the released model. |
|
|
| `transformers` declares: |
|
|
| ```python |
| _keys_to_ignore_on_load_unexpected = [r"^mtp.*"] |
| ``` |
|
|
| Therefore, the multi-token-prediction block is dropped by HF-based merging, including this release. |
|
|
| For speculative decoding, use an external draft model such as [z-lab/Qwen3.8-27B-DFlash2](https://huggingface.co/z-lab/Qwen3.8-27B-DFlash2) rather than NEXTN. |
|
|
| A draft model trained against the original weights may have a lower acceptance rate against the merged Kiwen weights. This does **not** affect correctness because speculative decoding uses rejection verification. |
|
|
| ### Benchmark Caveat |
|
|
| VMLU validation is publicly distributed with answer keys and may have some representation in pretraining data. |
|
|
| Its absolute score should therefore be interpreted cautiously. |
|
|
| However, the **base-versus-fine-tuned comparison remains informative**, since both models are exposed to the same potential contamination. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{kiwen27bk3, |
| title = {Kiwen-27B}, |
| author = {beyoru}, |
| year = {2026}, |
| url = {https://huggingface.co/beyoru/Kiwen-27B-K3} |
| } |
| ``` |
|
|
| ## License & Attribution |
|
|
| Built on: |
|
|
| * [Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) — Apache-2.0 |
|
|
| Kiwen-27B is released under the **Apache-2.0** license. |
