| --- |
| license: apache-2.0 |
| library_name: transformers |
| pipeline_tag: image-text-to-text |
| language: |
| - en |
| tags: |
| - quantized |
| - rfi |
| - mixed-precision |
| base_model: |
| - migtissera/Tess-4-27B |
| base_model_relation: quantized |
| --- |
| |
| > [!IMPORTANT] |
| > ## RFI/RFA hybrid quant of [migtissera/Tess-4-27B](https://huggingface.co/migtissera/Tess-4-27B) |
| > |
| > **Runtime:** requires [`tcclaviger/vllm:latest`](https://hub.docker.com/r/tcclaviger/vllm) — an **RDNA 4 (gfx12xx)** vLLM image and the only build with the RFI/RFA kernels; no other vLLM build loads these weights. **Not validated on any other hardware at this time.** |
|
|
| # Tess-27B-RFI |
|
|
| Mixed-precision (RFI/RFA hybrid) quantization of **Tess-4-27B** by Migel Tissera — |
| an agentic, thinking-native finetune of Qwen3.6-27B. All credit for the model to its |
| author; this repo only changes the numerics. |
|
|
| ## Quantization by component |
|
|
| - **Attention and all other linear layers** — 8-bit integer weights (RFI): group size 32, symmetric, Hadamard-32 rotation, block-float scales stored as int8 mantissa + int8 exponent, int8 activation compute path. |
| - **MTP speculative-decode head** — also 8-bit RFI: its fc, MLP, and attention projections are all int8-packed like the main attention; only its norms stay bf16. |
| - **MLP layers** — 4-bit float weights (RFA): IQ4_NL non-linear grid, group size 16, asymmetric, Hadamard-16 rotation, the same block-float int8 scale encoding. |
| - **Kept in bf16 (not quantized)** — vision encoder, linear-attention (GDN) blocks, embeddings, norms, and the lm_head. |
|
|
| ## Serving context — 512K via YaRN |
|
|
| All evaluation below was run while serving at **`--max-model-len 524288`** (512K tokens), |
| extended from the native 256K window with YaRN via `--hf-overrides`: |
|
|
| ```json |
| {"text_config": {"rope_parameters": {"rope_type": "yarn", "factor": 2.0, |
| "original_max_position_embeddings": 262144, "mrope_interleaved": true, |
| "mrope_section": [11, 11, 10], "partial_rotary_factor": 0.25, |
| "rope_theta": 10000000}}} |
| ``` |
|
|
| ## Evaluation results |
|
|
| Six builds measured with identical methodology, each against its own live vLLM endpoint (July 2026): |
| **Qwen3.6-27B** (bf16 base) → **Tess-4-27B** (the tune, bf16) → **Tess-27B-RFI** (this quant) → |
| [**Tess-27B-RFA**](https://huggingface.co/tcclaviger/Tess-27B-RFA) (all-attention 4-bit sibling) → |
| **Tess-FP8** (W8A8 block-128 FP8 sibling), |
| with **Qwen3.6-35B-A3B** (MoE, bf16) as a comparative reference point. |
| Bold marks the best score in each row (ties all bolded). |
|
|
| ### Tune impact — Qwen3.6-27B → Tess-4-27B |
|
|
| | Metric | Change | |
| |---|---| |
| | WikiText-2 perplexity | 7.056 → 6.669 (**−5.5%**) | |
| | Codeneedle overall recall | 97.8% → 97.7% (≈ flat) | |
| | MC accuracy (4 tasks) | ≈ flat (−0.1 to +1.6 pp) | |
| | Tool-eval (full 69, TC-61 excl) | 86 → 85 (−1) | |
| | GSM8K / MMLU (50q each) | 98→94% / 74→76% | |
| | Decode @ conc 1 (ISL 128) | 57.0 → 59.9 tok/s (+5%) | |
| | Decode @ conc 50 (ISL 128) | 556 → 528 tok/s (−5%) | |
|
|
| ### Quantization cost (RFI/RFA hybrid) — Tess-4-27B → Tess-27B-RFI |
|
|
| | Metric | Change | |
| |---|---| |
| | Checkpoint size | 55.6 → 28.9 GB (**−48%**) | |
| | WikiText-2 perplexity | 6.669 → 6.663 (≈ flat) | |
| | Codeneedle overall recall | 97.7% → 98.0% (≈ flat) | |
| | MC accuracy (4 tasks) | ≈ flat (−0.4 to +0.1 pp) | |
| | Tool-eval (full 69, TC-61 excl) | 85 → 87 (**+2**) | |
| | GSM8K / MMLU (50q each) | 94→98% / 76→82% (**both up**) | |
| | Decode @ conc 1 (ISL 128) | 59.9 → 75.3 tok/s (**+26%**) | |
| | Decode @ conc 50 (ISL 128) | 528 → 453 tok/s (−14%) | |
|
|
| ### Quality |
|
|
| | Metric | Qwen3.6-27B (base) | Tess-4-27B | **Tess-27B-RFI** | Tess-27B-RFA | Tess-FP8 | Qwen3.6-35B-A3B | |
| |---|---|---|---|---|---|---| |
| | WikiText-2 PPL (n_ctx 2048, lower is better) | 7.0559 | 6.6691 | 6.6632 | 6.6292 | 6.6627 | **6.5092** | |
| | ARC-Challenge (acc_norm) | 59.30% | **60.84%** | 60.41% | 60.32% | 60.49% | 55.20% | |
| | ARC-Easy (acc_norm) | 75.93% | 77.53% | 77.40% | **78.87%** | 77.82% | 71.13% | |
| | Winogrande (acc) | 77.51% | 77.43% | 77.51% | 76.80% | **77.66%** | 73.40% | |
| | HellaSwag (acc_norm) | 84.12% | 84.21% | **84.27%** | 84.05% | 84.13% | 82.95% | |
|
|
| Multiple-choice accuracy is lm-eval loglikelihood scoring, 0-shot. |
|
|
| ### Long-context positional recall (codeneedle) |
|
|
| Verbatim function recall under 10K–80K-token contexts. |
|
|
| | Corpus | Qwen3.6-27B (base) | Tess-4-27B | **Tess-27B-RFI** | Tess-27B-RFA | Tess-FP8 | Qwen3.6-35B-A3B | |
| |---|---|---|---|---|---|---| |
| | Python | **100%** | **100%** | **100%** | **100%** | 99.55% | 99.09% | |
| | C++ | 98.12% | 98.12% | 98.44% | **98.75%** | **98.75%** | 98.44% | |
| | Rust | **99.69%** | **99.69%** | **99.69%** | **99.69%** | **99.69%** | 99.38% | |
| | JS (~80K tokens) | 93.44% | 93.13% | **93.75%** | **93.75%** | 93.44% | 92.19% | |
| | Tools | 98.26% | **99.57%** | **99.57%** | **99.57%** | **99.57%** | 93.48% | |
| | **Overall recall** | 97.81% | 97.73% | 97.97% | **98.05%** | 97.86% | 97.28% | |
|
|
| ### Tool calling & accuracy benches |
|
|
| | Bench | Qwen3.6-27B (base) | Tess-4-27B | **Tess-27B-RFI** | Tess-27B-RFA | Tess-FP8 | Qwen3.6-35B-A3B | |
| |---|---|---|---|---|---|---| |
| | tool-eval final (full 69, TC-61 excl) | 86 | 85 | 87 | 86 | 87 | **90** | |
| | GSM8K (50q) | **98.0%** | 94.0% | **98.0%** | **98.0%** | **98.0%** | 96.0% | |
| | MMLU (50q) | 74.0% | 76.0% | **82.0%** | 80.0% | 76.0% | 64.0% | |
| | IFEval (20 prompts, prompt-level) | 90.0% | 90.0% | 90.0% | **95.0%** | 90.0% | 90.0% | |
|
|
| ### Decode throughput — tok/s output (ISL 128 / ISL 512) |
|
|
| `vllm bench serve`, random dataset, OSL 128, saturation, 4× R9700 (gfx1201), TP 4. |
|
|
| | Concurrency | Qwen3.6-27B (base) | Tess-4-27B | **Tess-27B-RFI** | Tess-27B-RFA | Tess-FP8 | Qwen3.6-35B-A3B | |
| |---|---|---|---|---|---|---| |
| | 1 | 57.0 / 61.4 | 59.9 / 63.8 | 75.3 / 69.3 | 67.9 / 58.2 | 86.0 / 85.1 | **91.9 / 114.4** | |
| | 10 | 304.5 / 233.6 | 289.6 / 227.8 | 281.5 / 219.2 | 292.2 / 194.8 | 280.6 / 269.2 | **434.9 / 440.3** | |
| | 25 | 424.4 / 321.7 | 492.0 / 341.1 | 429.1 / 284.0 | 369.0 / 260.3 | 533.2 / 355.4 | **688.9 / 563.7** | |
| | 50 | 556.0 / 349.8 | 527.9 / 345.6 | 452.8 / 295.0 | 422.6 / 278.4 | 560.3 / 394.1 | **889.8 / 702.9** | |
|
|
| ### MTP draft acceptance by work category |
|
|
| Measured from live serving logs, k=5 draft tokens, drafted-token-weighted aggregation. |
|
|
| | Work category | Overall acceptance | Pos 1 | Pos 2 | Pos 3 | Pos 4 | Pos 5 | |
| |---|---|---|---|---|---|---| |
| | JSON generation | **79.0%** | 92.5% | 85.9% | 79.3% | 71.6% | 65.6% | |
| | Math | 78.5% | 94.4% | 87.4% | 78.2% | 70.7% | 62.0% | |
| | Code | 64.2% | 88.7% | 74.5% | 63.2% | 51.2% | 43.6% | |
| | Creative English | 60.0% | 87.0% | 72.6% | 57.5% | 44.9% | 38.1% | |
|
|
| ## Notes |
|
|
| All builds serve on the [tcclaviger/vllm:latest](https://hub.docker.com/r/tcclaviger/vllm) image, which has kernel tunes baked in. |
| TunableOp is untuned — GEMMs run on default heuristic-determined values. |
| Base Qwen3.6-27B figures are the 2026-07-12 re-measurement on the same tcclaviger/vllm:latest image and thinking-OFF methodology as every other build, replacing an earlier non-comparable run. |
|
|
| ## Credits |
|
|
| - **Tess-4-27B** by Migel Tissera ([migtissera/Tess-4-27B](https://huggingface.co/migtissera/Tess-4-27B)) — the model quantized here: |
|
|
| ```bibtex |
| @misc{tissera2026tess4, |
| title = {Tess-4-27B}, |
| author = {Migel Tissera}, |
| year = {2026}, |
| howpublished = {\url{https://huggingface.co/migtissera/Tess-4-27B}} |
| } |
| ``` |
|
|
| - codeneedle (positional recall) originally by Alexander Ziskind, expanded test suite by tcclaviger (Rob Smith). |
| - Tool-calling scenarios (incl. TC-61) run on tool-eval-bench by SeraphimSerapis (Tim Messerschmidt), scenario methodology adapted from ToolCall-15 by stevibe. |
| - GSM8K, MMLU, and IFEval run via tool-eval-bench's built-in accuracy benchmarks at their defaults: GSM8K 8-shot CoT, MMLU 5-shot, IFEval zero-shot. |
|
|
