Granite 4.2 3B โ€” Uno drafter adapter

A rank-128 LoRA that turns ibm-granite/granite-4.2-3b into a lossless parallel drafter, using the recipe from Uno (arXiv 2609.04010). Greedy output is identical to plain Granite. The adapter can only change speed, never the answer.

This repo holds every form of the adapter:

Form Files Runtime
PEFT adapter adapter_config.json, adapter_model.safetensors, code/ PyTorch
NPU build model.q4nx, uno.q4nx, open_kernels/, config.json, tokenizer files OpenFlowLM-Next on the AMD XDNA 2 NPU
GGUF LoRA gguf/granite-4.2-3b-uno-f32.gguf None yet. See GGUF and llama.cpp.

Speed in one paragraph. On a GPU or CPU the adapter commits 1.77 tokens per forward pass at 8 rows. On the NPU a 4-row pass costs 1.5 to 1.7 decode steps, so the drafter alone is level with plain decode on short prompts and slower on chat. OpenFlowLM-Next therefore picks a method each cycle (prompt lookup, the drafter, or a plain step). In the runs below that choice was 0.99x of plain decode at worst and 2.39x at best.

How it works

Decoding runs in cycles of two passes over the same model.

  1. Draft. Feed the last committed token plus Sโˆ’1 random-noise rows. The LoRA is switched on for the noise rows only. One parallel pass proposes the next S tokens.
  2. Verify. Feed the proposal through the frozen base with the LoRA off. Keep the longest prefix that matches the base's greedy choice, plus the base's own token at the first miss.

The verifier is the unmodified base and acceptance is exact prefix matching. That is why the output equals plain autoregressive Granite.

TPF (tokens per forward) counts committed tokens per transformer pass, with both passes counted. It runs from 1.0 to (S+1)/2.

Results: tokens per forward

60-prompt core set (math 20, instruction-following 20, code 10, chat 10). Greedy, 256 new tokens, noise seeded per prompt. PyTorch on a GPU.

Setting All Math Code IFEval Chat Max possible
S=4, bf16 1.586 1.783 1.572 1.509 1.430 2.5
S=8, bf16 1.765 2.135 1.753 1.623 1.513 4.5
S=16, bf16 1.807 2.220 1.800 1.637 1.559 8.5
S=8, base fake-quantized to q4_1 1.763 2.149 1.727 1.644 1.478 4.5
S=8, base fake-quantized to Q4_K (approx.) 1.757 2.080 1.800 1.653 1.456 4.5
  • Losslessness: the two-pass output equals AR greedy on all 10 checked prompts in fp32 (128 tokens each).
    • In bf16, 5 of 10 diverge at a near-tie, because 8-row passes round differently from 1-row decoding. All 5 are identical in fp32.
  • q4 base weights cost under 0.5% TPF at S=8. No quantization-aware phase was needed.
  • S=8 is the size the adapter was trained for. Rows beyond 8 were never trained, so S=16 adds only ~2%.
  • TPF is not speed. It becomes a speedup only where a multi-row pass costs about one decode step. An earlier version of this card projected โ‰ˆ1.65ร— on the NPU from that assumption. The measured pass cost is higher, and the projection did not hold. The measured speeds are below.

Speed on the AMD XDNA 2 NPU

q4_1, 4 rows a pass, Ryzen AI 9 HX 370, quiet machine, OpenFlowLM-Next kernels of 2026-10-15 and 2026-10-16. Every run gave the same tokens as plain decode.

Plain decode is 23.4 / 22.3 / 21.4 / 19.7 tok/s after 0 / 512 / 1024 / 2048 tokens of context.

The drafter on every cycle

OFLM_SPEC_MODE=uno, 96 tokens, measured 2026-10-16.

context prompt tokens a cycle Uno plain decode speedup
64 chat 2.91 21.95 tok/s 23.0 tok/s 0.96x
512 chat 2.74 19.50 22.2 0.88x
1024 chat 2.59 17.28 21.4 0.81x
2048 chat 2.53 14.99 19.7 0.76x
64 code 3.43 25.83 23.1 1.12x
512 code 3.00 21.33 22.3 0.96x
1024 code 2.82 18.82 21.3 0.89x
2048 code 3.43 20.34 19.7 1.03x
  • Six short prompts (chat, code, math, reasoning): 3.02 tokens a cycle, geometric mean 1.00x.
  • A cycle costs 3.2 decode steps at short context and 3.3 at 2048 tokens. Chat commits about 2.5 to 2.9 tokens a cycle, so the cycle does not pay on chat. Math and code commit more and come out ahead at short context.

The per-cycle choice (what OFLM_SPECULATIVE=1 runs)

Each cycle the engine picks the method with the most tokens per millisecond expected: a prompt-lookup pass, a Uno cycle, or a plain step. 128 tokens (256 for the last four rows), measured 2026-10-15.

prompt context plain decode per-cycle choice speedup
chat short 23.7 tok/s 23.49 tok/s 0.99x
chat 2 short 23.8 24.11 1.01x
code short 23.7 27.94 1.18x
code 2 short 23.6 25.19 1.07x
math short 23.7 26.34 1.11x
reasoning short 23.6 26.54 1.12x
chat 1024 21.0 20.97 1.00x
code 1024 21.1 24.40 1.16x
chat 2048 18.8 19.75 1.05x
code 2048 18.8 20.67 1.10x
whole-file edit 1347 20.1 31.43 1.56x
whole-file bug fix 1390 20.1 48.03 2.39x
quote a document 1032 20.7 27.88 1.35x
answer from a document 728 21.4 32.05 1.49x
  • Most of this gain comes from the prompt-lookup cycles, which need no adapter. The drafter adds the most on math and code.
  • This table is one day older than the table above. The drafter pass became 5% to 8% faster after it was measured.

Run on the NPU

Needs OpenFlowLM-Next main at or after PR #209 (merged 2026-11-05).

  1. Download the NPU files into a models directory:
    hf download Cyronius/granite-4.2-3b-uno --exclude "adapter_model.safetensors" --exclude "gguf/*" --local-dir <root>/models/Granite-4.2-3B-Uno-NPU2
    
  2. Add an entry under "models" in a copy of OpenFlowLM's model_list.json:
    "granite4.2": {"3b": {
      "name": "Granite-4.2-3B-Uno-NPU2", "url": "", "file_url": "", "ms_url": "",
      "modified_at": "2026-11-12T00:00:00Z", "oflm_min_version": "0.9.22",
      "files": ["config.json", "model.q4nx", "tokenizer.json", "tokenizer_config.json", "chat_template.jinja", "uno.q4nx"],
      "default_context_length": 8192, "max_prefill_len": 4096,
      "details": {"family": "granite", "think": false, "think_toggleable": false,
                  "parameter_size": "3B", "quantization_level": "Q4_1"},
      "label": [], "footprint": 3.0}}
    
  3. Point oflm at both, turn speculation on, and run:
    set OFLM_CONFIG_PATH=<your model_list.json>
    set OFLM_MODEL_PATH=<root>
    set OFLM_SPECULATIVE=1
    oflm run granite4.2:3b
    
  • Without OFLM_SPECULATIVE=1 the same files run as plain Granite. The load log says that uno.q4nx is present but off.
  • Speculation serves greedy requests only, and Granite samples by default. Type /set topk 1 and /set rep-pen 1 first. Over oflm serve, send "top_k": 1, "repetition_penalty": 1.
  • OFLM_SPEC_MODE pins one method: uno, lookup, lookup-uno or plain.
  • The kernels in open_kernels/ are found beside the model. oflm add doesn't copy uno.q4nx or open_kernels/ yet, so use the steps above.

Use in PyTorch

This needs the custom Granite SDAR model class and the two-pass sampler in code/. Install the packages in code/requirements-droplet.txt first.

import sys

import torch
from huggingface_hub import snapshot_download
from peft import PeftModel

repo = snapshot_download("Cyronius/granite-4.2-3b-uno", allow_patterns=["adapter_*", "code/*"])
sys.path.insert(0, f"{repo}/code")
from training.constants import DEFAULT_MODEL_ID, DEFAULT_MODEL_REVISION, NOISE_HIGH
from training.lora import TokenwiseLoraRouter
from training.train import load_granite_uno
from uno_granite.sampler import uno_generate

tokenizer, base = load_granite_uno(snapshot_download(DEFAULT_MODEL_ID, revision=DEFAULT_MODEL_REVISION))
model = PeftModel.from_pretrained(base.cuda().eval(), repo, is_trainable=True).eval()
router = TokenwiseLoraRouter(model)  # gates the LoRA to draft rows
ids = tokenizer.apply_chat_template([{"role": "user", "content": "What is 17 * 23?"}], add_generation_prompt=True)
out, stats = uno_generate(model, ids, block_len=8, max_new_tokens=256,
                          eos_token_id=tokenizer.eos_token_id, noise_high=NOISE_HIGH, router=router)
print(tokenizer.decode(out), f"TPF {stats.tpf:.2f}")

allow_patterns skips the NPU and GGUF files (about 4 GB). is_trainable=True only exposes the LoRA tensors to the router; nothing is trained. To reproduce the TPF table, run code/scripts/eval_tpf.py.

GGUF and llama.cpp

  • Base model: use IBM's own files in ibm-granite/granite-4.2-3b-GGUF. Q4_1 is the quantization of the NPU build.
  • Adapter: gguf/granite-4.2-3b-uno-f32.gguf is the adapter as a GGUF LoRA (rank 128, alpha 8192, q/k/v/o/gate/up/down of all 40 layers, F32), made with llama.cpp's convert_lora_to_gguf.py.
  • The adapter does not give Uno decoding in llama.cpp. Uno needs the adapter on the noise rows only, with the base verifying the draft with the adapter off. llama.cpp applies a LoRA to every token and has no field for the noise ids (uniform in [1, 100256)). A runtime that routes the adapter per row is needed. Do not load this file as a model.

Training

Base ibm-granite/granite-4.2-3b @ e459acceac81e5fe67c07d9cfc72329a332e7eb1, frozen
Adapter LoRA r=128, ฮฑ=8192 (ฮฑ/r 64), on q, k, v, o, gate, up, down; 249.0M parameters. The router applies it to noise rows only.
Objective Total-variation distance between the drafter's and the frozen base's next-token distributions (Uno ฮณ=1). Uniform noise at rate 1.0 on all non-prompt tokens, block-diffusion attention layout.
Data 80% OpenThoughts3-1.2M @ 61bcf9d + 20% tulu-3-sft-mixture @ b14afda, rendered with Granite's chat template and packed to 4,096 tokens
Tokens 0.87B trained (about 0.24 of the 3.58B-token prepared mix)
Curriculum block size 2 for 165 steps โ†’ 4 for 248 โ†’ 8 for 1,238; 1,651 steps ร— 128 rows ร— 4,096 tokens
Optimizer AdamW, no weight decay, gradient clip 1.0, bf16 autocast. 60 warmup steps, LR 2e-5 constant, cosine decay to 0 over the last 300 steps.
Hardware 1ร— AMD Instinct MI350X (gfx950), ROCm 7.2, PyTorch 2.10, about 31 GPU-hours including one restart

Note on the learning rate. The run started at 5e-5 (the setting of IFM's K2-7B adapter). It diverged a dozen steps into block size 4, so the run was restarted from the step-165 boundary at 2e-5. The decay was added at step 1,160, after the block-size-8 loss had flattened.

uno_training_manifest.json records the exact data hashes, curriculum and settings.

Limitations

  • On the NPU the drafter alone does not beat plain decode on chat. It is level on short prompts and falls to 0.76x at 2048 tokens of context. Use the per-cycle choice, not OFLM_SPEC_MODE=uno.
  • The NPU runs 4 rows a pass. The adapter was trained for 8. On K2-7B, an 8-row pass cost about twice the matrix work of a 4-row pass, so 8 rows are not built for Granite.
  • Short training. About a quarter of one pass over the prepared data. Uno's published adapters train for one or more epochs, and the loss here was still falling.
  • Chat accepts least. The data is 80% math and code reasoning.
  • Greedy decoding only, in the included sampler and on the NPU. Sampling needs the standard speculative-sampling acceptance rule.
  • bf16 near-ties. In PyTorch, greedy output can differ from bf16 one-token decoding wherever the top two logits are within rounding error. It is identical in fp32.

Files

File Description
adapter_config.json, adapter_model.safetensors The final adapter (step 1,651), PEFT format
uno_training_manifest.json Data hashes, curriculum and optimizer settings
eval/ Raw results for every row of the TPF table
code/ The Granite port (uno_granite/), the training code adapted from Uno (training/, Apache-2.0, see NOTICE.uno) and the eval script
model.q4nx Granite 4.2 3B in OpenFlowLM's Q4NX format, q4_1, q_proj pre-scaled by 0.125 (the fold config.json records)
uno.q4nx The adapter for the NPU: rank-128 LoRA on all seven projections of the 40 layers, applied to draft rows only; noise ids [1, 100256)
open_kernels/ The NPU kernel set (decode, prefill, the 4-row pass and its LoRA variant, the LM heads), built on 2026-11-12 from OpenFlowLM-Next main at 96b915d (mlir-aie 1.4.2, Peano 21.0.0.2026080301, manifest_version 7)
config.json, tokenizer.json, tokenizer_config.json, chat_template.jinja Runtime config, tokenizer and Granite's chat template for the NPU build
gguf/granite-4.2-3b-uno-f32.gguf The adapter as a GGUF LoRA

Reproduce the NPU build

q4nx-build -i granite-4.2-3b-bf16.gguf -s <granite-4.2-3b dir> -f granite -t language --quant Q4_1 \
  --uno-adapter <this repo's adapter dir> --uno-noise-high 100256 -o Granite-4.2-3B-Uno-NPU2
python open_kernels/export_qwen36_kernels.py --model-dir Granite-4.2-3B-Uno-NPU2

The GGUF is ibm-granite/granite-4.2-3b-GGUF's bf16 file. The procedure and checks are requirement OPEN-FAMILY-GRANITE-UNO in OpenFlowLM-Next's specs/open-engine/spec.md.

Credits

  • Method and training code: Uno by IFM (Apache-2.0).
  • Base model: IBM Granite 4.2 3B (Apache-2.0).
  • Data: OpenThoughts3 and Tรผlu 3. See their dataset cards for licenses.
  • Runtime: OpenFlowLM-Next.
  • Compute: AMD Developer Cloud.
Downloads last month
37
GGUF
Model size
0.2B params
Architecture
granite
Hardware compatibility
Log In to add your hardware

32-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for Cyronius/granite-4.2-3b-uno

Adapter
(8)
this model