Instructions to use Cyronius/granite-4.2-3b-uno with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use Cyronius/granite-4.2-3b-uno with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("ibm-granite/granite-4.2-3b") model = PeftModel.from_pretrained(base_model, "Cyronius/granite-4.2-3b-uno") - Notebooks
- Google Colab
- Kaggle
Granite 4.2 3B โ Uno drafter adapter
A rank-128 LoRA that turns ibm-granite/granite-4.2-3b into a lossless parallel drafter, using the recipe from Uno (arXiv 2609.04010). Greedy output is identical to plain Granite. The adapter can only change speed, never the answer.
This repo holds every form of the adapter:
| Form | Files | Runtime |
|---|---|---|
| PEFT adapter | adapter_config.json, adapter_model.safetensors, code/ |
PyTorch |
| NPU build | model.q4nx, uno.q4nx, open_kernels/, config.json, tokenizer files |
OpenFlowLM-Next on the AMD XDNA 2 NPU |
| GGUF LoRA | gguf/granite-4.2-3b-uno-f32.gguf |
None yet. See GGUF and llama.cpp. |
Speed in one paragraph. On a GPU or CPU the adapter commits 1.77 tokens per forward pass at 8 rows. On the NPU a 4-row pass costs 1.5 to 1.7 decode steps, so the drafter alone is level with plain decode on short prompts and slower on chat. OpenFlowLM-Next therefore picks a method each cycle (prompt lookup, the drafter, or a plain step). In the runs below that choice was 0.99x of plain decode at worst and 2.39x at best.
How it works
Decoding runs in cycles of two passes over the same model.
- Draft. Feed the last committed token plus Sโ1 random-noise rows. The LoRA is switched on for the noise rows only. One parallel pass proposes the next S tokens.
- Verify. Feed the proposal through the frozen base with the LoRA off. Keep the longest prefix that matches the base's greedy choice, plus the base's own token at the first miss.
The verifier is the unmodified base and acceptance is exact prefix matching. That is why the output equals plain autoregressive Granite.
TPF (tokens per forward) counts committed tokens per transformer pass, with both passes counted. It runs from 1.0 to (S+1)/2.
Results: tokens per forward
60-prompt core set (math 20, instruction-following 20, code 10, chat 10). Greedy, 256 new tokens, noise seeded per prompt. PyTorch on a GPU.
| Setting | All | Math | Code | IFEval | Chat | Max possible |
|---|---|---|---|---|---|---|
| S=4, bf16 | 1.586 | 1.783 | 1.572 | 1.509 | 1.430 | 2.5 |
| S=8, bf16 | 1.765 | 2.135 | 1.753 | 1.623 | 1.513 | 4.5 |
| S=16, bf16 | 1.807 | 2.220 | 1.800 | 1.637 | 1.559 | 8.5 |
| S=8, base fake-quantized to q4_1 | 1.763 | 2.149 | 1.727 | 1.644 | 1.478 | 4.5 |
| S=8, base fake-quantized to Q4_K (approx.) | 1.757 | 2.080 | 1.800 | 1.653 | 1.456 | 4.5 |
- Losslessness: the two-pass output equals AR greedy on all 10 checked prompts in fp32 (128 tokens each).
- In bf16, 5 of 10 diverge at a near-tie, because 8-row passes round differently from 1-row decoding. All 5 are identical in fp32.
- q4 base weights cost under 0.5% TPF at S=8. No quantization-aware phase was needed.
- S=8 is the size the adapter was trained for. Rows beyond 8 were never trained, so S=16 adds only ~2%.
- TPF is not speed. It becomes a speedup only where a multi-row pass costs about one decode step. An earlier version of this card projected โ1.65ร on the NPU from that assumption. The measured pass cost is higher, and the projection did not hold. The measured speeds are below.
Speed on the AMD XDNA 2 NPU
q4_1, 4 rows a pass, Ryzen AI 9 HX 370, quiet machine, OpenFlowLM-Next kernels of 2026-10-15 and 2026-10-16. Every run gave the same tokens as plain decode.
Plain decode is 23.4 / 22.3 / 21.4 / 19.7 tok/s after 0 / 512 / 1024 / 2048 tokens of context.
The drafter on every cycle
OFLM_SPEC_MODE=uno, 96 tokens, measured 2026-10-16.
| context | prompt | tokens a cycle | Uno | plain decode | speedup |
|---|---|---|---|---|---|
| 64 | chat | 2.91 | 21.95 tok/s | 23.0 tok/s | 0.96x |
| 512 | chat | 2.74 | 19.50 | 22.2 | 0.88x |
| 1024 | chat | 2.59 | 17.28 | 21.4 | 0.81x |
| 2048 | chat | 2.53 | 14.99 | 19.7 | 0.76x |
| 64 | code | 3.43 | 25.83 | 23.1 | 1.12x |
| 512 | code | 3.00 | 21.33 | 22.3 | 0.96x |
| 1024 | code | 2.82 | 18.82 | 21.3 | 0.89x |
| 2048 | code | 3.43 | 20.34 | 19.7 | 1.03x |
- Six short prompts (chat, code, math, reasoning): 3.02 tokens a cycle, geometric mean 1.00x.
- A cycle costs 3.2 decode steps at short context and 3.3 at 2048 tokens. Chat commits about 2.5 to 2.9 tokens a cycle, so the cycle does not pay on chat. Math and code commit more and come out ahead at short context.
The per-cycle choice (what OFLM_SPECULATIVE=1 runs)
Each cycle the engine picks the method with the most tokens per millisecond expected: a prompt-lookup pass, a Uno cycle, or a plain step. 128 tokens (256 for the last four rows), measured 2026-10-15.
| prompt | context | plain decode | per-cycle choice | speedup |
|---|---|---|---|---|
| chat | short | 23.7 tok/s | 23.49 tok/s | 0.99x |
| chat 2 | short | 23.8 | 24.11 | 1.01x |
| code | short | 23.7 | 27.94 | 1.18x |
| code 2 | short | 23.6 | 25.19 | 1.07x |
| math | short | 23.7 | 26.34 | 1.11x |
| reasoning | short | 23.6 | 26.54 | 1.12x |
| chat | 1024 | 21.0 | 20.97 | 1.00x |
| code | 1024 | 21.1 | 24.40 | 1.16x |
| chat | 2048 | 18.8 | 19.75 | 1.05x |
| code | 2048 | 18.8 | 20.67 | 1.10x |
| whole-file edit | 1347 | 20.1 | 31.43 | 1.56x |
| whole-file bug fix | 1390 | 20.1 | 48.03 | 2.39x |
| quote a document | 1032 | 20.7 | 27.88 | 1.35x |
| answer from a document | 728 | 21.4 | 32.05 | 1.49x |
- Most of this gain comes from the prompt-lookup cycles, which need no adapter. The drafter adds the most on math and code.
- This table is one day older than the table above. The drafter pass became 5% to 8% faster after it was measured.
Run on the NPU
Needs OpenFlowLM-Next main at or after PR #209
(merged 2026-11-05).
- Download the NPU files into a models directory:
hf download Cyronius/granite-4.2-3b-uno --exclude "adapter_model.safetensors" --exclude "gguf/*" --local-dir <root>/models/Granite-4.2-3B-Uno-NPU2 - Add an entry under
"models"in a copy of OpenFlowLM'smodel_list.json:"granite4.2": {"3b": { "name": "Granite-4.2-3B-Uno-NPU2", "url": "", "file_url": "", "ms_url": "", "modified_at": "2026-11-12T00:00:00Z", "oflm_min_version": "0.9.22", "files": ["config.json", "model.q4nx", "tokenizer.json", "tokenizer_config.json", "chat_template.jinja", "uno.q4nx"], "default_context_length": 8192, "max_prefill_len": 4096, "details": {"family": "granite", "think": false, "think_toggleable": false, "parameter_size": "3B", "quantization_level": "Q4_1"}, "label": [], "footprint": 3.0}} - Point oflm at both, turn speculation on, and run:
set OFLM_CONFIG_PATH=<your model_list.json> set OFLM_MODEL_PATH=<root> set OFLM_SPECULATIVE=1 oflm run granite4.2:3b
- Without
OFLM_SPECULATIVE=1the same files run as plain Granite. The load log says thatuno.q4nxis present but off. - Speculation serves greedy requests only, and Granite samples by default. Type
/set topk 1and/set rep-pen 1first. Overoflm serve, send"top_k": 1, "repetition_penalty": 1. OFLM_SPEC_MODEpins one method:uno,lookup,lookup-unoorplain.- The kernels in
open_kernels/are found beside the model.oflm adddoesn't copyuno.q4nxoropen_kernels/yet, so use the steps above.
Use in PyTorch
This needs the custom Granite SDAR model class and the two-pass sampler in code/. Install the packages in
code/requirements-droplet.txt first.
import sys
import torch
from huggingface_hub import snapshot_download
from peft import PeftModel
repo = snapshot_download("Cyronius/granite-4.2-3b-uno", allow_patterns=["adapter_*", "code/*"])
sys.path.insert(0, f"{repo}/code")
from training.constants import DEFAULT_MODEL_ID, DEFAULT_MODEL_REVISION, NOISE_HIGH
from training.lora import TokenwiseLoraRouter
from training.train import load_granite_uno
from uno_granite.sampler import uno_generate
tokenizer, base = load_granite_uno(snapshot_download(DEFAULT_MODEL_ID, revision=DEFAULT_MODEL_REVISION))
model = PeftModel.from_pretrained(base.cuda().eval(), repo, is_trainable=True).eval()
router = TokenwiseLoraRouter(model) # gates the LoRA to draft rows
ids = tokenizer.apply_chat_template([{"role": "user", "content": "What is 17 * 23?"}], add_generation_prompt=True)
out, stats = uno_generate(model, ids, block_len=8, max_new_tokens=256,
eos_token_id=tokenizer.eos_token_id, noise_high=NOISE_HIGH, router=router)
print(tokenizer.decode(out), f"TPF {stats.tpf:.2f}")
allow_patterns skips the NPU and GGUF files (about 4 GB). is_trainable=True only exposes the LoRA tensors to the
router; nothing is trained. To reproduce the TPF table, run code/scripts/eval_tpf.py.
GGUF and llama.cpp
- Base model: use IBM's own files in
ibm-granite/granite-4.2-3b-GGUF.
Q4_1is the quantization of the NPU build. - Adapter:
gguf/granite-4.2-3b-uno-f32.ggufis the adapter as a GGUF LoRA (rank 128, alpha 8192, q/k/v/o/gate/up/down of all 40 layers, F32), made with llama.cpp'sconvert_lora_to_gguf.py. - The adapter does not give Uno decoding in llama.cpp. Uno needs the adapter on the noise rows only, with the base
verifying the draft with the adapter off. llama.cpp applies a LoRA to every token and has no field for the noise ids
(uniform in
[1, 100256)). A runtime that routes the adapter per row is needed. Do not load this file as a model.
Training
| Base | ibm-granite/granite-4.2-3b @ e459acceac81e5fe67c07d9cfc72329a332e7eb1, frozen |
| Adapter | LoRA r=128, ฮฑ=8192 (ฮฑ/r 64), on q, k, v, o, gate, up, down; 249.0M parameters. The router applies it to noise rows only. |
| Objective | Total-variation distance between the drafter's and the frozen base's next-token distributions (Uno ฮณ=1). Uniform noise at rate 1.0 on all non-prompt tokens, block-diffusion attention layout. |
| Data | 80% OpenThoughts3-1.2M @ 61bcf9d + 20% tulu-3-sft-mixture @ b14afda, rendered with Granite's chat template and packed to 4,096 tokens |
| Tokens | 0.87B trained (about 0.24 of the 3.58B-token prepared mix) |
| Curriculum | block size 2 for 165 steps โ 4 for 248 โ 8 for 1,238; 1,651 steps ร 128 rows ร 4,096 tokens |
| Optimizer | AdamW, no weight decay, gradient clip 1.0, bf16 autocast. 60 warmup steps, LR 2e-5 constant, cosine decay to 0 over the last 300 steps. |
| Hardware | 1ร AMD Instinct MI350X (gfx950), ROCm 7.2, PyTorch 2.10, about 31 GPU-hours including one restart |
Note on the learning rate. The run started at 5e-5 (the setting of IFM's K2-7B adapter). It diverged a dozen steps into block size 4, so the run was restarted from the step-165 boundary at 2e-5. The decay was added at step 1,160, after the block-size-8 loss had flattened.
uno_training_manifest.json records the exact data hashes, curriculum and settings.
Limitations
- On the NPU the drafter alone does not beat plain decode on chat. It is level on short prompts and falls to 0.76x at 2048 tokens of context. Use the per-cycle choice, not
OFLM_SPEC_MODE=uno. - The NPU runs 4 rows a pass. The adapter was trained for 8. On K2-7B, an 8-row pass cost about twice the matrix work of a 4-row pass, so 8 rows are not built for Granite.
- Short training. About a quarter of one pass over the prepared data. Uno's published adapters train for one or more epochs, and the loss here was still falling.
- Chat accepts least. The data is 80% math and code reasoning.
- Greedy decoding only, in the included sampler and on the NPU. Sampling needs the standard speculative-sampling acceptance rule.
- bf16 near-ties. In PyTorch, greedy output can differ from bf16 one-token decoding wherever the top two logits are within rounding error. It is identical in fp32.
Files
| File | Description |
|---|---|
adapter_config.json, adapter_model.safetensors |
The final adapter (step 1,651), PEFT format |
uno_training_manifest.json |
Data hashes, curriculum and optimizer settings |
eval/ |
Raw results for every row of the TPF table |
code/ |
The Granite port (uno_granite/), the training code adapted from Uno (training/, Apache-2.0, see NOTICE.uno) and the eval script |
model.q4nx |
Granite 4.2 3B in OpenFlowLM's Q4NX format, q4_1, q_proj pre-scaled by 0.125 (the fold config.json records) |
uno.q4nx |
The adapter for the NPU: rank-128 LoRA on all seven projections of the 40 layers, applied to draft rows only; noise ids [1, 100256) |
open_kernels/ |
The NPU kernel set (decode, prefill, the 4-row pass and its LoRA variant, the LM heads), built on 2026-11-12 from OpenFlowLM-Next main at 96b915d (mlir-aie 1.4.2, Peano 21.0.0.2026080301, manifest_version 7) |
config.json, tokenizer.json, tokenizer_config.json, chat_template.jinja |
Runtime config, tokenizer and Granite's chat template for the NPU build |
gguf/granite-4.2-3b-uno-f32.gguf |
The adapter as a GGUF LoRA |
Reproduce the NPU build
q4nx-build -i granite-4.2-3b-bf16.gguf -s <granite-4.2-3b dir> -f granite -t language --quant Q4_1 \
--uno-adapter <this repo's adapter dir> --uno-noise-high 100256 -o Granite-4.2-3B-Uno-NPU2
python open_kernels/export_qwen36_kernels.py --model-dir Granite-4.2-3B-Uno-NPU2
The GGUF is ibm-granite/granite-4.2-3b-GGUF's bf16 file. The procedure and checks are requirement
OPEN-FAMILY-GRANITE-UNO in OpenFlowLM-Next's specs/open-engine/spec.md.
Credits
- Method and training code: Uno by IFM (Apache-2.0).
- Base model: IBM Granite 4.2 3B (Apache-2.0).
- Data: OpenThoughts3 and Tรผlu 3. See their dataset cards for licenses.
- Runtime: OpenFlowLM-Next.
- Compute: AMD Developer Cloud.
- Downloads last month
- 37
32-bit
Model tree for Cyronius/granite-4.2-3b-uno
Base model
ibm-granite/granite-4.1-3b-base