README.md

---
license: apache-2.0
license_link: https://www.apache.org/licenses/LICENSE-2.0
language:
- en
base_model:
- ibm-granite/granite-4.1-8b
pipeline_tag: text-generation
library_name: transformers
tags:
- w8a8
- int8
- vllm
- compressed-tensors
- llm-compressor
- granite
---

# granite-4.1-8b-quantized.w8a8

INT8 (W8A8) `compressed-tensors` quantization of
[ibm-granite/granite-4.1-8b](https://huggingface.co/ibm-granite/granite-4.1-8b).

- **Weights:** INT8, symmetric, **per-channel**
- **Activations:** INT8, symmetric, **dynamic per-token**
- **Scope:** only `Linear` layers inside the transformer blocks; `lm_head` is
  left in BF16 (the base model has `tie_word_embeddings: true`, so quantizing it
  would also perturb the input embedding)
- **Method:** post-training, one-shot SmoothQuant → GPTQ via
  [llm-compressor](https://github.com/vllm-project/llm-compressor)
- **Size:** 8.96 GiB on disk. The linear weights halve; the tied
  embedding matrix, the norms and `lm_head` stay BF16, so the whole-checkpoint
  saving is smaller than 2× (and smaller the smaller the model, since the
  100k-entry vocab is a larger share of it)
- **Tooling:** llmcompressor 0.9.0.4, compressed-tensors 0.13.0,
  transformers 4.57.3

> **Purpose.** This checkpoint was produced for **inference-performance
> benchmarking** (INT8/AMX on Xeon and INT8 kernels on GPU). **No accuracy
> evaluation was run on it** — see [Accuracy](#accuracy) before using it for
> anything where quality matters.

## Deployment with vLLM

```bash
vllm serve devpramod-intel/granite-4.1-8b-quantized.w8a8 --max-model-len 32768
```

```python
from vllm import LLM, SamplingParams
from transformers import AutoTokenizer

model_id = "devpramod-intel/granite-4.1-8b-quantized.w8a8"
tokenizer = AutoTokenizer.from_pretrained(model_id)
llm = LLM(model=model_id, max_model_len=4096)

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Who are you? Please respond in pirate speak!"}],
    tokenize=False, add_generation_prompt=True,
)
print(llm.generate(prompt, SamplingParams(temperature=0.3, max_tokens=256))[0].outputs[0].text)
```

## Creation

```bash
python quantize_w8a8_granite41.py \
  --model-dir ibm-granite/granite-4.1-8b \
  --out granite-4.1-8b-quantized.w8a8 \
  --smoothing-strength 0.8 --dampening-frac 0.1 \
  --observer mse --num-samples 512
```

Recipe:

```yaml
quant_stage:
  quant_modifiers:
    SmoothQuantModifier:
      smoothing_strength: 0.8
      ignore: [lm_head]
      mappings:
      - - ['re:.*q_proj', 're:.*k_proj', 're:.*v_proj']
        - re:.*input_layernorm
      - - ['re:.*gate_proj', 're:.*up_proj']
        - re:.*post_attention_layernorm
      - - ['re:.*down_proj']
        - re:.*up_proj
    GPTQModifier:
      targets: [Linear]
      ignore: [lm_head]
      scheme: W8A8
      dampening_frac: 0.1
      weight_observer: mse
      sequential_targets: [GraniteDecoderLayer]
```

`recipe.yaml` in this repo is what llm-compressor actually applied and is
authoritative. It additionally shows `block_size: 128` and `actorder: static`,
which are llm-compressor 0.9.0.4 defaults rather than choices — the older
Granite cards predate `actorder` defaulting on, so this checkpoint is not
bit-identical to what their recipe produced in 2025.

Calibration: `neuralmagic/LLM_compression_calibration`, `train` split,
`shuffle(seed=42).select(512)`, the dataset's raw `text` field with
`add_special_tokens=True`, `max_seq_length=8192`.

## Recipe provenance

Every knob is taken from Red Hat AI's published `recipe.yaml` files for the
nearest architectural precedents — `ibm-granite/granite-4.1-8b` is a dense
`GraniteForCausalLM` with Llama-style blocks (q/k/v + gate/up/down, RMSNorm), so
the Granite 3.1 W8A8 recipes transfer directly.

| Precedent | Relationship | Knobs it contributes |
|---|---|---|
| [RedHatAI/granite-3.1-8b-instruct-quantized.w8a8](https://huggingface.co/RedHatAI/granite-3.1-8b-instruct-quantized.w8a8) | same family, same class, same size class | `smoothing_strength=0.8`, llama mappings, `dampening_frac=0.1`, weight observer `mse`, INT8 channel-weight / token-dynamic-activation config group |
| [RedHatAI/granite-3.1-2b-instruct-quantized.w8a8](https://huggingface.co/RedHatAI/granite-3.1-2b-instruct-quantized.w8a8) | smaller sibling | confirms the same structure at small scale (it uses 0.7 / 0.01) |
| [RedHatAI/granite-4.1-8b-fp8](https://huggingface.co/RedHatAI/granite-4.1-8b-fp8) | Red Hat's own quantization of this generation | confirms `targets=[Linear]`, `ignore=[lm_head]` is the whole story for granite-4.1 — no MoE/vision special-casing |

Deliberate deviations from those cards:

- **512 calibration samples** instead of the Granite cards' 3072 — W8A8 is far
  less calibration-sensitive than W4A16, and 512 is the llm-compressor default.
- **`max_seq_length=8192`**, not the `8196` printed on the Granite cards (a typo).
- **`sequential_targets` set** to the decoder-layer class, following current
  Red Hat cards; it lowers peak VRAM and does not change the result.

## Accuracy

**No accuracy benchmark was run on this checkpoint.** It exists to measure
throughput and latency. The figures below are *estimates by precedent*, not
measurements of this model, and should not be quoted as such:

| Evidence | Measured recovery vs BF16 |
|---|---|
| `granite-3.1-8b-instruct` W8A8, identical recipe (Red Hat card) | OpenLLM v1 **99.95%** (70.26 vs 70.30), OpenLLM v2 98.64%, HumanEval 99.3% |
| `granite-3.1-2b-instruct` W8A8 (Red Hat card) | OpenLLM v1 **99.52%** (61.68 vs 61.98) |
| a granite-**4.1**-8b derivative quantized with this exact script (internal, 7-dataset classification basket) | aggregate ≈**99.4%**, 46/48 byte-identical decodes on CPU |

On that basis the expected recovery here is **~99–100% on knowledge/reasoning
multiple-choice suites and ~98–99% on generative suites**. If you need a number
you can defend, run `lm-eval` against both this checkpoint and the BF16 base and
report the ratio.

## Verification performed

- `config.json` → `quantization_config`: `format: int-quantized`, weights
  `num_bits 8 / channel / symmetric / observer mse`, input activations
  `num_bits 8 / token / dynamic`, `ignore: ["lm_head"]`
- all quantized weights and scales checked finite (no NaN/Inf)
- checkpoint loads and generates coherent text