Instructions to use devpramod-intel/granite-4.1-8b-quantized.w8a8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use devpramod-intel/granite-4.1-8b-quantized.w8a8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="devpramod-intel/granite-4.1-8b-quantized.w8a8")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("devpramod-intel/granite-4.1-8b-quantized.w8a8")
model = AutoModelForCausalLM.from_pretrained("devpramod-intel/granite-4.1-8b-quantized.w8a8", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use devpramod-intel/granite-4.1-8b-quantized.w8a8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "devpramod-intel/granite-4.1-8b-quantized.w8a8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "devpramod-intel/granite-4.1-8b-quantized.w8a8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/devpramod-intel/granite-4.1-8b-quantized.w8a8

SGLang

How to use devpramod-intel/granite-4.1-8b-quantized.w8a8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "devpramod-intel/granite-4.1-8b-quantized.w8a8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "devpramod-intel/granite-4.1-8b-quantized.w8a8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "devpramod-intel/granite-4.1-8b-quantized.w8a8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "devpramod-intel/granite-4.1-8b-quantized.w8a8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use devpramod-intel/granite-4.1-8b-quantized.w8a8 with Docker Model Runner:
```
docker model run hf.co/devpramod-intel/granite-4.1-8b-quantized.w8a8
```

granite-4.1-8b-quantized.w8a8 / README.md

devpramod-intel

Update model card

050ff66 verified 7 days ago

preview code

Raw

History Blame Contribute Delete

6.34 kB

	---
	license: apache-2.0
	license_link: https://www.apache.org/licenses/LICENSE-2.0
	language:
	- en
	base_model:
	- ibm-granite/granite-4.1-8b
	pipeline_tag: text-generation
	library_name: transformers
	tags:
	- w8a8
	- int8
	- vllm
	- compressed-tensors
	- llm-compressor
	- granite
	---

	# granite-4.1-8b-quantized.w8a8

	INT8 (W8A8) `compressed-tensors` quantization of
	[ibm-granite/granite-4.1-8b](https://huggingface.co/ibm-granite/granite-4.1-8b).

	- Weights: INT8, symmetric, per-channel
	- Activations: INT8, symmetric, dynamic per-token
	- Scope: only `Linear` layers inside the transformer blocks; `lm_head` is
	left in BF16 (the base model has `tie_word_embeddings: true`, so quantizing it
	would also perturb the input embedding)
	- Method: post-training, one-shot SmoothQuant → GPTQ via
	[llm-compressor](https://github.com/vllm-project/llm-compressor)
	- Size: 8.96 GiB on disk. The linear weights halve; the tied
	embedding matrix, the norms and `lm_head` stay BF16, so the whole-checkpoint
	saving is smaller than 2× (and smaller the smaller the model, since the
	100k-entry vocab is a larger share of it)
	- Tooling: llmcompressor 0.9.0.4, compressed-tensors 0.13.0,
	transformers 4.57.3

	> Purpose. This checkpoint was produced for **inference-performance
	> benchmarking (INT8/AMX on Xeon and INT8 kernels on GPU). No accuracy
	> evaluation was run on it** — see [Accuracy](#accuracy) before using it for
	> anything where quality matters.

	## Deployment with vLLM

	```bash
	vllm serve devpramod-intel/granite-4.1-8b-quantized.w8a8 --max-model-len 32768
	```

	```python
	from vllm import LLM, SamplingParams
	from transformers import AutoTokenizer

	model_id = "devpramod-intel/granite-4.1-8b-quantized.w8a8"
	tokenizer = AutoTokenizer.from_pretrained(model_id)
	llm = LLM(model=model_id, max_model_len=4096)

	prompt = tokenizer.apply_chat_template(
	[{"role": "user", "content": "Who are you? Please respond in pirate speak!"}],
	tokenize=False, add_generation_prompt=True,
	)
	print(llm.generate(prompt, SamplingParams(temperature=0.3, max_tokens=256))[0].outputs[0].text)
	```

	## Creation

	```bash
	python quantize_w8a8_granite41.py \
	--model-dir ibm-granite/granite-4.1-8b \
	--out granite-4.1-8b-quantized.w8a8 \
	--smoothing-strength 0.8 --dampening-frac 0.1 \
	--observer mse --num-samples 512
	```

	Recipe:

	```yaml
	quant_stage:
	quant_modifiers:
	SmoothQuantModifier:
	smoothing_strength: 0.8
	ignore: [lm_head]
	mappings:
	- - ['re:.q_proj', 're:.k_proj', 're:.*v_proj']
	- re:.*input_layernorm
	- - ['re:.gate_proj', 're:.up_proj']
	- re:.*post_attention_layernorm
	- - ['re:.*down_proj']
	- re:.*up_proj
	GPTQModifier:
	targets: [Linear]
	ignore: [lm_head]
	scheme: W8A8
	dampening_frac: 0.1
	weight_observer: mse
	sequential_targets: [GraniteDecoderLayer]
	```

	`recipe.yaml` in this repo is what llm-compressor actually applied and is
	authoritative. It additionally shows `block_size: 128` and `actorder: static`,
	which are llm-compressor 0.9.0.4 defaults rather than choices — the older
	Granite cards predate `actorder` defaulting on, so this checkpoint is not
	bit-identical to what their recipe produced in 2025.

	Calibration: `neuralmagic/LLM_compression_calibration`, `train` split,
	`shuffle(seed=42).select(512)`, the dataset's raw `text` field with
	`add_special_tokens=True`, `max_seq_length=8192`.

	## Recipe provenance

	Every knob is taken from Red Hat AI's published `recipe.yaml` files for the
	nearest architectural precedents — `ibm-granite/granite-4.1-8b` is a dense
	`GraniteForCausalLM` with Llama-style blocks (q/k/v + gate/up/down, RMSNorm), so
	the Granite 3.1 W8A8 recipes transfer directly.

	\| Precedent \| Relationship \| Knobs it contributes \|
	\|---\|---\|---\|
	\| [RedHatAI/granite-3.1-8b-instruct-quantized.w8a8](https://huggingface.co/RedHatAI/granite-3.1-8b-instruct-quantized.w8a8) \| same family, same class, same size class \| `smoothing_strength=0.8`, llama mappings, `dampening_frac=0.1`, weight observer `mse`, INT8 channel-weight / token-dynamic-activation config group \|
	\| [RedHatAI/granite-3.1-2b-instruct-quantized.w8a8](https://huggingface.co/RedHatAI/granite-3.1-2b-instruct-quantized.w8a8) \| smaller sibling \| confirms the same structure at small scale (it uses 0.7 / 0.01) \|
	\| [RedHatAI/granite-4.1-8b-fp8](https://huggingface.co/RedHatAI/granite-4.1-8b-fp8) \| Red Hat's own quantization of this generation \| confirms `targets=[Linear]`, `ignore=[lm_head]` is the whole story for granite-4.1 — no MoE/vision special-casing \|

	Deliberate deviations from those cards:

	- 512 calibration samples instead of the Granite cards' 3072 — W8A8 is far
	less calibration-sensitive than W4A16, and 512 is the llm-compressor default.
	- `max_seq_length=8192`, not the `8196` printed on the Granite cards (a typo).
	- `sequential_targets` set to the decoder-layer class, following current
	Red Hat cards; it lowers peak VRAM and does not change the result.

	## Accuracy

	No accuracy benchmark was run on this checkpoint. It exists to measure
	throughput and latency. The figures below are estimates by precedent, not
	measurements of this model, and should not be quoted as such:

	\| Evidence \| Measured recovery vs BF16 \|
	\|---\|---\|
	\| `granite-3.1-8b-instruct` W8A8, identical recipe (Red Hat card) \| OpenLLM v1 99.95% (70.26 vs 70.30), OpenLLM v2 98.64%, HumanEval 99.3% \|
	\| `granite-3.1-2b-instruct` W8A8 (Red Hat card) \| OpenLLM v1 99.52% (61.68 vs 61.98) \|
	\| a granite-4.1-8b derivative quantized with this exact script (internal, 7-dataset classification basket) \| aggregate ≈99.4%, 46/48 byte-identical decodes on CPU \|

	On that basis the expected recovery here is **~99–100% on knowledge/reasoning
	multiple-choice suites and ~98–99% on generative suites**. If you need a number
	you can defend, run `lm-eval` against both this checkpoint and the BF16 base and
	report the ratio.

	## Verification performed

	- `config.json` → `quantization_config`: `format: int-quantized`, weights
	`num_bits 8 / channel / symmetric / observer mse`, input activations
	`num_bits 8 / token / dynamic`, `ignore: ["lm_head"]`
	- all quantized weights and scales checked finite (no NaN/Inf)
	- checkpoint loads and generates coherent text