Instructions to use chris320211/smoothquant-gemma2-2b-g52xlarge with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use chris320211/smoothquant-gemma2-2b-g52xlarge with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="chris320211/smoothquant-gemma2-2b-g52xlarge")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("chris320211/smoothquant-gemma2-2b-g52xlarge")
model = AutoModelForCausalLM.from_pretrained("chris320211/smoothquant-gemma2-2b-g52xlarge", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use chris320211/smoothquant-gemma2-2b-g52xlarge with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "chris320211/smoothquant-gemma2-2b-g52xlarge"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "chris320211/smoothquant-gemma2-2b-g52xlarge",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/chris320211/smoothquant-gemma2-2b-g52xlarge

SGLang

How to use chris320211/smoothquant-gemma2-2b-g52xlarge with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "chris320211/smoothquant-gemma2-2b-g52xlarge" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "chris320211/smoothquant-gemma2-2b-g52xlarge",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "chris320211/smoothquant-gemma2-2b-g52xlarge" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "chris320211/smoothquant-gemma2-2b-g52xlarge",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use chris320211/smoothquant-gemma2-2b-g52xlarge with Docker Model Runner:
```
docker model run hf.co/chris320211/smoothquant-gemma2-2b-g52xlarge
```

SmoothQuant google/gemma-2-2b-it

Quantized derivative of google/gemma-2-2b-it, made with QuantPort. Not Hub fp16. Weights are the packed/runtime artifact from job 20260921T132955Z-21a810.

WikiText-2 (test, 2048-token windows, 65504 tokens)

Measured on NVIDIA A10G (g5.2xlarge) against the original fp16 snapshot. Numbers copied from jobs/20260921T132955Z-21a810/benchmark.json.

	Quantized	fp16 snapshot
Perplexity	15.128110	14.809264
NLL loss	2.716555	2.695253
Tokens/s (2048 prefill, after warmup)	6322.2	5413.0
Peak VRAM (GB)	8.090	9.973

ppl_ratio: 1.021530 (quality_ok=True)
improved_throughput: True
improved_vram: True

Pull

from huggingface_hub import snapshot_download
path = snapshot_download("chris320211/smoothquant-gemma2-2b-g52xlarge")

This is not a drop-in AutoModelForCausalLM.from_pretrained checkpoint. Reload with the included quant_agent_inference_adapter.py after the method repo (and overlay, if any) is on QUANT_AGENT_METHOD_REPO. See quantization_config.json.

Serve with vLLM

A native vLLM build of these weights: chris320211/smoothquant-gemma2-2b-g52xlarge-vllm.

vllm serve chris320211/smoothquant-gemma2-2b-g52xlarge-vllm

vLLM PPL ratio 1.0285 against vLLM bf16; prefill 2.02×, decode 1.41× (batch 1), 1.39× (batch 16). Measured on NVIDIA GeForce RTX 3090 with vLLM 0.30.0.

vLLM results

Measured on NVIDIA GeForce RTX 3090: the vLLM build against vLLM bf16, and this repo's weights in plain Transformers against the original model.

	vLLM int8	vLLM bf16	Transformers int8	Transformers bf16
Perplexity	15.2446	14.8218	15.1281	14.8219
Prefill tok/s	30696.7	15225.4	13792.6	9814.8
Decode tok/s, batch 1	176.0	125.0	5.0	9.5
Decode tok/s, batch 16	2332.5	1677.2	75.5	149.8
Peak VRAM GB	–	–	8.09	9.97

Transformers columns: the native artifact chris320211/smoothquant-gemma2-2b-g52xlarge through its inference adapter (in fp16, as benchmarked), and google/gemma-2-2b-it in bf16, in Transformers 4.46.3 (torch 2.7.1+cu128) on the same GPU. Transformers prefills 8 windows per forward; vLLM schedules all 32.
Against Transformers bf16: vLLM int8 prefill 3.13×, decode 18.62× at batch 1, 15.57× at batch 16. The native artifact in Transformers: prefill 1.41×, decode 0.53× at batch 1, 0.50× at batch 16.

For serving, use chris320211/smoothquant-gemma2-2b-g52xlarge-vllm. In plain Transformers this artifact decodes slower than bf16; its gain there is memory.

License

Inherits the license of google/gemma-2-2b-it (gemma), plus the method repository license. Keep LICENSE / NOTICE.md from the snapshot when present.