Instructions to use QuantTrio/Qwen3-Coder-30B-A3B-Instruct-AWQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use QuantTrio/Qwen3-Coder-30B-A3B-Instruct-AWQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="QuantTrio/Qwen3-Coder-30B-A3B-Instruct-AWQ")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("QuantTrio/Qwen3-Coder-30B-A3B-Instruct-AWQ")
model = AutoModelForCausalLM.from_pretrained("QuantTrio/Qwen3-Coder-30B-A3B-Instruct-AWQ", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use QuantTrio/Qwen3-Coder-30B-A3B-Instruct-AWQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "QuantTrio/Qwen3-Coder-30B-A3B-Instruct-AWQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantTrio/Qwen3-Coder-30B-A3B-Instruct-AWQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/QuantTrio/Qwen3-Coder-30B-A3B-Instruct-AWQ

SGLang

How to use QuantTrio/Qwen3-Coder-30B-A3B-Instruct-AWQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "QuantTrio/Qwen3-Coder-30B-A3B-Instruct-AWQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantTrio/Qwen3-Coder-30B-A3B-Instruct-AWQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "QuantTrio/Qwen3-Coder-30B-A3B-Instruct-AWQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "QuantTrio/Qwen3-Coder-30B-A3B-Instruct-AWQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use QuantTrio/Qwen3-Coder-30B-A3B-Instruct-AWQ with Docker Model Runner:
```
docker model run hf.co/QuantTrio/Qwen3-Coder-30B-A3B-Instruct-AWQ
```

Independent Δbpb measurement against the bf16 original (0.05363, 95% CI [0.04071, 0.06796])

by yuuji-dev - opened 1 day ago

Discussion

yuuji-dev

1 day ago

An independent Δbpb measurement against the bf16 original

We measured this checkpoint against the original
Qwen/Qwen3-Coder-30B-A3B-Instruct (bf16) under identical conditions and
published the result as a public record.

Δbpb 0.05363, 95% CI [0.04071, 0.06796] (interval excludes zero)

Measured revision: c58857a7f41c0920f73d1b56678640f9c02017d7
Original: Qwen/Qwen3-Coder-30B-A3B-Instruct, revision b2cff646eb4bb1d68355c01b18ae02e7cf42d120, bf16
54,219 tokens, 53 sequences × 1,023, float16, H100
Paired bootstrap resampled at the sequence level, 10,000 iterations, seed 42

Δbpb is the difference in bits per byte required for the same text. A larger
value means the quantized version needs more bits, i.e. average prediction has
degraded.

The README warns that this model suffers significant loss under 4-bit
quantization. This measurement puts a number on that warning. It is not a
criticism of the publisher, who states the caveat openly and releases under
Apache-2.0.

Record: https://linea.takara-app.ai/records/no1/ (EN: /records/no1/en/, PDF available)

What this does not establish. This is one model, one quantized version, one
evaluation dataset. It says nothing about other models, other quantization
methods, or other evaluation data. It is not a comparison of quantization
methods. Our evaluation corpus is not public, so a third party cannot reproduce
the identical figure at present; its sha256 is recorded so that identity can be
confirmed if we publish it later. This record is not a certification,
accreditation, or conformity assessment by any third-party body.

We also measured QuantTrio/Qwen3-30B-A3B-Thinking-2507-AWQ from the same
publisher under identical settings and the same evaluation data. Its Δbpb is
0.01882 [0.01523, 0.02265] — the intervals do not overlap. The only variable
that differs is the model. So the size of the gap appears to depend on the
model, not on the quantization settings.
https://linea.takara-app.ai/records/no2/

— Treasure Planning LLC. We measure how compression changes MoE model quality
and record the conditions and scope of that measurement.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment