Instructions to use born2bewild/Qwen3.8-27B-W4A16-AutoRound-fast with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use born2bewild/Qwen3.8-27B-W4A16-AutoRound-fast with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="born2bewild/Qwen3.8-27B-W4A16-AutoRound-fast")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("born2bewild/Qwen3.8-27B-W4A16-AutoRound-fast")
model = AutoModelForMultimodalLM.from_pretrained("born2bewild/Qwen3.8-27B-W4A16-AutoRound-fast", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use born2bewild/Qwen3.8-27B-W4A16-AutoRound-fast with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "born2bewild/Qwen3.8-27B-W4A16-AutoRound-fast"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "born2bewild/Qwen3.8-27B-W4A16-AutoRound-fast",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/born2bewild/Qwen3.8-27B-W4A16-AutoRound-fast

SGLang

How to use born2bewild/Qwen3.8-27B-W4A16-AutoRound-fast with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "born2bewild/Qwen3.8-27B-W4A16-AutoRound-fast" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "born2bewild/Qwen3.8-27B-W4A16-AutoRound-fast",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "born2bewild/Qwen3.8-27B-W4A16-AutoRound-fast" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "born2bewild/Qwen3.8-27B-W4A16-AutoRound-fast",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use born2bewild/Qwen3.8-27B-W4A16-AutoRound-fast with Docker Model Runner:
```
docker model run hf.co/born2bewild/Qwen3.8-27B-W4A16-AutoRound-fast
```

Qwen3.8-27B-W4A16-AutoRound-fast

A complete, single-repository serving checkpoint for Qwen3.8-27B on Ampere GPUs. This is the assembled form of the syvai/qwen3.8-27b-3090-fast-variant companion package over a W4A16 AutoRound base, so users do not need a second model download or hardlink/overlay step.

Provenance

Base model: Qwen/Qwen3.8-27B.
Quantization: AutoRound-style symmetric INT4, group size 128, packaged as compressed-tensors (pack-quantized).
Companion optimization: syvai/qwen3.8-27b-3090-fast-variant.
External drafter used in the validated profile: syvai/Qwen3.8-27B-DFlash2-W4A16.

The fast variant keeps the GatedDeltaNet recurrence-control projections in BF16 and uses the companion's optimized INT4 MTP/lm-head tensors plus INT8 token embeddings. The vision tower remains higher precision. This repository contains all model shards, configuration, tokenizer, MTP extra tensors, and mtp_draft_vocab_ids.pt required by from_pretrained(); it is not an official Qwen release.

vLLM

This checkpoint is intended for vLLM with the DFlash2 backport and the Qwen3.8 compressed-tensors support. The Club3090 DFLASH15 profile uses:

TP=2, RTX 3090-class Ampere;
W4A16 weights, BF16 KV, FlashAttention;
SPEC_N=15, lookup-augmented DFlash2, and single-stream serving;
GPU_MEMORY_UTILIZATION=0.85 on WSL2/PCIe rigs.

The DFlash2 drafter is a separate checkpoint and is not embedded in this repository.

Local evidence

On one 2x RTX 3090 WSL2 rig, the target plus DFLASH15 measured approximately 105.94 narrative and 188.06 code decode tokens/s, with a validated 244,320-token ceiling. These are rig-specific measurements, not a universal performance claim. The exact launcher, patch pins, benchmark logs, and caveats are maintained in the Club3090 PR.

License and attribution

Apache-2.0. Redistribution must preserve this notice and the upstream Qwen and AutoRound attributions. Quantization and serving artifacts are provided as-is; please validate quality and hardware compatibility for your workload.

Downloads last month: 113

Safetensors

Model size

6B params

Tensor type

I32

BF16

Model tree for born2bewild/Qwen3.8-27B-W4A16-AutoRound-fast

Base model

Qwen/Qwen3.8-27B

Quantized

(760)

this model