Instructions to use HarimxChoi/WarpQuant-Qwen3.8-27B-R16E4H4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use HarimxChoi/WarpQuant-Qwen3.8-27B-R16E4H4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="HarimxChoi/WarpQuant-Qwen3.8-27B-R16E4H4")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("HarimxChoi/WarpQuant-Qwen3.8-27B-R16E4H4")
model = AutoModelForMultimodalLM.from_pretrained("HarimxChoi/WarpQuant-Qwen3.8-27B-R16E4H4", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use HarimxChoi/WarpQuant-Qwen3.8-27B-R16E4H4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "HarimxChoi/WarpQuant-Qwen3.8-27B-R16E4H4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HarimxChoi/WarpQuant-Qwen3.8-27B-R16E4H4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/HarimxChoi/WarpQuant-Qwen3.8-27B-R16E4H4

SGLang

How to use HarimxChoi/WarpQuant-Qwen3.8-27B-R16E4H4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "HarimxChoi/WarpQuant-Qwen3.8-27B-R16E4H4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HarimxChoi/WarpQuant-Qwen3.8-27B-R16E4H4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "HarimxChoi/WarpQuant-Qwen3.8-27B-R16E4H4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "HarimxChoi/WarpQuant-Qwen3.8-27B-R16E4H4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use HarimxChoi/WarpQuant-Qwen3.8-27B-R16E4H4 with Docker Model Runner:
```
docker model run hf.co/HarimxChoi/WarpQuant-Qwen3.8-27B-R16E4H4
```

WarpQuant-Qwen3.8-27B-R16E4H4 / warpquant_weak_columns.json

Harim Choi

Add files using upload-large-folder tool

f9e0eab verified 1 day ago

Raw

History Blame Contribute Delete

3 kB

	{
	"source": "/workspace/warpquant-qwen38-27b-20260815/source-bf16",
	"base": "/workspace/warpquant-qwen38-27b-20260815/base-3.50",
	"selection": "/workspace/warpquant-qwen38-27b-20260815/results/output-fisher-selection.json",
	"selection_sha256": "2a249ad8626308fef8f7cdbf85c49fac3438e8eb9b35a5639f62c3fae6eea3bb",
	"result_index": 1,
	"domain": "original",
	"base_selected_bpw": 3.5,
	"selected_values": 24326963200,
	"selected_average_bpw": 3.5499997474407325,
	"recovery_bits": 16,
	"group_size": 128,
	"recovery_scales": 0,
	"index_bits": 436224,
	"tensor_count": 368,
	"column_count": 13632,
	"corrected_values": 75994112,
	"auxiliary_bits": null,
	"embedding_bits": 4,
	"embedding_values": 1271398400,
	"embedding_scales": 9932800,
	"visual_bits": null,
	"visual_values": 0,
	"visual_scales": 0,
	"output_head_bits": 4,
	"output_head_values": 1271398400,
	"output_head_scales": 9932800,
	"remaining_bf16_values": 911667952,
	"total_values": 27781427952,
	"analytical_source_bpw": 4.011184646107352,
	"shard_sha256": {
	"model-00001-of-00018.safetensors": "42f3142024a7cdad6e987f368d2c020852c0e86e974f2f09d80bf5644c726d66",
	"model-00002-of-00018.safetensors": "a5e49f9221dbee5cf5350aa52fc722552f99bdeb8cb5bfa84fd757c5255b837d",
	"model-00003-of-00018.safetensors": "02d4a6140925795532a7471ee21c734bcb179cdd40745418f0016f79104ca4af",
	"model-00004-of-00018.safetensors": "396339f82b8bf05f68ab5db4d52df75c6ac76df1fa4ac1aab5bd31ad74ede7a4",
	"model-00005-of-00018.safetensors": "5b25d8b200e1a475830350bcbb3f9f41d81d81443f4f6f039e15ff4978284c6c",
	"model-00006-of-00018.safetensors": "d571d9f6738b87fd2c650ebb2d8cda878225259478a38ced2a8d332fb54100da",
	"model-00007-of-00018.safetensors": "fd6ed68be4d09828e260fd9d0254749e124a752992a10acee6befa5f384f6012",
	"model-00008-of-00018.safetensors": "ee82f733390aeab5eb33d11c1097661fd22c76c6fa1ded5f86f779309df04bf4",
	"model-00009-of-00018.safetensors": "2e598c54cd265acf68bbfa59abd25d8b09e6ad47d319f86c8b820e932e169810",
	"model-00010-of-00018.safetensors": "b42590fcb53bbb9eda7144e0784307ec30b36e8a9691103a5b319eb67c494260",
	"model-00011-of-00018.safetensors": "cf7547f9c6514db9592629dbee0b35c1933db23f3ce6bd294cd72355c86f1a64",
	"model-00012-of-00018.safetensors": "ce4feeb18988b1016a4ee0a1e6b4fa402aca94e9b3f75f01a68ba01a13cbb662",
	"model-00013-of-00018.safetensors": "48c32c3abc0d6bd5c2e22ac1c869e8d3b094efab26d69a5a9857ef26f6a74a6c",
	"model-00014-of-00018.safetensors": "e5c22aeb3fa710bcccd9d27dc049711d5867fecb54a62e7eabc7fc12616beef0",
	"model-00015-of-00018.safetensors": "653a37070687d2e432ab08d74b18390b5956855559909ec6a83ff827f51611ae",
	"model-00016-of-00018.safetensors": "0f69845b8f02054e1a1e3f41106edf50bb8e198918d14292637810adcf8cb558",
	"model-00017-of-00018.safetensors": "8d0532eef36ffa01d2f67dded0c709ce1e999d4999ef46630a74abb06c50e099",
	"model-00018-of-00018.safetensors": "82acdc59d0c90c309ce18107ff28dfc1b7580607897e81277f68b0f163ca92fd"
	}
	}