Instructions to use bullpoint/GLM-4.6-AWQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use bullpoint/GLM-4.6-AWQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="bullpoint/GLM-4.6-AWQ")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("bullpoint/GLM-4.6-AWQ")
model = AutoModelForCausalLM.from_pretrained("bullpoint/GLM-4.6-AWQ")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use bullpoint/GLM-4.6-AWQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "bullpoint/GLM-4.6-AWQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bullpoint/GLM-4.6-AWQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/bullpoint/GLM-4.6-AWQ

SGLang

How to use bullpoint/GLM-4.6-AWQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "bullpoint/GLM-4.6-AWQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bullpoint/GLM-4.6-AWQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "bullpoint/GLM-4.6-AWQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bullpoint/GLM-4.6-AWQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use bullpoint/GLM-4.6-AWQ with Docker Model Runner:
```
docker model run hf.co/bullpoint/GLM-4.6-AWQ
```

endless response

by ramidahbash - opened Dec 15, 2025

Discussion

ramidahbash

Dec 15, 2025

I have tried this AWQ version.
I deployed it using vllm 0.10.2 and 4 H100 GPUs and the response never ends, it looks like he in a conversation with itself so the response is the a question to himself and he answer it in a never ending loop.
Setting the temperature to 1.0 doesn't help.

bullpoint

Owner Dec 15, 2025

Does every prompt result in endless looping?

ramidahbash

Dec 15, 2025

yes

bullpoint

Owner Dec 15, 2025

Could you give me a sample prompt I could try to see what happens on my machine? It does not happen for any prompt I give.

ramidahbash

Dec 15, 2025

for every prompt I give, even "Hello how are you?", it happens.
is it deployed locally on your machine with vLLM?

bullpoint

Owner Dec 15, 2025

Yes -- with 4xRTX PRO 6000 -- so blackwell instead of hopper.

Here's my docker-compose.yaml using vllm's nightly. I just pulled it today, so maybe you could give the same a try and see if any errors when it starts up?

services:
  inference:
    image: vllm/vllm-openai:nightly
    container_name: inference
    privileged: true
    userns_mode: host
    ipc: host
    shm_size: "32gb"
    ulimits:
      memlock: -1
      stack: 67108864
    ports:
      - "0.0.0.0:8000:8000"
    deploy:
      resources:
        limits:
          memory: 32g
          cpus: '32'
        reservations:
          memory: 32g
          cpus: '32'
          devices:
            - driver: nvidia
              count: -1
              capabilities: [gpu]
    environment:
      - NVIDIA_VISIBLE_DEVICES=all
      - NVIDIA_DRIVER_CAPABILITIES=compute,utility
      - CUDA_LAUNCH_BLOCKING=0
      - NCCL_IB_DISABLE=1
      - NCCL_NVLS_ENABLE=0
      - NCCL_P2P_DISABLE=0
      - NCCL_SHM_DISABLE=0
      - VLLM_USE_V1=0
      - OMP_NUM_THREADS=8
      - TORCH_FLOAT32_MATMUL_PRECISION=high
    volumes:
      - /path/to/GLM-4.6-AWQ:/models/GLM-4.6-AWQ:ro
    entrypoint: ["/bin/bash", "-c"]
    command:
      - |
        # Run vLLM serve - TP=4, NO expert parallelism (for single-request speed)
        exec vllm serve /models/GLM-4.6-AWQ \
          --tensor-parallel-size 4 \
          --attention-backend FLASHINFER \
          --max-num-batched-tokens 16384 \
          --max-num-seqs 1 \
          --served-model-name GLM-4.6-AWQ \
          --enable-auto-tool-choice \
          --tool-call-parser glm45 \
          --reasoning-parser glm45 \
          --host 0.0.0.0 \
          --port 8000

ramidahbash

Dec 16, 2025

I configured my vllm the same as you, and it still doing this.

what version of vllm are you using? i tried 0.11.2 as well.

bullpoint

Owner Dec 16, 2025

I'm using vllm nightly, but v0.12 also works. Have you tried some other GLM 4.6 quants? https://huggingface.co/cyankiwi/GLM-4.6-AWQ-4bit is a good one.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment