Instructions to use ghostchoir/rebuttal_w4pc_cbq with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ghostchoir/rebuttal_w4pc_cbq with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ghostchoir/rebuttal_w4pc_cbq")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ghostchoir/rebuttal_w4pc_cbq")
model = AutoModelForCausalLM.from_pretrained("ghostchoir/rebuttal_w4pc_cbq", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ghostchoir/rebuttal_w4pc_cbq with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ghostchoir/rebuttal_w4pc_cbq"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ghostchoir/rebuttal_w4pc_cbq",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ghostchoir/rebuttal_w4pc_cbq

SGLang

How to use ghostchoir/rebuttal_w4pc_cbq with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ghostchoir/rebuttal_w4pc_cbq" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ghostchoir/rebuttal_w4pc_cbq",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ghostchoir/rebuttal_w4pc_cbq" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ghostchoir/rebuttal_w4pc_cbq",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ghostchoir/rebuttal_w4pc_cbq with Docker Model Runner:
```
docker model run hf.co/ghostchoir/rebuttal_w4pc_cbq
```

Qwen3-1.7B — CBQ W4-per-channel

Qwen/Qwen3-1.7B quantized to W4-per-channel with CBQ (Cross-Block Quantization, arXiv:2312.07950), then dequantized back to bf16.

This is a dense bf16 checkpoint, not a packed low-bit one. Every weight has been round-tripped through the W4-per-channel grid, so it carries the accuracy of W4-per-channel quantization, but the file itself is an ordinary Qwen3ForCausalLM in bfloat16 with the same tensor names and shapes as the base model. It loads in vLLM or transformers with no custom kernel, no quantization_config, and no code change — and it gives no speed or memory benefit over the base model. Use it to evaluate what W4-per-channel costs in quality.

from vllm import LLM
llm = LLM("ghostchoir/rebuttal_w4pc_cbq")            # dtype is bfloat16 per config.json

Configuration


bit-width	4
granularity	per-channel (`--w_group_size -1`)
scheme	asymmetric, weight-only (activations stay bf16)
CBQ settings	paper defaults
CBD window / overlap	2 / 1
epochs per window	3
LoRA-Rounding rank	5
CFP weight truncation	`tensor`
`Delta_W` hard-round fraction	0.1
held-out perplexity	2.6328 (bf16 baseline 2.0084, delta +0.6244)
mean per-token KL(FP \|\| quant)	0.30233 over 63406 tokens

Calibration: 4096 rows of open-r1/OpenR1-Math-220k rendered with the Qwen3 chat template and truncated to 2048 tokens. This is not the paper's calibration set (the paper uses 128 samples); it is matched to a sibling QAT framework so the two are comparable.

Perplexity above is on held-out rows of the same distribution and is a sanity metric only — it is not a reasoning-benchmark score.

Provenance and caveats

Produced by an unofficial CBQ implementation written from the paper text; CBQ has no official code release. The paper is ambiguous or self-inconsistent in several places (notably the interaction of Eq. 8 and Eq. 11 for Delta_W initialisation, and the direction of the Algorithm 1 objective), and every such choice is documented with the measurement behind it. Two settings differ from the paper deliberately: group-wise rather than per-channel quantization where noted, and the calibration set above.

Not affiliated with the CBQ authors or with Qwen. Inherits the base model's license.

Downloads last month: 22

Safetensors

Model size

2B params

Tensor type

BF16

Model tree for ghostchoir/rebuttal_w4pc_cbq

Base model

Qwen/Qwen3-1.7B-Base

Finetuned

Qwen/Qwen3-1.7B

Finetuned

(931)

this model

Paper for ghostchoir/rebuttal_w4pc_cbq

CBQ: Cross-Block Quantization for Large Language Models

Paper • 2312.07950 • Published Dec 13, 2023