Instructions to use prashantcp8/Qwen3.5-4B-INT8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use prashantcp8/Qwen3.5-4B-INT8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="prashantcp8/Qwen3.5-4B-INT8")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("prashantcp8/Qwen3.5-4B-INT8")
model = AutoModelForCausalLM.from_pretrained("prashantcp8/Qwen3.5-4B-INT8")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use prashantcp8/Qwen3.5-4B-INT8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "prashantcp8/Qwen3.5-4B-INT8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "prashantcp8/Qwen3.5-4B-INT8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/prashantcp8/Qwen3.5-4B-INT8

SGLang

How to use prashantcp8/Qwen3.5-4B-INT8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "prashantcp8/Qwen3.5-4B-INT8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "prashantcp8/Qwen3.5-4B-INT8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "prashantcp8/Qwen3.5-4B-INT8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "prashantcp8/Qwen3.5-4B-INT8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use prashantcp8/Qwen3.5-4B-INT8 with Docker Model Runner:
```
docker model run hf.co/prashantcp8/Qwen3.5-4B-INT8
```

Qwen3.5-4B-INT8

Author: Prashant Takale

Model Description

This is an INT8 quantized version of Qwen/Qwen3.5-4B using bitsandbytes LLM.int8() quantization.

What I Did

Quantized the original Qwen3.5-4B model from BF16 to INT8 precision using bitsandbytes
Reduced memory footprint by 42.4% while maintaining model quality
Benchmarked on WikiText-2 (perplexity) and GSM8K (math reasoning)

Why I Did This

Memory Efficiency: Reduce GPU memory requirements to run on smaller GPUs
Faster Inference: INT8 operations can be faster on compatible hardware
Accessibility: Enable deployment on consumer-grade hardware with limited VRAM

Benchmark Results

Metric	Baseline (BF16)	INT8	Change
Memory	8.41 GB	4.84 GB	-42.4%
Perplexity (WikiText-2)	12.59	12.75	+1.30%
GSM8K Accuracy	86.00%	~86%	Minimal degradation

Key Findings

1.74x memory reduction with minimal quality loss
Only +1.30% perplexity increase on WikiText-2
Math reasoning capabilities preserved on GSM8K benchmark

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "prashantcp8/Qwen3.5-4B-INT8",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("prashantcp8/Qwen3.5-4B-INT8")

messages = [{"role": "user", "content": "What is machine learning?"}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(response)

Technical Details

Quantization Method: bitsandbytes LLM.int8()
Base Model: Qwen/Qwen3.5-4B
Original Precision: BF16
Quantized Precision: INT8
Framework: Transformers + bitsandbytes

Limitations

Requires bitsandbytes library for optimal performance
Some operations may cast to FP16 during inference
Best suited for NVIDIA GPUs with INT8 tensor core support

Citation

If you use this model, please cite the original Qwen3.5 model:

@misc{qwen3.5,
  title={Qwen3.5 Technical Report},
  author={Qwen Team},
  year={2025},
  publisher={Alibaba}
}

Author

Prashant Takale

Note: This model is for testing and experimental purposes only. There are still some improvements required in terms of inference speed and accuracy. I plan to explore better quantization methods like AWQ or GPTQ in future iterations for optimized performance.

Downloads last month: 110

Safetensors

Model size

4B params

Tensor type

F32

BF16

Model tree for prashantcp8/Qwen3.5-4B-INT8

Base model

Qwen/Qwen3.5-4B-Base

Finetuned

Qwen/Qwen3.5-4B

Quantized

(175)

this model