Instructions to use pbhappliedsystems/Qwen2.5-32B-Instruct-bnb-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use pbhappliedsystems/Qwen2.5-32B-Instruct-bnb-4bit with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="pbhappliedsystems/Qwen2.5-32B-Instruct-bnb-4bit")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("pbhappliedsystems/Qwen2.5-32B-Instruct-bnb-4bit")
model = AutoModelForCausalLM.from_pretrained("pbhappliedsystems/Qwen2.5-32B-Instruct-bnb-4bit", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use pbhappliedsystems/Qwen2.5-32B-Instruct-bnb-4bit with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "pbhappliedsystems/Qwen2.5-32B-Instruct-bnb-4bit"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pbhappliedsystems/Qwen2.5-32B-Instruct-bnb-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/pbhappliedsystems/Qwen2.5-32B-Instruct-bnb-4bit

SGLang

How to use pbhappliedsystems/Qwen2.5-32B-Instruct-bnb-4bit with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "pbhappliedsystems/Qwen2.5-32B-Instruct-bnb-4bit" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pbhappliedsystems/Qwen2.5-32B-Instruct-bnb-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "pbhappliedsystems/Qwen2.5-32B-Instruct-bnb-4bit" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pbhappliedsystems/Qwen2.5-32B-Instruct-bnb-4bit",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use pbhappliedsystems/Qwen2.5-32B-Instruct-bnb-4bit with Docker Model Runner:
```
docker model run hf.co/pbhappliedsystems/Qwen2.5-32B-Instruct-bnb-4bit
```

Qwen2.5-32B-Instruct · BitsAndBytes 4-bit (NF4)

Quantized and evaluated by PBH Applied Systems, LLC — Applied AI/ML Consulting · LLM Optimization & Deployment · Quantized AI Infrastructure

🔬 This repository is part of a production-oriented evaluation series. Every model published under pbhappliedsystems has been independently evaluated using quant_eval v7.21 — a proprietary behavioral evaluation harness developed by PBH Applied Systems. Scores measure real agent-adjacent task performance across structured output, tool dispatch, multi-turn state retention, and multi-step planning families — not perplexity or leaderboard proxies.

⚠️ Single-runner evaluation. No FP16 baseline was evaluated for this model. The FP16 Qwen2.5-32B-Instruct (~65 GB) exceeds the VRAM capacity of the evaluation hardware (NVIDIA RTX 4090, 24 GB). All behavioral data in this card comes from the 4-bit quantized_bitsandbytes runner only.

🧩 Backend note. This is the BitsAndBytes NF4 variant — an HF-format safetensors checkpoint loaded directly through Transformers (or served via vLLM). It is a distinct backend from the GGUF Q4_K_M sibling. Where the two diverge in evaluation, it is called out explicitly below.

Try This Model in the Live AI Agent Demo

Launch the PBH Applied Systems AI Agent Demo →

This model is part of the PBH Applied Systems live AI Agent Demo, where visitors can test evaluated quantized open-weight models across production-style agent workflows: reasoning and analysis, document intelligence, and code automation.

The demo uses quant_eval results to show how model selection changes by task. A model that performs well for stateful multi-turn workflows or tool dispatch may not be the best choice for hard multi-step planning. Each deployed model is evaluated for practical agent behavior, including coherence, instruction following, reasoning, task completion, structured output reliability, tool-use behavior, and quantization impact.

For this repository, the NF4 4-bit variant represents a deployment-focused checkpoint: it fits on a single 24 GB GPU, loads directly with Transformers, and — unlike the GGUF path — emits clean output with no end-of-sequence token contamination. The evaluation results below explain where this quantized model preserves useful behavior, where 4-bit quantization introduces risk, and what guardrails are recommended before production deployment.

Model Description

This repository contains the BitsAndBytes 4-bit (NF4, double-quantized) checkpoint of Qwen/Qwen2.5-32B-Instruct, a 32-billion parameter instruction-tuned model from Alibaba Cloud. The 4-bit weights are stored in HF safetensors with the quantization configuration baked into config.json, so the model loads directly through Transformers with no separate quantization step.

Key Characteristics

Parameters: 32B (Qwen2ForCausalLM)
Format: BitsAndBytes 4-bit · NF4 · double-quant
Compute dtype: float16
Quant storage: uint8
File size: ~17.9 GB (4 × safetensors shards + quant_config.json)
Minimum VRAM (GPU inference): ~22–24 GB
Recommended GPU tier: RTX 4090 24 GB · A10G 24 GB · A100 40 GB
Context window: 32,768 tokens
Inference speed (eval hardware): avg 4.405 sec/case on RTX 4090
License: Apache 2.0
FP16 equivalent: ~65 GB — exceeds single-GPU 24 GB VRAM; requires multi-GPU or large-memory server

Quantization Configuration

{
  "quant_method": "bitsandbytes",
  "load_in_4bit": true,
  "bnb_4bit_quant_type": "nf4",
  "bnb_4bit_use_double_quant": true,
  "bnb_4bit_compute_dtype": "float16",
  "bnb_4bit_quant_storage": "uint8"
}

PBH Applied Systems Evaluation — quant_eval v7.21

Evaluation conducted by PBH Applied Systems, LLC using quant_eval v7.21 Run ID: 20260629_002808 · Fixtures: golden_oracle_fixtures_v7_21 (SHA256: 6d71a0b9147c...) · Seed: 42 Hardware: NVIDIA RTX 4090 · Runner: quantized_bitsandbytes (4-bit only) · Total rows: 42

Per-Family Pass Rates (NF4 4-bit)

Family	N	Pass Rate	Avg Secs	Bucket Score	Notes
json_multistep	5	0.400	13.509	1.600	Premature STOP — see Finding 1
stateful_followup	2	1.000	1.960	2.000	Both turns exact match
toolcall_only	2	1.000	2.015	2.000	Correct values; wrapper normalized — see Finding 4
mixed_brief_json	2	1.000	2.455	2.000	Clean ANSWER + JSON
toolcall	2	1.000	2.575	11.000	Stage-1 and final correct — no EOS contamination
json	4	n/a	4.845	10.000	All pass
fuzz	20	n/a	3.952	10.000	All 20 pass
mcq	5	n/a	0.210	1.000	5/5 perfect

Key Findings

Finding 1: Multi-Step Planning — Premature STOP (0.400)

json_multistep is the one family where this 4-bit checkpoint shows a genuine limitation. The format is always valid (schema_ok 1.000, stop_semantics_ok 1.000), but the model consistently under-plans, emitting STOP one step early or selecting the wrong placement on the hard case.

Case-level breakdown:

Case	Difficulty	Result	Secs	Got Plan	Expected	Failure
ms_easy_01	Easy	✅	10.84	`["A"]`	`["A"]`	—
ms_easy_02	Easy	❌	11.89	`["A","STOP"]`	`["A","A"]`	oracle_equiv_ok=0 only
ms_med_01	Medium	✅	13.41	`["A","B","C"]`	`["A","B","C"]`	—
ms_med_02	Medium	❌	13.79	`["A","B","STOP"]`	`["A","B","C"]`	oracle_equiv_ok=0 only
ms_hard_01	Hard	❌	17.62	`["A","A","B"]`	`["A","B","C"]`	cc=0, oe=0

On ms_easy_02 and ms_med_02 the model reasons consistently (checks_consistent_ok=1) but stops before completing the plan. On ms_hard_01 the intermediate reasoning is itself inconsistent (checks_consistent_ok=0) and the final plan is wrong.

What this means for production: do not depend on this model for unassisted multi-step planning. Wrap it with an external planner/oracle validation loop, or select a model with a stronger json_multistep profile if planning reliability is the primary requirement.

Finding 2: MCQ — Perfect 5/5

Case	Result	Raw
mcq_01	✅	`B`
mcq_02	✅	`B`
mcq_03	✅	`C`
mcq_04	✅	`B`
mcq_05	✅	`B`

Clean single-letter extraction, no A-bias, at 0.21 sec/case.

Finding 3: toolcall — Correct and Clean (No EOS Contamination)

Both toolcall cases pass stage-1 (valid dispatch) and stage-2 (correct final value) for a perfect bucket score of 11.000:

Case	Raw	Final	Result
tool_01	`{"tool_name": "add", "args": {"a": 2, "b": 3}}` ⏎ `5`	`5`	✅
tool_02	`{"tool_name": "add", "args": {"a": 10, "b": -4}}` ⏎ `6`	`6`	✅

This is a notable backend advantage: the GGUF Q4_K_M sibling passes stage-1 but fails stage-2 with final_mismatch caused by trailing <|im_end|> tokens. The Transformers/BnB path decodes with skip_special_tokens=True, so the final answer arrives clean — no EOS stripping required downstream.

Finding 4: toolcall_only — Correct Values, Wrapper Normalized (1.000)

The model emits the correct argument values but a non-standard outer wrapper:

Case	Raw
toolonly_01	`{"tool": "add", "params": {"a": 5, "b": 10}}`
toolonly_02	`{"tool": "add", "params": {"a": 25, "b": 75}}`

The argument value names ("a", "b") and values are correct; the model uses "tool"/"params" where the strict schema expects "tool_name"/"args". quant_eval v7.21 normalizes the common wrapper aliases (args / arguments / params / parameters) for both scoring and tool execution, so args_ok=1.000. If your downstream stack does not normalize wrappers, enforce the exact outer keys via system prompt (see Usage).

Finding 5: Stateful and Hybrid — Clean at 4-bit

stateful_followup (1.000) and mixed_brief_json (1.000) both pass cleanly, again with no EOS contamination:

Family	Case	Raw
stateful	state_01	`{"counter": 2}` → `{"counter": 5}`
stateful	state_02	`{"items": ["a","b"]}` → `{"items": ["a","b","c"]}`
mixed	mixed_01	`ANSWER: 13` ⏎ `{"a": 4, "b": 9, "sum": 13}`
mixed	mixed_02	`ANSWER: 6` ⏎ `{"a": -2, "b": 8, "sum": 6}`

Backend Comparison — NF4 4-bit vs GGUF Q4_K_M (same 32B)

Family	NF4 4-bit (this repo)	GGUF Q4_K_M	Note
json_multistep	0.400	0.600	NF4 adds the `ms_med_02` premature-STOP failure
stateful_followup	1.000	1.000	Both strong
mixed_brief_json	1.000	1.000	Both strong
mcq	5/5	5/5	Both perfect
json / fuzz	10.000	10.000	Both perfect
toolcall	11.000 (clean)	0.000 (EOS)	Transformers decode strips EOS; GGUF needs `<
toolcall_only	1.000	0.000¹	¹GGUF card predates v7.21 wrapper normalization; the model's raw wrapper is identical (`params`) on both backends

The two 4-bit backends are close on structured JSON, stateful, hybrid, and MCQ. The NF4 path is cleaner for tool-calling output (no EOS suffix) and faster per case in this run, while the GGUF Q4_K_M path is slightly stronger on multi-step planning. Choose NF4 when you want direct Transformers/vLLM serving and clean tool-call output; choose GGUF Q4_K_M when planning reliability is the deciding factor.

Signal-Level Diagnostics (NF4 4-bit)

json_multistep

Signal	Rate	Notes
schema_ok	1.000	Perfect
checks_consistent_ok	0.800	`ms_hard_01` fails
stop_semantics_ok	1.000	Perfect
oracle_equiv_ok	0.400	`ms_easy_02` + `ms_med_02` + `ms_hard_01`

toolcall_only

Signal	Rate	Notes
tool_name_ok	1.000	`"add"` recognized (`tool`/`tool_name` aliased)
args_ok	1.000	Values correct; `params`→`args` normalized

stateful_followup / mixed_brief_json

All gating signals (turn1/2_parse_ok, turn1/2_exact_match; answer_line_ok, json_parse_ok, schema_ok) = 1.000.

Recommended Use Cases

✅ Deploy with Confidence (NF4 4-bit)

Tool-calling agents — toolcall 11.000 (clean), toolcall_only 1.000. Best-in-series output cleanliness: no EOS stripping required.
Stateful multi-turn agents — 1.000 at both turns, clean JSON state.
Hybrid brief + JSON responses — mixed_brief_json 1.000 at ~2.5 sec/case.
Structured JSON outputs (single-step) — json and fuzz both bucket=10.000.
MCQ / single-choice extraction — 5/5 perfect at 0.21 sec/case.
Complex generation tasks — 32B parameter capacity for nuanced, long-form, or multilingual outputs.

⚠️ Use with Guardrails (NF4 4-bit)

Multi-step planning — 0.400 pass rate; consistent premature-STOP. Add an external planner/oracle validation loop, or prefer the GGUF Q4_K_M variant (0.600) if planning is the primary requirement.
Bare tool-call dispatch in non-normalizing stacks — the model emits tool/params wrappers. quant_eval normalizes these; if your runtime does not, enforce exact outer keys (tool_name/args) in the system prompt. Argument value names (a/b) are already correct.

Hardware Requirements

Configuration	VRAM Required	Notes
NF4 4-bit (this repo)	~22–24 GB	~17.9 GB weights + KV cache
NF4 4-bit · full context	~26–28 GB	Use A10G/A100 for 32K context headroom
FP16 (not in this repo)	~65 GB+	Multi-GPU or large-memory server

Usage

Installation

pip install "transformers>=4.46" "bitsandbytes>=0.46.1" accelerate huggingface_hub

BitsAndBytes 4-bit inference requires a CUDA GPU.

Python — Transformers (loads the baked-in 4-bit config)

from transformers import AutoTokenizer, AutoModelForCausalLM

REPO = "pbhappliedsystems/Qwen2.5-32B-Instruct-bnb-4bit"

tokenizer = AutoTokenizer.from_pretrained(REPO)
# No load_in_4bit needed — the quantization_config is baked into config.json.
model = AutoModelForCausalLM.from_pretrained(REPO, device_map="auto")

messages = [
    {"role": "system", "content": "You are a precise assistant. Follow instructions exactly and return structured outputs when requested."},
    {"role": "user", "content": "Return a JSON object with keys: summary, risk_level, action_items."},
]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

out = model.generate(**{"input_ids": inputs}, max_new_tokens=512, do_sample=False)
print(tokenizer.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))

Tip: use add_generation_prompt=True. Without it, the ChatML template can leak a leading assistant/user role label into the decoded text.

Bare tool-call dispatch — enforce outer keys if your stack does not normalize

import json

SYS = (
    'Respond ONLY with a JSON object using EXACTLY these keys:\n'
    '{"tool_name": "add", "args": {"a": <integer>, "b": <integer>}}\n'
    'No other text, no markdown.'
)

def call_tool(prompt: str, retries: int = 3) -> dict:
    for attempt in range(retries):
        msgs = [{"role": "system", "content": SYS}, {"role": "user", "content": prompt}]
        ids = tokenizer.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
        out = model.generate(**{"input_ids": ids}, max_new_tokens=64, do_sample=False)
        raw = tokenizer.decode(out[0][ids.shape[-1]:], skip_special_tokens=True).strip()
        try:
            parsed = json.loads(raw)
            assert "tool_name" in parsed and "args" in parsed
            assert "a" in parsed["args"] and "b" in parsed["args"]
            return parsed
        except (json.JSONDecodeError, AssertionError, KeyError):
            if attempt == retries - 1:
                raise ValueError(f"Tool call failed after {retries} attempts. Raw: {raw}")

print(call_tool("Add 5 and 10."))

Serving — vLLM (OpenAI-compatible)

vllm serve pbhappliedsystems/Qwen2.5-32B-Instruct-bnb-4bit \
  --quantization bitsandbytes \
  --max-model-len 8192

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-required")
resp = client.chat.completions.create(
    model="pbhappliedsystems/Qwen2.5-32B-Instruct-bnb-4bit",
    messages=[{"role": "user", "content": "Your prompt here"}],
    temperature=0.3,
)
print(resp.choices[0].message.content)

Recommended Generation Settings

temperature: 0.3 · top_p: 0.9 for general use; temperature: 0.0 (greedy) for deterministic structured/tool output.

Evaluation Artifacts

The full per-case evaluation CSV (comparison_results_v7_21_Qwen2.5_32B_Instruct_4bit_20260527_233818_20260629_002808.csv) and rollup.json are published in this repository for independent verification.

Artifact Provenance

Artifact	Format	Size (bytes)	SHA256
`model-00001-of-00004.safetensors`	BnB NF4	4,933,190,348	`b2a0e8a735e99b3a59bb3139541c444808aff3793a28c314c0f02bf17a00b5f7`
`model-00002-of-00004.safetensors`	BnB NF4	4,958,587,236	`fd4b028d13261c8da0e29ed57b95189d666f62f3e8d4ab232c17c4e4e131543a`
`model-00003-of-00004.safetensors`	BnB NF4	4,999,136,184	`0446d1c6da46a5daea91bed161fd62f2f48a658d879f58a14b7ab5528eb66935`
`model-00004-of-00004.safetensors`	BnB NF4	4,324,534,021	`39002c4ed64520809793fb2b2023caf9bdbf0914feb4786d553c418139457018`
`quant_config.json`	JSON	425	`479249ed811af17066e3cd591f9ce88127b6a530c7aa2b5ed5ca56dec9bdd083`

Quantized from Qwen/Qwen2.5-32B-Instruct using BitsAndBytes (NF4, double-quant, float16 compute). The FP16 base was not evaluated due to VRAM constraints.

Evaluation Methodology

quant_eval v7.21 — proprietary behavioral evaluation harness, PBH Applied Systems.

Fixture set: golden_oracle_fixtures_v7_21 (SHA256: 6d71a0b9147c079371b02a94f3c149eb78a6adc03dc16ff6833b964fbf4174f0)

Family	Description	Pass Signals
`fuzz`	Property-based regression; structured placement correctness	schema_ok, constraints_ok
`json`	Single-step structured JSON with constraint rules	schema_ok, constraints_ok
`json_multistep`	Multi-step planning with self-check and oracle verification	schema_ok, checks_consistent_ok, stop_semantics_ok, oracle_equiv_ok
`mcq`	Multiple-choice extraction	choice_ok
`stateful_followup`	Two-turn state tracking; turn-2 correct given turn-1	turn1/2_parse_ok, turn1/2_exact_match
`mixed_brief_json`	Hybrid: natural language answer + valid JSON block	answer_line_ok, json_parse_ok, schema_ok
`toolcall`	Tool call embedded in response; parse + schema validation + final value	stage1_tool_parse_ok, stage1_tool_schema_ok
`toolcall_only`	Bare schema-only tool call; strict tool name + args check (wrapper-normalized)	tool_name_ok, args_ok

Evaluation hardware: NVIDIA RTX 4090 (24 GB) · Runner: quantized_bitsandbytes (4-bit only) · Evaluation date: June 29, 2026 · Seed: 42

About PBH Applied Systems

PBH Applied Systems, LLC is an Oklahoma City–based applied machine learning and AI systems company specializing in production-grade model evaluation, quantization pipelines, agentic AI infrastructure, and scalable AI-driven application development.

Patrick Hill, M.S. — Founder · Data Scientist · AI/ML Engineer · Author of Applied Machine Learning: Concepts, Tools, and Case Studies (required reading, UAT CSC 373)

Core Service Areas: LLM Optimization & Deployment · AI Evaluation Frameworks · Agentic AI Infrastructure · Scalable AI Application Development · ML Pipeline Design & Analytics · Model & Agent Cataloging

🔬 About quant_eval & This Evaluation Series

quant_eval is a proprietary behavioral evaluation harness developed by PBH Applied Systems, LLC. It measures real agent-adjacent task performance across structured output, tool dispatch, multi-turn state retention, and multi-step planning — not perplexity or leaderboard proxies. Every model published under pbhappliedsystems has been independently evaluated using quant_eval before being recommended for any production role.

See it in action: Live AI Agent Demo →

Need a deployment recommendation? Not sure which quantization level — NF4, Q4_K_M, AWQ — is right for your hardware, latency target, or agent type? → pbhappliedsystems.com

Evaluated and published by PBH Applied Systems, LLC · patrick@pbhappliedsystems.com

📞 Work With PBH Applied Systems

This NF4 result is a concrete example of why backend-aware evaluation matters: the same 32B weights at 4-bit behave differently across BitsAndBytes and GGUF — cleaner tool-call output on one, stronger multi-step planning on the other. The evaluation tells you exactly which to deploy for which workload.

👉 Book a Scoping Call · 👉 Request an Evaluation Report — from $2,500

Connect


🌐	pbhappliedsystems.com
📧	patrick@pbhappliedsystems.com
💼	LinkedIn
▶️	YouTube
📸	Instagram
👍	Facebook

License

This repository inherits the license of the base model: Apache 2.0 — Qwen/Qwen2.5-32B-Instruct

The quant_eval evaluation methodology, fixture set, and scoring framework are proprietary to PBH Applied Systems, LLC and are not included in this repository.

BitsAndBytes quantization and behavioral evaluation performed by PBH Applied Systems, LLC · quant_eval v7.21 · Run ID: 20260629_002808

Downloads last month: 6

Safetensors

Model size

34B params

Tensor type

F32

F16

Model tree for pbhappliedsystems/Qwen2.5-32B-Instruct-bnb-4bit

Base model

Qwen/Qwen2.5-32B

Finetuned

Qwen/Qwen2.5-32B-Instruct

Quantized

(150)

this model