Instructions to use reality-interface/qwen3.8-27b-agentic-nvfp4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use reality-interface/qwen3.8-27b-agentic-nvfp4 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="reality-interface/qwen3.8-27b-agentic-nvfp4") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("reality-interface/qwen3.8-27b-agentic-nvfp4") model = AutoModelForMultimodalLM.from_pretrained("reality-interface/qwen3.8-27b-agentic-nvfp4", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=40) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use reality-interface/qwen3.8-27b-agentic-nvfp4 with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "reality-interface/qwen3.8-27b-agentic-nvfp4" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "reality-interface/qwen3.8-27b-agentic-nvfp4", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/reality-interface/qwen3.8-27b-agentic-nvfp4
- SGLang
How to use reality-interface/qwen3.8-27b-agentic-nvfp4 with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "reality-interface/qwen3.8-27b-agentic-nvfp4" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "reality-interface/qwen3.8-27b-agentic-nvfp4", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "reality-interface/qwen3.8-27b-agentic-nvfp4" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "reality-interface/qwen3.8-27b-agentic-nvfp4", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use reality-interface/qwen3.8-27b-agentic-nvfp4 with Docker Model Runner:
docker model run hf.co/reality-interface/qwen3.8-27b-agentic-nvfp4
fix(template): align Qwen reasoning parser state
Browse files- Make enable_thinking the authoritative reasoning-mode control used by the vLLM Qwen parser.\n- Remove message-level thinking toggles and prevent tools or repeated failures from silently changing parser state.\n- Add ultra-to-xhigh reasoning-effort compatibility while preserving the xhigh default.\n- Reject the unsafe legacy auto-disable option and update the README to use enable_thinking explicitly.\n- Preserve XML tools, OpenAI and Codex history, multimodal input, reasoning history, and agent error guidance.
- README.md +2 -2
- chat_template.jinja +14 -37
|
@@ -280,7 +280,6 @@ response = client.chat.completions.create(
|
|
| 280 |
extra_body={
|
| 281 |
"chat_template_kwargs": {
|
| 282 |
"enable_thinking": False,
|
| 283 |
-
"auto_disable_thinking_with_tools": True,
|
| 284 |
}
|
| 285 |
},
|
| 286 |
)
|
|
@@ -317,9 +316,10 @@ Always use the current `chat_template.jinja`. It is intentionally separate from
|
|
| 317 |
| `enable_thinking` | `true` | Enable or disable thinking output |
|
| 318 |
| `preserve_thinking` | `true` | Preserve earlier thinking in conversation history |
|
| 319 |
| `tool_call_format` | `xml` | Select `xml` or `json` tool-call output |
|
| 320 |
-
| `auto_disable_thinking_with_tools` | `false` | Disable thinking automatically when tools are present |
|
| 321 |
| `continue_final_message` | `false` | Continue an assistant message instead of opening a new turn |
|
| 322 |
|
|
|
|
|
|
|
| 323 |
## Client integrations
|
| 324 |
|
| 325 |
The model and template support developer/system instructions, reasoning history, OpenAI-style tools, parallel calls, tool results, and multimodal placeholders. The serving layer translates the native format for each client.
|
|
|
|
| 280 |
extra_body={
|
| 281 |
"chat_template_kwargs": {
|
| 282 |
"enable_thinking": False,
|
|
|
|
| 283 |
}
|
| 284 |
},
|
| 285 |
)
|
|
|
|
| 316 |
| `enable_thinking` | `true` | Enable or disable thinking output |
|
| 317 |
| `preserve_thinking` | `true` | Preserve earlier thinking in conversation history |
|
| 318 |
| `tool_call_format` | `xml` | Select `xml` or `json` tool-call output |
|
|
|
|
| 319 |
| `continue_final_message` | `false` | Continue an assistant message instead of opening a new turn |
|
| 320 |
|
| 321 |
+
Use `enable_thinking=false` explicitly for non-thinking tool calls. The legacy `auto_disable_thinking_with_tools` option is rejected because changing the prompt mode without changing vLLM's Qwen reasoning-parser state would misclassify reasoning as normal content.
|
| 322 |
+
|
| 323 |
## Client integrations
|
| 324 |
|
| 325 |
The model and template support developer/system instructions, reasoning history, OpenAI-style tools, parallel calls, tool results, and multimodal placeholders. The serving layer translates the native format for each client.
|
|
@@ -1,8 +1,8 @@
|
|
| 1 |
-
{%- set template_version = "qwen3.8-agentic-nvfp4-
|
| 2 |
{%- set requested_tool_format = tool_call_format if tool_call_format is defined and tool_call_format is not none else "xml" -%}
|
| 3 |
{%- set add_vision_id = add_vision_id if add_vision_id is defined else false -%}
|
| 4 |
-
{%- set thinking_was_explicit =
|
| 5 |
-
{%- set enable_thinking = enable_thinking if enable_thinking is defined else
|
| 6 |
{%- if preserve_reasoning is defined and preserve_reasoning is not none -%}
|
| 7 |
{%- set preserve_history_thinking = preserve_reasoning -%}
|
| 8 |
{%- elif preserve_thinking is defined and preserve_thinking is not none -%}
|
|
@@ -10,7 +10,6 @@
|
|
| 10 |
{%- else -%}
|
| 11 |
{%- set preserve_history_thinking = true -%}
|
| 12 |
{%- endif -%}
|
| 13 |
-
{%- set auto_disable_thinking_with_tools = auto_disable_thinking_with_tools if auto_disable_thinking_with_tools is defined else false -%}
|
| 14 |
{%- set max_tool_arg_chars = max_tool_arg_chars if max_tool_arg_chars is defined else 0 -%}
|
| 15 |
{%- set max_tool_response_chars = max_tool_response_chars if max_tool_response_chars is defined else 0 -%}
|
| 16 |
{%- set agent_error_hardening = agent_error_hardening if agent_error_hardening is defined else false -%}
|
|
@@ -18,8 +17,11 @@
|
|
| 18 |
{%- set image_counter = namespace(value=0) -%}
|
| 19 |
{%- set video_counter = namespace(value=0) -%}
|
| 20 |
{%- set state = namespace(thinking=enable_thinking, previous_role="", consecutive_failures=0) -%}
|
| 21 |
-
{
|
| 22 |
-
|
|
|
|
|
|
|
|
|
|
| 23 |
{%- endif -%}
|
| 24 |
|
| 25 |
{%- macro render_json_arguments(value) -%}
|
|
@@ -107,7 +109,7 @@
|
|
| 107 |
{%- set effort_raw = (reasoning_effort | string | lower) if reasoning_effort is defined and reasoning_effort is not none else "xhigh" -%}
|
| 108 |
{%- set effort = "" -%}
|
| 109 |
{%- if effort_raw in ("none", "off") -%}
|
| 110 |
-
{# Match current vLLM semantics: an explicit enable_thinking
|
| 111 |
{%- if not thinking_was_explicit -%}
|
| 112 |
{%- set state.thinking = false -%}
|
| 113 |
{%- endif -%}
|
|
@@ -115,27 +117,16 @@
|
|
| 115 |
{%- set effort = "low" -%}
|
| 116 |
{%- elif effort_raw == "medium" -%}
|
| 117 |
{%- set effort = "medium" -%}
|
| 118 |
-
{%- elif effort_raw in ("high", "xhigh", "max", "ultracode", "extreme") -%}
|
| 119 |
{%- set effort = "xhigh" -%}
|
| 120 |
{%- elif effort_raw -%}
|
| 121 |
{# Unknown future effort names keep the model's normal thinking behavior and do not hard-fail rendering. #}
|
| 122 |
{%- set effort = "" -%}
|
| 123 |
{%- endif -%}
|
| 124 |
|
| 125 |
-
{#
|
| 126 |
-
|
| 127 |
-
|
| 128 |
-
contradictory system prefix and generation scaffold. #}
|
| 129 |
-
{%- for control_message in messages -%}
|
| 130 |
-
{%- if control_message.role == "system" or control_message.role == "developer" or control_message.role == "user" -%}
|
| 131 |
-
{%- set control_text = render_content(control_message.content, false, false) -%}
|
| 132 |
-
{%- if "<|think_off|>" in control_text -%}
|
| 133 |
-
{%- set state.thinking = false -%}
|
| 134 |
-
{%- elif "<|think_on|>" in control_text -%}
|
| 135 |
-
{%- set state.thinking = true -%}
|
| 136 |
-
{%- endif -%}
|
| 137 |
-
{%- endif -%}
|
| 138 |
-
{%- endfor -%}
|
| 139 |
|
| 140 |
{# Keep the generation grammar Qwen-native and stable by default. Current vLLM
|
| 141 |
normalizes OpenAI wire-format argument strings into mappings before rendering.
|
|
@@ -165,12 +156,6 @@
|
|
| 165 |
{%- set system_acc = namespace(text="") -%}
|
| 166 |
{%- for message in messages[:head.count] -%}
|
| 167 |
{%- set part = render_content(message.content, false, true) | trim -%}
|
| 168 |
-
{%- if "<|think_off|>" in part -%}
|
| 169 |
-
{%- set state.thinking = false -%}
|
| 170 |
-
{%- elif "<|think_on|>" in part -%}
|
| 171 |
-
{%- set state.thinking = true -%}
|
| 172 |
-
{%- endif -%}
|
| 173 |
-
{%- set part = part | replace("<|think_off|>", "") | replace("<|think_on|>", "") | trim -%}
|
| 174 |
{%- if part -%}
|
| 175 |
{%- if system_acc.text -%}
|
| 176 |
{%- set system_acc.text = system_acc.text ~ "\n\n" ~ part -%}
|
|
@@ -250,14 +235,6 @@
|
|
| 250 |
{%- set content = rendered_content if tool_like else (rendered_content | trim) -%}
|
| 251 |
|
| 252 |
{%- set wrapped_tool_response = role == "user" and content[:15] == "<tool_response>" and content[-16:] == "</tool_response>" -%}
|
| 253 |
-
{%- if system_like or (role == "user" and not wrapped_tool_response) -%}
|
| 254 |
-
{%- if "<|think_off|>" in content -%}
|
| 255 |
-
{%- set state.thinking = false -%}
|
| 256 |
-
{%- elif "<|think_on|>" in content -%}
|
| 257 |
-
{%- set state.thinking = true -%}
|
| 258 |
-
{%- endif -%}
|
| 259 |
-
{%- set content = content | replace("<|think_off|>", "") | replace("<|think_on|>", "") | trim -%}
|
| 260 |
-
{%- endif -%}
|
| 261 |
|
| 262 |
{%- if system_like -%}
|
| 263 |
{{- "<|im_start|>system\n" ~ content ~ "<|im_end|>\n" -}}
|
|
@@ -429,7 +406,7 @@
|
|
| 429 |
{%- set continuing_final = continue_final_message is defined and continue_final_message and conversation and conversation[-1].role == "assistant" -%}
|
| 430 |
{%- if add_generation_prompt and not continuing_final -%}
|
| 431 |
{{- "<|im_start|>assistant\n" -}}
|
| 432 |
-
{%- if state.thinking
|
| 433 |
{{- "<think>\n" -}}
|
| 434 |
{%- else -%}
|
| 435 |
{# Qwen's own templates use this closed empty think marker to force non-thinking generation. #}
|
|
|
|
| 1 |
+
{%- set template_version = "qwen3.8-agentic-nvfp4-v1-stable" -%}
|
| 2 |
{%- set requested_tool_format = tool_call_format if tool_call_format is defined and tool_call_format is not none else "xml" -%}
|
| 3 |
{%- set add_vision_id = add_vision_id if add_vision_id is defined else false -%}
|
| 4 |
+
{%- set thinking_was_explicit = enable_thinking is defined -%}
|
| 5 |
+
{%- set enable_thinking = enable_thinking if enable_thinking is defined else true -%}
|
| 6 |
{%- if preserve_reasoning is defined and preserve_reasoning is not none -%}
|
| 7 |
{%- set preserve_history_thinking = preserve_reasoning -%}
|
| 8 |
{%- elif preserve_thinking is defined and preserve_thinking is not none -%}
|
|
|
|
| 10 |
{%- else -%}
|
| 11 |
{%- set preserve_history_thinking = true -%}
|
| 12 |
{%- endif -%}
|
|
|
|
| 13 |
{%- set max_tool_arg_chars = max_tool_arg_chars if max_tool_arg_chars is defined else 0 -%}
|
| 14 |
{%- set max_tool_response_chars = max_tool_response_chars if max_tool_response_chars is defined else 0 -%}
|
| 15 |
{%- set agent_error_hardening = agent_error_hardening if agent_error_hardening is defined else false -%}
|
|
|
|
| 17 |
{%- set image_counter = namespace(value=0) -%}
|
| 18 |
{%- set video_counter = namespace(value=0) -%}
|
| 19 |
{%- set state = namespace(thinking=enable_thinking, previous_role="", consecutive_failures=0) -%}
|
| 20 |
+
{# vLLM's Qwen parser reads only chat_template_kwargs.enable_thinking. A template-only
|
| 21 |
+
auto-disable would desynchronize parser and prompt state, so reject the legacy knob
|
| 22 |
+
instead of silently producing incorrectly parsed reasoning. #}
|
| 23 |
+
{%- if auto_disable_thinking_with_tools is defined and auto_disable_thinking_with_tools -%}
|
| 24 |
+
{{- raise_exception("auto_disable_thinking_with_tools is unsafe with the Qwen vLLM reasoning parser. Set chat_template_kwargs.enable_thinking=false explicitly instead.") -}}
|
| 25 |
{%- endif -%}
|
| 26 |
|
| 27 |
{%- macro render_json_arguments(value) -%}
|
|
|
|
| 109 |
{%- set effort_raw = (reasoning_effort | string | lower) if reasoning_effort is defined and reasoning_effort is not none else "xhigh" -%}
|
| 110 |
{%- set effort = "" -%}
|
| 111 |
{%- if effort_raw in ("none", "off") -%}
|
| 112 |
+
{# Match current vLLM semantics: an explicit enable_thinking kwarg wins. #}
|
| 113 |
{%- if not thinking_was_explicit -%}
|
| 114 |
{%- set state.thinking = false -%}
|
| 115 |
{%- endif -%}
|
|
|
|
| 117 |
{%- set effort = "low" -%}
|
| 118 |
{%- elif effort_raw == "medium" -%}
|
| 119 |
{%- set effort = "medium" -%}
|
| 120 |
+
{%- elif effort_raw in ("high", "xhigh", "max", "ultra", "ultracode", "extreme") -%}
|
| 121 |
{%- set effort = "xhigh" -%}
|
| 122 |
{%- elif effort_raw -%}
|
| 123 |
{# Unknown future effort names keep the model's normal thinking behavior and do not hard-fail rendering. #}
|
| 124 |
{%- set effort = "" -%}
|
| 125 |
{%- endif -%}
|
| 126 |
|
| 127 |
+
{# `enable_thinking` is the single source of truth for generation mode. Do not
|
| 128 |
+
mutate reasoning state from magic strings embedded in conversation content: vLLM's
|
| 129 |
+
Qwen parser cannot see those mutations and would parse the output in the wrong state. #}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 130 |
|
| 131 |
{# Keep the generation grammar Qwen-native and stable by default. Current vLLM
|
| 132 |
normalizes OpenAI wire-format argument strings into mappings before rendering.
|
|
|
|
| 156 |
{%- set system_acc = namespace(text="") -%}
|
| 157 |
{%- for message in messages[:head.count] -%}
|
| 158 |
{%- set part = render_content(message.content, false, true) | trim -%}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 159 |
{%- if part -%}
|
| 160 |
{%- if system_acc.text -%}
|
| 161 |
{%- set system_acc.text = system_acc.text ~ "\n\n" ~ part -%}
|
|
|
|
| 235 |
{%- set content = rendered_content if tool_like else (rendered_content | trim) -%}
|
| 236 |
|
| 237 |
{%- set wrapped_tool_response = role == "user" and content[:15] == "<tool_response>" and content[-16:] == "</tool_response>" -%}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 238 |
|
| 239 |
{%- if system_like -%}
|
| 240 |
{{- "<|im_start|>system\n" ~ content ~ "<|im_end|>\n" -}}
|
|
|
|
| 406 |
{%- set continuing_final = continue_final_message is defined and continue_final_message and conversation and conversation[-1].role == "assistant" -%}
|
| 407 |
{%- if add_generation_prompt and not continuing_final -%}
|
| 408 |
{{- "<|im_start|>assistant\n" -}}
|
| 409 |
+
{%- if state.thinking -%}
|
| 410 |
{{- "<think>\n" -}}
|
| 411 |
{%- else -%}
|
| 412 |
{# Qwen's own templates use this closed empty think marker to force non-thinking generation. #}
|