Instructions to use MESHIVEAI/Qwen3.8-Flash-Next-NVFP4-W4A16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use MESHIVEAI/Qwen3.8-Flash-Next-NVFP4-W4A16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="MESHIVEAI/Qwen3.8-Flash-Next-NVFP4-W4A16")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("MESHIVEAI/Qwen3.8-Flash-Next-NVFP4-W4A16")
model = AutoModelForMultimodalLM.from_pretrained("MESHIVEAI/Qwen3.8-Flash-Next-NVFP4-W4A16", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use MESHIVEAI/Qwen3.8-Flash-Next-NVFP4-W4A16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "MESHIVEAI/Qwen3.8-Flash-Next-NVFP4-W4A16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MESHIVEAI/Qwen3.8-Flash-Next-NVFP4-W4A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/MESHIVEAI/Qwen3.8-Flash-Next-NVFP4-W4A16

SGLang

How to use MESHIVEAI/Qwen3.8-Flash-Next-NVFP4-W4A16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "MESHIVEAI/Qwen3.8-Flash-Next-NVFP4-W4A16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MESHIVEAI/Qwen3.8-Flash-Next-NVFP4-W4A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "MESHIVEAI/Qwen3.8-Flash-Next-NVFP4-W4A16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "MESHIVEAI/Qwen3.8-Flash-Next-NVFP4-W4A16",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use MESHIVEAI/Qwen3.8-Flash-Next-NVFP4-W4A16 with Docker Model Runner:
```
docker model run hf.co/MESHIVEAI/Qwen3.8-Flash-Next-NVFP4-W4A16
```

Qwen3.8-Flash-Next-NVFP4-W4A16

Built with Qwen.

Weight-only NVFP4 quantization of the routed MoE experts of Qwen/Qwen3.8-Flash-Next. Activations stay at 16-bit.

What is quantized


Scheme	NVFP4, 4-bit weights, group size 16, BF16 activations (W4A16)
Scope	routed experts of the 48 text MoE layers — 120.8B params
Tool	NVIDIA TensorRT Model Optimizer 0.46.0
Calibration	data-free (`algorithm: max`)
Size	169 GiB, down from 336 GiB BF16

Everything else stays in BF16: attention, Gated DeltaNet linear attention, QSA indexer, shared experts, routers, hyper-connections, PLE n-gram tables, MTP head, vision tower, and lm_head.

Parameter breakdown

The model is 48 text layers, hidden size 2560, 512 experts per layer with top-10 routing and moe_intermediate_size 640. The routed experts are stored as fused 3-D tensors (mlp.experts.gate_up_proj (512, 1280, 2560) and mlp.experts.down_proj (512, 2560, 640)) and account for 120.8B of the ~125B main-model parameters — 96.6% of it.

Separately, the per-layer-embedding (PLE) n-gram tables are 128 shards of (2500012, 160), or 51.2B parameters / 97.7 GiB in BF16. They are not quantized here and are copied through byte-for-byte from the source checkpoint.

The parameter count shown in the sidebar is derived from the packed tensors and understates the real total.

Serving

SGLang has day-0 support for qwen4_exp and can serve this architecture today. vLLM support is not merged yet (vllm-project/vllm#53896).

Single-GPU serving needs the 97.7 GiB PLE table offloaded to host RAM (vllm-project/vllm#53899 adds VLLM_PLE_CPU_OFFLOAD=1 for vLLM). With PLE offloaded, the on-GPU weights are roughly 60 GiB.

We have not validated serving end to end yet. Measured configurations — GPU, tensor-parallel size, achievable max-model-len, and required engine flags — will be published here once we have run them.

Reproduction

Quantized with modelopt's NVFP4_EXPERTS_ONLY_CFG, modified to disable the expert input quantizers so activations remain 16-bit. The model is loaded with accelerate offload so that export_hf_checkpoint takes modelopt's streaming writer path, mtq.compress is applied before export, and the PLE tables are excluded from the export and restored afterwards from the source shards. This keeps peak host memory low enough to run the whole job on a single 96 GB GPU node with 160 GB of RAM.

License

Released under the Qwen Community License (see LICENSE). Built with Qwen.

Downloads last month: 70

Safetensors

Model size

117B params

Tensor type

I64

BF16

Model tree for MESHIVEAI/Qwen3.8-Flash-Next-NVFP4-W4A16

Base model

Qwen/Qwen3.8-Flash-Next

Quantized

(161)

this model