Instructions to use modal-labs/Inkling-NVFP4-DFlash with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use modal-labs/Inkling-NVFP4-DFlash with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="modal-labs/Inkling-NVFP4-DFlash", trust_remote_code=True)

# Load model directly
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("modal-labs/Inkling-NVFP4-DFlash", trust_remote_code=True)
model = AutoModel.from_pretrained("modal-labs/Inkling-NVFP4-DFlash", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use modal-labs/Inkling-NVFP4-DFlash with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "modal-labs/Inkling-NVFP4-DFlash"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "modal-labs/Inkling-NVFP4-DFlash",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/modal-labs/Inkling-NVFP4-DFlash

SGLang

How to use modal-labs/Inkling-NVFP4-DFlash with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "modal-labs/Inkling-NVFP4-DFlash" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "modal-labs/Inkling-NVFP4-DFlash",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "modal-labs/Inkling-NVFP4-DFlash" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "modal-labs/Inkling-NVFP4-DFlash",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use modal-labs/Inkling-NVFP4-DFlash with Docker Model Runner:
```
docker model run hf.co/modal-labs/Inkling-NVFP4-DFlash
```

Inkling-NVFP4-DFlash / README.md

dcw2

Update README.md

f2149ce verified 18 days ago

preview code

Raw

History Blame Contribute Delete

4.17 kB

	---
	pipeline_tag: text-generation
	library_name: transformers
	base_model:
	- thinkingmachines/Inkling-NVFP4
	license: apache-2.0
	inference: false
	tags:
	- dflash
	- speculative-decoding
	- speculative-decoding-draft
	- block-diffusion
	- draft-model
	- diffusion-language-model
	- efficiency
	- inkling
	- modelopt
	- nvfp4
	- mxfp8
	- fp8
	- sglang
	---

	# Inkling-NVFP4-DFlash

	[Paper](https://arxiv.org/abs/2602.06036) \| [Github](https://github.com/z-lab/dflash) \| [Blog](https://z-lab.ai/projects/dflash)

	This repository contains a DFlash draft model for [`thinkingmachines/Inkling-NVFP4`](https://huggingface.co/thinkingmachines/Inkling-NVFP4). It is not a standalone language model. It is intended to be paired with the target model in a speculative decoding server.

	This is an early preview release, and the drafter is still training. It uses all causal sliding-window attention (SWA) layers.

	DFlash uses a lightweight block diffusion draft model to propose multiple tokens in parallel. The target model verifies those proposals, improving serving throughput while preserving the target model's output distribution.

	## Quick Start

	Here is an example deployment. Some options, including `trtllm_mha` for draft attention, are pending upstream SGLang support.

	```bash
	sglang serve \
	--model-path thinkingmachines/Inkling-NVFP4 \
	--tp 8 \
	--trust-remote-code \
	--quantization modelopt_fp4 \
	--fp4-gemm-backend flashinfer_trtllm \
	--moe-runner-backend flashinfer_trtllm_routed \
	--enable-torch-symm-mem \
	--attention-backend fa4 \
	--mamba-radix-cache-strategy extra_buffer \
	--disable-custom-all-reduce \
	--page-size 128 \
	--reasoning-parser inkling \
	--tool-call-parser inkling \
	--enable-multimodal \
	--cuda-graph-backend-prefill breakable \
	--enable-scattered-sconv \
	--mem-fraction-static 0.78 \
	--max-running-requests 32 \
	--swa-full-tokens-ratio 0.10 \
	--mamba-full-memory-ratio 0.10 \
	--chunked-prefill-size 16384 \
	--watchdog-timeout 900 \
	--weight-loader-prefetch-checkpoints \
	--weight-loader-prefetch-num-threads 8 \
	--cuda-graph-max-bs-decode 32 \
	--cuda-graph-max-bs-prefill 4096 \
	--cuda-graph-bs-prefill 128 256 384 512 768 1024 1536 2048 2560 3072 3584 4096 \
	--kv-cache-dtype mxfp8 \
	--speculative-algorithm DFLASH \
	--speculative-draft-model-path modal-labs/Inkling-DFlash \
	--speculative-dflash-block-size 16 \
	--speculative-draft-model-quantization fp8 \
	--speculative-draft-attention-backend trtllm_mha \
	--speculative-draft-kv-cache-dtype fp8_e4m3 \
	--speculative-draft-window-size 4096 \
	--host 0.0.0.0 \
	--port 30000
	```

	## Benchmark Results

	This preview includes a preliminary accept-length evaluation at concurrency 1 and block size 16. A full benchmark suite, including throughput and higher-concurrency measurements, will follow.

	### Setup

	- Runtime: SGLang on 8x NVIDIA B300 GPUs, tensor parallel size 8
	- Target model: `thinkingmachines/Inkling-NVFP4`
	- Target weights and KV cache: ModelOpt NVFP4 weights with an `mxfp8` KV cache
	- Backends: `fa4` target attention and `trtllm_mha` DFlash draft attention
	- Draft model: FP8 weights with an `fp8_e4m3` KV cache and a 4096-token draft window
	- Workloads: GSM8K, MATH500, HumanEval, MBPP, and MT-Bench with Inkling's native chat renderer
	- Decoding: greedy, reasoning effort 0.9, max output length 2048 tokens
	- Measurement: DFlash block size 16 at concurrency 1, with up to 64 measured generation requests per workload; warmup and warmdown requests are excluded
	- Accept length: `completion_tokens / spec_verify_ct` per generation turn, averaged across generation turns

	### Accept Length

	Mean DFlash accept length at concurrency 1.

	\| Workload \| DFlash block=16 \|
	\| --- \| --- \|
	\| GSM8K \| 4.562 \|
	\| MATH500 \| 4.712 \|
	\| HumanEval \| 4.959 \|
	\| MBPP \| 3.907 \|
	\| MT-Bench \| 2.914 \|

	## Citation

	If you find DFlash useful, please cite the original paper:

	```bibtex
	@article{chen2026dflash,
	title = {{DFlash: Block Diffusion for Flash Speculative Decoding}},
	author = {Chen, Jian and Liang, Yesheng and Liu, Zhijian},
	journal = {arXiv preprint arXiv:2602.06036},
	year = {2026}
	}
	```