Instructions to use vroomfondel/Ornith-1.0-9B-NVFP4-ModelOpt with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use vroomfondel/Ornith-1.0-9B-NVFP4-ModelOpt with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="vroomfondel/Ornith-1.0-9B-NVFP4-ModelOpt")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("vroomfondel/Ornith-1.0-9B-NVFP4-ModelOpt")
model = AutoModelForMultimodalLM.from_pretrained("vroomfondel/Ornith-1.0-9B-NVFP4-ModelOpt")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use vroomfondel/Ornith-1.0-9B-NVFP4-ModelOpt with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "vroomfondel/Ornith-1.0-9B-NVFP4-ModelOpt"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "vroomfondel/Ornith-1.0-9B-NVFP4-ModelOpt",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/vroomfondel/Ornith-1.0-9B-NVFP4-ModelOpt

SGLang

How to use vroomfondel/Ornith-1.0-9B-NVFP4-ModelOpt with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "vroomfondel/Ornith-1.0-9B-NVFP4-ModelOpt" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "vroomfondel/Ornith-1.0-9B-NVFP4-ModelOpt",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "vroomfondel/Ornith-1.0-9B-NVFP4-ModelOpt" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "vroomfondel/Ornith-1.0-9B-NVFP4-ModelOpt",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use vroomfondel/Ornith-1.0-9B-NVFP4-ModelOpt with Docker Model Runner:
```
docker model run hf.co/vroomfondel/Ornith-1.0-9B-NVFP4-ModelOpt
```

Ornith-1.0-9B NVFP4 (ModelOpt)

NVFP4 (W4A4) quantization of deepreinforce-ai/Ornith-1.0-9B, produced with the NVIDIA TensorRT Model Optimizer (ModelOpt). This is a derivative checkpoint, not the original model; see the base model card for the model's capabilities, training, and general usage.


Base model	deepreinforce-ai/Ornith-1.0-9B (Qwen3.5 dense, agentic coding, MIT)
Architecture	`Qwen3_5ForConditionalGeneration`, multimodal, Gated-DeltaNet hybrid-mamba (dense, 9B, 32 layers)
Precision	uniform W4A4 NVFP4: both the dense FFN AND the (linear + full) attention are quantized; FP8 KV cache. Vision tower kept BF16.
Size	~18 GB BF16 source
Producer	NVIDIA ModelOpt

Recipe note: this is the uniform W4A4 variant, which also quantizes the Gated-DeltaNet linear-attention path. With the required serving fixes applied (see below) it is coherent and scores GSM8K 94.0% (see Benchmarks). Some other NVFP4 builds keep the linear-attention path in BF16; this one does not.

Serving (SGLang)

Serving this checkpoint correctly on SGLang needs two fixes, both applied at launch by the serving launch script:

The NVFP4 scalar-scale fix, sgl-project/sglang#29151 (merged to main, NOT in 0.5.15). Without it the fused Gated-DeltaNet in_proj_qkv NVFP4 scale loads into only one slot, corrupting the global alpha, and the model emits word-salad. This is the critical one for this build.
The quantized-attention loader changes in qwen3_5.py, sgl-project/sglang#31220: enable quantized attention for modelopt_fp4 + load the baked FP8 KV-cache scales.

Serve on Blackwell (e.g. DGX Spark GB10 / sm121) with:

--quantization modelopt_fp4 \
--attention-backend flashinfer \
--fp4-gemm-backend flashinfer_cutlass \
--kv-cache-dtype fp8_e4m3 \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder

See the Notes section below for the exact patch details.

Quantization details (auto-generated)

source model: deepreinforce-ai/Ornith-1.0-9B
qformat: nvfp4 kv_cache: fp8
calibration: 512 samples from each of pg19, cnn_dailymail
producer: NVIDIA ModelOpt 0.45.0
generated: 2026-07-15T10:23:44Z

Before/after sample generation was skipped for this run (SKIP_GENERATE=1).

Notes

Serving requires an attention-quant loader patch

This checkpoint quantizes attention as well as the FFN (uniform W4A4), unlike NVIDIA's own Qwen3.5 NVFP4 exports which keep attention in BF16. SGLang's qwen3_5.py hardcodes attention to unquantized for any modelopt_fp4 checkpoint, so it must be patched at launch time. This is applied via quantizer/sglang_launch_sm121.sh (a Python heredoc that flips the forced-None override so ModelOptFp4Config.is_layer_excluded() decides per-prefix instead). Upstreamed in sgl-project/sglang PR #31220. The patch is a no-op on NVIDIA/attention-BF16 checkpoints.

Serving requires the NVFP4 scalar-scale fix (sglang PR

This build quantizes the Gated-DeltaNet linear-attention path, whose fused in_proj_qkv loads its per-tensor NVFP4 scale (weight_scale_2 / input_scale) via a tuple shard id. SGLang before PR #29151 (commit cfc3d0555e, merged to main, NOT in 0.5.15) hardcodes shard_id=0 for that scalar in MergedColumnParallelLinear / QKVParallelLinear.weight_loader_v2, leaving the other logical slots uninitialized; the ModelOpt NVFP4 path then .max()es over them so a garbage slot becomes the runtime global alpha, producing word-salad. Serve on a SGLang that includes PR #29151, or apply it at launch (this build's launch script does). Confirmed against a BF16 baseline and a linear-attn-excluded NVFP4 build (both coherent): the failure is this scale-loading path, not the recipe.

Uniform W4A4 recipe, validated

This is the uniform W4A4 variant: the dense FFN, the full attention, AND the Gated-DeltaNet linear-attention path are all NVFP4 (FP8 KV cache). It has cleared a coherence smoke-serve (capital-of-Australia, arithmetic, code generation all correct) and GSM8K (94.0%, see Benchmarks), with the two serving fixes above applied. Unlike some other NVFP4 Qwen3.5 builds, it does quantize the linear-attention path.

Multimodal, vision tower kept BF16

Calibration is text-only, so the vision tower is excluded from quantization and stays BF16 (avoiding the amax=0 degenerate-quant failure mode). The export ships the processor configs (preprocessor_config.json etc.) needed to serve this as a proper multimodal checkpoint.

Benchmarks

Task	Metric	Value	Setup	Hardware	Date	Notes
GSM8K	accuracy (strict == flexible)	94.0%	chat + reasoning (qwen3 think), 4-shot, temperature 0.6, max_tokens 4500, N=400	DGX Spark GB10 / sm121; SGLang 0.5.15 + NVFP4 scalar-scale fix (PR #29151) + qwen3_5 attention/KV patches (PR #31220)	2026-07-15	376/400 correct; strict == flexible; 11/400 (2.75%) empty (generation-length cap). Uniform W4A4 (dense FFN + full attention + Gated-DeltaNet linear-attn NVFP4), FP8 KV cache.

Downloads last month: -

Safetensors

Model size

6B params

Tensor type

BF16

F8_E4M3

Model tree for vroomfondel/Ornith-1.0-9B-NVFP4-ModelOpt

Base model

deepreinforce-ai/Ornith-1.0-9B

Quantized

(77)

this model