Instructions to use vroomfondel/Ornith-1.0-35B-NVFP4-ModelOpt with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use vroomfondel/Ornith-1.0-35B-NVFP4-ModelOpt with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="vroomfondel/Ornith-1.0-35B-NVFP4-ModelOpt")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("vroomfondel/Ornith-1.0-35B-NVFP4-ModelOpt")
model = AutoModelForMultimodalLM.from_pretrained("vroomfondel/Ornith-1.0-35B-NVFP4-ModelOpt")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use vroomfondel/Ornith-1.0-35B-NVFP4-ModelOpt with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "vroomfondel/Ornith-1.0-35B-NVFP4-ModelOpt"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "vroomfondel/Ornith-1.0-35B-NVFP4-ModelOpt",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/vroomfondel/Ornith-1.0-35B-NVFP4-ModelOpt

SGLang

How to use vroomfondel/Ornith-1.0-35B-NVFP4-ModelOpt with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "vroomfondel/Ornith-1.0-35B-NVFP4-ModelOpt" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "vroomfondel/Ornith-1.0-35B-NVFP4-ModelOpt",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "vroomfondel/Ornith-1.0-35B-NVFP4-ModelOpt" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "vroomfondel/Ornith-1.0-35B-NVFP4-ModelOpt",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use vroomfondel/Ornith-1.0-35B-NVFP4-ModelOpt with Docker Model Runner:
```
docker model run hf.co/vroomfondel/Ornith-1.0-35B-NVFP4-ModelOpt
```

Ornith-1.0-35B NVFP4 (ModelOpt)

NVFP4 (W4A4) quantization of deepreinforce-ai/Ornith-1.0-35B, produced with the NVIDIA TensorRT Model Optimizer (ModelOpt). This is a derivative checkpoint, not the original model; see the base model card for the model's capabilities, training, and general usage.


Base model	deepreinforce-ai/Ornith-1.0-35B (Qwen3.5-MoE, agentic coding, MIT)
Architecture	`Qwen3_5MoeForConditionalGeneration`, multimodal, Gated-DeltaNet hybrid-mamba MoE (35B total, ~3B active)
Precision	uniform W4A4 NVFP4: both the MoE experts and the (linear + full) attention are quantized; FP8 KV cache. Vision tower kept BF16.
Size	~21 GB (down from ~70 GB BF16)
Producer	NVIDIA ModelOpt 0.45.0

Recipe note: this is the uniform W4A4 variant, which also quantizes the Gated-DeltaNet linear-attention path. That is more aggressive than NVIDIA's own Qwen3.5-NVFP4 exports (MoE-only, attention kept in BF16). It has been validated for coherent output and math reasoning (GSM8K 96.75%, see Benchmarks below), but is offered on that basis.

Serving (SGLang)

Because this checkpoint quantizes attention, a stock SGLang build needs three small loader changes in qwen3_5.py (enable quantized attention for modelopt_fp4, plus load the baked FP8 KV-cache scales). These are upstreamed as sgl-project/sglang#31220 and applied at launch. Serve on Blackwell (e.g. DGX Spark GB10 / sm121) with:

--quantization modelopt_fp4 \
--attention-backend flashinfer \
--moe-runner-backend flashinfer_cutlass \
--fp4-gemm-backend flashinfer_cutlass \
--kv-cache-dtype fp8_e4m3 \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder

See the Notes section below for the exact patch details and validation.

Quantization details (auto-generated)

source model: deepreinforce-ai/Ornith-1.0-35B
qformat: nvfp4 kv_cache: fp8
calibration: 512 samples from each of pg19, cnn_dailymail
producer: NVIDIA ModelOpt 0.45.0
generated: 2026-07-14T19:05:05Z

Before/after sample generation was skipped for this run (SKIP_GENERATE=1).

Notes

Serving requires an attention-quant loader patch

This checkpoint quantizes attention as well as the MoE experts (uniform W4A4), unlike NVIDIA's own Qwen3.5 NVFP4 exports which quantize the MoE only and keep attention in BF16. SGLang's qwen3_5.py hardcodes attention to unquantized for any modelopt_fp4 checkpoint, so it must be patched at launch time. This is applied via quantizer/sglang_launch_sm121.sh (a Python heredoc that flips the forced-None override so ModelOptFp4Config.is_layer_excluded() decides per-prefix instead). The patch is a no-op on NVIDIA/MoE-only checkpoints.

Deliberately-risky recipe, not the default

This is the uniform W4A4 variant (attention AND experts quantized) of a Gated-DeltaNet hybrid-mamba MoE, kept for a quality A/B against the safe default. Prefer ornith-1.0-35b_nvfp4-mlp-only.yaml (experts-only, attention high precision, matches NVIDIA's own Qwen3.5-NVFP4 recipe shape) unless this uniform build has cleared a real smoke-serve quality check.

Validated coherent output on SGLang GB10/sm121

Smoke-served on SGLang GB10/sm121 with the attention-quant loader patch applied, and produced coherent, correct output on both a German-language prompt (capital of Australia: Canberra) and an arithmetic prompt (17 x 24 = 408). No word-salad was observed on the Gated-DeltaNet linear-attention path despite it being quantized to W4A4.

FP8 KV-cache scales load via the serving launch patch

A stock SGLang build would drop this checkpoint's baked per-layer FP8 KV scales, because qwen3_5 builds RadixAttention without a quant_config (so the KV scales fall back to 1.0). The serving launch patch (sglang_launch_sm121.sh) passes quant_config to RadixAttention and remaps/loads the baked k_proj.k_scale/v_proj.v_scale onto attn.k_scale/attn.v_scale. Verified: the runtime k_scale_float holds the calibrated per-layer values (~0.01 to 0.045), not 1.0. Upstreamed in sgl-project/sglang PR #31220.

Multimodal, vision tower kept BF16

Calibration is text-only, so the vision tower is excluded from quantization and stays BF16 (avoiding the amax=0 degenerate-quant failure mode). The export ships the processor configs (preprocessor_config.json etc.) needed to serve this as a proper multimodal checkpoint.

Benchmarks

Task	Metric	Value	Setup	Hardware	Date	Notes
GSM8K	accuracy (strict == flexible)	96.75%	chat + reasoning (qwen3 think), 4-shot, temperature 0.6, max_tokens 4500, N=400	DGX Spark GB10 / sm121; SGLang flashinfer attention + flashinfer_cutlass MoE	2026-07-15	387/400 correct; strict == flexible; 4/400 (1%) empty (generation-length cap). Uniform W4A4 (attention + experts NVFP4), FP8 KV cache.

Downloads last month: 352

Safetensors

Model size

20B params

Tensor type

BF16

F8_E4M3

Model tree for vroomfondel/Ornith-1.0-35B-NVFP4-ModelOpt

Base model

deepreinforce-ai/Ornith-1.0-35B

Quantized

(141)

this model