Instructions to use whpthomas/Ornith-1.0-35B-int4-AutoRound with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use whpthomas/Ornith-1.0-35B-int4-AutoRound with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="whpthomas/Ornith-1.0-35B-int4-AutoRound")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("whpthomas/Ornith-1.0-35B-int4-AutoRound")
model = AutoModelForMultimodalLM.from_pretrained("whpthomas/Ornith-1.0-35B-int4-AutoRound")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use whpthomas/Ornith-1.0-35B-int4-AutoRound with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "whpthomas/Ornith-1.0-35B-int4-AutoRound"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "whpthomas/Ornith-1.0-35B-int4-AutoRound",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/whpthomas/Ornith-1.0-35B-int4-AutoRound

SGLang

How to use whpthomas/Ornith-1.0-35B-int4-AutoRound with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "whpthomas/Ornith-1.0-35B-int4-AutoRound" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "whpthomas/Ornith-1.0-35B-int4-AutoRound",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "whpthomas/Ornith-1.0-35B-int4-AutoRound" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "whpthomas/Ornith-1.0-35B-int4-AutoRound",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use whpthomas/Ornith-1.0-35B-int4-AutoRound with Docker Model Runner:
```
docker model run hf.co/whpthomas/Ornith-1.0-35B-int4-AutoRound
```

Ornith-1.0-35B-A3B-int4-AutoRound

A 4-bit AutoRound quantized version of deepreinforce-ai/Ornith-1.0-35B, optimized for efficient inference on consumer hardware including the NVIDIA DGX Spark (GB10).

Model Details

Property	Value
Base Model	deepreinforce-ai/Ornith-1.0-35B
Architecture	Qwen3.5-MoE (Mixture of Experts)
Quantization	int4 AutoRound (W4A16)
Bits	4-bit
Group Size	128
Symmetric	Yes
Calibration Dataset	NVIDIA OpenCodeInstruct
Quantization Tool	spark-auto-round v0.14.3
Context Length	196,608 tokens
Model Size	~19 GB (quantized)

Architecture

Type: Qwen3.5-MoE with Vision (multimodal)
Hidden Size: 2,048
Num Layers: 40
Num Experts: 256
Experts Per Token: 8
Attention Heads: 16 (full attention every 4th layer, linear attention otherwise)
Vocab Size: 248,320

Quantization Details

This model was quantized using AutoRound, an advanced quantization technique from Intel that uses signed gradient descent to jointly optimize weight rounding and clipping ranges.

Key Settings

--batch_size 8
--iters 1000
--nsamples 512
--seqlen 2048
--dataset opencode-instruct
--group_size 128

Quantization Quality

Metric	Value
Peak RAM	111.31 GB
Peak VRAM	28.00 GB
Layers Passed	22/40 (55%)
Layers Warning	18/40 (45%)

The model retains strong quality with all layers meeting minimum thresholds. Layers 22-39 show slightly elevated sensitivity (cosine similarity 0.986-0.990) which is typical for deeper MoE layers.

Note: Shared expert gates are preserved in FP16 to maintain MoE routing accuracy.

Usage with Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model_path = "whpthomas/Ornith-1.0-35B-int4-AutoRound"

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_path)

# Load quantized model
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    device_map="auto",
    torch_dtype="auto",
)

# Generate text
messages = [
    {"role": "user", "content": "Explain quantum computing in simple terms."}
]

text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Usage with vLLM

For optimal performance, use vLLM with the recommended serving configuration:

Installation

pip install vllm

Basic Serving

vllm serve whpthomas/Ornith-1.0-35B-int4-AutoRound \
  --served-model-name ornith-1.0-35b \
  --max-model-len 196608 \
  --gpu-memory-utilization 0.55 \
  --load-format auto \
  --attention-backend flashinfer \
  --moe-backend marlin \
  --enable-prefix-caching \
  --enable-chunked-prefill

Recommended Production Configuration

For DGX Spark (GB10) with optimal performance:

vllm serve whpthomas/Ornith-1.0-35B-int4-AutoRound \
  --served-model-name ornith-1.0-35b \
  --max-model-len 196608 \
  --gpu-memory-utilization 0.55 \
  --max-num-batched-tokens 16384 \
  --max-num-seqs 8 \
  --optimization-level 3 \
  --performance-mode throughput \
  --load-format instanttensor \
  --attention-backend flashinfer \
  --moe-backend marlin \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --default-chat-template-kwargs '{"preserve_thinking":true}' \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --generation-config auto \
  --override-generation-config '{"temperature":0.7,"top_p":0.95,"top_k":-1,"min_p":0.0,"presence_penalty":0.0,"repetition_penalty":1.0}'

Environment Variables (DGX Spark)

export TORCH_MATMUL_PRECISION=high
export NVIDIA_FORWARD_COMPAT=1
export NVIDIA_DISABLE_REQUIRE=1
export CUDA_DEVICE_MAX_CONNECTIONS=1
export VLLM_MARLIN_USE_ATOMIC_ADD=1
export FLASHINFER_DISABLE_VERSION_CHECK=1

Tool Calling

This model supports tool calling with the Qwen3 coder parser. When using vLLM with --enable-auto-tool-choice --tool-call-parser qwen3_coder, the model can invoke tools and return structured function calls.

Reasoning

The model supports extended thinking with the Qwen3 reasoning parser. Use --reasoning-parser qwen3 and set preserve_thinking: true in chat template kwargs to enable reasoning traces.

Performance Characteristics

Based on testing with the Qwen3.5-MoE architecture on DGX Spark:

Metric	Value
Throughput (MTP)	~26-30 t/s
Throughput (DFlash)	~35-40 t/s
Latency (Time to First Token)	~100-200ms

Performance varies based on context length, batch size, and hardware configuration.

Citation

If you use this quantized model, please cite both the base model and the quantization tool:

@misc{ornith2025,
  title={Ornith-1.0-35B},
  author={DeepReinforce AI},
  year={2025},
  publisher={HuggingFace},
  url={https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B}
}

@misc{sparkautaround2025,
  title={Spark Auto Round},
  author={whpthomas},
  year={2025},
  publisher={GitHub},
  url={https://github.com/whpthomas/spark-auto-round}
}

Acknowledgments

Base Model: deepreinforce-ai/Ornith-1.0-35B - The original bf16 model
Quantization: Intel auto-round - The underlying quantization framework
Spark Auto Round: spark-auto-round - GB10-optimized quantization wrapper
vLLM Serving: spark-vllm-docker - Community vLLM Docker configuration for DGX Spark

License

The quantized model inherits the license of the base model. Please refer to the original model card for license details.

Model Type

This is an int4 AutoRound quantized model. It requires:

A CUDA-capable GPU
vLLM (recommended) or transformers with auto-round support
Sufficient GPU memory (~20GB+ depending on context length)

Downloads last month: 679

Safetensors

Model size

1B params

Tensor type

I32

BF16

F16

Model tree for whpthomas/Ornith-1.0-35B-int4-AutoRound

Base model

deepreinforce-ai/Ornith-1.0-35B

Quantized

(142)

this model