README.md

---
language:
- en
- multilingual
base_model:
- deepreinforce-ai/Ornith-1.0-9B
pipeline_tag: text-generation
tags:
- qwen
- qwen3.5
- fp8
- vllm
- conversational
- text-generation-inference
- reasoning
- agentic
- tool-calling
license: apache-2.0
---

## Model Overview
- **Model Architecture:** Qwen3_5ForCausalLM
  - **Input:** Text
  - **Output:** Text
- **Model Optimizations:**
  - **Activation quantization:** FP8
  - **Weight quantization:** FP8
- **Intended Use Cases:** Intended for commercial and research use. Similarly to the base model, this quantized version is intended for agentic coding tasks, terminal-based workflows, and tool-calling applications.
- **Out-of-scope:** Use in any manner that violates applicable laws or regulations (including trade compliance laws).
- **Version:** 1.0
- **Model Developers:** inference-optimization (Neural Magic / RedHat)

### Model Optimizations

This model was obtained by quantizing activations and weights of [deepreinforce-ai/Ornith-1.0-9B](https://huggingface.co/deepreinforce-ai/Ornith-1.0-9B) to FP8 data type.
This optimization reduces the number of bits used to represent weights and activations from 16 to 8, reducing GPU memory requirements (by approximately 50%) and increasing matrix-multiply compute throughput (by approximately 2x).
Weight quantization also reduces disk size requirements by approximately 50%.

Only weights and activations of the linear operators within transformers blocks are quantized.
Weights are quantized with a symmetric static per-channel scheme, whereas activations are quantized with a symmetric dynamic per-token scheme.
The [llm-compressor](https://github.com/vllm-project/llm-compressor) library is used for quantization.

## Deployment

### Using Transformers

This model can be deployed using the Hugging Face Transformers library:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "inference-optimization/Ornith-1.0-9B-FP8-Dynamic"

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

prompt = "Explain quantum computing in simple terms."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=200,
        temperature=0.6,
        do_sample=True,
        top_p=0.95,
        top_k=20
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

### Using vLLM (Note: Compatibility Issue)

**Important:** Due to architectural differences between the original multimodal Ornith model and the text-only quantized version, this model currently has compatibility issues with vLLM when the model architecture expects vision components. We recommend using the Transformers library for deployment until this is resolved.

## Model Details

- **Base Model:** deepreinforce-ai/Ornith-1.0-9B (built on Qwen 3.5 and Gemma 4)
- **Model Size:** ~9B parameters
- **Disk Size:** ~11 GB (down from ~19 GB in bf16)
- **Context Length:** Up to 262,144 tokens (reduced context recommended for faster inference)
- **Reasoning:** Supports `<think>...</think>` blocks for chain-of-thought reasoning
- **Tool Calling:** XML-based function calls compatible with OpenAI format

## Performance

This FP8 quantized model provides:
- **~50% memory reduction** compared to the bf16 base model
- **~2x faster inference** for matrix multiplication operations
- **Maintained accuracy** for reasoning and coding tasks

## Limitations

- The quantization process removes multimodal (vision) capabilities present in the base model
- This is a text-only model suitable for language tasks, coding, and reasoning
- vLLM compatibility is currently limited due to vision config expectations

## Creation

<details>
  <summary>Creation details</summary>

  This model was created with [llm-compressor](https://github.com/vllm-project/llm-compressor) by running the code snippet below.

  ```python
from transformers import AutoModelForCausalLM, AutoTokenizer

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

MODEL_ID = "deepreinforce-ai/Ornith-1.0-9B"

# Load model.
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    low_cpu_mem_usage=True,
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

recipe = QuantizationModifier(
    targets=["Linear"],
    scheme="FP8_DYNAMIC",
    ignore=["lm_head"],
)

# Apply quantization.
oneshot(model=model, recipe=recipe)

# Save to disk in compressed-tensors format.
SAVE_DIR = "Ornith-1.0-9B-FP8-Dynamic"
model.save_pretrained(SAVE_DIR, save_compressed=True)
tokenizer.save_pretrained(SAVE_DIR)
  ```
</details>

## Citation

If you use this model, please cite:

```bibtex
@software{ornith_fp8_dynamic,
  author = {inference-optimization},
  title = {Ornith-1.0-9B-FP8-Dynamic},
  year = {2026},
  publisher = {HuggingFace},
  url = {https://huggingface.co/inference-optimization/Ornith-1.0-9B-FP8-Dynamic}
}
```

Original base model:
```bibtex
@software{ornith,
  author = {DeepReinforce AI},
  title = {Ornith-1.0-9B},
  year = {2025},
  publisher = {HuggingFace},
  url = {https://huggingface.co/deepreinforce-ai/Ornith-1.0-9B}
}
```