Document FP8 quantization method + point serving examples to FP8 repo
Browse files
README.md
CHANGED
|
@@ -1,5 +1,6 @@
|
|
| 1 |
---
|
| 2 |
-
base_model:
|
|
|
|
| 3 |
library_name: transformers
|
| 4 |
pipeline_tag: text-generation
|
| 5 |
license: apache-2.0
|
|
@@ -10,6 +11,14 @@ license_link: https://huggingface.co/dnotitia/DNA3.0-397B-A17B/blob/main/LICENSE
|
|
| 10 |
<img src="./dna-3.0-logo.png" width="400" style="margin: 40px auto;">
|
| 11 |
</p>
|
| 12 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 13 |
# DNA 3.0
|
| 14 |
|
| 15 |
We introduce **DNA 3.0**, a large-scale Mixture-of-Experts model that builds upon the **Qwen3.5/3.6** base model with enhanced capabilities for Korean and enterprise scenarios. By applying an **Uncensored Training** methodology together with **Persona Training** (deep grounding in Dnotitia's corporate knowledge and product context), we've created a model that excels in analytical reasoning, agentic coding, and multimodal understanding while maintaining genuinely open, enterprise-aware conversational capabilities.
|
|
@@ -69,19 +78,19 @@ DNA 3.0 is compatible with the Hugging Face Transformers ecosystem as well as po
|
|
| 69 |
|
| 70 |
```shell
|
| 71 |
# Standard (multimodal) serving on 8xH200 GPUs
|
| 72 |
-
vllm serve dnotitia/DNA3.0-397B-A17B \
|
| 73 |
--tensor-parallel-size 8 \
|
| 74 |
--reasoning-parser qwen3
|
| 75 |
|
| 76 |
# Tool-calling enabled
|
| 77 |
-
vllm serve dnotitia/DNA3.0-397B-A17B \
|
| 78 |
--tensor-parallel-size 8 \
|
| 79 |
--reasoning-parser qwen3 \
|
| 80 |
--enable-auto-tool-choice \
|
| 81 |
--tool-call-parser qwen3_coder
|
| 82 |
|
| 83 |
# Text-only mode (skip vision encoder to free KV-cache memory) serving on 4xH200 GPUs
|
| 84 |
-
vllm serve dnotitia/DNA3.0-397B-A17B \
|
| 85 |
--tensor-parallel-size 4 \
|
| 86 |
--reasoning-parser qwen3 \
|
| 87 |
--language-model-only
|
|
@@ -96,7 +105,7 @@ $ curl https://demo-api.dnotitia.ai/v1/chat/completions \
|
|
| 96 |
-H "Content-Type: application/json" \
|
| 97 |
-H "Authorization: Bearer dna-router_xxxx" \
|
| 98 |
-d '{
|
| 99 |
-
"model": "DNA3.0-397B-A17B",
|
| 100 |
"messages": [
|
| 101 |
{
|
| 102 |
"role": "user",
|
|
@@ -123,7 +132,7 @@ $ curl https://demo-api.dnotitia.ai/v1/chat/completions \
|
|
| 123 |
-H "Content-Type: application/json" \
|
| 124 |
-H "Authorization: Bearer dna-router_xxxx" \
|
| 125 |
-d '{
|
| 126 |
-
"model": "DNA3.0-397B-A17B",
|
| 127 |
"messages": [
|
| 128 |
{
|
| 129 |
"role": "user",
|
|
|
|
| 1 |
---
|
| 2 |
+
base_model: dnotitia/DNA3.0-397B-A17B
|
| 3 |
+
quantized_by: llm-compressor (FP8_DYNAMIC W8A8 E4M3)
|
| 4 |
library_name: transformers
|
| 5 |
pipeline_tag: text-generation
|
| 6 |
license: apache-2.0
|
|
|
|
| 11 |
<img src="./dna-3.0-logo.png" width="400" style="margin: 40px auto;">
|
| 12 |
</p>
|
| 13 |
|
| 14 |
+
> [!Note]
|
| 15 |
+
> This repository contains FP8-quantized model weights and configuration files for the post-trained model in the Hugging Face Transformers format.
|
| 16 |
+
>
|
| 17 |
+
> These artifacts are compatible with Hugging Face Transformers, vLLM, SGLang, KTransformers, etc.
|
| 18 |
+
>
|
| 19 |
+
> The quantization method is per-channel FP8 W8A8 quantization (E4M3) with per-token dynamic activations, produced with [llm-compressor](https://github.com/vllm-project/llm-compressor) using the `FP8_DYNAMIC` scheme (no calibration data required). To preserve quality on this Mixture-of-Experts model, the token embeddings, LM head, MoE router gates, the shared expert, the small Gated-DeltaNet (linear-attention) parameters, and the vision tower are kept in BF16, while the routed experts and the remaining large linear projections are quantized to FP8. Its performance metrics are nearly identical to those of the original model.
|
| 20 |
+
|
| 21 |
+
|
| 22 |
# DNA 3.0
|
| 23 |
|
| 24 |
We introduce **DNA 3.0**, a large-scale Mixture-of-Experts model that builds upon the **Qwen3.5/3.6** base model with enhanced capabilities for Korean and enterprise scenarios. By applying an **Uncensored Training** methodology together with **Persona Training** (deep grounding in Dnotitia's corporate knowledge and product context), we've created a model that excels in analytical reasoning, agentic coding, and multimodal understanding while maintaining genuinely open, enterprise-aware conversational capabilities.
|
|
|
|
| 78 |
|
| 79 |
```shell
|
| 80 |
# Standard (multimodal) serving on 8xH200 GPUs
|
| 81 |
+
vllm serve dnotitia/DNA3.0-397B-A17B-FP8 \
|
| 82 |
--tensor-parallel-size 8 \
|
| 83 |
--reasoning-parser qwen3
|
| 84 |
|
| 85 |
# Tool-calling enabled
|
| 86 |
+
vllm serve dnotitia/DNA3.0-397B-A17B-FP8 \
|
| 87 |
--tensor-parallel-size 8 \
|
| 88 |
--reasoning-parser qwen3 \
|
| 89 |
--enable-auto-tool-choice \
|
| 90 |
--tool-call-parser qwen3_coder
|
| 91 |
|
| 92 |
# Text-only mode (skip vision encoder to free KV-cache memory) serving on 4xH200 GPUs
|
| 93 |
+
vllm serve dnotitia/DNA3.0-397B-A17B-FP8 \
|
| 94 |
--tensor-parallel-size 4 \
|
| 95 |
--reasoning-parser qwen3 \
|
| 96 |
--language-model-only
|
|
|
|
| 105 |
-H "Content-Type: application/json" \
|
| 106 |
-H "Authorization: Bearer dna-router_xxxx" \
|
| 107 |
-d '{
|
| 108 |
+
"model": "DNA3.0-397B-A17B-FP8",
|
| 109 |
"messages": [
|
| 110 |
{
|
| 111 |
"role": "user",
|
|
|
|
| 132 |
-H "Content-Type: application/json" \
|
| 133 |
-H "Authorization: Bearer dna-router_xxxx" \
|
| 134 |
-d '{
|
| 135 |
+
"model": "DNA3.0-397B-A17B-FP8",
|
| 136 |
"messages": [
|
| 137 |
{
|
| 138 |
"role": "user",
|