Update CQ branding to Coded Quantization and ThousandML
Browse files
README.md
CHANGED
|
@@ -16,17 +16,17 @@ pipeline_tag: text-generation
|
|
| 16 |
|
| 17 |
# Qwen3-Coder-Next-CQ-4bit
|
| 18 |
|
| 19 |
-
> **80B-parameter Mixture-of-Experts coding model · 3B activated parameters ·
|
| 20 |
>
|
| 21 |
> **Note on Hugging Face's model-size display:** Hugging Face may report a smaller stored parameter count for this repository because CQ stores the weights in a packed 4-bit representation. The underlying model is **Qwen3-Coder-Next**, with approximately **80B total parameters and 3B activated parameters per token**. The smaller number reflects the packed storage tensors counted by the Hub, not the parameter count of the original model architecture.
|
| 22 |
|
| 23 |
-
**Qwen3-Coder-Next quantized to 4-bit with
|
| 24 |
|
| 25 |
-
**
|
| 26 |
|
| 27 |
CQ is a **calibration-free 4-bit quantization method** designed to preserve the behavior of the original model without requiring a calibration dataset. Because CQ does not optimize the quantization around a fixed calibration corpus, it is designed to generalize beyond calibration-specific data and workloads.
|
| 28 |
|
| 29 |
-
This repository contains **Qwen3-Coder-Next** quantized with **
|
| 30 |
|
| 31 |
## Why CQ?
|
| 32 |
|
|
@@ -45,7 +45,7 @@ This avoids tying the quantization procedure to a particular calibration corpus
|
|
| 45 |
| **Base model** | `Qwen/Qwen3-Coder-Next` |
|
| 46 |
| **Original model parameters** | **~80B total / ~3B activated** |
|
| 47 |
| **Model type** | Mixture-of-Experts coding model |
|
| 48 |
-
| **Quantization** | **
|
| 49 |
| **Calibration** | **None — calibration-free** |
|
| 50 |
| **Runtime** | vLLM + CQ custom kernels |
|
| 51 |
| **Quantization name** | `cq` |
|
|
@@ -143,11 +143,11 @@ On WSL2, combine it with:
|
|
| 143 |
-e VLLM_WSL2_ENABLE_PIN_MEMORY=1
|
| 144 |
```
|
| 145 |
|
| 146 |
-
## About
|
| 147 |
|
| 148 |
Learn more at **https://thousandml.com**.
|
| 149 |
|
| 150 |
-
**
|
| 151 |
|
| 152 |
Traditional post-training quantization methods can depend on a calibration dataset to estimate quantization parameters. The resulting quantization can therefore be influenced by the distribution and composition of that calibration data.
|
| 153 |
|
|
|
|
| 16 |
|
| 17 |
# Qwen3-Coder-Next-CQ-4bit
|
| 18 |
|
| 19 |
+
> **80B-parameter Mixture-of-Experts coding model · 3B activated parameters · Coded Quantization (CQ) 4-bit**
|
| 20 |
>
|
| 21 |
> **Note on Hugging Face's model-size display:** Hugging Face may report a smaller stored parameter count for this repository because CQ stores the weights in a packed 4-bit representation. The underlying model is **Qwen3-Coder-Next**, with approximately **80B total parameters and 3B activated parameters per token**. The smaller number reflects the packed storage tensors counted by the Hub, not the parameter count of the original model architecture.
|
| 22 |
|
| 23 |
+
**Qwen3-Coder-Next quantized to 4-bit with Coded Quantization (CQ).**
|
| 24 |
|
| 25 |
+
**ThousandML:** https://thousandml.com
|
| 26 |
|
| 27 |
CQ is a **calibration-free 4-bit quantization method** designed to preserve the behavior of the original model without requiring a calibration dataset. Because CQ does not optimize the quantization around a fixed calibration corpus, it is designed to generalize beyond calibration-specific data and workloads.
|
| 28 |
|
| 29 |
+
This repository contains **Qwen3-Coder-Next** quantized with **Coded Quantization (CQ) 4-bit**.
|
| 30 |
|
| 31 |
## Why CQ?
|
| 32 |
|
|
|
|
| 45 |
| **Base model** | `Qwen/Qwen3-Coder-Next` |
|
| 46 |
| **Original model parameters** | **~80B total / ~3B activated** |
|
| 47 |
| **Model type** | Mixture-of-Experts coding model |
|
| 48 |
+
| **Quantization** | **Coded Quantization (CQ) 4-bit** |
|
| 49 |
| **Calibration** | **None — calibration-free** |
|
| 50 |
| **Runtime** | vLLM + CQ custom kernels |
|
| 51 |
| **Quantization name** | `cq` |
|
|
|
|
| 143 |
-e VLLM_WSL2_ENABLE_PIN_MEMORY=1
|
| 144 |
```
|
| 145 |
|
| 146 |
+
## About Coded Quantization (CQ)
|
| 147 |
|
| 148 |
Learn more at **https://thousandml.com**.
|
| 149 |
|
| 150 |
+
**Coded Quantization (CQ)** is a calibration-free quantization approach for efficient LLM inference.
|
| 151 |
|
| 152 |
Traditional post-training quantization methods can depend on a calibration dataset to estimate quantization parameters. The resulting quantization can therefore be influenced by the distribution and composition of that calibration data.
|
| 153 |
|