Instructions to use KaedeTai/dflash2-mlx-quantized-draft with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use KaedeTai/dflash2-mlx-quantized-draft with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] huggingface-cli download --local-dir dflash2-mlx-quantized-draft KaedeTai/dflash2-mlx-quantized-draft
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
fix: 16.1 GB resident (measured), not 15.5 GB of disk weights
Browse files
README.md
CHANGED
|
@@ -52,7 +52,9 @@ Stacked against no speculation at all, on the same target and machine:
|
|
| 52 |
| + bf16 drafter | 61.4 | 1.84× |
|
| 53 |
| **+ Q4 drafter** | **76.1** | **2.29×** |
|
| 54 |
|
| 55 |
-
A 27B **dense** model at 76 tok/s in
|
|
|
|
|
|
|
| 56 |
|
| 57 |
## Target-model comparison (same protocol)
|
| 58 |
|
|
|
|
| 52 |
| + bf16 drafter | 61.4 | 1.84× |
|
| 53 |
| **+ Q4 drafter** | **76.1** | **2.29×** |
|
| 54 |
|
| 55 |
+
A 27B **dense** model at 76 tok/s in **16.1 GB resident** (measured RSS with the server
|
| 56 |
+
loaded and generating; 14 GB target + 1.2 GB drafter on disk, plus KV cache — it grows
|
| 57 |
+
with context).
|
| 58 |
|
| 59 |
## Target-model comparison (same protocol)
|
| 60 |
|