Upload folder using huggingface_hub
Browse filesThis view is limited to 50 files because it contains too many changes. See raw diff
- DeepSeek_V4.pdf +2 -2
- README.md +224 -41
- config.json +937 -1
- inference/README.md +4 -3
- inference/generate.py +0 -4
- inference/model.py +16 -181
- inference/requirements.txt +1 -2
- model-00002-of-00046.safetensors +2 -2
- model-00003-of-00046.safetensors +2 -2
- model-00004-of-00046.safetensors +2 -2
- model-00005-of-00046.safetensors +2 -2
- model-00006-of-00046.safetensors +2 -2
- model-00007-of-00046.safetensors +2 -2
- model-00008-of-00046.safetensors +2 -2
- model-00009-of-00046.safetensors +2 -2
- model-00010-of-00046.safetensors +2 -2
- model-00011-of-00046.safetensors +2 -2
- model-00012-of-00046.safetensors +2 -2
- model-00013-of-00046.safetensors +2 -2
- model-00014-of-00046.safetensors +2 -2
- model-00015-of-00046.safetensors +2 -2
- model-00016-of-00046.safetensors +2 -2
- model-00017-of-00046.safetensors +2 -2
- model-00018-of-00046.safetensors +2 -2
- model-00019-of-00046.safetensors +2 -2
- model-00020-of-00046.safetensors +2 -2
- model-00021-of-00046.safetensors +2 -2
- model-00022-of-00046.safetensors +2 -2
- model-00023-of-00046.safetensors +2 -2
- model-00024-of-00046.safetensors +2 -2
- model-00025-of-00046.safetensors +2 -2
- model-00026-of-00046.safetensors +2 -2
- model-00027-of-00046.safetensors +2 -2
- model-00028-of-00046.safetensors +2 -2
- model-00029-of-00046.safetensors +2 -2
- model-00030-of-00046.safetensors +2 -2
- model-00031-of-00046.safetensors +2 -2
- model-00032-of-00046.safetensors +2 -2
- model-00033-of-00046.safetensors +2 -2
- model-00034-of-00046.safetensors +2 -2
- model-00035-of-00046.safetensors +2 -2
- model-00036-of-00046.safetensors +2 -2
- model-00037-of-00046.safetensors +2 -2
- model-00038-of-00046.safetensors +2 -2
- model-00039-of-00046.safetensors +2 -2
- model-00040-of-00046.safetensors +2 -2
- model-00041-of-00046.safetensors +2 -2
- model-00042-of-00046.safetensors +2 -2
- model-00043-of-00046.safetensors +2 -2
- model-00044-of-00046.safetensors +2 -2
DeepSeek_V4.pdf
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:8a03dadae71894de1515da33e296f0df1dbeed3e7f4bf0badd087f9af77f29e9
|
| 3 |
+
size 4480407
|
README.md
CHANGED
|
@@ -1,57 +1,240 @@
|
|
| 1 |
---
|
| 2 |
license: mit
|
| 3 |
library_name: transformers
|
| 4 |
-
base_model:
|
| 5 |
-
- deepseek-ai/DeepSeek-V4-Flash
|
| 6 |
---
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 12 |
|
| 13 |
## How to Run Locally
|
| 14 |
|
| 15 |
-
**vLLM and Sglang is not supported currently: https://huggingface.co/Intel/DeepSeek-V4-Flash-W4A16-AutoRound/discussions/1**
|
| 16 |
-
|
| 17 |
Please refer to the [inference](inference/README.md) folder for detailed instructions on running DeepSeek-V4 locally, including model weight conversion and interactive chat demos.
|
| 18 |
|
| 19 |
For local deployment, we recommend setting the sampling parameters to `temperature = 1.0, top_p = 1.0`. For the Think Max reasoning mode, we recommend setting the context window to at least **384K** tokens.
|
| 20 |
|
|
|
|
| 21 |
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
~~~bash
|
| 25 |
-
auto-round deepseek-ai/DeepSeek-V4-Flash \
|
| 26 |
-
--model_free \
|
| 27 |
-
--scheme W4A16 \
|
| 28 |
-
--ignore_layers compressor,indexer.weights_proj \
|
| 29 |
-
--layer_config "{'wo_a':{bits:16}}" \
|
| 30 |
-
--output_dir "./DeepSeek-V4-Flash-W4A16"
|
| 31 |
-
~~~
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
## Ethical Considerations and Limitations
|
| 36 |
-
|
| 37 |
-
The model can produce factually incorrect output, and should not be relied on to produce factually accurate information. Because of the limitations of the pretrained model and the finetuning datasets, it is possible that this model could generate lewd, biased or otherwise offensive outputs.
|
| 38 |
-
|
| 39 |
-
Therefore, before deploying any applications of the model, developers should perform safety testing.
|
| 40 |
-
|
| 41 |
-
## Caveats and Recommendations
|
| 42 |
-
|
| 43 |
-
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
|
| 44 |
-
|
| 45 |
-
Here are a couple of useful links to learn more about Intel's AI software:
|
| 46 |
-
|
| 47 |
-
- [Intel Neural Compressor](https://github.com/intel/neural-compressor)
|
| 48 |
-
|
| 49 |
-
## Disclaimer
|
| 50 |
|
| 51 |
-
|
| 52 |
|
| 53 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 54 |
|
| 55 |
-
|
| 56 |
|
| 57 |
-
|
|
|
|
| 1 |
---
|
| 2 |
license: mit
|
| 3 |
library_name: transformers
|
|
|
|
|
|
|
| 4 |
---
|
| 5 |
+
# DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
|
| 6 |
+
|
| 7 |
+
<!-- markdownlint-disable first-line-h1 -->
|
| 8 |
+
<!-- markdownlint-disable html -->
|
| 9 |
+
<!-- markdownlint-disable no-duplicate-header -->
|
| 10 |
+
|
| 11 |
+
<div align="center">
|
| 12 |
+
<img src="https://github.com/deepseek-ai/DeepSeek-V2/blob/main/figures/logo.svg?raw=true" width="60%" alt="DeepSeek-V4" />
|
| 13 |
+
</div>
|
| 14 |
+
<hr>
|
| 15 |
+
<div align="center" style="line-height: 1;">
|
| 16 |
+
<a href="https://www.deepseek.com/" target="_blank" style="margin: 2px;">
|
| 17 |
+
<img alt="Homepage" src="https://github.com/deepseek-ai/DeepSeek-V2/blob/main/figures/badge.svg?raw=true" style="display: inline-block; vertical-align: middle;"/>
|
| 18 |
+
</a>
|
| 19 |
+
<a href="https://chat.deepseek.com/" target="_blank" style="margin: 2px;">
|
| 20 |
+
<img alt="Chat" src="https://img.shields.io/badge/🤖%20Chat-DeepSeek%20V4-536af5?color=536af5&logoColor=white" style="display: inline-block; vertical-align: middle;"/>
|
| 21 |
+
</a>
|
| 22 |
+
</div>
|
| 23 |
+
<div align="center" style="line-height: 1;">
|
| 24 |
+
<a href="https://huggingface.co/deepseek-ai" target="_blank" style="margin: 2px;">
|
| 25 |
+
<img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-DeepSeek%20AI-ffc107?color=ffc107&logoColor=white" style="display: inline-block; vertical-align: middle;"/>
|
| 26 |
+
</a>
|
| 27 |
+
<a href="https://twitter.com/deepseek_ai" target="_blank" style="margin: 2px;">
|
| 28 |
+
<img alt="Twitter Follow" src="https://img.shields.io/badge/Twitter-deepseek_ai-white?logo=x&logoColor=white" style="display: inline-block; vertical-align: middle;"/>
|
| 29 |
+
</a>
|
| 30 |
+
</div>
|
| 31 |
+
<div align="center" style="line-height: 1;">
|
| 32 |
+
<a href="LICENSE" style="margin: 2px;">
|
| 33 |
+
<img alt="License" src="https://img.shields.io/badge/License-MIT-f5de53?&color=f5de53" style="display: inline-block; vertical-align: middle;"/>
|
| 34 |
+
</a>
|
| 35 |
+
</div>
|
| 36 |
+
|
| 37 |
+
<p align="center">
|
| 38 |
+
<a href="https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/DeepSeek_V4.pdf"><b>Technical Report</b>👁️</a>
|
| 39 |
+
</p>
|
| 40 |
+
|
| 41 |
+
## Introduction
|
| 42 |
+
|
| 43 |
+
We present a preview version of **DeepSeek-V4** series, including two strong Mixture-of-Experts (MoE) language models — **DeepSeek-V4-Pro** with 1.6T parameters (49B activated) and **DeepSeek-V4-Flash** with 284B parameters (13B activated) — both supporting a context length of **one million tokens**.
|
| 44 |
+
|
| 45 |
+
DeepSeek-V4 series incorporate several key upgrades in architecture and optimization:
|
| 46 |
+
|
| 47 |
+
1. **Hybrid Attention Architecture:** We design a hybrid attention mechanism combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to dramatically improve long-context efficiency. In the 1M-token context setting, DeepSeek-V4-Pro requires only **27% of single-token inference FLOPs** and **10% of KV cache** compared with DeepSeek-V3.2.
|
| 48 |
+
2. **Manifold-Constrained Hyper-Connections (mHC):** We incorporate mHC to strengthen conventional residual connections, enhancing stability of signal propagation across layers while preserving model expressivity.
|
| 49 |
+
3. **Muon Optimizer:** We employ the Muon optimizer for faster convergence and greater training stability.
|
| 50 |
+
|
| 51 |
+
We pre-train both models on more than **32T** diverse and high-quality tokens, followed by a comprehensive post-training pipeline. The post-training features a two-stage paradigm: independent cultivation of domain-specific experts (through SFT and RL with GRPO), followed by unified model consolidation via on-policy distillation, integrating distinct proficiencies across diverse domains into a single model.
|
| 52 |
+
|
| 53 |
+
**DeepSeek-V4-Pro-Max**, the maximum reasoning effort mode of DeepSeek-V4-Pro, significantly advances the knowledge capabilities of open-source models, firmly establishing itself as the best open-source model available today. It achieves top-tier performance in coding benchmarks and significantly bridges the gap with leading closed-source models on reasoning and agentic tasks. Meanwhile, **DeepSeek-V4-Flash-Max** achieves comparable reasoning performance to the Pro version when given a larger thinking budget, though its smaller parameter scale naturally places it slightly behind on pure knowledge tasks and the most complex agentic workflows.
|
| 54 |
+
|
| 55 |
+
<div align="center">
|
| 56 |
+
<img src="assets/dsv4_performance.png" >
|
| 57 |
+
</div>
|
| 58 |
+
|
| 59 |
+
## Model Downloads
|
| 60 |
+
|
| 61 |
+
<div align="center">
|
| 62 |
+
|
| 63 |
+
| **Model** | **#Total Params** | **#Activated Params** | **Context Length** | **Precision** | **Download** |
|
| 64 |
+
| :---: | :---: | :---: | :---: | :---: | :---: |
|
| 65 |
+
| DeepSeek-V4-Flash-Base | 284B | 13B | 1M | FP8 Mixed | [HuggingFace](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Base) \| [ModelScope](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Flash-Base) |
|
| 66 |
+
| DeepSeek-V4-Flash | 284B | 13B | 1M | FP4 + FP8 Mixed* | [HuggingFace](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) \| [ModelScope](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Flash) |
|
| 67 |
+
| DeepSeek-V4-Pro-Base | 1.6T | 49B | 1M | FP8 Mixed | [HuggingFace](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-Base) \| [ModelScope](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Pro-Base) |
|
| 68 |
+
| DeepSeek-V4-Pro | 1.6T | 49B | 1M | FP4 + FP8 Mixed* | [HuggingFace](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro) \| [ModelScope](https://modelscope.cn/models/deepseek-ai/DeepSeek-V4-Pro) |
|
| 69 |
+
|
| 70 |
+
</div>
|
| 71 |
+
|
| 72 |
+
*\*FP4 + FP8 Mixed: MoE expert parameters use FP4 precision; most other parameters use FP8.*
|
| 73 |
+
|
| 74 |
+
## Evaluation Results
|
| 75 |
+
|
| 76 |
+
### Base Model
|
| 77 |
+
|
| 78 |
+
<div align="center">
|
| 79 |
+
|
| 80 |
+
| Benchmark (Metric) | # Shots | DeepSeek-V3.2-Base | DeepSeek-V4-Flash-Base | DeepSeek-V4-Pro-Base |
|
| 81 |
+
| :--- | :---: | :---: | :---: | :---: |
|
| 82 |
+
| Architecture | - | MoE | MoE | MoE |
|
| 83 |
+
| # Activated Params | - | 37B | 13B | 49B |
|
| 84 |
+
| # Total Params | - | 671B | 284B | 1.6T |
|
| 85 |
+
| **World Knowledge** | | | | |
|
| 86 |
+
| AGIEval (EM) | 0-shot | 80.1 | 82.6 | **83.1** |
|
| 87 |
+
| MMLU (EM) | 5-shot | 87.8 | 88.7 | **90.1** |
|
| 88 |
+
| MMLU-Redux (EM) | 5-shot | 87.5 | 89.4 | **90.8** |
|
| 89 |
+
| MMLU-Pro (EM) | 5-shot | 65.5 | 68.3 | **73.5** |
|
| 90 |
+
| MMMLU (EM) | 5-shot | 87.9 | 88.8 | **90.3** |
|
| 91 |
+
| C-Eval (EM) | 5-shot | 90.4 | 92.1 | **93.1** |
|
| 92 |
+
| CMMLU (EM) | 5-shot | 88.9 | 90.4 | **90.8** |
|
| 93 |
+
| MultiLoKo (EM) | 5-shot | 38.7 | 42.2 | **51.1** |
|
| 94 |
+
| Simple-QA verified (EM) | 25-shot | 28.3 | 30.1 | **55.2** |
|
| 95 |
+
| SuperGPQA (EM) | 5-shot | 45.0 | 46.5 | **53.9** |
|
| 96 |
+
| FACTS Parametric (EM) | 25-shot | 27.1 | 33.9 | **62.6** |
|
| 97 |
+
| TriviaQA (EM) | 5-shot | 83.3 | 82.8 | **85.6** |
|
| 98 |
+
| **Language & Reasoning** | | | | |
|
| 99 |
+
| BBH (EM) | 3-shot | **87.6** | 86.9 | 87.5 |
|
| 100 |
+
| DROP (F1) | 1-shot | 88.2 | 88.6 | **88.7** |
|
| 101 |
+
| HellaSwag (EM) | 0-shot | 86.4 | 85.7 | **88.0** |
|
| 102 |
+
| WinoGrande (EM) | 0-shot | 78.9 | 79.5 | **81.5** |
|
| 103 |
+
| CLUEWSC (EM) | 5-shot | 83.5 | 82.2 | **85.2** |
|
| 104 |
+
| **Code & Math** | | | | |
|
| 105 |
+
| BigCodeBench (Pass@1) | 3-shot | **63.9** | 56.8 | 59.2 |
|
| 106 |
+
| HumanEval (Pass@1) | 0-shot | 62.8 | 69.5 | **76.8** |
|
| 107 |
+
| GSM8K (EM) | 8-shot | 91.1 | 90.8 | **92.6** |
|
| 108 |
+
| MATH (EM) | 4-shot | 60.5 | 57.4 | **64.5** |
|
| 109 |
+
| MGSM (EM) | 8-shot | 81.3 | **85.7** | 84.4 |
|
| 110 |
+
| CMath (EM) | 3-shot | 92.6 | **93.6** | 90.9 |
|
| 111 |
+
| **Long Context** | | | | |
|
| 112 |
+
| LongBench-V2 (EM) | 1-shot | 40.2 | 44.7 | **51.5** |
|
| 113 |
+
|
| 114 |
+
</div>
|
| 115 |
+
|
| 116 |
+
### Instruct Model
|
| 117 |
+
|
| 118 |
+
DeepSeek-V4-Pro and DeepSeek-V4-Flash both support three reasoning effort modes:
|
| 119 |
+
|
| 120 |
+
| Reasoning Mode | Characteristics | Typical Use Cases | Response Format |
|
| 121 |
+
| :--- | :--- | :--- | :--- |
|
| 122 |
+
| Non-think | Fast, intuitive responses | Routine daily tasks, low-risk decisions | `</think>` summary |
|
| 123 |
+
| Think High | Conscious logical analysis, slower but more accurate | Complex problem-solving, planning | `<think>` thinking `</think>` summary |
|
| 124 |
+
| Think Max | Push reasoning to its fullest extent | Exploring the boundary of model reasoning capability | Special system prompt + `<think>` thinking `</think>` summary |
|
| 125 |
+
|
| 126 |
+
#### DeepSeek-V4-Pro-Max vs Frontier Models
|
| 127 |
+
|
| 128 |
+
<div align="center">
|
| 129 |
+
|
| 130 |
+
| Benchmark (Metric) | Opus-4.6 Max | GPT-5.4 xHigh | Gemini-3.1-Pro High | K2.6 Thinking | GLM-5.1 Thinking | DS-V4-Pro Max |
|
| 131 |
+
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 132 |
+
| **Knowledge & Reasoning** | | | | | | |
|
| 133 |
+
| MMLU-Pro (EM) | 89.1 | 87.5 | **91.0** | 87.1 | 86.0 | 87.5 |
|
| 134 |
+
| SimpleQA-Verified (Pass@1) | 46.2 | 45.3 | **75.6** | 36.9 | 38.1 | 57.9 |
|
| 135 |
+
| Chinese-SimpleQA (Pass@1) | 76.4 | 76.8 | **85.9** | 75.9 | 75.0 | 84.4 |
|
| 136 |
+
| GPQA Diamond (Pass@1) | 91.3 | 93.0 | **94.3** | 90.5 | 86.2 | 90.1 |
|
| 137 |
+
| HLE (Pass@1) | 40.0 | 39.8 | **44.4** | 36.4 | 34.7 | 37.7 |
|
| 138 |
+
| LiveCodeBench (Pass@1) | 88.8 | - | 91.7 | 89.6 | - | **93.5** |
|
| 139 |
+
| Codeforces (Rating) | - | 3168 | 3052 | - | - | **3206** |
|
| 140 |
+
| HMMT 2026 Feb (Pass@1) | 96.2 | **97.7** | 94.7 | 92.7 | 89.4 | 95.2 |
|
| 141 |
+
| IMOAnswerBench (Pass@1) | 75.3 | **91.4** | 81.0 | 86.0 | 83.8 | 89.8 |
|
| 142 |
+
| Apex (Pass@1) | 34.5 | 54.1 | **60.9** | 24.0 | 11.5 | 38.3 |
|
| 143 |
+
| Apex Shortlist (Pass@1) | 85.9 | 78.1 | 89.1 | 75.5 | 72.4 | **90.2** |
|
| 144 |
+
| **Long Context** | | | | | | |
|
| 145 |
+
| MRCR 1M (MMR) | **92.9** | - | 76.3 | - | - | 83.5 |
|
| 146 |
+
| CorpusQA 1M (ACC) | **71.7** | - | 53.8 | - | - | 62.0 |
|
| 147 |
+
| **Agentic** | | | | | | |
|
| 148 |
+
| Terminal Bench 2.0 (Acc) | 65.4 | **75.1** | 68.5 | 66.7 | 63.5 | 67.9 |
|
| 149 |
+
| SWE Verified (Resolved) | **80.8** | - | 80.6 | 80.2 | - | 80.6 |
|
| 150 |
+
| SWE Pro (Resolved) | 57.3 | 57.7 | 54.2 | **58.6** | 58.4 | 55.4 |
|
| 151 |
+
| SWE Multilingual (Resolved) | **77.5** | - | - | 76.7 | 73.3 | 76.2 |
|
| 152 |
+
| BrowseComp (Pass@1) | 83.7 | 82.7 | **85.9** | 83.2 | 79.3 | 83.4 |
|
| 153 |
+
| HLE w/ tools (Pass@1) | 53.1 | 52.0 | 51.6 | **54.0** | 50.4 | 48.2 |
|
| 154 |
+
| GDPval-AA (Elo) | 1619 | **1674** | 1314 | 1482 | 1535 | 1554 |
|
| 155 |
+
| MCPAtlas Public (Pass@1) | **73.8** | 67.2 | 69.2 | 66.6 | 71.8 | 73.6 |
|
| 156 |
+
| Toolathlon (Pass@1) | 47.2 | **54.6** | 48.8 | 50.0 | 40.7 | 51.8 |
|
| 157 |
+
|
| 158 |
+
</div>
|
| 159 |
+
|
| 160 |
+
#### Comparison across Modes
|
| 161 |
+
|
| 162 |
+
<div align="center">
|
| 163 |
+
|
| 164 |
+
| Benchmark (Metric) | V4-Flash Non-Think | V4-Flash High | V4-Flash Max | V4-Pro Non-Think | V4-Pro High | V4-Pro Max |
|
| 165 |
+
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 166 |
+
| **Knowledge & Reasoning** | | | | | | |
|
| 167 |
+
| MMLU-Pro (EM) | 83.0 | 86.4 | 86.2 | 82.9 | 87.1 | **87.5** |
|
| 168 |
+
| SimpleQA-Verified (Pass@1) | 23.1 | 28.9 | 34.1 | 45.0 | 46.2 | **57.9** |
|
| 169 |
+
| Chinese-SimpleQA (Pass@1) | 71.5 | 73.2 | 78.9 | 75.8 | 77.7 | **84.4** |
|
| 170 |
+
| GPQA Diamond (Pass@1) | 71.2 | 87.4 | 88.1 | 72.9 | 89.1 | **90.1** |
|
| 171 |
+
| HLE (Pass@1) | 8.1 | 29.4 | 34.8 | 7.7 | 34.5 | **37.7** |
|
| 172 |
+
| LiveCodeBench (Pass@1) | 55.2 | 88.4 | 91.6 | 56.8 | 89.8 | **93.5** |
|
| 173 |
+
| Codeforces (Rating) | - | 2816 | 3052 | - | 2919 | **3206** |
|
| 174 |
+
| HMMT 2026 Feb (Pass@1) | 40.8 | 91.9 | 94.8 | 31.7 | 94.0 | **95.2** |
|
| 175 |
+
| IMOAnswerBench (Pass@1) | 41.9 | 85.1 | 88.4 | 35.3 | 88.0 | **89.8** |
|
| 176 |
+
| Apex (Pass@1) | 1.0 | 19.1 | 33.0 | 0.4 | 27.4 | **38.3** |
|
| 177 |
+
| Apex Shortlist (Pass@1) | 9.3 | 72.1 | 85.7 | 9.2 | 85.5 | **90.2** |
|
| 178 |
+
| **Long Context** | | | | | | |
|
| 179 |
+
| MRCR 1M (MMR) | 37.5 | 76.9 | 78.7 | 44.7 | 83.3 | **83.5** |
|
| 180 |
+
| CorpusQA 1M (ACC) | 15.5 | 59.3 | 60.5 | 35.6 | 56.5 | **62.0** |
|
| 181 |
+
| **Agentic** | | | | | | |
|
| 182 |
+
| Terminal Bench 2.0 (Acc) | 49.1 | 56.6 | 56.9 | 59.1 | 63.3 | **67.9** |
|
| 183 |
+
| SWE Verified (Resolved) | 73.7 | 78.6 | 79.0 | 73.6 | 79.4 | **80.6** |
|
| 184 |
+
| SWE Pro (Resolved) | 49.1 | 52.3 | 52.6 | 52.1 | 54.4 | **55.4** |
|
| 185 |
+
| SWE Multilingual (Resolved) | 69.7 | 70.2 | 73.3 | 69.8 | 74.1 | **76.2** |
|
| 186 |
+
| BrowseComp (Pass@1) | - | 53.5 | 73.2 | - | 80.4 | **83.4** |
|
| 187 |
+
| HLE w/ tools (Pass@1) | - | 40.3 | 45.1 | - | 44.7 | **48.2** |
|
| 188 |
+
| MCPAtlas (Pass@1) | 64.0 | 67.4 | 69.0 | 69.4 | **74.2** | 73.6 |
|
| 189 |
+
| GDPval-AA (Elo) | - | - | 1395 | - | - | **1554** |
|
| 190 |
+
| Toolathlon (Pass@1) | 40.7 | 43.5 | 47.8 | 46.3 | 49.0 | **51.8** |
|
| 191 |
+
|
| 192 |
+
</div>
|
| 193 |
+
|
| 194 |
+
## Chat Template
|
| 195 |
+
|
| 196 |
+
This release does not include a Jinja-format chat template. Instead, we provide a dedicated `encoding` folder with Python scripts and test cases demonstrating how to encode messages in OpenAI-compatible format into input strings for the model, and how to parse the model's text output. Please refer to the [`encoding`](encoding/README.md) folder for full documentation.
|
| 197 |
+
|
| 198 |
+
A brief example:
|
| 199 |
+
|
| 200 |
+
```python
|
| 201 |
+
from encoding_dsv4 import encode_messages, parse_message_from_completion_text
|
| 202 |
+
|
| 203 |
+
messages = [
|
| 204 |
+
{"role": "user", "content": "hello"},
|
| 205 |
+
{"role": "assistant", "content": "Hello! I am DeepSeek.", "reasoning_content": "thinking..."},
|
| 206 |
+
{"role": "user", "content": "1+1=?"}
|
| 207 |
+
]
|
| 208 |
+
|
| 209 |
+
# messages -> string
|
| 210 |
+
prompt = encode_messages(messages, thinking_mode="thinking")
|
| 211 |
+
|
| 212 |
+
# string -> tokens
|
| 213 |
+
import transformers
|
| 214 |
+
tokenizer = transformers.AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-V4-Pro")
|
| 215 |
+
tokens = tokenizer.encode(prompt)
|
| 216 |
+
```
|
| 217 |
|
| 218 |
## How to Run Locally
|
| 219 |
|
|
|
|
|
|
|
| 220 |
Please refer to the [inference](inference/README.md) folder for detailed instructions on running DeepSeek-V4 locally, including model weight conversion and interactive chat demos.
|
| 221 |
|
| 222 |
For local deployment, we recommend setting the sampling parameters to `temperature = 1.0, top_p = 1.0`. For the Think Max reasoning mode, we recommend setting the context window to at least **384K** tokens.
|
| 223 |
|
| 224 |
+
## License
|
| 225 |
|
| 226 |
+
This repository and the model weights are licensed under the [MIT License](LICENSE).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 227 |
|
| 228 |
+
## Citation
|
| 229 |
|
| 230 |
+
```
|
| 231 |
+
@misc{deepseekai2026deepseekv4,
|
| 232 |
+
title={DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence},
|
| 233 |
+
author={DeepSeek-AI},
|
| 234 |
+
year={2026},
|
| 235 |
+
}
|
| 236 |
+
```
|
| 237 |
|
| 238 |
+
## Contact
|
| 239 |
|
| 240 |
+
If you have any questions, please raise an issue or contact us at [service@deepseek.com](service@deepseek.com).
|
config.json
CHANGED
|
@@ -6,6 +6,7 @@
|
|
| 6 |
"attention_dropout": 0.0,
|
| 7 |
"bos_token_id": 0,
|
| 8 |
"eos_token_id": 1,
|
|
|
|
| 9 |
"hc_eps": 1e-06,
|
| 10 |
"hc_mult": 4,
|
| 11 |
"hc_sinkhorn_iters": 20,
|
|
@@ -41,15 +42,950 @@
|
|
| 41 |
"data_type": "int",
|
| 42 |
"iters": 0,
|
| 43 |
"model_free": true,
|
| 44 |
-
"autoround_version": "0.
|
| 45 |
"extra_config": {
|
|
|
|
|
|
|
|
|
|
| 46 |
"embed": {
|
| 47 |
"bits": 16,
|
| 48 |
"data_type": "float"
|
| 49 |
},
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 50 |
"head": {
|
| 51 |
"bits": 16,
|
| 52 |
"data_type": "float"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 53 |
}
|
| 54 |
}
|
| 55 |
},
|
|
|
|
| 6 |
"attention_dropout": 0.0,
|
| 7 |
"bos_token_id": 0,
|
| 8 |
"eos_token_id": 1,
|
| 9 |
+
"expert_dtype": "fp4",
|
| 10 |
"hc_eps": 1e-06,
|
| 11 |
"hc_mult": 4,
|
| 12 |
"hc_sinkhorn_iters": 20,
|
|
|
|
| 42 |
"data_type": "int",
|
| 43 |
"iters": 0,
|
| 44 |
"model_free": true,
|
| 45 |
+
"autoround_version": "0.14.0",
|
| 46 |
"extra_config": {
|
| 47 |
+
"wo_a": {
|
| 48 |
+
"bits": 16
|
| 49 |
+
},
|
| 50 |
"embed": {
|
| 51 |
"bits": 16,
|
| 52 |
"data_type": "float"
|
| 53 |
},
|
| 54 |
+
"layers.0.attn.wo_a": {
|
| 55 |
+
"bits": 16,
|
| 56 |
+
"data_type": "float"
|
| 57 |
+
},
|
| 58 |
+
"layers.0.ffn.gate": {
|
| 59 |
+
"bits": 16,
|
| 60 |
+
"data_type": "float"
|
| 61 |
+
},
|
| 62 |
+
"layers.1.attn.wo_a": {
|
| 63 |
+
"bits": 16,
|
| 64 |
+
"data_type": "float"
|
| 65 |
+
},
|
| 66 |
+
"layers.1.ffn.gate": {
|
| 67 |
+
"bits": 16,
|
| 68 |
+
"data_type": "float"
|
| 69 |
+
},
|
| 70 |
+
"layers.2.attn.wo_a": {
|
| 71 |
+
"bits": 16,
|
| 72 |
+
"data_type": "float"
|
| 73 |
+
},
|
| 74 |
+
"layers.2.attn.compressor.wgate": {
|
| 75 |
+
"bits": 16,
|
| 76 |
+
"data_type": "float"
|
| 77 |
+
},
|
| 78 |
+
"layers.2.attn.compressor.wkv": {
|
| 79 |
+
"bits": 16,
|
| 80 |
+
"data_type": "float"
|
| 81 |
+
},
|
| 82 |
+
"layers.2.attn.indexer.compressor.wgate": {
|
| 83 |
+
"bits": 16,
|
| 84 |
+
"data_type": "float"
|
| 85 |
+
},
|
| 86 |
+
"layers.2.attn.indexer.compressor.wkv": {
|
| 87 |
+
"bits": 16,
|
| 88 |
+
"data_type": "float"
|
| 89 |
+
},
|
| 90 |
+
"layers.2.attn.indexer.weights_proj": {
|
| 91 |
+
"bits": 16,
|
| 92 |
+
"data_type": "float"
|
| 93 |
+
},
|
| 94 |
+
"layers.2.ffn.gate": {
|
| 95 |
+
"bits": 16,
|
| 96 |
+
"data_type": "float"
|
| 97 |
+
},
|
| 98 |
+
"layers.3.attn.wo_a": {
|
| 99 |
+
"bits": 16,
|
| 100 |
+
"data_type": "float"
|
| 101 |
+
},
|
| 102 |
+
"layers.3.attn.compressor.wgate": {
|
| 103 |
+
"bits": 16,
|
| 104 |
+
"data_type": "float"
|
| 105 |
+
},
|
| 106 |
+
"layers.3.attn.compressor.wkv": {
|
| 107 |
+
"bits": 16,
|
| 108 |
+
"data_type": "float"
|
| 109 |
+
},
|
| 110 |
+
"layers.3.ffn.gate": {
|
| 111 |
+
"bits": 16,
|
| 112 |
+
"data_type": "float"
|
| 113 |
+
},
|
| 114 |
+
"layers.4.attn.wo_a": {
|
| 115 |
+
"bits": 16,
|
| 116 |
+
"data_type": "float"
|
| 117 |
+
},
|
| 118 |
+
"layers.4.attn.compressor.wgate": {
|
| 119 |
+
"bits": 16,
|
| 120 |
+
"data_type": "float"
|
| 121 |
+
},
|
| 122 |
+
"layers.4.attn.compressor.wkv": {
|
| 123 |
+
"bits": 16,
|
| 124 |
+
"data_type": "float"
|
| 125 |
+
},
|
| 126 |
+
"layers.4.attn.indexer.compressor.wgate": {
|
| 127 |
+
"bits": 16,
|
| 128 |
+
"data_type": "float"
|
| 129 |
+
},
|
| 130 |
+
"layers.4.attn.indexer.compressor.wkv": {
|
| 131 |
+
"bits": 16,
|
| 132 |
+
"data_type": "float"
|
| 133 |
+
},
|
| 134 |
+
"layers.4.attn.indexer.weights_proj": {
|
| 135 |
+
"bits": 16,
|
| 136 |
+
"data_type": "float"
|
| 137 |
+
},
|
| 138 |
+
"layers.4.ffn.gate": {
|
| 139 |
+
"bits": 16,
|
| 140 |
+
"data_type": "float"
|
| 141 |
+
},
|
| 142 |
+
"layers.5.attn.wo_a": {
|
| 143 |
+
"bits": 16,
|
| 144 |
+
"data_type": "float"
|
| 145 |
+
},
|
| 146 |
+
"layers.5.attn.compressor.wgate": {
|
| 147 |
+
"bits": 16,
|
| 148 |
+
"data_type": "float"
|
| 149 |
+
},
|
| 150 |
+
"layers.5.attn.compressor.wkv": {
|
| 151 |
+
"bits": 16,
|
| 152 |
+
"data_type": "float"
|
| 153 |
+
},
|
| 154 |
+
"layers.5.ffn.gate": {
|
| 155 |
+
"bits": 16,
|
| 156 |
+
"data_type": "float"
|
| 157 |
+
},
|
| 158 |
+
"layers.6.attn.wo_a": {
|
| 159 |
+
"bits": 16,
|
| 160 |
+
"data_type": "float"
|
| 161 |
+
},
|
| 162 |
+
"layers.6.attn.compressor.wgate": {
|
| 163 |
+
"bits": 16,
|
| 164 |
+
"data_type": "float"
|
| 165 |
+
},
|
| 166 |
+
"layers.6.attn.compressor.wkv": {
|
| 167 |
+
"bits": 16,
|
| 168 |
+
"data_type": "float"
|
| 169 |
+
},
|
| 170 |
+
"layers.6.attn.indexer.compressor.wgate": {
|
| 171 |
+
"bits": 16,
|
| 172 |
+
"data_type": "float"
|
| 173 |
+
},
|
| 174 |
+
"layers.6.attn.indexer.compressor.wkv": {
|
| 175 |
+
"bits": 16,
|
| 176 |
+
"data_type": "float"
|
| 177 |
+
},
|
| 178 |
+
"layers.6.attn.indexer.weights_proj": {
|
| 179 |
+
"bits": 16,
|
| 180 |
+
"data_type": "float"
|
| 181 |
+
},
|
| 182 |
+
"layers.6.ffn.gate": {
|
| 183 |
+
"bits": 16,
|
| 184 |
+
"data_type": "float"
|
| 185 |
+
},
|
| 186 |
+
"layers.7.attn.wo_a": {
|
| 187 |
+
"bits": 16,
|
| 188 |
+
"data_type": "float"
|
| 189 |
+
},
|
| 190 |
+
"layers.7.attn.compressor.wgate": {
|
| 191 |
+
"bits": 16,
|
| 192 |
+
"data_type": "float"
|
| 193 |
+
},
|
| 194 |
+
"layers.7.attn.compressor.wkv": {
|
| 195 |
+
"bits": 16,
|
| 196 |
+
"data_type": "float"
|
| 197 |
+
},
|
| 198 |
+
"layers.7.ffn.gate": {
|
| 199 |
+
"bits": 16,
|
| 200 |
+
"data_type": "float"
|
| 201 |
+
},
|
| 202 |
+
"layers.8.attn.wo_a": {
|
| 203 |
+
"bits": 16,
|
| 204 |
+
"data_type": "float"
|
| 205 |
+
},
|
| 206 |
+
"layers.8.attn.compressor.wgate": {
|
| 207 |
+
"bits": 16,
|
| 208 |
+
"data_type": "float"
|
| 209 |
+
},
|
| 210 |
+
"layers.8.attn.compressor.wkv": {
|
| 211 |
+
"bits": 16,
|
| 212 |
+
"data_type": "float"
|
| 213 |
+
},
|
| 214 |
+
"layers.8.attn.indexer.compressor.wgate": {
|
| 215 |
+
"bits": 16,
|
| 216 |
+
"data_type": "float"
|
| 217 |
+
},
|
| 218 |
+
"layers.8.attn.indexer.compressor.wkv": {
|
| 219 |
+
"bits": 16,
|
| 220 |
+
"data_type": "float"
|
| 221 |
+
},
|
| 222 |
+
"layers.8.attn.indexer.weights_proj": {
|
| 223 |
+
"bits": 16,
|
| 224 |
+
"data_type": "float"
|
| 225 |
+
},
|
| 226 |
+
"layers.8.ffn.gate": {
|
| 227 |
+
"bits": 16,
|
| 228 |
+
"data_type": "float"
|
| 229 |
+
},
|
| 230 |
+
"layers.9.attn.wo_a": {
|
| 231 |
+
"bits": 16,
|
| 232 |
+
"data_type": "float"
|
| 233 |
+
},
|
| 234 |
+
"layers.9.attn.compressor.wgate": {
|
| 235 |
+
"bits": 16,
|
| 236 |
+
"data_type": "float"
|
| 237 |
+
},
|
| 238 |
+
"layers.9.attn.compressor.wkv": {
|
| 239 |
+
"bits": 16,
|
| 240 |
+
"data_type": "float"
|
| 241 |
+
},
|
| 242 |
+
"layers.9.ffn.gate": {
|
| 243 |
+
"bits": 16,
|
| 244 |
+
"data_type": "float"
|
| 245 |
+
},
|
| 246 |
+
"layers.10.attn.wo_a": {
|
| 247 |
+
"bits": 16,
|
| 248 |
+
"data_type": "float"
|
| 249 |
+
},
|
| 250 |
+
"layers.10.attn.compressor.wgate": {
|
| 251 |
+
"bits": 16,
|
| 252 |
+
"data_type": "float"
|
| 253 |
+
},
|
| 254 |
+
"layers.10.attn.compressor.wkv": {
|
| 255 |
+
"bits": 16,
|
| 256 |
+
"data_type": "float"
|
| 257 |
+
},
|
| 258 |
+
"layers.10.attn.indexer.compressor.wgate": {
|
| 259 |
+
"bits": 16,
|
| 260 |
+
"data_type": "float"
|
| 261 |
+
},
|
| 262 |
+
"layers.10.attn.indexer.compressor.wkv": {
|
| 263 |
+
"bits": 16,
|
| 264 |
+
"data_type": "float"
|
| 265 |
+
},
|
| 266 |
+
"layers.10.attn.indexer.weights_proj": {
|
| 267 |
+
"bits": 16,
|
| 268 |
+
"data_type": "float"
|
| 269 |
+
},
|
| 270 |
+
"layers.10.ffn.gate": {
|
| 271 |
+
"bits": 16,
|
| 272 |
+
"data_type": "float"
|
| 273 |
+
},
|
| 274 |
+
"layers.11.attn.wo_a": {
|
| 275 |
+
"bits": 16,
|
| 276 |
+
"data_type": "float"
|
| 277 |
+
},
|
| 278 |
+
"layers.11.attn.compressor.wgate": {
|
| 279 |
+
"bits": 16,
|
| 280 |
+
"data_type": "float"
|
| 281 |
+
},
|
| 282 |
+
"layers.11.attn.compressor.wkv": {
|
| 283 |
+
"bits": 16,
|
| 284 |
+
"data_type": "float"
|
| 285 |
+
},
|
| 286 |
+
"layers.11.ffn.gate": {
|
| 287 |
+
"bits": 16,
|
| 288 |
+
"data_type": "float"
|
| 289 |
+
},
|
| 290 |
+
"layers.12.attn.wo_a": {
|
| 291 |
+
"bits": 16,
|
| 292 |
+
"data_type": "float"
|
| 293 |
+
},
|
| 294 |
+
"layers.12.attn.compressor.wgate": {
|
| 295 |
+
"bits": 16,
|
| 296 |
+
"data_type": "float"
|
| 297 |
+
},
|
| 298 |
+
"layers.12.attn.compressor.wkv": {
|
| 299 |
+
"bits": 16,
|
| 300 |
+
"data_type": "float"
|
| 301 |
+
},
|
| 302 |
+
"layers.12.attn.indexer.compressor.wgate": {
|
| 303 |
+
"bits": 16,
|
| 304 |
+
"data_type": "float"
|
| 305 |
+
},
|
| 306 |
+
"layers.12.attn.indexer.compressor.wkv": {
|
| 307 |
+
"bits": 16,
|
| 308 |
+
"data_type": "float"
|
| 309 |
+
},
|
| 310 |
+
"layers.12.attn.indexer.weights_proj": {
|
| 311 |
+
"bits": 16,
|
| 312 |
+
"data_type": "float"
|
| 313 |
+
},
|
| 314 |
+
"layers.12.ffn.gate": {
|
| 315 |
+
"bits": 16,
|
| 316 |
+
"data_type": "float"
|
| 317 |
+
},
|
| 318 |
+
"layers.13.attn.wo_a": {
|
| 319 |
+
"bits": 16,
|
| 320 |
+
"data_type": "float"
|
| 321 |
+
},
|
| 322 |
+
"layers.13.attn.compressor.wgate": {
|
| 323 |
+
"bits": 16,
|
| 324 |
+
"data_type": "float"
|
| 325 |
+
},
|
| 326 |
+
"layers.13.attn.compressor.wkv": {
|
| 327 |
+
"bits": 16,
|
| 328 |
+
"data_type": "float"
|
| 329 |
+
},
|
| 330 |
+
"layers.13.ffn.gate": {
|
| 331 |
+
"bits": 16,
|
| 332 |
+
"data_type": "float"
|
| 333 |
+
},
|
| 334 |
+
"layers.14.attn.wo_a": {
|
| 335 |
+
"bits": 16,
|
| 336 |
+
"data_type": "float"
|
| 337 |
+
},
|
| 338 |
+
"layers.14.attn.compressor.wgate": {
|
| 339 |
+
"bits": 16,
|
| 340 |
+
"data_type": "float"
|
| 341 |
+
},
|
| 342 |
+
"layers.14.attn.compressor.wkv": {
|
| 343 |
+
"bits": 16,
|
| 344 |
+
"data_type": "float"
|
| 345 |
+
},
|
| 346 |
+
"layers.14.attn.indexer.compressor.wgate": {
|
| 347 |
+
"bits": 16,
|
| 348 |
+
"data_type": "float"
|
| 349 |
+
},
|
| 350 |
+
"layers.14.attn.indexer.compressor.wkv": {
|
| 351 |
+
"bits": 16,
|
| 352 |
+
"data_type": "float"
|
| 353 |
+
},
|
| 354 |
+
"layers.14.attn.indexer.weights_proj": {
|
| 355 |
+
"bits": 16,
|
| 356 |
+
"data_type": "float"
|
| 357 |
+
},
|
| 358 |
+
"layers.14.ffn.gate": {
|
| 359 |
+
"bits": 16,
|
| 360 |
+
"data_type": "float"
|
| 361 |
+
},
|
| 362 |
+
"layers.15.attn.wo_a": {
|
| 363 |
+
"bits": 16,
|
| 364 |
+
"data_type": "float"
|
| 365 |
+
},
|
| 366 |
+
"layers.15.attn.compressor.wgate": {
|
| 367 |
+
"bits": 16,
|
| 368 |
+
"data_type": "float"
|
| 369 |
+
},
|
| 370 |
+
"layers.15.attn.compressor.wkv": {
|
| 371 |
+
"bits": 16,
|
| 372 |
+
"data_type": "float"
|
| 373 |
+
},
|
| 374 |
+
"layers.15.ffn.gate": {
|
| 375 |
+
"bits": 16,
|
| 376 |
+
"data_type": "float"
|
| 377 |
+
},
|
| 378 |
+
"layers.16.attn.wo_a": {
|
| 379 |
+
"bits": 16,
|
| 380 |
+
"data_type": "float"
|
| 381 |
+
},
|
| 382 |
+
"layers.16.attn.compressor.wgate": {
|
| 383 |
+
"bits": 16,
|
| 384 |
+
"data_type": "float"
|
| 385 |
+
},
|
| 386 |
+
"layers.16.attn.compressor.wkv": {
|
| 387 |
+
"bits": 16,
|
| 388 |
+
"data_type": "float"
|
| 389 |
+
},
|
| 390 |
+
"layers.16.attn.indexer.compressor.wgate": {
|
| 391 |
+
"bits": 16,
|
| 392 |
+
"data_type": "float"
|
| 393 |
+
},
|
| 394 |
+
"layers.16.attn.indexer.compressor.wkv": {
|
| 395 |
+
"bits": 16,
|
| 396 |
+
"data_type": "float"
|
| 397 |
+
},
|
| 398 |
+
"layers.16.attn.indexer.weights_proj": {
|
| 399 |
+
"bits": 16,
|
| 400 |
+
"data_type": "float"
|
| 401 |
+
},
|
| 402 |
+
"layers.16.ffn.gate": {
|
| 403 |
+
"bits": 16,
|
| 404 |
+
"data_type": "float"
|
| 405 |
+
},
|
| 406 |
+
"layers.17.attn.wo_a": {
|
| 407 |
+
"bits": 16,
|
| 408 |
+
"data_type": "float"
|
| 409 |
+
},
|
| 410 |
+
"layers.17.attn.compressor.wgate": {
|
| 411 |
+
"bits": 16,
|
| 412 |
+
"data_type": "float"
|
| 413 |
+
},
|
| 414 |
+
"layers.17.attn.compressor.wkv": {
|
| 415 |
+
"bits": 16,
|
| 416 |
+
"data_type": "float"
|
| 417 |
+
},
|
| 418 |
+
"layers.17.ffn.gate": {
|
| 419 |
+
"bits": 16,
|
| 420 |
+
"data_type": "float"
|
| 421 |
+
},
|
| 422 |
+
"layers.18.attn.wo_a": {
|
| 423 |
+
"bits": 16,
|
| 424 |
+
"data_type": "float"
|
| 425 |
+
},
|
| 426 |
+
"layers.18.attn.compressor.wgate": {
|
| 427 |
+
"bits": 16,
|
| 428 |
+
"data_type": "float"
|
| 429 |
+
},
|
| 430 |
+
"layers.18.attn.compressor.wkv": {
|
| 431 |
+
"bits": 16,
|
| 432 |
+
"data_type": "float"
|
| 433 |
+
},
|
| 434 |
+
"layers.18.attn.indexer.compressor.wgate": {
|
| 435 |
+
"bits": 16,
|
| 436 |
+
"data_type": "float"
|
| 437 |
+
},
|
| 438 |
+
"layers.18.attn.indexer.compressor.wkv": {
|
| 439 |
+
"bits": 16,
|
| 440 |
+
"data_type": "float"
|
| 441 |
+
},
|
| 442 |
+
"layers.18.attn.indexer.weights_proj": {
|
| 443 |
+
"bits": 16,
|
| 444 |
+
"data_type": "float"
|
| 445 |
+
},
|
| 446 |
+
"layers.18.ffn.gate": {
|
| 447 |
+
"bits": 16,
|
| 448 |
+
"data_type": "float"
|
| 449 |
+
},
|
| 450 |
+
"layers.19.attn.wo_a": {
|
| 451 |
+
"bits": 16,
|
| 452 |
+
"data_type": "float"
|
| 453 |
+
},
|
| 454 |
+
"layers.19.attn.compressor.wgate": {
|
| 455 |
+
"bits": 16,
|
| 456 |
+
"data_type": "float"
|
| 457 |
+
},
|
| 458 |
+
"layers.19.attn.compressor.wkv": {
|
| 459 |
+
"bits": 16,
|
| 460 |
+
"data_type": "float"
|
| 461 |
+
},
|
| 462 |
+
"layers.19.ffn.gate": {
|
| 463 |
+
"bits": 16,
|
| 464 |
+
"data_type": "float"
|
| 465 |
+
},
|
| 466 |
+
"layers.20.attn.wo_a": {
|
| 467 |
+
"bits": 16,
|
| 468 |
+
"data_type": "float"
|
| 469 |
+
},
|
| 470 |
+
"layers.20.attn.compressor.wgate": {
|
| 471 |
+
"bits": 16,
|
| 472 |
+
"data_type": "float"
|
| 473 |
+
},
|
| 474 |
+
"layers.20.attn.compressor.wkv": {
|
| 475 |
+
"bits": 16,
|
| 476 |
+
"data_type": "float"
|
| 477 |
+
},
|
| 478 |
+
"layers.20.attn.indexer.compressor.wgate": {
|
| 479 |
+
"bits": 16,
|
| 480 |
+
"data_type": "float"
|
| 481 |
+
},
|
| 482 |
+
"layers.20.attn.indexer.compressor.wkv": {
|
| 483 |
+
"bits": 16,
|
| 484 |
+
"data_type": "float"
|
| 485 |
+
},
|
| 486 |
+
"layers.20.attn.indexer.weights_proj": {
|
| 487 |
+
"bits": 16,
|
| 488 |
+
"data_type": "float"
|
| 489 |
+
},
|
| 490 |
+
"layers.20.ffn.gate": {
|
| 491 |
+
"bits": 16,
|
| 492 |
+
"data_type": "float"
|
| 493 |
+
},
|
| 494 |
+
"layers.21.attn.wo_a": {
|
| 495 |
+
"bits": 16,
|
| 496 |
+
"data_type": "float"
|
| 497 |
+
},
|
| 498 |
+
"layers.21.attn.compressor.wgate": {
|
| 499 |
+
"bits": 16,
|
| 500 |
+
"data_type": "float"
|
| 501 |
+
},
|
| 502 |
+
"layers.21.attn.compressor.wkv": {
|
| 503 |
+
"bits": 16,
|
| 504 |
+
"data_type": "float"
|
| 505 |
+
},
|
| 506 |
+
"layers.21.ffn.gate": {
|
| 507 |
+
"bits": 16,
|
| 508 |
+
"data_type": "float"
|
| 509 |
+
},
|
| 510 |
+
"layers.22.attn.wo_a": {
|
| 511 |
+
"bits": 16,
|
| 512 |
+
"data_type": "float"
|
| 513 |
+
},
|
| 514 |
+
"layers.22.attn.compressor.wgate": {
|
| 515 |
+
"bits": 16,
|
| 516 |
+
"data_type": "float"
|
| 517 |
+
},
|
| 518 |
+
"layers.22.attn.compressor.wkv": {
|
| 519 |
+
"bits": 16,
|
| 520 |
+
"data_type": "float"
|
| 521 |
+
},
|
| 522 |
+
"layers.22.attn.indexer.compressor.wgate": {
|
| 523 |
+
"bits": 16,
|
| 524 |
+
"data_type": "float"
|
| 525 |
+
},
|
| 526 |
+
"layers.22.attn.indexer.compressor.wkv": {
|
| 527 |
+
"bits": 16,
|
| 528 |
+
"data_type": "float"
|
| 529 |
+
},
|
| 530 |
+
"layers.22.attn.indexer.weights_proj": {
|
| 531 |
+
"bits": 16,
|
| 532 |
+
"data_type": "float"
|
| 533 |
+
},
|
| 534 |
+
"layers.22.ffn.gate": {
|
| 535 |
+
"bits": 16,
|
| 536 |
+
"data_type": "float"
|
| 537 |
+
},
|
| 538 |
+
"layers.23.attn.wo_a": {
|
| 539 |
+
"bits": 16,
|
| 540 |
+
"data_type": "float"
|
| 541 |
+
},
|
| 542 |
+
"layers.23.attn.compressor.wgate": {
|
| 543 |
+
"bits": 16,
|
| 544 |
+
"data_type": "float"
|
| 545 |
+
},
|
| 546 |
+
"layers.23.attn.compressor.wkv": {
|
| 547 |
+
"bits": 16,
|
| 548 |
+
"data_type": "float"
|
| 549 |
+
},
|
| 550 |
+
"layers.23.ffn.gate": {
|
| 551 |
+
"bits": 16,
|
| 552 |
+
"data_type": "float"
|
| 553 |
+
},
|
| 554 |
+
"layers.24.attn.wo_a": {
|
| 555 |
+
"bits": 16,
|
| 556 |
+
"data_type": "float"
|
| 557 |
+
},
|
| 558 |
+
"layers.24.attn.compressor.wgate": {
|
| 559 |
+
"bits": 16,
|
| 560 |
+
"data_type": "float"
|
| 561 |
+
},
|
| 562 |
+
"layers.24.attn.compressor.wkv": {
|
| 563 |
+
"bits": 16,
|
| 564 |
+
"data_type": "float"
|
| 565 |
+
},
|
| 566 |
+
"layers.24.attn.indexer.compressor.wgate": {
|
| 567 |
+
"bits": 16,
|
| 568 |
+
"data_type": "float"
|
| 569 |
+
},
|
| 570 |
+
"layers.24.attn.indexer.compressor.wkv": {
|
| 571 |
+
"bits": 16,
|
| 572 |
+
"data_type": "float"
|
| 573 |
+
},
|
| 574 |
+
"layers.24.attn.indexer.weights_proj": {
|
| 575 |
+
"bits": 16,
|
| 576 |
+
"data_type": "float"
|
| 577 |
+
},
|
| 578 |
+
"layers.24.ffn.gate": {
|
| 579 |
+
"bits": 16,
|
| 580 |
+
"data_type": "float"
|
| 581 |
+
},
|
| 582 |
+
"layers.25.attn.wo_a": {
|
| 583 |
+
"bits": 16,
|
| 584 |
+
"data_type": "float"
|
| 585 |
+
},
|
| 586 |
+
"layers.25.attn.compressor.wgate": {
|
| 587 |
+
"bits": 16,
|
| 588 |
+
"data_type": "float"
|
| 589 |
+
},
|
| 590 |
+
"layers.25.attn.compressor.wkv": {
|
| 591 |
+
"bits": 16,
|
| 592 |
+
"data_type": "float"
|
| 593 |
+
},
|
| 594 |
+
"layers.25.ffn.gate": {
|
| 595 |
+
"bits": 16,
|
| 596 |
+
"data_type": "float"
|
| 597 |
+
},
|
| 598 |
+
"layers.26.attn.wo_a": {
|
| 599 |
+
"bits": 16,
|
| 600 |
+
"data_type": "float"
|
| 601 |
+
},
|
| 602 |
+
"layers.26.attn.compressor.wgate": {
|
| 603 |
+
"bits": 16,
|
| 604 |
+
"data_type": "float"
|
| 605 |
+
},
|
| 606 |
+
"layers.26.attn.compressor.wkv": {
|
| 607 |
+
"bits": 16,
|
| 608 |
+
"data_type": "float"
|
| 609 |
+
},
|
| 610 |
+
"layers.26.attn.indexer.compressor.wgate": {
|
| 611 |
+
"bits": 16,
|
| 612 |
+
"data_type": "float"
|
| 613 |
+
},
|
| 614 |
+
"layers.26.attn.indexer.compressor.wkv": {
|
| 615 |
+
"bits": 16,
|
| 616 |
+
"data_type": "float"
|
| 617 |
+
},
|
| 618 |
+
"layers.26.attn.indexer.weights_proj": {
|
| 619 |
+
"bits": 16,
|
| 620 |
+
"data_type": "float"
|
| 621 |
+
},
|
| 622 |
+
"layers.26.ffn.gate": {
|
| 623 |
+
"bits": 16,
|
| 624 |
+
"data_type": "float"
|
| 625 |
+
},
|
| 626 |
+
"layers.27.attn.wo_a": {
|
| 627 |
+
"bits": 16,
|
| 628 |
+
"data_type": "float"
|
| 629 |
+
},
|
| 630 |
+
"layers.27.attn.compressor.wgate": {
|
| 631 |
+
"bits": 16,
|
| 632 |
+
"data_type": "float"
|
| 633 |
+
},
|
| 634 |
+
"layers.27.attn.compressor.wkv": {
|
| 635 |
+
"bits": 16,
|
| 636 |
+
"data_type": "float"
|
| 637 |
+
},
|
| 638 |
+
"layers.27.ffn.gate": {
|
| 639 |
+
"bits": 16,
|
| 640 |
+
"data_type": "float"
|
| 641 |
+
},
|
| 642 |
+
"layers.28.attn.wo_a": {
|
| 643 |
+
"bits": 16,
|
| 644 |
+
"data_type": "float"
|
| 645 |
+
},
|
| 646 |
+
"layers.28.attn.compressor.wgate": {
|
| 647 |
+
"bits": 16,
|
| 648 |
+
"data_type": "float"
|
| 649 |
+
},
|
| 650 |
+
"layers.28.attn.compressor.wkv": {
|
| 651 |
+
"bits": 16,
|
| 652 |
+
"data_type": "float"
|
| 653 |
+
},
|
| 654 |
+
"layers.28.attn.indexer.compressor.wgate": {
|
| 655 |
+
"bits": 16,
|
| 656 |
+
"data_type": "float"
|
| 657 |
+
},
|
| 658 |
+
"layers.28.attn.indexer.compressor.wkv": {
|
| 659 |
+
"bits": 16,
|
| 660 |
+
"data_type": "float"
|
| 661 |
+
},
|
| 662 |
+
"layers.28.attn.indexer.weights_proj": {
|
| 663 |
+
"bits": 16,
|
| 664 |
+
"data_type": "float"
|
| 665 |
+
},
|
| 666 |
+
"layers.28.ffn.gate": {
|
| 667 |
+
"bits": 16,
|
| 668 |
+
"data_type": "float"
|
| 669 |
+
},
|
| 670 |
+
"layers.29.attn.wo_a": {
|
| 671 |
+
"bits": 16,
|
| 672 |
+
"data_type": "float"
|
| 673 |
+
},
|
| 674 |
+
"layers.29.attn.compressor.wgate": {
|
| 675 |
+
"bits": 16,
|
| 676 |
+
"data_type": "float"
|
| 677 |
+
},
|
| 678 |
+
"layers.29.attn.compressor.wkv": {
|
| 679 |
+
"bits": 16,
|
| 680 |
+
"data_type": "float"
|
| 681 |
+
},
|
| 682 |
+
"layers.29.ffn.gate": {
|
| 683 |
+
"bits": 16,
|
| 684 |
+
"data_type": "float"
|
| 685 |
+
},
|
| 686 |
+
"layers.30.attn.wo_a": {
|
| 687 |
+
"bits": 16,
|
| 688 |
+
"data_type": "float"
|
| 689 |
+
},
|
| 690 |
+
"layers.30.attn.compressor.wgate": {
|
| 691 |
+
"bits": 16,
|
| 692 |
+
"data_type": "float"
|
| 693 |
+
},
|
| 694 |
+
"layers.30.attn.compressor.wkv": {
|
| 695 |
+
"bits": 16,
|
| 696 |
+
"data_type": "float"
|
| 697 |
+
},
|
| 698 |
+
"layers.30.attn.indexer.compressor.wgate": {
|
| 699 |
+
"bits": 16,
|
| 700 |
+
"data_type": "float"
|
| 701 |
+
},
|
| 702 |
+
"layers.30.attn.indexer.compressor.wkv": {
|
| 703 |
+
"bits": 16,
|
| 704 |
+
"data_type": "float"
|
| 705 |
+
},
|
| 706 |
+
"layers.30.attn.indexer.weights_proj": {
|
| 707 |
+
"bits": 16,
|
| 708 |
+
"data_type": "float"
|
| 709 |
+
},
|
| 710 |
+
"layers.30.ffn.gate": {
|
| 711 |
+
"bits": 16,
|
| 712 |
+
"data_type": "float"
|
| 713 |
+
},
|
| 714 |
+
"layers.31.attn.wo_a": {
|
| 715 |
+
"bits": 16,
|
| 716 |
+
"data_type": "float"
|
| 717 |
+
},
|
| 718 |
+
"layers.31.attn.compressor.wgate": {
|
| 719 |
+
"bits": 16,
|
| 720 |
+
"data_type": "float"
|
| 721 |
+
},
|
| 722 |
+
"layers.31.attn.compressor.wkv": {
|
| 723 |
+
"bits": 16,
|
| 724 |
+
"data_type": "float"
|
| 725 |
+
},
|
| 726 |
+
"layers.31.ffn.gate": {
|
| 727 |
+
"bits": 16,
|
| 728 |
+
"data_type": "float"
|
| 729 |
+
},
|
| 730 |
+
"layers.32.attn.wo_a": {
|
| 731 |
+
"bits": 16,
|
| 732 |
+
"data_type": "float"
|
| 733 |
+
},
|
| 734 |
+
"layers.32.attn.compressor.wgate": {
|
| 735 |
+
"bits": 16,
|
| 736 |
+
"data_type": "float"
|
| 737 |
+
},
|
| 738 |
+
"layers.32.attn.compressor.wkv": {
|
| 739 |
+
"bits": 16,
|
| 740 |
+
"data_type": "float"
|
| 741 |
+
},
|
| 742 |
+
"layers.32.attn.indexer.compressor.wgate": {
|
| 743 |
+
"bits": 16,
|
| 744 |
+
"data_type": "float"
|
| 745 |
+
},
|
| 746 |
+
"layers.32.attn.indexer.compressor.wkv": {
|
| 747 |
+
"bits": 16,
|
| 748 |
+
"data_type": "float"
|
| 749 |
+
},
|
| 750 |
+
"layers.32.attn.indexer.weights_proj": {
|
| 751 |
+
"bits": 16,
|
| 752 |
+
"data_type": "float"
|
| 753 |
+
},
|
| 754 |
+
"layers.32.ffn.gate": {
|
| 755 |
+
"bits": 16,
|
| 756 |
+
"data_type": "float"
|
| 757 |
+
},
|
| 758 |
+
"layers.33.attn.wo_a": {
|
| 759 |
+
"bits": 16,
|
| 760 |
+
"data_type": "float"
|
| 761 |
+
},
|
| 762 |
+
"layers.33.attn.compressor.wgate": {
|
| 763 |
+
"bits": 16,
|
| 764 |
+
"data_type": "float"
|
| 765 |
+
},
|
| 766 |
+
"layers.33.attn.compressor.wkv": {
|
| 767 |
+
"bits": 16,
|
| 768 |
+
"data_type": "float"
|
| 769 |
+
},
|
| 770 |
+
"layers.33.ffn.gate": {
|
| 771 |
+
"bits": 16,
|
| 772 |
+
"data_type": "float"
|
| 773 |
+
},
|
| 774 |
+
"layers.34.attn.wo_a": {
|
| 775 |
+
"bits": 16,
|
| 776 |
+
"data_type": "float"
|
| 777 |
+
},
|
| 778 |
+
"layers.34.attn.compressor.wgate": {
|
| 779 |
+
"bits": 16,
|
| 780 |
+
"data_type": "float"
|
| 781 |
+
},
|
| 782 |
+
"layers.34.attn.compressor.wkv": {
|
| 783 |
+
"bits": 16,
|
| 784 |
+
"data_type": "float"
|
| 785 |
+
},
|
| 786 |
+
"layers.34.attn.indexer.compressor.wgate": {
|
| 787 |
+
"bits": 16,
|
| 788 |
+
"data_type": "float"
|
| 789 |
+
},
|
| 790 |
+
"layers.34.attn.indexer.compressor.wkv": {
|
| 791 |
+
"bits": 16,
|
| 792 |
+
"data_type": "float"
|
| 793 |
+
},
|
| 794 |
+
"layers.34.attn.indexer.weights_proj": {
|
| 795 |
+
"bits": 16,
|
| 796 |
+
"data_type": "float"
|
| 797 |
+
},
|
| 798 |
+
"layers.34.ffn.gate": {
|
| 799 |
+
"bits": 16,
|
| 800 |
+
"data_type": "float"
|
| 801 |
+
},
|
| 802 |
+
"layers.35.attn.wo_a": {
|
| 803 |
+
"bits": 16,
|
| 804 |
+
"data_type": "float"
|
| 805 |
+
},
|
| 806 |
+
"layers.35.attn.compressor.wgate": {
|
| 807 |
+
"bits": 16,
|
| 808 |
+
"data_type": "float"
|
| 809 |
+
},
|
| 810 |
+
"layers.35.attn.compressor.wkv": {
|
| 811 |
+
"bits": 16,
|
| 812 |
+
"data_type": "float"
|
| 813 |
+
},
|
| 814 |
+
"layers.35.ffn.gate": {
|
| 815 |
+
"bits": 16,
|
| 816 |
+
"data_type": "float"
|
| 817 |
+
},
|
| 818 |
+
"layers.36.attn.wo_a": {
|
| 819 |
+
"bits": 16,
|
| 820 |
+
"data_type": "float"
|
| 821 |
+
},
|
| 822 |
+
"layers.36.attn.compressor.wgate": {
|
| 823 |
+
"bits": 16,
|
| 824 |
+
"data_type": "float"
|
| 825 |
+
},
|
| 826 |
+
"layers.36.attn.compressor.wkv": {
|
| 827 |
+
"bits": 16,
|
| 828 |
+
"data_type": "float"
|
| 829 |
+
},
|
| 830 |
+
"layers.36.attn.indexer.compressor.wgate": {
|
| 831 |
+
"bits": 16,
|
| 832 |
+
"data_type": "float"
|
| 833 |
+
},
|
| 834 |
+
"layers.36.attn.indexer.compressor.wkv": {
|
| 835 |
+
"bits": 16,
|
| 836 |
+
"data_type": "float"
|
| 837 |
+
},
|
| 838 |
+
"layers.36.attn.indexer.weights_proj": {
|
| 839 |
+
"bits": 16,
|
| 840 |
+
"data_type": "float"
|
| 841 |
+
},
|
| 842 |
+
"layers.36.ffn.gate": {
|
| 843 |
+
"bits": 16,
|
| 844 |
+
"data_type": "float"
|
| 845 |
+
},
|
| 846 |
+
"layers.37.attn.wo_a": {
|
| 847 |
+
"bits": 16,
|
| 848 |
+
"data_type": "float"
|
| 849 |
+
},
|
| 850 |
+
"layers.37.attn.compressor.wgate": {
|
| 851 |
+
"bits": 16,
|
| 852 |
+
"data_type": "float"
|
| 853 |
+
},
|
| 854 |
+
"layers.37.attn.compressor.wkv": {
|
| 855 |
+
"bits": 16,
|
| 856 |
+
"data_type": "float"
|
| 857 |
+
},
|
| 858 |
+
"layers.37.ffn.gate": {
|
| 859 |
+
"bits": 16,
|
| 860 |
+
"data_type": "float"
|
| 861 |
+
},
|
| 862 |
+
"layers.38.attn.wo_a": {
|
| 863 |
+
"bits": 16,
|
| 864 |
+
"data_type": "float"
|
| 865 |
+
},
|
| 866 |
+
"layers.38.attn.compressor.wgate": {
|
| 867 |
+
"bits": 16,
|
| 868 |
+
"data_type": "float"
|
| 869 |
+
},
|
| 870 |
+
"layers.38.attn.compressor.wkv": {
|
| 871 |
+
"bits": 16,
|
| 872 |
+
"data_type": "float"
|
| 873 |
+
},
|
| 874 |
+
"layers.38.attn.indexer.compressor.wgate": {
|
| 875 |
+
"bits": 16,
|
| 876 |
+
"data_type": "float"
|
| 877 |
+
},
|
| 878 |
+
"layers.38.attn.indexer.compressor.wkv": {
|
| 879 |
+
"bits": 16,
|
| 880 |
+
"data_type": "float"
|
| 881 |
+
},
|
| 882 |
+
"layers.38.attn.indexer.weights_proj": {
|
| 883 |
+
"bits": 16,
|
| 884 |
+
"data_type": "float"
|
| 885 |
+
},
|
| 886 |
+
"layers.38.ffn.gate": {
|
| 887 |
+
"bits": 16,
|
| 888 |
+
"data_type": "float"
|
| 889 |
+
},
|
| 890 |
+
"layers.39.attn.wo_a": {
|
| 891 |
+
"bits": 16,
|
| 892 |
+
"data_type": "float"
|
| 893 |
+
},
|
| 894 |
+
"layers.39.attn.compressor.wgate": {
|
| 895 |
+
"bits": 16,
|
| 896 |
+
"data_type": "float"
|
| 897 |
+
},
|
| 898 |
+
"layers.39.attn.compressor.wkv": {
|
| 899 |
+
"bits": 16,
|
| 900 |
+
"data_type": "float"
|
| 901 |
+
},
|
| 902 |
+
"layers.39.ffn.gate": {
|
| 903 |
+
"bits": 16,
|
| 904 |
+
"data_type": "float"
|
| 905 |
+
},
|
| 906 |
+
"layers.40.attn.wo_a": {
|
| 907 |
+
"bits": 16,
|
| 908 |
+
"data_type": "float"
|
| 909 |
+
},
|
| 910 |
+
"layers.40.attn.compressor.wgate": {
|
| 911 |
+
"bits": 16,
|
| 912 |
+
"data_type": "float"
|
| 913 |
+
},
|
| 914 |
+
"layers.40.attn.compressor.wkv": {
|
| 915 |
+
"bits": 16,
|
| 916 |
+
"data_type": "float"
|
| 917 |
+
},
|
| 918 |
+
"layers.40.attn.indexer.compressor.wgate": {
|
| 919 |
+
"bits": 16,
|
| 920 |
+
"data_type": "float"
|
| 921 |
+
},
|
| 922 |
+
"layers.40.attn.indexer.compressor.wkv": {
|
| 923 |
+
"bits": 16,
|
| 924 |
+
"data_type": "float"
|
| 925 |
+
},
|
| 926 |
+
"layers.40.attn.indexer.weights_proj": {
|
| 927 |
+
"bits": 16,
|
| 928 |
+
"data_type": "float"
|
| 929 |
+
},
|
| 930 |
+
"layers.40.ffn.gate": {
|
| 931 |
+
"bits": 16,
|
| 932 |
+
"data_type": "float"
|
| 933 |
+
},
|
| 934 |
+
"layers.41.attn.wo_a": {
|
| 935 |
+
"bits": 16,
|
| 936 |
+
"data_type": "float"
|
| 937 |
+
},
|
| 938 |
+
"layers.41.attn.compressor.wgate": {
|
| 939 |
+
"bits": 16,
|
| 940 |
+
"data_type": "float"
|
| 941 |
+
},
|
| 942 |
+
"layers.41.attn.compressor.wkv": {
|
| 943 |
+
"bits": 16,
|
| 944 |
+
"data_type": "float"
|
| 945 |
+
},
|
| 946 |
+
"layers.41.ffn.gate": {
|
| 947 |
+
"bits": 16,
|
| 948 |
+
"data_type": "float"
|
| 949 |
+
},
|
| 950 |
+
"layers.42.attn.wo_a": {
|
| 951 |
+
"bits": 16,
|
| 952 |
+
"data_type": "float"
|
| 953 |
+
},
|
| 954 |
+
"layers.42.attn.compressor.wgate": {
|
| 955 |
+
"bits": 16,
|
| 956 |
+
"data_type": "float"
|
| 957 |
+
},
|
| 958 |
+
"layers.42.attn.compressor.wkv": {
|
| 959 |
+
"bits": 16,
|
| 960 |
+
"data_type": "float"
|
| 961 |
+
},
|
| 962 |
+
"layers.42.attn.indexer.compressor.wgate": {
|
| 963 |
+
"bits": 16,
|
| 964 |
+
"data_type": "float"
|
| 965 |
+
},
|
| 966 |
+
"layers.42.attn.indexer.compressor.wkv": {
|
| 967 |
+
"bits": 16,
|
| 968 |
+
"data_type": "float"
|
| 969 |
+
},
|
| 970 |
+
"layers.42.attn.indexer.weights_proj": {
|
| 971 |
+
"bits": 16,
|
| 972 |
+
"data_type": "float"
|
| 973 |
+
},
|
| 974 |
+
"layers.42.ffn.gate": {
|
| 975 |
+
"bits": 16,
|
| 976 |
+
"data_type": "float"
|
| 977 |
+
},
|
| 978 |
"head": {
|
| 979 |
"bits": 16,
|
| 980 |
"data_type": "float"
|
| 981 |
+
},
|
| 982 |
+
"mtp.0.attn.wo_a": {
|
| 983 |
+
"bits": 16,
|
| 984 |
+
"data_type": "float"
|
| 985 |
+
},
|
| 986 |
+
"mtp.0.ffn.gate": {
|
| 987 |
+
"bits": 16,
|
| 988 |
+
"data_type": "float"
|
| 989 |
}
|
| 990 |
}
|
| 991 |
},
|
inference/README.md
CHANGED
|
@@ -3,9 +3,9 @@
|
|
| 3 |
First convert huggingface model weight files to the format of this project.
|
| 4 |
```bash
|
| 5 |
export EXPERTS=256
|
| 6 |
-
export MP=
|
| 7 |
-
export CONFIG=
|
| 8 |
-
python
|
| 9 |
```
|
| 10 |
|
| 11 |
Then chat with DeepSeek model at will!
|
|
@@ -23,3 +23,4 @@ Or multi nodes inference.
|
|
| 23 |
torchrun --nnodes ${NODES} --nproc-per-node $((MP / NODES)) --node-rank $RANK --master-addr $ADDR generate.py --ckpt-path ${SAVE_PATH} --config ${CONFIG} --input-file ${FILE}
|
| 24 |
```
|
| 25 |
|
|
|
|
|
|
| 3 |
First convert huggingface model weight files to the format of this project.
|
| 4 |
```bash
|
| 5 |
export EXPERTS=256
|
| 6 |
+
export MP=4
|
| 7 |
+
export CONFIG=config.json
|
| 8 |
+
python convert.py --hf-ckpt-path ${HF_CKPT_PATH} --save-path ${SAVE_PATH} --n-experts ${EXPERTS} --model-parallel ${MP}
|
| 9 |
```
|
| 10 |
|
| 11 |
Then chat with DeepSeek model at will!
|
|
|
|
| 23 |
torchrun --nnodes ${NODES} --nproc-per-node $((MP / NODES)) --node-rank $RANK --master-addr $ADDR generate.py --ckpt-path ${SAVE_PATH} --config ${CONFIG} --input-file ${FILE}
|
| 24 |
```
|
| 25 |
|
| 26 |
+
If you want to use fp8, just remove `"expert_dtype": "fp4"` in `config.json` and specify `--expert-dtype fp8` in `convert.py`.
|
inference/generate.py
CHANGED
|
@@ -1,6 +1,4 @@
|
|
| 1 |
import os
|
| 2 |
-
# Prevent gptqmodel from setting CUDA_DEVICE_ORDER=PCI_BUS_ID (breaks multi-GPU on some systems)
|
| 3 |
-
os.environ.setdefault("CUDA_DEVICE_ORDER", "FASTEST_FIRST")
|
| 4 |
import json
|
| 5 |
import sys
|
| 6 |
from argparse import ArgumentParser
|
|
@@ -102,8 +100,6 @@ def main(
|
|
| 102 |
tokenizer = AutoTokenizer.from_pretrained(ckpt_path)
|
| 103 |
print("load model")
|
| 104 |
load_model(model, os.path.join(ckpt_path, f"model{rank}-mp{world_size}.safetensors"), strict=False)
|
| 105 |
-
if args.dtype == "w4a16":
|
| 106 |
-
model.init_woq_layers()
|
| 107 |
torch.set_default_device("cuda")
|
| 108 |
print("I'm DeepSeek 👋")
|
| 109 |
|
|
|
|
| 1 |
import os
|
|
|
|
|
|
|
| 2 |
import json
|
| 3 |
import sys
|
| 4 |
from argparse import ArgumentParser
|
|
|
|
| 100 |
tokenizer = AutoTokenizer.from_pretrained(ckpt_path)
|
| 101 |
print("load model")
|
| 102 |
load_model(model, os.path.join(ckpt_path, f"model{rank}-mp{world_size}.safetensors"), strict=False)
|
|
|
|
|
|
|
| 103 |
torch.set_default_device("cuda")
|
| 104 |
print("I'm DeepSeek 👋")
|
| 105 |
|
inference/model.py
CHANGED
|
@@ -16,35 +16,9 @@ world_size = 1
|
|
| 16 |
rank = 0
|
| 17 |
block_size = 128
|
| 18 |
fp4_block_size = 32
|
| 19 |
-
w4a16_group_size = 128
|
| 20 |
default_dtype = torch.bfloat16
|
| 21 |
scale_fmt = None
|
| 22 |
scale_dtype = torch.float32
|
| 23 |
-
w4a16_mode = False # set in Transformer.__init__ when args.dtype == "w4a16"
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
def dequantize_w4a16(qweight: torch.Tensor, qzeros: torch.Tensor, scales: torch.Tensor,
|
| 27 |
-
group_size: int = 128) -> torch.Tensor:
|
| 28 |
-
"""Auto-round / auto_gptq W4A16 packing -> BF16 weight [out, in].
|
| 29 |
-
|
| 30 |
-
qweight: int32 [in/8, out], LSB-first 4-bit packed along input dim
|
| 31 |
-
qzeros : int32 [in/g, out/8], LSB-first 4-bit packed along output dim
|
| 32 |
-
scales : bf16 [in/g, out]
|
| 33 |
-
"""
|
| 34 |
-
in_packed, out_features = qweight.shape
|
| 35 |
-
in_features = in_packed * 8
|
| 36 |
-
n_groups = scales.shape[0]
|
| 37 |
-
device = qweight.device
|
| 38 |
-
shifts = torch.arange(0, 32, 4, device=device, dtype=torch.int32)
|
| 39 |
-
w = (qweight.unsqueeze(1) >> shifts.view(1, 8, 1)) & 0xF # [in/8, 8, out]
|
| 40 |
-
w = w.reshape(in_features, out_features).to(torch.float32)
|
| 41 |
-
z = (qzeros.unsqueeze(2) >> shifts.view(1, 1, 8)) & 0xF # [in/g, out/8, 8]
|
| 42 |
-
z = z.reshape(n_groups, out_features).to(torch.float32) + 1.0 # GPTQ stores zero - 1
|
| 43 |
-
s = scales.to(torch.float32)
|
| 44 |
-
w = w.view(n_groups, group_size, out_features)
|
| 45 |
-
deq = (w - z.unsqueeze(1)) * s.unsqueeze(1)
|
| 46 |
-
deq = deq.view(in_features, out_features)
|
| 47 |
-
return deq.t().contiguous().to(torch.bfloat16)
|
| 48 |
|
| 49 |
|
| 50 |
@contextmanager
|
|
@@ -62,7 +36,7 @@ class ModelArgs:
|
|
| 62 |
"""Model hyperparameters. Field names match the config JSON keys."""
|
| 63 |
max_batch_size: int = 4
|
| 64 |
max_seq_len: int = 4096
|
| 65 |
-
dtype: Literal["bf16", "fp8"
|
| 66 |
scale_fmt: Literal[None, "ue8m0"] = "ue8m0"
|
| 67 |
expert_dtype: Literal[None, "fp4"] = None
|
| 68 |
scale_dtype: Literal["fp32", "fp8"] = "fp8"
|
|
@@ -153,34 +127,8 @@ class Linear(nn.Module):
|
|
| 153 |
super().__init__()
|
| 154 |
self.in_features = in_features
|
| 155 |
self.out_features = out_features
|
| 156 |
-
|
| 157 |
-
|
| 158 |
-
# embed, attn_sink, etc.) are NOT instances of `Linear`, so they are
|
| 159 |
-
# untouched.
|
| 160 |
-
if w4a16_mode:
|
| 161 |
-
dtype = "w4a16"
|
| 162 |
-
else:
|
| 163 |
-
dtype = dtype or default_dtype
|
| 164 |
-
self.is_w4a16 = (dtype == "w4a16")
|
| 165 |
-
if self.is_w4a16:
|
| 166 |
-
assert in_features % 8 == 0 and in_features % w4a16_group_size == 0
|
| 167 |
-
assert out_features % 8 == 0
|
| 168 |
-
self.group_size = w4a16_group_size
|
| 169 |
-
self.qweight = nn.Parameter(
|
| 170 |
-
torch.empty(in_features // 8, out_features, dtype=torch.int32),
|
| 171 |
-
requires_grad=False,
|
| 172 |
-
)
|
| 173 |
-
self.qzeros = nn.Parameter(
|
| 174 |
-
torch.empty(in_features // self.group_size, out_features // 8, dtype=torch.int32),
|
| 175 |
-
requires_grad=False,
|
| 176 |
-
)
|
| 177 |
-
self.scales = nn.Parameter(
|
| 178 |
-
torch.empty(in_features // self.group_size, out_features, dtype=torch.bfloat16),
|
| 179 |
-
requires_grad=False,
|
| 180 |
-
)
|
| 181 |
-
self.register_parameter("weight", None)
|
| 182 |
-
self.register_parameter("scale", None)
|
| 183 |
-
elif dtype == torch.float4_e2m1fn_x2:
|
| 184 |
# FP4: weight is [out, in//2] in float4_e2m1fn_x2, logically [out, in] in fp4
|
| 185 |
# Scale is [out, in//32] in float8_e8m0fnu (1 scale per 32 fp4 elements along K)
|
| 186 |
self.weight = nn.Parameter(torch.empty(out_features, in_features // 2, dtype=torch.float4_e2m1fn_x2))
|
|
@@ -200,51 +148,7 @@ class Linear(nn.Module):
|
|
| 200 |
else:
|
| 201 |
self.register_parameter("bias", None)
|
| 202 |
|
| 203 |
-
def init_woq(self, QuantLinear):
|
| 204 |
-
"""Create a QuantLinear from loaded GPTQ parameters."""
|
| 205 |
-
if not self.is_w4a16:
|
| 206 |
-
return
|
| 207 |
-
# Marlin requires out_features % 64 == 0; fall back to manual dequant
|
| 208 |
-
if self.out_features % 64 != 0:
|
| 209 |
-
self._woq = None
|
| 210 |
-
return
|
| 211 |
-
dev = self.qweight.device
|
| 212 |
-
layer = QuantLinear(
|
| 213 |
-
bits=4, group_size=self.group_size,
|
| 214 |
-
in_features=self.in_features, out_features=self.out_features,
|
| 215 |
-
bias=False, desc_act=False, sym=True, register_buffers=True,
|
| 216 |
-
)
|
| 217 |
-
layer = layer.to(dev)
|
| 218 |
-
layer.qweight.copy_(self.qweight.data)
|
| 219 |
-
layer.qzeros.copy_(self.qzeros.data)
|
| 220 |
-
layer.scales.copy_(self.scales.to(layer.scales.dtype).data)
|
| 221 |
-
layer.g_idx.copy_(torch.arange(self.in_features, dtype=torch.int32, device=dev) // self.group_size)
|
| 222 |
-
layer.post_init()
|
| 223 |
-
self._woq = layer
|
| 224 |
-
# Free original parameters to save memory
|
| 225 |
-
self.qweight = None
|
| 226 |
-
self.qzeros = None
|
| 227 |
-
self.scales = None
|
| 228 |
-
|
| 229 |
-
def get_weight(self) -> torch.Tensor:
|
| 230 |
-
"""Return the dequantised BF16 weight [out, in]. For non-W4A16 modes
|
| 231 |
-
returns ``self.weight`` unchanged. Used only for wo_a einsum path."""
|
| 232 |
-
if self.is_w4a16:
|
| 233 |
-
if self._woq is not None:
|
| 234 |
-
return dequantize_w4a16(self._woq.qweight, self._woq.qzeros, self._woq.scales, self.group_size)
|
| 235 |
-
return dequantize_w4a16(self.qweight, self.qzeros, self.scales, self.group_size)
|
| 236 |
-
return self.weight
|
| 237 |
-
|
| 238 |
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
| 239 |
-
if self.is_w4a16:
|
| 240 |
-
if hasattr(self, '_woq') and self._woq is not None:
|
| 241 |
-
y = self._woq(x.to(torch.bfloat16))
|
| 242 |
-
else:
|
| 243 |
-
w = dequantize_w4a16(self.qweight, self.qzeros, self.scales, self.group_size)
|
| 244 |
-
y = F.linear(x.to(w.dtype), w)
|
| 245 |
-
if self.bias is not None:
|
| 246 |
-
y = y + self.bias
|
| 247 |
-
return y.type_as(x)
|
| 248 |
return linear(x, self.weight, self.bias)
|
| 249 |
|
| 250 |
|
|
@@ -256,8 +160,6 @@ class ColumnParallelLinear(Linear):
|
|
| 256 |
super().__init__(in_features, self.part_out_features, bias, dtype)
|
| 257 |
|
| 258 |
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
| 259 |
-
if self.is_w4a16:
|
| 260 |
-
return Linear.forward(self, x)
|
| 261 |
return linear(x, self.weight, self.bias)
|
| 262 |
|
| 263 |
|
|
@@ -269,14 +171,7 @@ class RowParallelLinear(Linear):
|
|
| 269 |
super().__init__(self.part_in_features, out_features, bias, dtype)
|
| 270 |
|
| 271 |
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
| 272 |
-
|
| 273 |
-
if hasattr(self, '_woq') and self._woq is not None:
|
| 274 |
-
y = self._woq(x.to(torch.bfloat16))
|
| 275 |
-
else:
|
| 276 |
-
w = dequantize_w4a16(self.qweight, self.qzeros, self.scales, self.group_size)
|
| 277 |
-
y = F.linear(x.to(w.dtype), w)
|
| 278 |
-
else:
|
| 279 |
-
y = linear(x, self.weight, None)
|
| 280 |
if world_size > 1:
|
| 281 |
y = y.float()
|
| 282 |
dist.all_reduce(y)
|
|
@@ -638,12 +533,12 @@ class Attention(nn.Module):
|
|
| 638 |
o = sparse_attn(q, self.kv_cache[:bsz], self.attn_sink, topk_idxs, self.softmax_scale)
|
| 639 |
apply_rotary_emb(o[..., -rd:], freqs_cis, True)
|
| 640 |
|
| 641 |
-
# o
|
| 642 |
-
# Flatten groups into the feature dim, call wo_a as a normal linear, then reshape back.
|
| 643 |
-
# Equivalent to the per-group einsum when wo_a weight is block-diagonal across groups
|
| 644 |
-
# (always true here since n_local_groups = n_groups/world_size = 1 for 8-GPU deploy).
|
| 645 |
o = o.view(bsz, seqlen, self.n_local_groups, -1)
|
| 646 |
-
|
|
|
|
|
|
|
|
|
|
| 647 |
x = self.wo_b(o.flatten(2))
|
| 648 |
return x
|
| 649 |
|
|
@@ -659,59 +554,15 @@ class Gate(nn.Module):
|
|
| 659 |
self.score_func = args.score_func
|
| 660 |
self.route_scale = args.route_scale
|
| 661 |
self.hash = layer_id < args.n_hash_layers
|
| 662 |
-
self.
|
| 663 |
-
if self.is_w4a16:
|
| 664 |
-
in_f, out_f = args.dim, args.n_routed_experts
|
| 665 |
-
assert in_f % w4a16_group_size == 0 and out_f % 8 == 0
|
| 666 |
-
self.group_size = w4a16_group_size
|
| 667 |
-
self.qweight = nn.Parameter(
|
| 668 |
-
torch.empty(in_f // 8, out_f, dtype=torch.int32), requires_grad=False)
|
| 669 |
-
self.qzeros = nn.Parameter(
|
| 670 |
-
torch.empty(in_f // self.group_size, out_f // 8, dtype=torch.int32), requires_grad=False)
|
| 671 |
-
self.scales = nn.Parameter(
|
| 672 |
-
torch.empty(in_f // self.group_size, out_f, dtype=torch.bfloat16), requires_grad=False)
|
| 673 |
-
self.register_parameter("weight", None)
|
| 674 |
-
else:
|
| 675 |
-
self.weight = nn.Parameter(torch.empty(args.n_routed_experts, args.dim))
|
| 676 |
if self.hash:
|
| 677 |
self.tid2eid = nn.Parameter(torch.empty(args.vocab_size, args.n_activated_experts, dtype=torch.int32), requires_grad=False)
|
| 678 |
self.bias = None
|
| 679 |
else:
|
| 680 |
self.bias = nn.Parameter(torch.empty(args.n_routed_experts, dtype=torch.float32))
|
| 681 |
|
| 682 |
-
def init_woq(self, QuantLinear):
|
| 683 |
-
if not self.is_w4a16:
|
| 684 |
-
return
|
| 685 |
-
dev = self.qweight.device
|
| 686 |
-
in_f, out_f = self.dim, self.qweight.shape[1]
|
| 687 |
-
if out_f % 64 != 0:
|
| 688 |
-
self._woq = None
|
| 689 |
-
return
|
| 690 |
-
layer = QuantLinear(
|
| 691 |
-
bits=4, group_size=self.group_size,
|
| 692 |
-
in_features=in_f, out_features=out_f,
|
| 693 |
-
bias=False, desc_act=False, sym=True, register_buffers=True,
|
| 694 |
-
)
|
| 695 |
-
layer = layer.to(dev)
|
| 696 |
-
layer.qweight.copy_(self.qweight.data)
|
| 697 |
-
layer.qzeros.copy_(self.qzeros.data)
|
| 698 |
-
layer.scales.copy_(self.scales.to(layer.scales.dtype).data)
|
| 699 |
-
layer.g_idx.copy_(torch.arange(in_f, dtype=torch.int32, device=dev) // self.group_size)
|
| 700 |
-
layer.post_init()
|
| 701 |
-
self._woq = layer
|
| 702 |
-
self.qweight = None
|
| 703 |
-
self.qzeros = None
|
| 704 |
-
self.scales = None
|
| 705 |
-
|
| 706 |
def forward(self, x: torch.Tensor, input_ids: Optional[torch.Tensor] = None) -> Tuple[torch.Tensor, torch.Tensor]:
|
| 707 |
-
|
| 708 |
-
if hasattr(self, '_woq') and self._woq is not None:
|
| 709 |
-
scores = self._woq(x.to(torch.bfloat16)).float()
|
| 710 |
-
else:
|
| 711 |
-
w = dequantize_w4a16(self.qweight, self.qzeros, self.scales, self.group_size)
|
| 712 |
-
scores = F.linear(x.to(w.dtype), w).float()
|
| 713 |
-
else:
|
| 714 |
-
scores = linear(x.float(), self.weight.float())
|
| 715 |
if self.score_func == "softmax":
|
| 716 |
scores = scores.softmax(dim=-1)
|
| 717 |
elif self.score_func == "sigmoid":
|
|
@@ -773,7 +624,6 @@ class MoE(nn.Module):
|
|
| 773 |
self.experts = nn.ModuleList([Expert(args.dim, args.moe_inter_dim, dtype=expert_dtype, swiglu_limit=args.swiglu_limit) if self.experts_start_idx <= i < self.experts_end_idx else None
|
| 774 |
for i in range(self.n_routed_experts)])
|
| 775 |
assert args.n_shared_experts == 1
|
| 776 |
-
# no swiglu_limit
|
| 777 |
self.shared_experts = Expert(args.dim, args.moe_inter_dim, swiglu_limit=args.swiglu_limit)
|
| 778 |
|
| 779 |
def forward(self, x: torch.Tensor, input_ids: torch.Tensor) -> torch.Tensor:
|
|
@@ -859,7 +709,7 @@ class ParallelHead(nn.Module):
|
|
| 859 |
self.norm_eps = norm_eps
|
| 860 |
self.hc_eps = hc_eps
|
| 861 |
self.part_vocab_size = (vocab_size // world_size)
|
| 862 |
-
# lm_head
|
| 863 |
self.weight = nn.Parameter(torch.empty(self.part_vocab_size, self.dim, dtype=torch.float32))
|
| 864 |
|
| 865 |
def get_logits(self, x):
|
|
@@ -920,18 +770,12 @@ class Transformer(nn.Module):
|
|
| 920 |
"""Full DeepSeek-V4 model: embed -> HC-expand -> N blocks -> HC-head -> logits.
|
| 921 |
Sets global state (world_size, rank, default_dtype, scale_fmt, scale_dtype) in __init__."""
|
| 922 |
def __init__(self, args: ModelArgs):
|
| 923 |
-
global world_size, rank, default_dtype, scale_fmt, scale_dtype
|
| 924 |
world_size = dist.get_world_size() if dist.is_initialized() else 1
|
| 925 |
rank = dist.get_rank() if dist.is_initialized() else 0
|
| 926 |
-
|
| 927 |
-
if
|
| 928 |
-
|
| 929 |
-
scale_fmt = None
|
| 930 |
-
scale_dtype = torch.float32
|
| 931 |
-
else:
|
| 932 |
-
default_dtype = torch.float8_e4m3fn if args.dtype == "fp8" else torch.bfloat16
|
| 933 |
-
scale_fmt = "ue8m0" if args.scale_dtype == "fp8" else args.scale_fmt
|
| 934 |
-
scale_dtype = torch.float8_e8m0fnu if args.scale_dtype == "fp8" else torch.float32
|
| 935 |
super().__init__()
|
| 936 |
self.max_seq_len = args.max_seq_len
|
| 937 |
self.norm_eps = args.norm_eps
|
|
@@ -954,15 +798,6 @@ class Transformer(nn.Module):
|
|
| 954 |
self.hc_head_base = nn.Parameter(torch.empty(hc_mult))
|
| 955 |
self.hc_head_scale = nn.Parameter(torch.empty(1))
|
| 956 |
|
| 957 |
-
def init_woq_layers(self):
|
| 958 |
-
"""After load_model(), convert all W4A16 parameters into QuantLinear layers."""
|
| 959 |
-
# from gptqmodel.nn_modules.qlinear.tritonv2 import TritonV2QuantLinear as QuantLinear
|
| 960 |
-
from gptqmodel.nn_modules.qlinear.marlin import MarlinQuantLinear as QuantLinear
|
| 961 |
-
for module in self.modules():
|
| 962 |
-
if hasattr(module, 'init_woq') and module is not self:
|
| 963 |
-
module.init_woq(QuantLinear)
|
| 964 |
-
torch.cuda.empty_cache()
|
| 965 |
-
|
| 966 |
@torch.inference_mode()
|
| 967 |
def forward(self, input_ids: torch.Tensor, start_pos: int = 0):
|
| 968 |
h = self.embed(input_ids)
|
|
|
|
| 16 |
rank = 0
|
| 17 |
block_size = 128
|
| 18 |
fp4_block_size = 32
|
|
|
|
| 19 |
default_dtype = torch.bfloat16
|
| 20 |
scale_fmt = None
|
| 21 |
scale_dtype = torch.float32
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 22 |
|
| 23 |
|
| 24 |
@contextmanager
|
|
|
|
| 36 |
"""Model hyperparameters. Field names match the config JSON keys."""
|
| 37 |
max_batch_size: int = 4
|
| 38 |
max_seq_len: int = 4096
|
| 39 |
+
dtype: Literal["bf16", "fp8"] = "fp8"
|
| 40 |
scale_fmt: Literal[None, "ue8m0"] = "ue8m0"
|
| 41 |
expert_dtype: Literal[None, "fp4"] = None
|
| 42 |
scale_dtype: Literal["fp32", "fp8"] = "fp8"
|
|
|
|
| 127 |
super().__init__()
|
| 128 |
self.in_features = in_features
|
| 129 |
self.out_features = out_features
|
| 130 |
+
dtype = dtype or default_dtype
|
| 131 |
+
if dtype == torch.float4_e2m1fn_x2:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 132 |
# FP4: weight is [out, in//2] in float4_e2m1fn_x2, logically [out, in] in fp4
|
| 133 |
# Scale is [out, in//32] in float8_e8m0fnu (1 scale per 32 fp4 elements along K)
|
| 134 |
self.weight = nn.Parameter(torch.empty(out_features, in_features // 2, dtype=torch.float4_e2m1fn_x2))
|
|
|
|
| 148 |
else:
|
| 149 |
self.register_parameter("bias", None)
|
| 150 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 151 |
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 152 |
return linear(x, self.weight, self.bias)
|
| 153 |
|
| 154 |
|
|
|
|
| 160 |
super().__init__(in_features, self.part_out_features, bias, dtype)
|
| 161 |
|
| 162 |
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
|
|
|
|
|
|
| 163 |
return linear(x, self.weight, self.bias)
|
| 164 |
|
| 165 |
|
|
|
|
| 171 |
super().__init__(self.part_in_features, out_features, bias, dtype)
|
| 172 |
|
| 173 |
def forward(self, x: torch.Tensor) -> torch.Tensor:
|
| 174 |
+
y = linear(x, self.weight, None)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 175 |
if world_size > 1:
|
| 176 |
y = y.float()
|
| 177 |
dist.all_reduce(y)
|
|
|
|
| 533 |
o = sparse_attn(q, self.kv_cache[:bsz], self.attn_sink, topk_idxs, self.softmax_scale)
|
| 534 |
apply_rotary_emb(o[..., -rd:], freqs_cis, True)
|
| 535 |
|
| 536 |
+
# o
|
|
|
|
|
|
|
|
|
|
| 537 |
o = o.view(bsz, seqlen, self.n_local_groups, -1)
|
| 538 |
+
wo_a = self.wo_a.weight.view(self.n_local_groups, self.o_lora_rank, -1)
|
| 539 |
+
# NOTE: wo_a is FP8 in checkpoint; could do FP8 einsum here for better perf,
|
| 540 |
+
# but using BF16 for simplicity.
|
| 541 |
+
o = torch.einsum("bsgd,grd->bsgr", o, wo_a)
|
| 542 |
x = self.wo_b(o.flatten(2))
|
| 543 |
return x
|
| 544 |
|
|
|
|
| 554 |
self.score_func = args.score_func
|
| 555 |
self.route_scale = args.route_scale
|
| 556 |
self.hash = layer_id < args.n_hash_layers
|
| 557 |
+
self.weight = nn.Parameter(torch.empty(args.n_routed_experts, args.dim))
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 558 |
if self.hash:
|
| 559 |
self.tid2eid = nn.Parameter(torch.empty(args.vocab_size, args.n_activated_experts, dtype=torch.int32), requires_grad=False)
|
| 560 |
self.bias = None
|
| 561 |
else:
|
| 562 |
self.bias = nn.Parameter(torch.empty(args.n_routed_experts, dtype=torch.float32))
|
| 563 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 564 |
def forward(self, x: torch.Tensor, input_ids: Optional[torch.Tensor] = None) -> Tuple[torch.Tensor, torch.Tensor]:
|
| 565 |
+
scores = linear(x.float(), self.weight.float())
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 566 |
if self.score_func == "softmax":
|
| 567 |
scores = scores.softmax(dim=-1)
|
| 568 |
elif self.score_func == "sigmoid":
|
|
|
|
| 624 |
self.experts = nn.ModuleList([Expert(args.dim, args.moe_inter_dim, dtype=expert_dtype, swiglu_limit=args.swiglu_limit) if self.experts_start_idx <= i < self.experts_end_idx else None
|
| 625 |
for i in range(self.n_routed_experts)])
|
| 626 |
assert args.n_shared_experts == 1
|
|
|
|
| 627 |
self.shared_experts = Expert(args.dim, args.moe_inter_dim, swiglu_limit=args.swiglu_limit)
|
| 628 |
|
| 629 |
def forward(self, x: torch.Tensor, input_ids: torch.Tensor) -> torch.Tensor:
|
|
|
|
| 709 |
self.norm_eps = norm_eps
|
| 710 |
self.hc_eps = hc_eps
|
| 711 |
self.part_vocab_size = (vocab_size // world_size)
|
| 712 |
+
# lm_head in the checkpoint is stored in bf16, while the parameter here is stored in fp32 for easier computation of logits later.
|
| 713 |
self.weight = nn.Parameter(torch.empty(self.part_vocab_size, self.dim, dtype=torch.float32))
|
| 714 |
|
| 715 |
def get_logits(self, x):
|
|
|
|
| 770 |
"""Full DeepSeek-V4 model: embed -> HC-expand -> N blocks -> HC-head -> logits.
|
| 771 |
Sets global state (world_size, rank, default_dtype, scale_fmt, scale_dtype) in __init__."""
|
| 772 |
def __init__(self, args: ModelArgs):
|
| 773 |
+
global world_size, rank, default_dtype, scale_fmt, scale_dtype
|
| 774 |
world_size = dist.get_world_size() if dist.is_initialized() else 1
|
| 775 |
rank = dist.get_rank() if dist.is_initialized() else 0
|
| 776 |
+
default_dtype = torch.float8_e4m3fn if args.dtype == "fp8" else torch.bfloat16
|
| 777 |
+
scale_fmt = "ue8m0" if args.scale_dtype == "fp8" else args.scale_fmt
|
| 778 |
+
scale_dtype = torch.float8_e8m0fnu if args.scale_dtype == "fp8" else torch.float32
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 779 |
super().__init__()
|
| 780 |
self.max_seq_len = args.max_seq_len
|
| 781 |
self.norm_eps = args.norm_eps
|
|
|
|
| 798 |
self.hc_head_base = nn.Parameter(torch.empty(hc_mult))
|
| 799 |
self.hc_head_scale = nn.Parameter(torch.empty(1))
|
| 800 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 801 |
@torch.inference_mode()
|
| 802 |
def forward(self, input_ids: torch.Tensor, start_pos: int = 0):
|
| 803 |
h = self.embed(input_ids)
|
inference/requirements.txt
CHANGED
|
@@ -2,5 +2,4 @@ torch>=2.10.0
|
|
| 2 |
transformers>=5.0.0
|
| 3 |
safetensors>=0.7.0
|
| 4 |
fast_hadamard_transform
|
| 5 |
-
tilelang==0.1.8
|
| 6 |
-
gptqmodel==6.0.3
|
|
|
|
| 2 |
transformers>=5.0.0
|
| 3 |
safetensors>=0.7.0
|
| 4 |
fast_hadamard_transform
|
| 5 |
+
tilelang==0.1.8
|
|
|
model-00002-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:6618a5361a98e3320b904118ec44b24344d80e5431dafd2b711555ccde44d366
|
| 3 |
+
size 3477095680
|
model-00003-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:9d0a73917baa25d1c01e3a42df95aba1dc2cb68c558e518bce9d0e1a196110fe
|
| 3 |
+
size 3477095680
|
model-00004-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3589581d62d9281b8ad1da35b95bfedef84cfcb923bc8577a048952dd2e98a59
|
| 3 |
+
size 3502972736
|
model-00005-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a4206f35baed1f28eaef83248318a6cb69af37dd6d26c57fb300d1f7eaed419f
|
| 3 |
+
size 3479543432
|
model-00006-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d559bcb1d22582a78c746c7922e7e7e2be793106409b0036b289adfe10dbdb0a
|
| 3 |
+
size 3496768264
|
model-00007-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:9c7cbd22f0499b820bb97dff5c0d5be431ca5b77e154651ec897ab12d01eb2eb
|
| 3 |
+
size 3479543432
|
model-00008-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b32125218dffc7048a6e198da9782e09e6d810d3cec5759c38d08d7d5ed0684a
|
| 3 |
+
size 3496768264
|
model-00009-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d86d105bb06228e43730db3b9e374da67ba4b54ba53e4531b7662d4d9f872f70
|
| 3 |
+
size 3479543432
|
model-00010-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e292a801f3fa6a94e27aa0b531e7bffa7a4d8881defaeb5de615903dc791dea6
|
| 3 |
+
size 3496768264
|
model-00011-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:24105aec63df32300dba0d2aae2e8b3a4bc8e1a36261b68b81e9898f001d0a85
|
| 3 |
+
size 3479543432
|
model-00012-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:27cd663d032f20849d3d0fe4a033569429d49ca5be2255b81e6ddf4c5c62cc13
|
| 3 |
+
size 3496770616
|
model-00013-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:cf26562d7ac818e48b09be54c940620f7d326ba2072d8ca55f3a5926db3570fd
|
| 3 |
+
size 3479545776
|
model-00014-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a977b89d8b041bf73f3811971a3d834521dbf96f20ce5c37cbd8446a5ccf354a
|
| 3 |
+
size 3496770616
|
model-00015-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:04ca5679a66fc7743360f3a8bc9a49f38132f00b3aacf17e68b0753948ef91c9
|
| 3 |
+
size 3479545776
|
model-00016-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:7983e469cadec35df9a3353d21ef80d5ff81aee56ac340ea19b5aa8aab95726f
|
| 3 |
+
size 3496770616
|
model-00017-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:a8b0fc6a864e2deb724b24b54860985d59bdaba67b3908854f962c87ee805f45
|
| 3 |
+
size 3479545776
|
model-00018-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:bbb00df23b6e2a91706b44a16b3cd85b53c805c23840beabd80cf8318f580b1a
|
| 3 |
+
size 3496770616
|
model-00019-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2a2e0858ccb9d60bdb2ecc8547ee83437b8875168b7fe7259815083c805e43bd
|
| 3 |
+
size 3479545776
|
model-00020-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:cf8bb4d99f392a2106f6602518feb48c2627bb076ebec657594dad071766e754
|
| 3 |
+
size 3496770616
|
model-00021-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:be8e098683557b6e44374cb7b8ca1b7d754be3034a9758b7e9d87288251f8283
|
| 3 |
+
size 3479545776
|
model-00022-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:63e6f9ff02eb0ca214408c353727ae6ddacd9a443df6893063a39fad5bc8da02
|
| 3 |
+
size 3496770616
|
model-00023-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:4e2bc0b3988718211272fba9abdb5fe4f133f1b6ccc7492b1bdfada5e405da7c
|
| 3 |
+
size 3479545776
|
model-00024-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:eacf61cb808296c33e3bfc8d8876bb1465e113bf1a0146ed89d5debd3be9ec21
|
| 3 |
+
size 3496770616
|
model-00025-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f2c45ce2ff4642ccc223dad6901471a0567269aef8bbf565860f3e08add4e413
|
| 3 |
+
size 3479545776
|
model-00026-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ade2380ab274963f77ffb0d5f1a196e6b073af01492a819bcc5225a7cf73e27d
|
| 3 |
+
size 3496770616
|
model-00027-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0f95d38eee88befab5d33275b7f0c1d3bc2d8bbbc165e4a0218f03deaf05001b
|
| 3 |
+
size 3479545776
|
model-00028-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0644622a898b12accbd66ae648da69abbc55e707ac2ebf0c242f16e5022c165e
|
| 3 |
+
size 3496770616
|
model-00029-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:310c3af9c0b3f8d56167c8095c75ad6c69786571a0b8e050cb9187e5df4bede9
|
| 3 |
+
size 3479545776
|
model-00030-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:182e5de93c63c5e9d50efae769bc1e7f64c130c4e6e802bc529ca541607d3a53
|
| 3 |
+
size 3496770616
|
model-00031-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b621a2e8540c459e08b6285bb2c25df1c4d6cad486f8c3abb5abc312dc6aa306
|
| 3 |
+
size 3479545776
|
model-00032-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:f8cc5af0a73949f00442412e1a299b97637af208f80cb07cae1eb48cf243a42c
|
| 3 |
+
size 3496770616
|
model-00033-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:afda71aac8f47c2c18615e0ddf244da3146d3550f2baee2edbbf5c5b614d6937
|
| 3 |
+
size 3479545776
|
model-00034-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:21ecf6ab3a1ff7c6962540a33b01d7e6bee7f42d24a3588ddb66602618da6dca
|
| 3 |
+
size 3496770616
|
model-00035-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:781ddbbf31f30a59cfd2b9a0d27ada5596b229779af7969921aea008f228d615
|
| 3 |
+
size 3479545776
|
model-00036-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:2b4cdbb8be17e42c8eb19f370df4b06e5262aae07a9ea348947effb78db25c58
|
| 3 |
+
size 3496770616
|
model-00037-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:4e2b208a2a4887aa434ea02143fcca905d2209cb0953c0258e8e930b50e2d6cf
|
| 3 |
+
size 3479545776
|
model-00038-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:e91f54779818309a096a4abeadbbbd36617317706d65aa4e31362fa4b371aa51
|
| 3 |
+
size 3496770616
|
model-00039-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b5879afa52366dac9af42da20acb117222373067612f107042832c8ed96e4963
|
| 3 |
+
size 3479545776
|
model-00040-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b2ca6d4421ebec09f14830273b95fa7102a97cbe60733046a5c2612c750dd1bd
|
| 3 |
+
size 3496770616
|
model-00041-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:54680691e6eb715f82f31522af745045c913705f23ef5730a059332fad2ef77b
|
| 3 |
+
size 3479545776
|
model-00042-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:ecd566bb6f100bcaf728a198c0cd0299995bdbf5e041650f47d48ec43264d2f8
|
| 3 |
+
size 3496770616
|
model-00043-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:db7a7d5a4e627f536b37afa7e679b520222d9dc7996271f2cb28782a48c0e704
|
| 3 |
+
size 3479545776
|
model-00044-of-00046.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:4b8128fd119dc6eaa64de3b1d18b0a46f80b21c87b06bbc373756cb3ed2094f1
|
| 3 |
+
size 3496770616
|