diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000000000000000000000000000000000000..17787c9eb02b68f55955503341d62a1ec51eb17e --- /dev/null +++ b/.gitattributes @@ -0,0 +1,40 @@ +*.7z filter=lfs diff=lfs merge=lfs -text +*.arrow filter=lfs diff=lfs merge=lfs -text +*.bin filter=lfs diff=lfs merge=lfs -text +*.bz2 filter=lfs diff=lfs merge=lfs -text +*.ckpt filter=lfs diff=lfs merge=lfs -text +*.ftz filter=lfs diff=lfs merge=lfs -text +*.gz filter=lfs diff=lfs merge=lfs -text +*.h5 filter=lfs diff=lfs merge=lfs -text +*.joblib filter=lfs diff=lfs merge=lfs -text +*.lfs.* filter=lfs diff=lfs merge=lfs -text +*.mlmodel filter=lfs diff=lfs merge=lfs -text +*.model filter=lfs diff=lfs merge=lfs -text +*.msgpack filter=lfs diff=lfs merge=lfs -text +*.npy filter=lfs diff=lfs merge=lfs -text +*.npz filter=lfs diff=lfs merge=lfs -text +*.onnx filter=lfs diff=lfs merge=lfs -text +*.ot filter=lfs diff=lfs merge=lfs -text +*.parquet filter=lfs diff=lfs merge=lfs -text +*.pb filter=lfs diff=lfs merge=lfs -text +*.pickle filter=lfs diff=lfs merge=lfs -text +*.pkl filter=lfs diff=lfs merge=lfs -text +*.pt filter=lfs diff=lfs merge=lfs -text +*.pth filter=lfs diff=lfs merge=lfs -text +*.rar filter=lfs diff=lfs merge=lfs -text +*.safetensors filter=lfs diff=lfs merge=lfs -text +saved_model/**/* filter=lfs diff=lfs merge=lfs -text +*.tar.* filter=lfs diff=lfs merge=lfs -text +*.tar filter=lfs diff=lfs merge=lfs -text +*.tflite filter=lfs diff=lfs merge=lfs -text +*.tgz filter=lfs diff=lfs merge=lfs -text +*.wasm filter=lfs diff=lfs merge=lfs -text +*.xz filter=lfs diff=lfs merge=lfs -text +*.zip filter=lfs diff=lfs merge=lfs -text +*.zst filter=lfs diff=lfs merge=lfs -text +*tfevents* filter=lfs diff=lfs merge=lfs -text +assets/benchmark.png filter=lfs diff=lfs merge=lfs -text +assets/dfly2.png filter=lfs diff=lfs merge=lfs -text +assets/performance-cost-open-benchmarks.png filter=lfs diff=lfs merge=lfs -text +assets/performance-cost-workindex.png filter=lfs diff=lfs merge=lfs -text +pretrained-model-bpb.png filter=lfs diff=lfs merge=lfs -text diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000000000000000000000000000000000000..76d8ade724bbb184088e750d6c99cd408144cdd9 --- /dev/null +++ b/.gitignore @@ -0,0 +1,40 @@ +# Python +__pycache__/ +*.py[cod] +*$py.class +*.so +*.egg-info/ +dist/ +build/ +*.egg + +# Virtual environments +.venv/ +venv/ +env/ + +# IDE +.idea/ +.vscode/ +*.swp +*.swo + +# OS +.DS_Store +Thumbs.db + +# Training outputs +output/ +saves/ + +# Generated by the multi-node launch scripts +env.txt +hostfile +pssh.hosts + +# Logs +*.log +logs/ + +# Cache +.cache/ diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000000000000000000000000000000000000..2be6f88597077cc42c495586eaf4fb6c6587e33c --- /dev/null +++ b/LICENSE @@ -0,0 +1,245 @@ + Tencent Hy License, Version 1.0 + + This License is derived from the Apache License, Version 2.0 + (http://www.apache.org/licenses/LICENSE-2.0), with modifications, mainly + the additional commercial terms for large Model-as-a-Service ("MaaS") + businesses in Section 10. This License is not the Apache License, + Version 2.0, and is not endorsed by or affiliated with the Apache + Software Foundation. + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 10 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). For the avoidance + of doubt, the Work includes the model weights, parameters, + configuration files, inference and training code, and documentation + made available by Licensor under this License. + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. For the avoidance of doubt, + Derivative Works include any fine-tuned, quantized, distilled, merged, + pruned, or otherwise modified versions of the model weights. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. Notwithstanding the + foregoing, any license terms You provide for the Work or any + Derivative Works must not remove, limit, or conflict with Section 10, + and Section 10 shall continue to apply to every recipient of the + Work or any Derivative Works. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + 10. Additional Commercial Terms. + + 10.1 MaaS. "MaaS" means giving third parties remote access to model + inference or fine-tuning (e.g., via API), allowing them to + submit inputs or specify parameters or training data. This + excludes (i) end-user products with model capabilities confined + to specific features or workflows, without general-purpose + model access, or (ii) merely relaying requests to an + independently operated third-party model service. + + 10.2 Separate License Required. Notwithstanding Sections 2 and 3, + and except as provided in Section 10.3, if You operate a MaaS + business, You must obtain a separate written license to use the + Work or any Derivative Works from Licensor (including by email), + in either of the following cases: + + (a) if Your aggregate worldwide gross revenue exceeds + US$50 million (or the equivalent in other currencies) in + any consecutive 12-month period, whether or not such + revenue is derived from the Work or any Derivative Works, + before using (or continuing to use) the Work or any + Derivative Works; or + + (b) before using the Work or any Derivative Works in or for any + product or service provided by You that has more than + 100 million monthly active users in any calendar month. + + 10.3 Internal Use. Section 10.2 does not apply to Your use of the + Work or any Derivative Works solely for internal purposes, + provided that neither the Work, any Derivative Works, nor their + outputs are made available to third parties. + + 10.4 Contact. For licensing requests or questions, please contact + hy_partners@tencent.com. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Tencent Hy License to your work. + + To apply this License to a Derivative Work, attach the following + notice, with the fields enclosed by brackets "[]" replaced with your + own identifying information. (Don't include the brackets!) + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Tencent Hy License, Version 1.0 (the "License"), + which is based on the Apache License, Version 2.0, with additional + commercial terms. You may not use this file except in compliance + with the License. You may obtain a copy of the License in the + LICENSE file distributed with this work. + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied. See the License for the specific language governing + permissions and limitations under the License. diff --git a/README.md b/README.md new file mode 100644 index 0000000000000000000000000000000000000000..fb97b75cab89e5d7076b71750246fea8fcbc0646 --- /dev/null +++ b/README.md @@ -0,0 +1,396 @@ +--- +license: other +license_name: tencent-hy-license +license_link: https://huggingface.co/tencent/Hy4-flash/blob/main/LICENSE +library_name: transformers +pipeline_tag: image-text-to-text +tags: +- hunyuan +- hy4-flash +- moe +- multimodal +- instruct +--- + +

+ 中文 | English +

+
+ +

+ Tencent Hy +

+ +
+ +[![License](https://img.shields.io/badge/License-Tencent%20Hy%20License-blue)](#license) +   +[![Hugging Face](https://img.shields.io/badge/Hugging%20Face-Tencent%20Hy-ffc107?color=ffc107&logoColor=white)](https://huggingface.co/tencent/Hy4-flash) +   +[![ModelScope](https://img.shields.io/badge/ModelScope-Tencent%20Hy-624aff)](https://modelscope.cn/models/Tencent-Hunyuan/Hy4-flash) +   +[![GitCode](https://img.shields.io/badge/GitCode-Tencent%20Hy-red?logoColor=white)](https://ai.gitcode.com/tencent_hunyuan/Hy4-flash) + +
+ +

+ Official Website  |   + GitHub  |   + GGUF +

+ +--- + +## Table of Contents + +- [Model Introduction](#model-introduction) +- [Pretrained model capability](#pretrained-model-capability) +- [Practicality and Cost Efficiency](#practicality-and-cost-efficiency) + - [Optimization Approach](#optimization-approach) + - [Task Performance and Cost](#task-performance-and-cost) + - [User Blind Evaluation](#user-blind-evaluation) +- [Appendix: Benchmark results](#appendix-benchmark-results) +- [News](#news) +- [Model Links](#model-links) +- [Quickstart](#quickstart) + - [Transformers](#transformers) + - [OpenAI-compatible API](#openai-compatible-api) +- [Deployment](#deployment) + - [DFly2 for Speculative Decoding](#dfly2-for-speculative-decoding) + - [Quantization](#quantization) + - [Inference Frameworks](#inference-frameworks) + - [vLLM](#vllm) + - [SGLang](#sglang) +- [Finetuning](#finetuning) +- [License](#license) +- [Contact Us](#contact-us) + +## Model Introduction + +**Hy4 flash** is our most cost-effective Hy model to date, featuring 220B total parameters (including 64B Engram parameters) with just 12B active parameters per token. It natively accepts text and image inputs, generates text, and supports a context window of up to 1M tokens. + +Its 42-layer language backbone consists of one dense feed-forward layer and 41 MoE layers, with identity Hyper-Connections (iHC) supporting inter-layer information flow. A vision encoder (adapted from SigLIP2-So400m-Patch16-Res512) is connected to the backbone through an MLP projector. In addition to the backbone, a built-in 5-layer drafter, [DFly2](#dfly2-for-speculative-decoding) (0.85B parameters), reuses the target model's KV cache and sparse indices to support long-context speculative decoding at a low, constant cost. + +Compared with Hy4, Hy4 Flash introduces three notable architectural changes: + +1. **Hybrid attention.** Sparse attention (IndexPool) and sliding-window attention (SWA) are interleaved at a 1:3 ratio. +2. **LatentMoE.** Routed experts operate in a lower-dimensional latent space. Each token activates 16 of 512 routed experts, alongside one shared expert. +3. **N-gram embedding (Engram).** The backbone is augmented with learned bigram and trigram representations, retrieved via multi-head hash lookups. + +| Property | Value | +| --- | --- | +| **Overview** | | +| Architecture | Mixture-of-Experts (MoE) | +| Total Parameters (excluding Engram, ViT, and DFly2) | 155.5B | +| Engram Parameters | 63.5B | +| ViT Parameters | 0.43B | +| DFly2 Parameters | 0.85B | +| Activated Parameters per Token | Text 11.3B + Vision 0.43B | +| Context Length | 1M (1,048,576 tokens) | +| **Language Backbone** | | +| Backbone Layers | 42 — 1 dense layer and 41 MoE layers | +| Hidden Size | 3,072 | +| Vocabulary Size | 120,832 | +| Residual Connections | iHC (identity Hyper-Connections) with 4 residual streams | +| **Attention — Hybrid Layout** | | +| Attention Architecture | Sparse attention + sliding-window attention (SWA) | +| Backbone Layer Pattern | 2 × Sparse → 10 × (3 × SWA → Sparse) | +| Attention Formulation | Multi-head Latent Attention (MLA) | +| Multimodal Position Encoding | Interleaved MRoPE | +| **Attention — Layers** | **Sparse / SWA** | +| Backbone Layers | 12 / 30 | +| Attention Heads | 32 / 64 | +| MLA q_lora_rank | 1,024 / 1,024 | +| MLA kv_lora_rank | 512 / 512 | +| QK Head Dimension (non-RoPE + RoPE) | 192 + 64 / 192 + 64 | +| Value Head Dimension | 256 / 256 | +| RoPE Base | 10,000,000 / 10,000 | +| Sparse Context Selection | Up to 2,048 tokens via group-level selection | +| SWA Context Window | 512-token causal window (511 preceding + current) | +| **Feed-Forward Networks** | | +| Dense FFN Intermediate Size | 13,824 | +| Routed Experts per MoE Layer | 512 | +| Shared Experts per MoE Layer | 1 | +| Activated Routed Experts per Token | 16 | +| MoE Latent Dimension | 1,536 | +| Expert Intermediate Size | 1,536 | +| **N-gram Embedding** | | +| N-gram Orders | Bigrams and trigrams | +| Backbone Layer Indices (starting from 0) | 1, 11, 21, 31 | +| Hash Heads per N-gram Order | 8 | +| Hash Vocabulary Size per Head | 10.3M | +| Embedding Dimension per N-gram Order | 768 (8 heads × 96 dimensions) | +| Embedding Table Parameters | 63.4B | +| **Vision Encoder** | | +| Vision Encoder Layers | 27 | +| Vision Encoder Hidden Size | 1,152 | +| Vision Encoder Intermediate Size | 4,304 | +| Vision Encoder Attention Heads | 16 | +| Vision Patch Size | 16 × 16 | +| Vision Position Embedding Size | 128 × 128 | +| Visual Feature Merging | 2 x 2 | + +## Pretrained model capability + +Bits-per-byte (BPB) on internal code & agent validation, lower is better: + +

+ Bits-per-byte on internal code and agent validation versus active parameters; lower is better +

+ +Few-shot pretrain benchmark evaluation: + +| Category | Benchmark | #Shots | Hy4 flash Base | Hy4 Base | Hy3 preview Base | DeepSeek-V4-flash-Base | DeepSeek-V4-Pro-Base | +| --- | --- | :---: | ---: | ---: | ---: | ---: | ---: | +| **Knowledge** | MMLU† | 5 | 88.7 | **91.6** | 87.5 | 89.7 | 91.1 | +| | MMLU-Pro† | 5 | 71.1 | **73.9** | 65.8 | 68.6 | 73.4 | +| | MMLU-Redux† | 5 | 87.6 | **90.6** | 87.0 | 88.8 | 90.2 | +| | SuperGPQA† | 5 | 57.8 | **62.7** | 52.3 | 52.5 | 54.8 | +| | C-Eval† | 5 | 92.0 | **95.0** | 90.1 | 91.9 | 93.4 | +| | CMMLU† | 5 | 91.6 | **93.1** | 89.6 | 89.8 | 90.7 | +| | SimpleQA | 5 | 37.5 | **57.2** | 26.4 | 31.3 | 52.1 | +| | Chinese-SimpleQA† | 5 | 74.6 | **83.9** | 69.6 | 71.9 | 76.6 | +| | AGIEval (English)† | 3 | 75.4 | **79.3** | 76.0 | 76.3 | 78.2 | +| | AGIEval (Chinese)† | 3 | 90.6 | **91.2** | 87.8 | 88.5 | 89.8 | +| **Reasoning** | HellaSwag (3,000-item subset) | 10 | 85.2 | **87.1** | 85.7 | 85.1 | 86.8 | +| | PIQA | 4 | 84.4 | **85.3** | 84.6 | 83.8 | 84.9 | +| | ARC-Challenge | 0 | 95.0 | **97.3** | 96.0 | **97.3** | 97.0 | +| | Social IQa | 0 | 80.1 | **82.7** | 80.5 | 72.5 | 73.1 | +| | KOR-Bench | 3 | 54.7 | **57.6** | 54.6 | 53.4 | 55.4 | +| | C³ | 0 | 77.3 | **82.4** | 75.9 | 74.1 | 78.3 | +| **Math** | U-MATH | 5 | 69.8 | **71.6** | 60.9 | 57.4 | 64.8 | +| | OCW Courses | 5 | 64.0 | 66.7 | 62.6 | 63.3 | **68.2** | +| **Code** | MBPP-plus | 3 | 73.7 | 74.0 | 77.4 | 76.8 | **79.1** | +| **Multilingual** | Belebele (5-language subset)† | 5 | 93.8 | 95.0 | 94.2 | 94.4 | **95.2** | +| | INCLUDE† | 5 | 82.8 | **88.1** | 84.3 | 84.0 | 84.3 | +| **Multimodal** | MMMU-Pro (Standard, 10 opts) | 4 | 52.0 | **59.6** | — | — | — | +| | SimpleVQA | 4 | 48.5 | **54.5** | — | — | — | +| | CVBench | 4 | 83.4 | **86.7** | — | — | — | +| | RealWorldQA | 4 | 71.1 | **72.3** | — | — | — | +| | TextVQA | 4 | 73.4 | **78.2** | — | — | — | +| | CountBenchQA | 4 | 86.6 | **90.6** | — | — | — | + +† Macro-averaged accuracy for the marked text benchmarks. Other text rows use the accuracy metric reported by our evaluation suite. Bold marks the highest available numerical score in each row; it does not indicate statistical significance. + +Multimodal results are reported only for Hy4 flash Base and Hy4 Base. + +ARC-Challenge and HellaSwag use 299-item and 3,000-item subsets, respectively. INCLUDE uses a 5,000-example Lite subset covering 44 languages. Belebele covers German, Korean, Vietnamese, Spanish, and Russian (4,475 examples in total). + +**Disclaimer: Because our base model was exposed to a large amount of synthetic instruction data during mid-training, it occasionally fails to follow the expected answer format in few-shot evaluations, which may distort the scores.** + +## Practicality and Cost Efficiency + +### Optimization Approach + +Across training, evaluation, and inference, we aggressively optimized for cost efficiency while preserving intelligence: + +- **Pre-training**: Hy4 flash integrates hybrid attention, LatentMoE, and Engram into its architecture—reducing KV cache size to just **1/24 of Hy3** and **1/6 of Hy4**, with extensive scaling ladder experiments ensuring performance meets expectations. +- **Post-training**: Self-distillation from Hy4 establishes strong general intelligence, while reinforcement learning cuts average end-to-end output tokens by **50%**. +- **Evaluation**: We built WorkIndex, an internal benchmark grounded in typical WorkBuddy tasks, and paired it with human evaluation—focusing on **deliverable quality, single-turn output length, and task turnaround time** rather than public benchmark scores detached from everyday work. +- **Inference**: Combining megakernels, 4-bit quantization, and [DFly2](#dfly2-for-speculative-decoding) pushes decoding speed past **150 tokens/s**, enabling low-cost serving at scale. + +### Task Performance and Cost + +**Hy4 flash delivers overall task performance slightly above Hy4 preview at a much lower cost per task.** On WorkIndex, it scores **42.1**, compared with **39.7** for Hy4 preview, while its average cost per task is just **1/13 of Hy4 preview**, **1/20 of Hy4**, and **1/8 of DeepSeek-V4.1-Flash-1017** (Max tier, peak-hour pricing). + +

+ Performance and cost per task on WorkIndex for Hy4 flash and comparison models + Performance and cost per task on Open Benchmarks for Hy4 flash and comparison models +

+

Figure 1: Performance and cost across internal and external tasks. WorkIndex (internal) comprises 8 evaluation sets covering slide creation, slide editing, human-AI slide co-writing, Markdown human-AI co-writing, document editing, spreadsheet repair, audio/video processing, and visual understanding.
Open Benchmarks includes 7 public agent benchmarks: DeepSWE, ProgramBench, Terminal-Bench 4.0, APEX-Agents, AutomationBench, HLE, and OSWorld 2.1.

+ +

+ Average cost per task on WorkIndex for Hy4 flash, Hy4 preview, and Hy4 +

+

Figure 2: Average cost per task on WorkIndex. Task scores are 42.1 for Hy4 flash, 39.7 for Hy4 preview, and 51.4 for Hy4.

+ +We recommend Hy4 flash for most everyday productivity tasks, including **deep search, document processing, slide creation, simple frontend demos, and computer use**. For complex tasks such as long-horizon software engineering, we recommend Hy4 for its higher intelligence ceiling. + +### User Blind Evaluation + +In the blind evaluation, **Hy4 flash achieved user satisfaction close to Claude Haiku 5.5, while recording the lowest cost per task and shortest median end-to-end completion time among the evaluated models**. Its user satisfaction score was **2.79/4** (Claude Haiku 5.5: **2.81/4**), with a median end-to-end completion time of **6.7 minutes** and a cost of **RMB 0.26 per task**. + +

+ Blind evaluation comparing user satisfaction, median end-to-end completion time, and cost per task +

+

Figure 3: Blind evaluation with 85 internal WorkBuddy users across 259 tasks spanning 27 categories.

+ +## Appendix: Benchmark results + +

+ Full benchmark scores, task costs, and evaluation notes for Hy4 flash and comparison models +

+ +## News + +* We release **Hy4 flash**, **Hy4 flash-FP8**, and the pre-trained **Hy4 flash-base** as open-weight models on [Hugging Face](https://huggingface.co/tencent/Hy4-flash), [ModelScope](https://modelscope.cn/models/Tencent-Hunyuan/Hy4-flash), and [GitCode](https://ai.gitcode.com/tencent_hunyuan/Hy4-flash). + +## Model Links + +| Model Name | Description | Hugging Face | ModelScope | GitCode | +|:---|:---|:---:|:---:|:---:| +| Hy4 flash | Instruct model | [Model](https://huggingface.co/tencent/Hy4-flash) | [Model](https://modelscope.cn/models/Tencent-Hunyuan/Hy4-flash) | [Model](https://ai.gitcode.com/tencent_hunyuan/Hy4-flash) | +| Hy4 flash-FP8 | FP8 quantized instruct model | [Model](https://huggingface.co/tencent/Hy4-flash-FP8) | [Model](https://modelscope.cn/models/Tencent-Hunyuan/Hy4-flash-FP8) | [Model](https://ai.gitcode.com/tencent_hunyuan/Hy4-flash-FP8) | +| Hy4 flash-base | Pre-trained base model | [Model](https://huggingface.co/tencent/Hy4-flash-base) | [Model](https://modelscope.cn/models/Tencent-Hunyuan/Hy4-flash-base) | [Model](https://ai.gitcode.com/tencent_hunyuan/Hy4-flash-base) | + +Hy4 flash-base is a pre-trained base model, not an instruction-tuned chat model. The chat and reasoning examples below apply to Hy4 flash and Hy4 flash-FP8. + +## Quickstart + +Use [Transformers](#transformers) to load the model for local inference, or deploy a server and access it through the [OpenAI-compatible API](#openai-compatible-api). + +> **Reasoning mode (instruct models)**: `reasoning_effort` supports `"high"` (default, thinking enabled) and `"no_think"` (direct responses). Both examples below use `"no_think"`. +> +> **Recommended sampling parameters**: `temperature=0.9`, `top_p=1.0`. + +### Transformers + +> Support for Hy4 flash will be merged into the `transformers` main branch soon. Until then, install from the commit below. + +```bash +pip install "transformers[torch,vision] @ git+https://github.com/huggingface/transformers.git@refs/pull/50903/head" +``` + +The following example generates a brief image description: + +```python +from transformers import AutoModelForImageTextToText, AutoProcessor + +model_id = "tencent/Hy4-flash" +processor = AutoProcessor.from_pretrained(model_id) +model = AutoModelForImageTextToText.from_pretrained(model_id, device_map="auto") + +messages = [ + { + "role": "user", + "content": [ + { + "type": "image", + "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg", + }, + {"type": "text", "text": "Describe this image in one sentence."}, + ], + } +] + +inputs = processor.apply_chat_template( + messages, + add_generation_prompt=True, + tokenize=True, + return_dict=True, + return_tensors="pt", + reasoning_effort="no_think", +).to(model.device) + +outputs = model.generate(**inputs, max_new_tokens=256) +print(processor.decode(outputs[0, inputs["input_ids"].shape[1] :], skip_special_tokens=True)) +``` + +### OpenAI-compatible API + +For API access, start a server using the [vLLM deployment instructions](#vllm), then run the following client example: + +```python +from openai import OpenAI + +client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY") + +response = client.chat.completions.create( + model="hy4-flash", + messages=[ + {"role": "user", "content": "Hello! Can you briefly introduce yourself?"}, + ], + temperature=0.9, + top_p=1.0, + extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}}, +) +print(response.choices[0].message.content) +``` + +## Deployment + +### DFly2 for Speculative Decoding + +DFly2 is Hy4 flash's speculative-decoding drafter, trained with [AngelSpec](https://github.com/Tencent/AngelSpec) and built directly on the target model's **Top-k Sparse KV**: each draft layer shares the KV cache (**KV Share**) and the compressed indexer's selection of up to 2,048 positions (**Index Share**) with its corresponding sparse attention layer. This yields three advantages: + +- **Constant cost**: each draft token attends to at most ~2K keys, regardless of context length. +- **Zero extra KV memory**: no separate KV cache is maintained for the drafter. +- **PD-disaggregation ready**: not involved in prefill; the prefill-to-decode transfer adds only one last-layer hidden state. + +

+ DFly2 architecture: each draft layer reuses the KV cache and the indexer-selected positions of a sparse attention layer of Hy4 flash +

+ +### Quantization + +We provide [AngelSlim](https://github.com/tencent/AngelSlim), a more accessible, comprehensive, and efficient toolkit for large model compression. AngelSlim supports a comprehensive suite of compression tools for large-scale multimodal models, including common quantization algorithms, low-bit quantization, and speculative sampling. + +For this model, we have provided a [GGUF](https://huggingface.co/AngelSlim/Hy4-flash-GGUF) compressed version for end-to-end testing purposes. + +### Inference Frameworks + +For production serving, we recommend using [vLLM](https://github.com/vllm-project/vllm) or [SGLang](https://docs.sglang.io/). Please refer to the recipes: +- [Hy4 flash vLLM Recipe](https://recipes.vllm.ai/tencent/Hy4-flash) +- [Hy4 flash SGLang Cookbook](https://docs.sglang.io/cookbook/autoregressive/Tencent/Hy4-flash) + +#### vLLM + +Use official prebuilt image `vllm/vllm-openai:hy4-flash`: + +```bash +docker run --gpus all \ + -p 8000:8000 \ + --ipc=host \ + -v ~/.cache/huggingface:/root/.cache/huggingface \ + vllm/vllm-openai:hy4-flash tencent/Hy4-flash-FP8 \ + --tensor-parallel-size 8 \ + --speculative-config '{"num_speculative_tokens":5,"method":"dfly"}' \ + --attention-backend FLASHMLA_SPARSE \ + --tool-call-parser hy_v4 \ + --reasoning-parser hy_v4 \ + --enable-auto-tool-choice \ + --port 8000 \ + --served-model-name hy4-flash +``` + +#### SGLang + +Use the official prebuilt image `lmsysorg/sglang:hy4-flash` (multi-arch, x86 and Arm): + +```bash +docker pull lmsysorg/sglang:hy4-flash + +docker run --gpus all \ + -p 8000:8000 \ + --ipc=host \ + lmsysorg/sglang:hy4-flash \ + sglang serve \ + --trust-remote-code \ + --model-path tencent/Hy4-flash-FP8 \ + --tp 8 \ + --cuda-graph-backend-prefill disabled \ + --speculative-algorithm DFLY \ + --speculative-dfly-block-size 6 \ + --host 127.0.0.1 \ + --port 8000 +``` + +## Finetuning + +Hy4 flash provides a complete model finetuning pipeline. For detailed documentation, please refer to: [Finetuning Guide](https://huggingface.co/tencent/Hy4-flash/blob/main/finetune/README.md) + +## License + +Hy4 flash is released under the **Tencent Hy License**. See [LICENSE](https://huggingface.co/tencent/Hy4-flash/blob/main/LICENSE) for details. + +## Contact Us + +If you have any questions or suggestions, feel free to reach out to our R&D and product teams via email: + +📧 **hunyuan_opensource@tencent.com** + +--- + +

+ Hy4 flash is developed by the Tencent Hy Team. +

diff --git a/README_CN.md b/README_CN.md new file mode 100644 index 0000000000000000000000000000000000000000..5ced1363f3b5431ab67a67da23c8d9b689c0226f --- /dev/null +++ b/README_CN.md @@ -0,0 +1,385 @@ +

+ English | 中文 +

+
+ +

+ 腾讯混元 +

+ +
+ +[![License](https://img.shields.io/badge/License-Tencent%20Hy%20License-blue)](#许可证) +   +[![Hugging Face](https://img.shields.io/badge/Hugging%20Face-Tencent%20Hy-ffc107?color=ffc107&logoColor=white)](https://huggingface.co/tencent/Hy4-flash) +   +[![ModelScope](https://img.shields.io/badge/ModelScope-Tencent%20Hy-624aff)](https://modelscope.cn/models/Tencent-Hunyuan/Hy4-flash) +   +[![GitCode](https://img.shields.io/badge/GitCode-Tencent%20Hy-red?logoColor=white)](https://ai.gitcode.com/tencent_hunyuan/Hy4-flash) + +
+ +

+ 官方网站  |   + GitHub  |   + GGUF +

+ +--- + +## 目录 + +- [模型介绍](#模型介绍) +- [预训练模型能力](#预训练模型能力) +- [实用性与成本效率](#实用性与成本效率) + - [优化思路](#优化思路) + - [任务效果与成本](#任务效果与成本) + - [用户盲测](#用户盲测) +- [Benchmark 附录](#benchmark-附录) +- [新闻](#新闻) +- [模型链接](#模型链接) +- [快速开始](#快速开始) + - [Transformers](#transformers) + - [OpenAI 兼容 API](#openai-兼容-api) +- [推理和部署](#推理和部署) + - [DFly2 投机解码](#dfly2-投机解码) + - [量化工具](#量化工具) + - [推理框架](#推理框架) + - [vLLM](#vllm) + - [SGLang](#sglang) +- [模型微调](#模型微调) +- [许可证](#许可证) +- [联系我们](#联系我们) + +## 模型介绍 + +**Hy4 flash** 是混元系列迄今性价比最高的多模态模型,原生支持文本和图像输入,可生成文本,上下文长度可达 1M token。模型总参数量约为 2200 亿,其中 Engram 约占 640 亿。处理每个 token 时,只需激活约 120 亿个参数。 + +语言模型主干由 1 层稠密前馈网络和 41 层混合专家网络(MoE)组成,共 42 层。各层通过 identity Hyper-Connections(iHC)传递残差信息。视觉编码器由 SigLIP2-So400m-Patch16-Res512 训练而来,提取的图像特征经 MLP 投影后送入语言模型主干。主干之外还内置了 5 层草稿模型 [DFly2](#dfly2-投机解码)(约 8.5 亿参数),直接复用主模型的 KV cache 与稀疏索引,以固定的低开销支持长上下文投机解码。 + +相较于 Hy4,Hy4 flash 在架构上有三处主要变化。 + +1. **混合注意力(Hybrid attention)** 用滑动窗口注意力(SWA)捕捉相邻 token 之间的联系,用 IndexPool 稀疏注意力从更长的上下文中选取相关片段。稀疏注意力与 SWA 以 1:3 的模式交替组合。 +2. **LatentMoE** 将 token 的表示从 3072 维投影到 1536 维,让路由专家在更低维的空间里完成计算。每个 MoE 层会为当前 token 从 512 个路由专家中选出 16 个,另有 1 个共享专家参与每个 token 的计算。 +3. **Engram(N-gram embedding)** 为模型引入了按需查表的记忆机制。模型根据最近两个或三个 token 的组合,通过多头哈希取出训练中学到的片段表示,并将其融入当前 token 的隐藏状态。 + +| 规格项 | 取值 | +| --- | --- | +| **模型概览** | | +| Architecture | Mixture-of-Experts (MoE) | +| Total Parameters (excluding Engram, ViT, and DFly2) | 155.5B | +| Engram Parameters | 63.5B | +| ViT Parameters | 0.43B | +| DFly2 Parameters | 0.85B | +| Activated Parameters per Token | Text 11.3B + Vision 0.43B | +| Context Length | 1M (1,048,576 tokens) | +| **语言模型主干** | | +| Backbone Layers | 42 — 1 dense layer and 41 MoE layers | +| Hidden Size | 3,072 | +| Vocabulary Size | 120,832 | +| Residual Connections | iHC (identity Hyper-Connections) with 4 residual streams | +| **混合注意力布局** | | +| Attention Architecture | Sparse attention + sliding-window attention (SWA) | +| Backbone Layer Pattern | 2 × Sparse → 10 × (3 × SWA → Sparse) | +| Attention Formulation | Multi-head Latent Attention (MLA) | +| Multimodal Position Encoding | Interleaved MRoPE | +| **注意力层配置** | **稀疏注意力 / 滑动窗口注意力** | +| Backbone Layers | 12 / 30 | +| Attention Heads | 32 / 64 | +| MLA q_lora_rank | 1,024 / 1,024 | +| MLA kv_lora_rank | 512 / 512 | +| QK Head Dimension (non-RoPE + RoPE) | 192 + 64 / 192 + 64 | +| Value Head Dimension | 256 / 256 | +| RoPE Base | 10,000,000 / 10,000 | +| Sparse Context Selection | Up to 2,048 tokens via group-level selection | +| SWA Context Window | 512-token causal window (511 preceding + current) | +| **前馈网络** | | +| Dense FFN Intermediate Size | 13,824 | +| Routed Experts per MoE Layer | 512 | +| Shared Experts per MoE Layer | 1 | +| Activated Routed Experts per Token | 16 | +| MoE Latent Dimension | 1,536 | +| Expert Intermediate Size | 1,536 | +| **N-gram 嵌入(Engram)** | | +| N-gram Orders | Bigrams and trigrams | +| Backbone Layer Indices (starting from 0) | 1, 11, 21, 31 | +| Hash Heads per N-gram Order | 8 | +| Hash Vocabulary Size per Head | 10.3M | +| Embedding Dimension per N-gram Order | 768 (8 heads × 96 dimensions) | +| Embedding Table Parameters | 63.4B | +| **视觉编码器** | | +| Vision Encoder Layers | 27 | +| Vision Encoder Hidden Size | 1,152 | +| Vision Encoder Intermediate Size | 4,304 | +| Vision Encoder Attention Heads | 16 | +| Vision Patch Size | 16 × 16 | +| Vision Position Embedding Size | 128 × 128 | +| Visual Feature Merging | 2 x 2 | + +## 预训练模型能力 + +内部代码与 Agent 验证集上的 BPB(Bits-per-byte,每字节比特数),数值越低越好: + +

+ 内部代码与 Agent 验证集上的 BPB 与激活参数量对比,BPB 越低越好 +

+ +预训练模型的 few-shot 基准评测结果: + +| 类别 | 评测基准 | #Shots | Hy4 flash Base | Hy4 Base | Hy3 preview Base | DeepSeek-V4-flash-Base | DeepSeek-V4-Pro-Base | +| --- | --- | :---: | ---: | ---: | ---: | ---: | ---: | +| **知识** | MMLU† | 5 | 88.7 | **91.6** | 87.5 | 89.7 | 91.1 | +| | MMLU-Pro† | 5 | 71.1 | **73.9** | 65.8 | 68.6 | 73.4 | +| | MMLU-Redux† | 5 | 87.6 | **90.6** | 87.0 | 88.8 | 90.2 | +| | SuperGPQA† | 5 | 57.8 | **62.7** | 52.3 | 52.5 | 54.8 | +| | C-Eval† | 5 | 92.0 | **95.0** | 90.1 | 91.9 | 93.4 | +| | CMMLU† | 5 | 91.6 | **93.1** | 89.6 | 89.8 | 90.7 | +| | SimpleQA | 5 | 37.5 | **57.2** | 26.4 | 31.3 | 52.1 | +| | Chinese-SimpleQA† | 5 | 74.6 | **83.9** | 69.6 | 71.9 | 76.6 | +| | AGIEval (English)† | 3 | 75.4 | **79.3** | 76.0 | 76.3 | 78.2 | +| | AGIEval (Chinese)† | 3 | 90.6 | **91.2** | 87.8 | 88.5 | 89.8 | +| **推理** | HellaSwag (3,000-item subset) | 10 | 85.2 | **87.1** | 85.7 | 85.1 | 86.8 | +| | PIQA | 4 | 84.4 | **85.3** | 84.6 | 83.8 | 84.9 | +| | ARC-Challenge | 0 | 95.0 | **97.3** | 96.0 | **97.3** | 97.0 | +| | Social IQa | 0 | 80.1 | **82.7** | 80.5 | 72.5 | 73.1 | +| | KOR-Bench | 3 | 54.7 | **57.6** | 54.6 | 53.4 | 55.4 | +| | C³ | 0 | 77.3 | **82.4** | 75.9 | 74.1 | 78.3 | +| **数学** | U-MATH | 5 | 69.8 | **71.6** | 60.9 | 57.4 | 64.8 | +| | OCW Courses | 5 | 64.0 | 66.7 | 62.6 | 63.3 | **68.2** | +| **代码** | MBPP-plus | 3 | 73.7 | 74.0 | 77.4 | 76.8 | **79.1** | +| **多语言** | Belebele (5-language subset)† | 5 | 93.8 | 95.0 | 94.2 | 94.4 | **95.2** | +| | INCLUDE† | 5 | 82.8 | **88.1** | 84.3 | 84.0 | 84.3 | +| **多模态** | MMMU-Pro (Standard, 10 opts) | 4 | 52.0 | **59.6** | — | — | — | +| | SimpleVQA | 4 | 48.5 | **54.5** | — | — | — | +| | CVBench | 4 | 83.4 | **86.7** | — | — | — | +| | RealWorldQA | 4 | 71.1 | **72.3** | — | — | — | +| | TextVQA | 4 | 73.4 | **78.2** | — | — | — | +| | CountBenchQA | 4 | 86.6 | **90.6** | — | — | — | + +† 标注的文本评测采用宏平均准确率,其余文本评测采用本评测框架报告的准确率。每行最高分以粗体标示;加粗不代表差异具有统计显著性。 + +多模态结果仅列出 Hy4 flash Base 和 Hy4 Base。 + +ARC-Challenge 和 HellaSwag 分别使用包含 299 道和 3,000 道题目的子集。INCLUDE 使用覆盖 44 种语言、包含 5,000 个样本的 Lite 子集。Belebele 覆盖德语、韩语、越南语、西班牙语和俄语,共 4,475 个样本。 + +**免责声明:我们的基座模型在中期训练中使用了大量合成指令数据,因此在 few-shot 评测中偶尔无法遵循预期的答案格式,可能导致得分失真。** + +## 实用性与成本效率 + +### 优化思路 + +我们通过一系列工作在极致优化成本的同时兼顾智能水平和实用性: + +- 在预训练阶段,Hy4 flash 加入了混合注意力、LatentMoE、Engram 等架构特性,KV cache 大小仅为 Hy3 的 **1/24**、Hy4 的 **1/6**,且通过大量 scaling ladder 实验确保效果符合预期; +- 在后训练阶段,我们通过自蒸馏 Hy4 获得了较强的综合智能水平,再通过强化学习使端到端任务输出长度降低 **50%**; +- 在评估阶段,我们构建基于 WorkBuddy 典型任务的内部评测集 WorkIndex,并结合人评,聚焦日常工作的**产物效果、单轮输出长度、端到端用时**,而非远离日常工作的公开榜单分数; +- 在推理阶段,我们结合 megakernel、4bit 量化、[DFly2](#dfly2-投机解码) 等手段极致优化,使 Hy4 flash 解码速度超过 **150 tps**,且可以大规模低成本部署运营。 + +### 任务效果与成本 + +**Hy4 flash 的综合任务效果略优于 Hy4 preview,同时显著降低了完成任务的成本。** 在 WorkIndex 上,其得分为 **42.1**,高于 Hy4 preview 的 **39.7**;平均单任务成本仅为 **Hy4 preview 的 1/13、Hy4 的 1/20、DeepSeek-V4.1-Flash-1017 的 1/8**(DeepSeek 按 `max` 档位、忙时价格计算)。 + +

+ Hy4 flash 与对比模型在 WorkIndex 上的得分和单任务成本 + Hy4 flash 与对比模型在公开评测上的得分和单任务成本 +

+

图 1:内外部任务上的效果与成本。WorkIndex (internal) 包含 8 个内部评测集:PPT 制作、PPT 修改、PPT 人机双写、Markdown 人机双写、文档编辑、表格修复、音视频处理和视觉理解。
Open Benchmarks 包含 7 个公开 Agent 评测集:DeepSWE、ProgramBench、Terminal-Bench 4.0、APEX-Agents、AutomationBench、HLE 和 OSWorld 2.1。

+ +

+ Hy4 flash、Hy4 preview 和 Hy4 的 WorkIndex 平均单任务成本与得分 +

+

图 2:WorkIndex 平均单任务成本对比;Hy4 flash、Hy4 preview、Hy4 的任务得分分别为 42.1、39.7、51.4。

+ +我们推荐在**深度搜索、文档处理、PPT 制作、简单前端展示和 computer use** 等大部分日常任务中使用 Hy4 flash;对于长程软件工程等复杂任务,仍推荐智能上限更高的 Hy4。 + +### 用户盲测 + +**在真实工作任务盲测中,Hy4 flash 的满意度接近 Claude Haiku 5.5,同时在参评模型中实现了最低的单任务成本和最短的任务耗时中位数。** Hy4 flash 的用户满意度为 **2.79/4**(Claude Haiku 5.5 为 **2.81/4**),端到端任务时长中位数为 **6.7 分钟**,单任务成本为 **0.26 元**。 + +

+ 用户盲测中的满意度、端到端任务时长中位数和单任务成本对比 +

+

图 3:85 位 WorkBuddy 内部用户在 27 类、259 个日常工作任务上的盲测结果。

+ +## Benchmark 附录 + +

+ Hy4 flash 与对比模型的完整评测得分、单任务成本和测试说明 +

+ +## 新闻 + +* 我们在 [Hugging Face](https://huggingface.co/tencent/Hy4-flash)、[ModelScope](https://modelscope.cn/models/Tencent-Hunyuan/Hy4-flash) 和 [GitCode](https://ai.gitcode.com/tencent_hunyuan/Hy4-flash) 开源 **Hy4 flash**、**Hy4 flash-FP8** 和预训练模型 **Hy4 flash-base** 的权重。 + +## 模型链接 + +| 模型名 | 简介 | Hugging Face | ModelScope | GitCode | +|:---|:---|:---:|:---:|:---:| +| Hy4 flash | Instruct 模型 | [Model](https://huggingface.co/tencent/Hy4-flash) | [Model](https://modelscope.cn/models/Tencent-Hunyuan/Hy4-flash) | [Model](https://ai.gitcode.com/tencent_hunyuan/Hy4-flash) | +| Hy4 flash-FP8 | FP8 量化 Instruct 模型 | [Model](https://huggingface.co/tencent/Hy4-flash-FP8) | [Model](https://modelscope.cn/models/Tencent-Hunyuan/Hy4-flash-FP8) | [Model](https://ai.gitcode.com/tencent_hunyuan/Hy4-flash-FP8) | +| Hy4 flash-base | 预训练模型 | [Model](https://huggingface.co/tencent/Hy4-flash-base) | [Model](https://modelscope.cn/models/Tencent-Hunyuan/Hy4-flash-base) | [Model](https://ai.gitcode.com/tencent_hunyuan/Hy4-flash-base) | + +Hy4 flash-base 为预训练底座,未经过指令微调;下方对话与推理模式示例适用于 Hy4 flash 和 Hy4 flash-FP8。 + +## 快速开始 + +可使用 [Transformers](#transformers) 直接加载模型进行本地推理,或部署服务后通过 [OpenAI 兼容 API](#openai-兼容-api) 调用。 + +> **推理模式(Instruct 模型)**:`reasoning_effort` 支持 `"high"`(默认,启用思考)和 `"no_think"`(直接回复)。以下两个示例均使用 `"no_think"`。 +> +> **推荐采样参数**:`temperature=0.9`,`top_p=1.0`。 + +### Transformers + +> Hy4 flash 的支持即将合入 `transformers` 主分支,在此之前请从下面的提交安装。 + +```bash +pip install "transformers[torch,vision] @ git+https://github.com/huggingface/transformers.git@refs/pull/50903/head" +``` + +下面以图片描述为例: + +```python +from transformers import AutoModelForImageTextToText, AutoProcessor + +model_id = "tencent/Hy4-flash" +processor = AutoProcessor.from_pretrained(model_id) +model = AutoModelForImageTextToText.from_pretrained(model_id, device_map="auto") + +messages = [ + { + "role": "user", + "content": [ + { + "type": "image", + "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg", + }, + {"type": "text", "text": "Describe this image in one sentence."}, + ], + } +] + +inputs = processor.apply_chat_template( + messages, + add_generation_prompt=True, + tokenize=True, + return_dict=True, + return_tensors="pt", + reasoning_effort="no_think", +).to(model.device) + +outputs = model.generate(**inputs, max_new_tokens=256) +print(processor.decode(outputs[0, inputs["input_ids"].shape[1] :], skip_special_tokens=True)) +``` + +### OpenAI 兼容 API + +选择 API 调用时,先按 [vLLM 部署说明](#vllm) 启动服务,再运行以下客户端示例: + +```python +from openai import OpenAI + +client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY") + +response = client.chat.completions.create( + model="hy4-flash", + messages=[ + {"role": "user", "content": "你好!请简单介绍一下你自己。"}, + ], + temperature=0.9, + top_p=1.0, + extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}}, +) +print(response.choices[0].message.content) +``` + +## 推理和部署 + +### DFly2 投机解码 + +DFly2 是基于 [AngelSpec](https://github.com/Tencent/AngelSpec) 框架训练的 Hy4 flash 投机解码草稿模型,直接复用主模型的 **Top-k Sparse KV**:每个 draft 层与主模型对应的稀疏注意力层共享 KV cache(**KV Share**)及压缩索引器选出的最多 2,048 个位置(**Index Share**)。由此带来三点优势: + +- **计算开销固定**:每个草稿 token 至多关注约 2K 个 key,注意力复杂度与上下文长度无关。 +- **零额外 KV 显存**:无需维护草稿模型独立的 KV cache。 +- **适配 PD 分离**:不参与 prefill,P→D 仅额外传输一个末层隐状态。 + +

+ DFly2 结构图:每个 draft 层复用 Hy4 flash 对应稀疏注意力层的 KV cache 与压缩索引器选出的位置 +

+ +### 量化工具 + +我们提供了 [AngelSlim](https://github.com/tencent/AngelSlim)——一套易用、全面、高效的大模型压缩工具包,涵盖常用量化算法、低比特量化和投机采样等能力。 + +对于本模型我们提供了[GGUF](https://huggingface.co/AngelSlim/Hy4-flash-GGUF) 压缩版本以供端测使用。 + +### 推理框架 + +对于生产环境部署,我们建议使用 [vLLM](https://github.com/vllm-project/vllm) 或 [SGLang](https://docs.sglang.io/)。请参考部署方案: +- [Hy4 flash vLLM Recipe](https://recipes.vllm.ai/tencent/Hy4-flash) +- [Hy4 flash SGLang Cookbook](https://docs.sglang.io/cookbook/autoregressive/Tencent/Hy4-flash) + + + +#### vLLM + +使用社区官方镜像 `vllm/vllm-openai:hy4-flash` 部署: + +```bash +docker run --gpus all \ + -p 8000:8000 \ + --ipc=host \ + -v ~/.cache/huggingface:/root/.cache/huggingface \ + vllm/vllm-openai:hy4-flash tencent/Hy4-flash-FP8 \ + --tensor-parallel-size 8 \ + --speculative-config '{"num_speculative_tokens":5,"method":"dfly"}' \ + --attention-backend FLASHMLA_SPARSE \ + --tool-call-parser hy_v4 \ + --reasoning-parser hy_v4 \ + --enable-auto-tool-choice \ + --port 8000 \ + --served-model-name hy4-flash +``` + + +#### SGLang + +使用社区官方镜像 `lmsysorg/sglang:hy4-flash` 部署,(支持 x86 和 Arm)。 + +```bash +docker pull lmsysorg/sglang:hy4-flash + +docker run --gpus all \ + -p 8000:8000 \ + --ipc=host \ + lmsysorg/sglang:hy4-flash \ + sglang serve \ + --trust-remote-code \ + --model-path tencent/Hy4-flash-FP8 \ + --tp 8 \ + --cuda-graph-backend-prefill disabled \ + --speculative-algorithm DFLY \ + --speculative-dfly-block-size 6 \ + --host 127.0.0.1 \ + --port 8000 +``` + +## 模型微调 + +Hy4 flash 提供了完整的模型微调流程,详细的微调文档请参考:[模型微调指南](https://huggingface.co/tencent/Hy4-flash/blob/main/finetune/README_CN.md) + +## 许可证 + +Hy4 flash 基于 **Tencent Hy License** 发布。详情请参阅 [LICENSE](https://huggingface.co/tencent/Hy4-flash/blob/main/LICENSE)。 + +## 联系我们 + +如有问题或建议,欢迎通过邮件联系我们的研发和产品团队: + +📧 **hunyuan_opensource@tencent.com** + +--- + +

+ Hy4 flash 由腾讯混元团队研发。 +

diff --git a/assets/benchmark.png b/assets/benchmark.png new file mode 100644 index 0000000000000000000000000000000000000000..3368850c8b60926b7ba5d4e17b6c7f48bb4f8d58 --- /dev/null +++ b/assets/benchmark.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:2aab42905fa871f41086bdbd474a7d850970b5614cd18e7eadb4656606ae113f +size 937522 diff --git a/assets/blind-evaluation-en.png b/assets/blind-evaluation-en.png new file mode 100644 index 0000000000000000000000000000000000000000..f9c4b23afc56c9b27ba9c0dde498adbd7ab46d80 Binary files /dev/null and b/assets/blind-evaluation-en.png differ diff --git a/assets/blind-evaluation-zh.png b/assets/blind-evaluation-zh.png new file mode 100644 index 0000000000000000000000000000000000000000..adb5a89f57652e1e7c665d7dba70fc29952bcc92 Binary files /dev/null and b/assets/blind-evaluation-zh.png differ diff --git a/assets/dfly2.png b/assets/dfly2.png new file mode 100644 index 0000000000000000000000000000000000000000..dbd9a8b1f7a13ae4330b4411e1a3ad444f026431 --- /dev/null +++ b/assets/dfly2.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1a38561973c67c7834cbacf44df02d7d6ea9e569e50514b6bd58ac28a6e94ae9 +size 906016 diff --git a/assets/logo-en.png b/assets/logo-en.png new file mode 100644 index 0000000000000000000000000000000000000000..836258358f0904150d09dc896f300715368937d4 Binary files /dev/null and b/assets/logo-en.png differ diff --git a/assets/logo-zh.png b/assets/logo-zh.png new file mode 100644 index 0000000000000000000000000000000000000000..ff8436092e09b52a2ea06f237713fe64141f475c Binary files /dev/null and b/assets/logo-zh.png differ diff --git a/assets/performance-cost-open-benchmarks.png b/assets/performance-cost-open-benchmarks.png new file mode 100644 index 0000000000000000000000000000000000000000..2cef3ff37b716ef9373342b5c94540bcf1b9dd64 --- /dev/null +++ b/assets/performance-cost-open-benchmarks.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:825c7f7994b6c16f69232a277d34ee6e51c7db57d4b8f1a8ef3ef8e45858182a +size 119011 diff --git a/assets/performance-cost-workindex.png b/assets/performance-cost-workindex.png new file mode 100644 index 0000000000000000000000000000000000000000..83bbab26c3abda5c80c394290dd5a1be743ef528 --- /dev/null +++ b/assets/performance-cost-workindex.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:74ab28aff592ad549606d26ae40295135a5873908b76c67eab938e948f2a9552 +size 114997 diff --git a/assets/workindex-cost.png b/assets/workindex-cost.png new file mode 100644 index 0000000000000000000000000000000000000000..1b967388dfe81cfd896ba8883c4407c8ab4560c9 Binary files /dev/null and b/assets/workindex-cost.png differ diff --git a/chat_template.jinja b/chat_template.jinja new file mode 100644 index 0000000000000000000000000000000000000000..8bb8ebcdf8b687c0edd5c5a746f38ab4a98efc9b --- /dev/null +++ b/chat_template.jinja @@ -0,0 +1,355 @@ +{#- ----------‑‑‑ special token variables ‑‑‑---------- -#} +{%- set HYTK = ':opensource' %} +{%- set hy_start_token = '<|hy_start{}|>'.format(HYTK) %} +{%- set hy_middle_token = '<|hy_middle{}|>'.format(HYTK) %} +{%- set hy_end_token = '<|hy_end{}|>'.format(HYTK) %} +{%- set image_start_token = '<|hy_image▁start{}|>'.format(HYTK) %} +{%- set image_pad_token = '<|hy_image▁pad{}|>'.format(HYTK) %} +{%- set image_end_token = '<|hy_image▁end{}|>'.format(HYTK) %} +{%- set video_start_token = '<|hy_video▁start{}|>'.format(HYTK) %} +{%- set video_pad_token = '<|hy_video▁pad{}|>'.format(HYTK) %} +{%- set video_end_token = '<|hy_video▁end{}|>'.format(HYTK) %} +{%- set think_begin_token = ''.format(HYTK) %} +{%- set think_end_token = ''.format(HYTK) %} +{%- set toolcalls_begin_token = ''.format(HYTK) %} +{%- set toolcalls_end_token = ''.format(HYTK) %} +{%- set toolcall_begin_token = ''.format(HYTK) %} +{%- set toolcall_end_token = ''.format(HYTK) %} +{%- set argkey_begin_token = ''.format(HYTK) %} +{%- set argkey_end_token = ''.format(HYTK) %} +{%- set argvalue_begin_token = ''.format(HYTK) %} +{%- set argvalue_end_token = ''.format(HYTK) %} +{%- set toolresponse_begin_token = ''.format(HYTK) %} +{%- set toolresponse_end_token = ''.format(HYTK) %} +{%- set reasoning_mode_token = '<|reasoning_mode{}|>'.format(HYTK) %} + +{#- ----------‑‑‑ hyperparameters variables ‑‑‑---------- -#} +{%- if not add_generation_prompt is defined %} + {%- set add_generation_prompt = false %} +{%- endif %} +{%- if not preserved_thinking is defined %} + {%- if not tools %} + {%- set preserved_thinking = false %} + {%- else %} + {%- set preserved_thinking = true %} + {%- endif %} +{%- endif %} + +{%- if not reasoning_effort is defined %} + {%- set reasoning_effort = 'high' %} +{%- elif reasoning_effort not in ['max', 'high', 'no_think'] %} + {%- if reasoning_effort is none %} + {{- raise_exception('reasoning_effort error : None, should be no_think/high/max') }} + {%- else %} + {{- raise_exception('reasoning_effort error : ' + reasoning_effort + ', should be no_think/high/max') }} + {%- endif %} +{%- endif %} + +{%- if fallback_strategy is defined and fallback_strategy == 'reasoning_toolcall_retry' %} + {%- set reasoning_effort = 'high' %} + {%- set add_generation_prompt = false %} +{%- endif %} +{%- if not raw_last_assistant is defined %} + {%- set raw_last_assistant = false %} +{%- endif %} +{%- if not add_vision_id is defined %} + {%- set add_vision_id = false %} +{%- endif %} +{%- set image_count = namespace(value=0) %} +{%- set video_count = namespace(value=0) %} + +{%- macro tool_to_json(tool) -%} + {%- set ns_tool = namespace(first=true) -%} + {{- '{' -}} + {%- for k, v in tool.items() -%} + {%- if k != 'defer_loading' and k != 'strict' -%} + {%- if not ns_tool.first -%}{{- ', ' -}}{%- endif -%} + {%- set ns_tool.first = false -%} + {{- '"' ~ k ~ '": ' ~ (v | tojson(ensure_ascii=False)) -}} + {%- endif -%} + {%- endfor -%} + {{- '}' -}} +{%- endmacro -%} + +{%- macro render_content(content, do_vision_count=true, is_system_content=false) -%} + {%- if content is string -%} + {{- content -}} + {%- elif content is iterable and content is not mapping -%} + {%- for item in content -%} + {%- if item is mapping and item.type == 'text' -%} + {{- item.text -}} + {%- elif item is mapping and ('image' in item or 'image_url' in item or item.type in ['image', 'image_url']) -%} + {%- if is_system_content -%} + {{- raise_exception('System message cannot contain images.') -}} + {%- endif -%} + {%- if do_vision_count -%} + {%- set image_count.value = image_count.value + 1 -%} + {%- endif -%} + {%- if add_vision_id -%} + {{- 'Picture ' ~ image_count.value ~ ': ' -}} + {%- endif -%} + {{- image_start_token ~ image_pad_token ~ image_end_token -}} + {%- elif item is mapping and ('video' in item or 'video_url' in item or item.type in ['video', 'video_url']) -%} + {%- if is_system_content -%} + {{- raise_exception('System message cannot contain videos.') -}} + {%- endif -%} + {%- if do_vision_count -%} + {%- set video_count.value = video_count.value + 1 -%} + {%- endif -%} + {%- if add_vision_id -%} + {{- 'Video ' ~ video_count.value ~ ': ' -}} + {%- endif -%} + {{- video_start_token ~ video_pad_token ~ video_end_token -}} + {%- elif item is string -%} + {{- item -}} + {%- else -%} + {{- item | string -}} + {%- endif -%} + {%- endfor -%} + {%- elif content is none -%} + {{- '' -}} + {%- else -%} + {{- content | string -}} + {%- endif -%} +{%- endmacro -%} + +{%- macro render_tools_prompt() -%} + {{- '# Tools\n\nYou may call one or more functions to assist with the user query.' -}} + {{- '\n\nYou are provided with function signatures within XML tags:' -}} + {{- '\n\n' -}} + {%- set tool_ns = namespace(first=true) -%} + {%- for tool in tools -%} + {%- set t = tool['function'] if tool is mapping and 'function' in tool else tool -%} + {%- if t.defer_loading is not defined or not t.defer_loading -%} + {%- if not tool_ns.first -%}{{- '\n' -}}{%- endif -%} + {%- set tool_ns.first = false -%} + {{- tool_to_json(t) -}} + {%- endif -%} + {%- endfor -%} + {{- '\n\n\n' -}} + {{- 'For function call returns, you should first print ' ~ toolcalls_begin_token -}} + {{- '\nFor each function call, you should return object like:\n' -}} + {{- toolcall_begin_token ~ '{function-name}' -}} + {{- argkey_begin_token ~ '{arg-key-1}' ~ argkey_end_token -}} + {{- argvalue_begin_token ~ '{arg-value-1}' ~ argvalue_end_token -}} + {{- argkey_begin_token ~ '{arg-key-2}' ~ argkey_end_token -}} + {{- argvalue_begin_token ~ '{arg-value-2}' ~ argvalue_end_token -}} + {{- '...' -}} + {{- toolcall_end_token -}} + {{- '\nAt the end of function call returns, you should print ' ~ toolcalls_end_token -}} +{%- endmacro -%} + +{#- adjacent tool_search_output parts share one block, so a tool search result has the same shape as the tools prompt -#} +{%- macro render_tool_response(message) -%} + {%- set content = message['content'] -%} + {{- toolresponse_begin_token -}} + {%- if content is string -%} + {{- content -}} + {%- elif content is iterable and content is not mapping -%} + {%- set tools_ns = namespace(open=false) -%} + {%- for item in content -%} + {%- set is_search_output = item is mapping and item.type is defined and item.type == 'tool_search_output' -%} + {%- if is_search_output and not tools_ns.open -%} + {{- '\n' -}} + {%- set tools_ns.open = true -%} + {%- elif not is_search_output and tools_ns.open -%} + {{- '' -}} + {%- set tools_ns.open = false -%} + {%- endif -%} + {%- if is_search_output -%} + {%- if item.tools is defined and item.tools is iterable -%} + {%- for tool in item.tools -%} + {%- set t = tool['function'] if tool is mapping and 'function' in tool else tool -%} + {{- tool_to_json(t) ~ '\n' -}} + {%- endfor -%} + {%- endif -%} + {%- elif item is mapping and item.output is defined -%} + {{- render_content(item.output) -}} + {%- else -%} + {{- render_content([item]) -}} + {%- endif -%} + {%- endfor -%} + {%- if tools_ns.open -%} + {{- '' -}} + {%- endif -%} + {%- else -%} + {{- render_content(content) -}} + {%- endif -%} + {{- toolresponse_end_token -}} +{%- endmacro -%} + +{#- consecutive tool messages are clustered into a single tool message whose content is the list of the original tool messages; the result is written to merged_ns.messages (a macro cannot return a list) -#} +{%- set merged_ns = namespace(messages=[]) %} +{%- macro merge_tool_responses(messages) -%} + {%- set merge_ns = namespace(result=[], cluster=[]) -%} + {%- for message in messages -%} + {%- if message['role'] == 'tool' -%} + {%- set merge_ns.cluster = merge_ns.cluster + [message] -%} + {%- else -%} + {%- if merge_ns.cluster -%} + {%- set merge_ns.result = merge_ns.result + [{'role': 'tool', 'content': merge_ns.cluster}] -%} + {%- set merge_ns.cluster = [] -%} + {%- endif -%} + {%- set merge_ns.result = merge_ns.result + [message] -%} + {%- endif -%} + {%- endfor -%} + {%- if merge_ns.cluster -%} + {%- set merge_ns.result = merge_ns.result + [{'role': 'tool', 'content': merge_ns.cluster}] -%} + {%- endif -%} + {%- set merged_ns.messages = merge_ns.result -%} +{%- endmacro -%} + +{#- render one tool cluster, reordered to follow the tool_call order of the preceding assistant turn; any missing / ambiguous id falls back to the original tool message order for the whole cluster -#} +{%- macro render_tool_responses(tool_messages, tool_call_ids) -%} + {%- set order_ns = namespace(ordered=[], matched=true) -%} + {%- if tool_call_ids and tool_call_ids | length == tool_messages | length -%} + {%- for tool_call_id in tool_call_ids -%} + {%- set hit_ns = namespace(count=0, message=none) -%} + {%- for tool_message in tool_messages -%} + {%- set message_id = tool_message['tool_call_id'] if tool_message['tool_call_id'] is defined else tool_message['id'] -%} + {%- if message_id is defined and message_id == tool_call_id -%} + {%- set hit_ns.count = hit_ns.count + 1 -%} + {%- set hit_ns.message = tool_message -%} + {%- endif -%} + {%- endfor -%} + {%- if hit_ns.count == 1 -%} + {%- set order_ns.ordered = order_ns.ordered + [hit_ns.message] -%} + {%- else -%} + {%- set order_ns.matched = false -%} + {%- endif -%} + {%- endfor -%} + {%- else -%} + {%- set order_ns.matched = false -%} + {%- endif -%} + {%- for tool_message in (order_ns.ordered if order_ns.matched else tool_messages) -%} + {{- render_tool_response(tool_message) -}} + {%- endfor -%} +{%- endmacro -%} + +{%- set _ = merge_tool_responses(messages) %} +{%- set normed_messages = merged_ns.messages %} + +{%- set ns = namespace(last_user_index=-1, has_leading_system=false) %} +{%- if normed_messages and normed_messages[0].role == 'system' %} + {%- set ns.has_leading_system = true %} +{%- endif %} +{%- for message in normed_messages %} + {%- if message['role'] == 'user' %} + {%- set ns.last_user_index = loop.index0 %} + {%- endif %} +{%- endfor %} + +{#- no leading system: synthesize one for tools and/or reasoning_mode -#} +{%- if not ns.has_leading_system %} + {{- hy_start_token ~ 'system' ~ hy_middle_token -}} + {%- if tools %} + {{- render_tools_prompt() -}} + {%- endif %} + {{- reasoning_mode_token ~ 'reasoning_effort:' ~ reasoning_effort -}} + {{- hy_end_token -}} +{%- endif %} + +{%- set last_ns = namespace(last_is_assistant=false) %} +{%- set prev_ns = namespace(tool_call_ids=[]) %} +{%- for message in normed_messages %} + {%- if message['role'] == 'system' %} + {{- hy_start_token ~ 'system' ~ hy_middle_token -}} + {#- tools / reasoning_mode only attach to the leading system (messages[0]) -#} + {%- if loop.first %} + {%- if tools %} + {{- render_tools_prompt() -}} + {%- endif %} + {%- set content = render_content(message['content']) -%} + {%- if tools and content -%} + {{- '\n\n' -}} + {%- endif -%} + {{- content -}} + {{- reasoning_mode_token ~ 'reasoning_effort:' ~ reasoning_effort -}} + {%- else %} + {{- render_content(message['content']) -}} + {%- endif %} + {{- hy_end_token -}} + {%- elif message['role'] == 'user' %} + {{- hy_start_token ~ 'user' ~ hy_middle_token -}} + {{- render_content(message['content']) -}} + {{- hy_end_token -}} + {%- elif message['role'] == 'assistant' %} + {%- set content_body = render_content(message['content']) -%} + {#- 'reasoning' takes precedence, fall back to 'reasoning_content' when it is empty -#} + {%- set reasoning_text = '' %} + {%- if message['reasoning'] is defined and message['reasoning'] is string and message['reasoning'] %} + {%- set reasoning_text = message['reasoning'] %} + {%- elif message['reasoning_content'] is defined and message['reasoning_content'] is string and message['reasoning_content'] %} + {%- set reasoning_text = message['reasoning_content'] %} + {%- endif %} + {#- no_think overrides preserved_thinking: history is always emitted with empty think tags -#} + {%- if reasoning_effort != 'no_think' and (preserved_thinking or loop.index0 > ns.last_user_index) and reasoning_text %} + {%- set content = think_begin_token ~ reasoning_text ~ think_end_token ~ content_body %} + {%- else %} + {%- set content = think_begin_token ~ think_end_token ~ content_body %} + {%- endif %} + {{- hy_start_token ~ 'assistant' ~ hy_middle_token -}} + {%- if message['tool_calls'] is defined and message['tool_calls'] %} + {#- remember the tool_call ids of this turn to order the following tool responses; an incomplete id set means "do not reorder" -#} + {%- set ids_ns = namespace(ids=[], complete=true) %} + {%- for tool in message['tool_calls'] %} + {%- set tool_call_id = tool['id'] if tool['id'] is defined and tool['id'] else tool['tool_call_id'] %} + {%- if tool_call_id is defined and tool_call_id %} + {%- set ids_ns.ids = ids_ns.ids + [tool_call_id] %} + {%- else %} + {%- set ids_ns.complete = false %} + {%- endif %} + {%- endfor %} + {%- set prev_ns.tool_call_ids = ids_ns.ids if ids_ns.complete else [] %} + {{- content -}} + {{- toolcalls_begin_token -}} + {%- for tool in message['tool_calls'] -%} + {%- set func = tool['function'] if tool is mapping and 'function' in tool else tool -%} + {%- set arguments = func['arguments'] -%} + {{- toolcall_begin_token ~ func['name'] -}} + {%- for key, value in arguments.items() -%} + {{- argkey_begin_token ~ key ~ argkey_end_token -}} + {%- if value is not string -%} + {%- set value = value | tojson(ensure_ascii=False) -%} + {%- endif -%} + {{- argvalue_begin_token ~ value ~ argvalue_end_token -}} + {%- endfor -%} + {{- toolcall_end_token -}} + {%- endfor -%} + {{- toolcalls_end_token -}} + {%- else %} + {%- if loop.last and raw_last_assistant %} + {{- content_body -}} + {%- else %} + {{- content -}} + {%- endif %} + {%- endif %} + {#- continuation / prefill: last assistant is still open, do not close with hy_end -#} + {%- if not (loop.last and raw_last_assistant) and not (loop.last and fallback_strategy is defined and fallback_strategy == 'reasoning_toolcall_retry') %} + {{- hy_end_token -}} + {%- endif %} + {%- elif message['role'] == 'tool' %} + {{- hy_start_token ~ 'tool' ~ hy_middle_token -}} + {{- render_tool_responses(message['content'], prev_ns.tool_call_ids) -}} + {{- hy_end_token -}} + {%- else %} + {{- hy_start_token ~ message['role'] ~ hy_middle_token -}} + {{- render_content(message['content']) -}} + {{- hy_end_token -}} + {%- endif %} + {%- if message['role'] != 'assistant' or not (message['tool_calls'] is defined and message['tool_calls']) %} + {%- set prev_ns.tool_call_ids = [] %} + {%- endif %} + {%- if loop.last and message['role'] == 'assistant' %} + {%- set last_ns.last_is_assistant = true %} + {%- endif %} +{%- endfor %} +{%- if add_generation_prompt %} + {%- if not last_ns.last_is_assistant %} + {%- if reasoning_effort == 'no_think' %} + {{- hy_start_token ~ 'assistant' ~ hy_middle_token ~ think_begin_token ~ think_end_token -}} + {%- else %} + {{- hy_start_token ~ 'assistant' ~ hy_middle_token ~ think_begin_token -}} + {%- endif %} + {%- endif %} +{%- endif %} diff --git a/config.json b/config.json new file mode 100644 index 0000000000000000000000000000000000000000..53b66f45537b2220dc882deed899b26a1b5356ab --- /dev/null +++ b/config.json @@ -0,0 +1,494 @@ +{ + "architectures": [ + "HYV4FlashForConditionalGeneration" + ], + "dtype": "bfloat16", + "image_end_token_id": 120119, + "image_newline_token_id": 120121, + "image_start_token_id": 120118, + "image_token_id": 120120, + "model_type": "hy_v4_flash", + "processor_class": "HYV4FlashProcessor", + "spatial_merge_size": 2, + "text_config": { + "architectures": [ + "HYV4FlashForCausalLM" + ], + "attention_bias": false, + "attention_dropout": 0.0, + "bitwise_backward_align": true, + "bos_token_id": 120000, + "compressed_indexer_candidate_topk": 2048, + "compressed_indexer_compress_ratio": 4, + "compressed_indexer_group_topk": 512, + "compressed_indexer_head_dim": 128, + "compressed_indexer_key_pooling": "gated", + "compressed_indexer_num_attention_heads": 16, + "compressed_indexer_reuse_fine_indexer": false, + "compressed_indexer_skip_fine_select": true, + "compressed_indexer_stage2_query_block_size": 8192, + "compressed_indexer_teacher_pooling_type": "max", + "compressed_indexer_triton_pooling_min_groups": 1, + "compressed_indexer_use_triton_pooling": true, + "dsa_skip_attn_post_norm": false, + "enable_ihc": true, + "enable_lm_head_fp32": true, + "engram_embed_random_seed": 1234, + "engram_fp8_embedding": true, + "engram_isolate_fffd_namespace": false, + "engram_layer_head_vocab_sizes": { + "12": [ + [ + 10312277, + 10312283, + 10312301, + 10312343, + 10312373, + 10312397, + 10312409, + 10312453 + ], + [ + 10312457, + 10312483, + 10312507, + 10312513, + 10312531, + 10312543, + 10312567, + 10312571 + ] + ], + "2": [ + [ + 10312007, + 10312021, + 10312039, + 10312069, + 10312091, + 10312121, + 10312123, + 10312147 + ], + [ + 10312177, + 10312187, + 10312207, + 10312213, + 10312229, + 10312241, + 10312249, + 10312259 + ] + ], + "22": [ + [ + 10312613, + 10312619, + 10312633, + 10312657, + 10312681, + 10312699, + 10312723, + 10312747 + ], + [ + 10312759, + 10312789, + 10312811, + 10312823, + 10312867, + 10312873, + 10312889, + 10312903 + ] + ], + "32": [ + [ + 10312909, + 10312919, + 10312927, + 10312957, + 10312979, + 10313011, + 10313021, + 10313057 + ], + [ + 10313063, + 10313117, + 10313123, + 10313129, + 10313161, + 10313179, + 10313189, + 10313201 + ] + ] + }, + "engram_layer_ids": [ + 2, + 12, + 22, + 32 + ], + "engram_max_ngram_size": 3, + "engram_n_embed_per_ngram": 768, + "engram_n_head_per_ngram": 8, + "engram_pad_id": 120002, + "engram_seed": 0, + "engram_tokenizer_name_or_path": null, + "engram_vocab_map_path": null, + "engram_vocab_size": [ + 10312000, + 10312000 + ], + "eos_token_id": 120025, + "fast_op_fp32_linear_gemm": true, + "first_k_dense_replace": 1, + "full_nope": false, + "gate_groups": 1, + "gate_lora_op": "none", + "gate_lora_rank": null, + "gated_mla": true, + "gating_type": "elementwise", + "hc_eps": 1e-06, + "hc_magnitude": 2.0, + "hc_mult": 4, + "head_dim": 64, + "hidden_act": "silu", + "hidden_size": 3072, + "ihc_kernel_backend": "naive", + "index_head_dim": 128, + "index_n_heads": 16, + "index_topk": 2048, + "indexer_types": [ + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full", + "full" + ], + "initializer_range": 0.006, + "intermediate_size": 13824, + "kv_lora_rank": 512, + "layer_types": [ + "deepseek_sparse_attention", + "deepseek_sparse_attention", + "sliding_attention", + "sliding_attention", + "sliding_attention", + "deepseek_sparse_attention", + "sliding_attention", + "sliding_attention", + "sliding_attention", + "deepseek_sparse_attention", + "sliding_attention", + "sliding_attention", + "sliding_attention", + "deepseek_sparse_attention", + "sliding_attention", + "sliding_attention", + "sliding_attention", + "deepseek_sparse_attention", + "sliding_attention", + "sliding_attention", + "sliding_attention", + "deepseek_sparse_attention", + "sliding_attention", + "sliding_attention", + "sliding_attention", + "deepseek_sparse_attention", + "sliding_attention", + "sliding_attention", + "sliding_attention", + "deepseek_sparse_attention", + "sliding_attention", + "sliding_attention", + "sliding_attention", + "deepseek_sparse_attention", + "sliding_attention", + "sliding_attention", + "sliding_attention", + "deepseek_sparse_attention", + "sliding_attention", + "sliding_attention", + "sliding_attention", + "deepseek_sparse_attention" + ], + "learnable_sink": true, + "learnable_sink_init": 0.0, + "max_position_embeddings": 1048576, + "mlp_layer_types": [ + "dense", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse", + "sparse" + ], + "model_type": "hy_v4_flash", + "moe_aux_loss_coeff": 0.0, + "moe_intermediate_size": 1536, + "moe_router_enable_expert_bias": true, + "moe_router_use_sigmoid": true, + "n_group": 1, + "n_routed_experts": 512, + "n_shared_experts": 1, + "norm_topk_prob": true, + "num_attention_heads": 32, + "num_experts_per_tok": 16, + "num_hidden_layers": 42, + "num_key_value_heads": 32, + "num_nextn_predict_layers": 0, + "o_groups": 1, + "o_lora_op": "none", + "o_lora_rank": null, + "output_router_logits": false, + "pad_token_id": 120002, + "q_lora_rank": 1024, + "qk_head_dim": 256, + "qk_nope_head_dim": 192, + "qk_norm": true, + "qk_rope_head_dim": 64, + "return_bos_score": true, + "return_max_logit": true, + "rms_norm_eps": 1e-05, + "rope_parameters": { + "sliding_attention": { + "mrope_interleaved": true, + "mrope_section": [ + 12, + 10, + 10 + ], + "rope_theta": 10000, + "rope_type": "default" + }, + "deepseek_sparse_attention": { + "mrope_interleaved": true, + "mrope_section": [ + 12, + 10, + 10 + ], + "rope_theta": 10000000, + "rope_type": "default" + } + }, + "routed_expert_hidden_size": 1536, + "routed_scaling_factor": 3.998, + "sandwich_norm": true, + "sliding_window": 512, + "swa_num_attention_heads": 64, + "swa_pattern": [ + 0, + 0, + 1, + 1, + 1, + 0, + 1, + 1, + 1, + 0, + 1, + 1, + 1, + 0, + 1, + 1, + 1, + 0, + 1, + 1, + 1, + 0, + 1, + 1, + 1, + 0, + 1, + 1, + 1, + 0, + 1, + 1, + 1, + 0, + 1, + 1, + 1, + 0, + 1, + 1, + 1, + 0 + ], + "swa_qk_head_dim": 256, + "swa_qk_nope_head_dim": 192, + "swa_qk_rope_head_dim": 64, + "swa_v_head_dim": 256, + "swiglu_limit": 10.0, + "tie_word_embeddings": false, + "topk_group": 1, + "use_cache": true, + "use_compressed_indexer": true, + "use_dsa": true, + "use_grouped_mm": false, + "use_mla": true, + "v_head_dim": 256, + "vocab_size": 120832, + "window_size": [ + 511, + 0 + ] + }, + "transformers_version": "5.23.0", + "video_end_token_id": 120123, + "video_start_token_id": 120122, + "video_token_id": 120124, + "vision_config": { + "add_patchemb_bias": true, + "anyres_vit_max_image_size": 2048, + "attention_backend": "hf_align_sdpa", + "attention_dropout": 0.0, + "bias_activation_fusion": true, + "cat_extra_token": 0, + "hidden_act": "gelu", + "hidden_dropout": 0.0, + "hidden_size": 1152, + "img_max_token_num": 4096, + "intermediate_size": 4304, + "interpolate_mode": "bilinear", + "learnable_mlp_pooling_size": 0, + "max_image_size": 2048, + "max_vit_seq_len": 16384, + "min_image_size": 512, + "model_type": "hy_v4_flash", + "num_attention_heads": 16, + "num_channels": 3, + "num_hidden_layers": 27, + "num_key_value_heads": 16, + "out_hidden_size": 3072, + "packed_seq_alignment": 128, + "patch_size": 16, + "perceive_post_norm": false, + "perceive_pre_norm": true, + "projector_type": "rmsnorm_mlp_adapter", + "remove_prenorm": true, + "resize_resolution": 2048, + "rms_norm_eps": 1e-05, + "rope_2d_percent": 1.0, + "rope_2d_theta": 10000.0, + "spatial_merge_size": 2, + "spatial_patch_size": 1, + "temporal_patch_size": 1, + "text_hidden_size": 3072, + "video_max_image_size": 768, + "video_min_image_size": 256, + "video_spatial_compression": false, + "video_temporal_compression": false, + "vision_full_attention": false + }, + "vision_start_token_id": 120118, + "speculative_config": { + "method": "dfly", + "model": "dfly", + "num_speculative_tokens": 5, + "draft_sample_method": "probabilistic" + }, + "dfly_mtp_fusion": "slot0", + "dfly_target_index_topk": 2048, + "dfly_context_local_window": null, + "dfly_enable_hidden_correction": true, + "dfly_num_hidden_layers": 5, + "dfly_mask_token_id": 120009, + "dfly_target_index_layer": [ + 25, + 29, + 33, + 37, + 41 + ], + "dfly_target_kv_layers": [ + 25, + 29, + 33, + 37, + 41 + ] +} diff --git a/finetune/README.md b/finetune/README.md new file mode 100644 index 0000000000000000000000000000000000000000..02ec5d4468346d99c1a6a43afd37b7d540bb5067 --- /dev/null +++ b/finetune/README.md @@ -0,0 +1,592 @@ +

+ 中文 | English +

+ +# Model Fine-tuning + +Hy4 flash provides processes related to model fine-tuning. This section details how to process training data for model fine-tuning purposes. + +## Training Data Format and Processing + +**Hy4 flash supports two thinking modes, controlled by the `reasoning_effort` parameter (only `no_think` and `high` are accepted; any other value raises an error). The available options are**: + +| `reasoning_effort` | Mode | Description | +|---|---|---| +| `no_think` | Fast thinking | No reasoning process; the model directly outputs the answer. | +| `high` | Deep thinking | Thorough reasoning with a detailed thinking process. | + +The training data should be formatted as a list of messages shown below. By default, the system prompt for both training and inference is empty; customize it as needed. + +For `high` mode, the assistant's reasoning process can be provided in the `reasoning_content` field (the `chat_template` also accepts a `reasoning` field, which takes precedence); in `no_think` mode the template always emits empty `` tags, so no reasoning field is needed. + +```jsonl +# Fast thinking pattern (no_think) — no reasoning field +{"reasoning_effort": "no_think", "messages": [{"content": "You are a helpful assistant.\nThe current time is 2026-01-01 13:26:12 Thursday", "role": "system"}, {"content": "1+1=?", "role": "user"}, {"role": "assistant", "content": "1+1=2"}]} + +# Deep thinking pattern (high) — with reasoning content +{"reasoning_effort": "high", "messages": [{"content": "You are a helpful assistant.\nThe current time is 2026-01-01 13:26:12 Thursday", "role": "system"}, {"content": "1+1=?", "role": "user"}, {"role": "assistant", "content": "1+1=2", "reasoning_content": "The user is asking for the result of 1 + 1. In basic decimal arithmetic, 1 + 1 equals 2."}]} +``` + +> **`reasoning_effort` default values:** +> - **During training**: If `reasoning_effort` is not specified in the training data, the training script defaults to `no_think` (fast thinking). This is because samples without a reasoning field are treated as non-thinking-mode data by the training script. +> - **During inference**: If `reasoning_effort` is not passed when calling `apply_chat_template`, the `chat_template` defaults to `high` (deep thinking). + +Example of using `apply_chat_template` to tokenize: + +```python +from transformers import AutoTokenizer +tokenizer = AutoTokenizer.from_pretrained("./models", use_fast=False, trust_remote_code=True) + +messages = [ + {"content": "You are a helpful assistant.", "role": "system"}, + {"content": "1+1=?", "role": "user"}, + {"role": "assistant", "content": "1+1=2"} +] +ids = tokenizer.apply_chat_template(messages, tokenize=True, reasoning_effort="no_think") +``` + +### Multimodal Data Format (Image / Video) + +Hy4 flash supports mixed training with images, videos, and pure text. The data format is JSONL, with one sample per line. + +#### Image Data + +Use the `` tag in the user message to mark the image position, and provide the image path list in the `images` field: + +```json +{"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Please describe this image in detail."}, {"role": "assistant", "content": "The image shows a beautiful sunset over the ocean."}], "images": ["test_images/sunset.jpg"]} +``` + +Multiple images are supported — use multiple `` tags and provide the corresponding paths in the `images` list (in order): + +```json +{"messages": [{"role": "user", "content": "Compare the first image with the second one."}], "images": ["img1.jpg", "img2.jpg"]} +``` + +#### Video Data + +Use the `