File size: 623 Bytes
81a6e00 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | ---
license: apache-2.0
datasets:
- HuggingFaceFW/fineweb-edu
language:
- en
base_model:
- Qwen/Qwen3-1.7B
tags:
- linear-attention
- hybrid
- rnn
- distillation
---
Links:
- GitHub repo: <https://github.com/thunlp/hybrid-linear-attention>
- Paper: <https://arxiv.org/abs/2601.22156>
This is the final HypeNet-2B checkpoint from the paper [Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts](https://arxiv.org/pdf/2601.22156), distilled from Qwen3-1.7B using the HALO pipeline proposed in our paper. For more information, please refer to our GitHub repo.
|