---
license: apache-2.0
datasets:
- HuggingFaceFW/fineweb-edu
language:
- en
base_model:
- Qwen/Qwen3-1.7B
tags:
- linear-attention
- hybrid
- rnn
- distillation
---

Links:

- GitHub repo: <https://github.com/thunlp/hybrid-linear-attention>
- Paper: <https://arxiv.org/abs/2601.22156>

This is the final HypeNet-2B checkpoint from the paper [Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts](https://arxiv.org/pdf/2601.22156), distilled from Qwen3-1.7B using the HALO pipeline proposed in our paper. For more information, please refer to our GitHub repo.