metadata
license: apache-2.0
datasets:
- HuggingFaceFW/fineweb-edu
language:
- en
base_model:
- Qwen/Qwen3-4B
tags:
- linear-attention
- hybrid
- rnn
- distillation
Links:
- GitHub repo: https://github.com/thunlp/hybrid-linear-attention
- Paper: https://arxiv.org/abs/2601.22156
This is the final HypeNet-5B checkpoint from the paper Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts, distilled from Qwen3-4B using the HALO pipeline proposed in our paper. For more information, please refer to our GitHub repo.