README.md

---
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
  - fla
  - gated-deltanet
  - hybrid-linear-attention
  - massive-activations
---

# Massive Activations in Hybrid Linear Attention Models

Controlled-pretraining checkpoints accompanying **Massive Activations in Hybrid
Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike
Plateaus**.

> **Legacy aggregate repository.** This repository is retained permanently for
> compatibility with existing links, citations, caches, and workflows that use
> `repo_id="startlux-models/Massive-Activations-HLA"` together with a
> `subfolder`. For new downloads, code, and citations, use the
> [Hugging Face Collection](https://huggingface.co/collections/startlux-models/massive-activations-in-hybrid-linear-attention-models-6a7d272591824bcb9cd17d5e)
> and the independent model repositories listed below. New model releases will
> be published as independent repositories rather than added here.

**Resources:**
[Model Collection](https://huggingface.co/collections/startlux-models/massive-activations-in-hybrid-linear-attention-models-6a7d272591824bcb9cd17d5e) ·
[Official analysis repository](https://github.com/StartluxLabs/Massive-Activations-HLA) ·
[Quickstart](https://github.com/StartluxLabs/Massive-Activations-HLA/blob/main/QUICKSTART.md) ·
[Installation](https://github.com/StartluxLabs/Massive-Activations-HLA/blob/main/INSTALL.md) ·
[Reproduction guide](https://github.com/StartluxLabs/Massive-Activations-HLA/blob/main/docs/reproduction.md)

The Collection is the recommended entry point. Each checkpoint now has an
independent public model repository, with its files at the repository root for
direct download. No existing files have been removed from this aggregate
repository, so legacy links and subfolder-based workflows continue to work.

## Model collection

| Model repository | Scale | Experiment | Full-attention layers |
| --- | ---: | --- | --- |
| [`gdn-340m-pas-fa-layer04-10b`](https://huggingface.co/startlux-models/gdn-340m-pas-fa-layer04-10b) | 340M | PAS placement | 4 |
| [`gdn-340m-pas-fa-layer12-10b`](https://huggingface.co/startlux-models/gdn-340m-pas-fa-layer12-10b) | 340M | PAS placement | 12 |
| [`gdn-340m-pas-fa-layer20-10b`](https://huggingface.co/startlux-models/gdn-340m-pas-fa-layer20-10b) | 340M | PAS placement | 20 |
| [`gdn-nooutgate-340m-pas-fa-layer12-10b`](https://huggingface.co/startlux-models/gdn-nooutgate-340m-pas-fa-layer12-10b) | 340M | PAS gating ablation | 12 |
| [`gdn-gatedfa-340m-pas-fa-layer12-10b`](https://huggingface.co/startlux-models/gdn-gatedfa-340m-pas-fa-layer12-10b) | 340M | PAS gated-FA ablation | 12 |
| [`gdn-1.3b-pas-fa-layer12-50b`](https://huggingface.co/startlux-models/gdn-1.3b-pas-fa-layer12-50b) | 1.3B | PAS scale study | 12 |
| [`gdn-340m-isp-hybrid-3to1-10b`](https://huggingface.co/startlux-models/gdn-340m-isp-hybrid-3to1-10b) | 340M | ISP 3:1 hybrid | 3, 6, 9, 12, 15, 18, 21, 24 |
| [`gdn-nooutgate-340m-isp-hybrid-3to1-10b`](https://huggingface.co/startlux-models/gdn-nooutgate-340m-isp-hybrid-3to1-10b) | 340M | ISP gating ablation | 3, 6, 9, 12, 15, 18, 21, 24 |
| [`gdn-gatedfa-340m-isp-hybrid-3to1-10b`](https://huggingface.co/startlux-models/gdn-gatedfa-340m-isp-hybrid-3to1-10b) | 340M | ISP gated-FA ablation | 3, 6, 9, 12, 15, 18, 21, 24 |
| [`gdn-1.3b-isp-hybrid-3to1-50b`](https://huggingface.co/startlux-models/gdn-1.3b-isp-hybrid-3to1-50b) | 1.3B | ISP scale study | 3, 6, 9, 12, 15, 18, 21, 24 |

Layer numbers in this table are one-based for readability.

## Compatibility and reproducibility scope

The baseline and `gdn-nooutgate-*` checkpoints load with the public, pinned
environment documented in the GitHub repository:

```bash
conda create -n ma-hla python=3.12 -y
conda activate ma-hla
bash scripts/install_released_gdn_cu126.sh
```

The two `gdn-gatedfa-*` checkpoints are **weights-only research artifacts**.
Their full-attention layers use a post-SDPA, head-specific sigmoid output gate
inspired by the G1 design in [Gated Attention for Large Language Models:
Non-linearity, Sparsity, and Attention-Sink-Free](https://arxiv.org/abs/2505.06708)
([official code](https://github.com/qiuzh20/gated_attention)). The exact
GatedDeltaNet integration is not distributed. These two checkpoints are not
part of the public from-scratch quickstart.

The repository loader raises an explicit compatibility message for gated-FA
models unless a user deliberately configures a compatible local implementation.

## Loading a baseline checkpoint

Register the public FLA architecture before using Transformers directly. The
independent repositories do not require a `subfolder` argument:

```python
import fla.models.gated_deltanet  # registers the custom config/model
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "startlux-models/gdn-340m-pas-fa-layer12-10b"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    torch_dtype="auto",
)
```

For analysis, prefer the GitHub registry and scripts because they validate the
FLA version and recover full-attention layer metadata consistently.

## Reproducibility notes

- Public FLA: v0.5.2, commit `9c8e42e762fce087c27b673af4922795d9edb85e`.
- Exact A800/CUDA 12.6 package versions are recorded in
  `requirements/released-gdn-cu126.txt` in the code repository.
- The Summer prompt and inline five-domain examples reproduce the public smoke
  pipeline. Exact input-level regeneration of every paper panel requires the
  sampled JSONL inputs used for that panel.
- Model weights are Apache-2.0; repository analysis code is MIT.

## Limitations

These are research checkpoints, not instruction-tuned or safety-tuned models.
They have not been validated for production use. The gated-FA variants require
an undistributed compatibility implementation as described above.