Instructions to use startlux-models/gdn-340m-pas-fa-layer20-10b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use startlux-models/gdn-340m-pas-fa-layer20-10b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="startlux-models/gdn-340m-pas-fa-layer20-10b")

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("startlux-models/gdn-340m-pas-fa-layer20-10b", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use startlux-models/gdn-340m-pas-fa-layer20-10b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "startlux-models/gdn-340m-pas-fa-layer20-10b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "startlux-models/gdn-340m-pas-fa-layer20-10b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/startlux-models/gdn-340m-pas-fa-layer20-10b

SGLang

How to use startlux-models/gdn-340m-pas-fa-layer20-10b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "startlux-models/gdn-340m-pas-fa-layer20-10b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "startlux-models/gdn-340m-pas-fa-layer20-10b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "startlux-models/gdn-340m-pas-fa-layer20-10b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "startlux-models/gdn-340m-pas-fa-layer20-10b",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use startlux-models/gdn-340m-pas-fa-layer20-10b with Docker Model Runner:
```
docker model run hf.co/startlux-models/gdn-340m-pas-fa-layer20-10b
```

gdn-340m-pas-fa-layer20-10b

This is a 340M controlled-pretraining checkpoint released for the paper Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus.

It is one member of the Massive Activations HLA model release and the corresponding Hugging Face Collection. The official analysis and reproducibility code is available at StartluxLabs/Massive-Activations-HLA.

Checkpoint details

Field	Value
Model	`gdn-340m-pas-fa-layer20-10b`
Scale	340M
Training tokens	10B
Experiment	PAS
Full-attention layers	20 (one-based)
Output-gating variant	Baseline
Final training step	19073
Weight format	Safetensors

Compatibility and reproducibility scope

This checkpoint loads with the public, pinned environment documented in the GitHub repository:

conda create -n ma-hla python=3.12 -y
conda activate ma-hla
bash scripts/install_released_gdn_cu126.sh

Public FLA is pinned to v0.5.2, commit 9c8e42e762fce087c27b673af4922795d9edb85e. Exact A800/CUDA 12.6 package versions are recorded in requirements/released-gdn-cu126.txt.

Loading

REPO_ID = "startlux-models/gdn-340m-pas-fa-layer20-10b"

import fla.models.gated_deltanet  # registers the custom config/model
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(REPO_ID)
model = AutoModelForCausalLM.from_pretrained(
    REPO_ID,
    torch_dtype="auto",
)

For analysis, prefer the GitHub registry and scripts because they validate the FLA version and recover full-attention layer metadata consistently.

Reproduction

Use the official GitHub repository for the tested analysis workflow, PAS/ISP metrics, lifecycle atlases, installation instructions, and model registry:

Intended use and limitations

This checkpoint is a research artifact for studying massive activations, hybrid linear-attention architectures, PAS/ISP morphology, attention placement, output gating, and scale. It is not instruction-tuned, safety-tuned, or validated for production deployment. It has not been comprehensively evaluated for downstream accuracy, factuality, bias, robustness, privacy, or safety.

The checkpoint uses the custom Transformers architecture GatedDeltaNetForCausalLM (model_type="gated_deltanet").

Training data

The model was trained from scratch on open data. A dataset-level composition and sampling breakdown is not included in this release; users should not infer language or domain coverage beyond the published project materials.

Citation

Please cite the accompanying paper. The arXiv link and final BibTeX entry will be added after the preprint metadata is public.

License

The released model artifacts are available under the Apache License 2.0. See LICENSE. Third-party software and datasets retain their own licenses and terms.

Downloads last month: 176

Safetensors

Model size

0.4B params

Tensor type

F32

Collection including startlux-models/gdn-340m-pas-fa-layer20-10b

Massive Activations in Hybrid Linear Attention Models

Collection

Official PAS, ISP, gating-ablation, and scale-study checkpoints for Massive Activations in Hybrid Linear Attention Models. • 10 items • Updated 2 days ago