README.md

---
license: apache-2.0
language:
  - en
tags:
  - innomium
  - continuum
  - causal-lm
  - linear-attention
  - long-context
  - reasoning
  - math
library_name: transformers
pipeline_tag: text-generation
thumbnail: continuum_banner.png
---

<div align="center">

# Continuum1-9B

<img src="continuum_banner.png" width="900" alt="Continuum1-9B by Innomium"/>

<br/><br/>

### Fully linear foundation model for extreme long-context reasoning.

**Innomium · ~8.6B parameters · 2M native context · BF16**

[![Organization](https://img.shields.io/badge/Organization-Innomium-1565FF?style=for-the-badge)](https://huggingface.co/innomium)
[![Parameters](https://img.shields.io/badge/Parameters-~8.6B-001530?style=for-the-badge)](#model-overview)
[![Context](https://img.shields.io/badge/Context-2M_tokens-1565FF?style=for-the-badge)](#model-overview)

</div>

---

## Model Overview

**Continuum1-9B** is a foundation model developed by **Innomium**, built for extreme long-context reasoning at production scale. It combines a hybrid **Gated Linear Attention (GLA)** architecture with **No Positional Embeddings (NOPE)** to deliver million-token context with linear compute cost.

| | |
|---|---|
| **Model** | Continuum1-9B |
| **Organization** | [Innomium](https://huggingface.co/innomium) |
| **Parameters** | ~8.6B |
| **Precision** | BF16 |
| **Context length** | 2,097,152 tokens |
| **Vocabulary** | 248,320 tokens |
| **Architecture** | Hybrid GLA + Gated DeltaNet |
| **Custom code** | Required (`trust_remote_code=True`) |

---

## Architecture

Continuum1-9B is a **32-layer** decoder that alternates three linear-attention layers with one full-attention layer per block.

**Gated DeltaNet** handles the majority of layers with efficient recurrent state updates. **Gated Linear Attention (GLA)** powers the full-attention layers for global anchoring. **NOPE** removes rotary positional embeddings so the model relies on internal state trajectories rather than explicit position signals — enabling stable extrapolation to sequences far beyond the training window.

| Spec | Value |
|------|-------|
| Hidden size | 4,096 |
| Intermediate size | 12,288 |
| Attention heads | 16 (GQA, 4 KV heads) |
| Head dimension | 256 |
| Weight format | Safetensors (4 shards) |

---

## Training

Continuum1-9B was trained in two stages:

1. **Structural distillation** — 10B tokens. Layer-wise transfer of pretrained knowledge into linear attention units via a hybrid MSE + cross-entropy objective.
2. **Long-context expansion** — 20B tokens at native 2M sequence length with full NOPE.

## Training Data

Continuum1-9B was trained on a multi-source corpus spanning math, STEM, reasoning, code, and curated web text. Additional training data sources are proprietary and not publicly disclosed.

---

## Benchmarks

| Benchmark | Score |
|-----------|-------|
| MMLU | **75.0%** |
| BBH | **48.7%** |
| ARC-C | **62.9%** |
| TruthfulQA | **48.5%** |
| WinoGrande | **75.8%** |
| GPQA | **33.9%** |
| ARC-E | **84.6%** |
| PIQA | **82.6%** |
| HellaSwag | **78.2%** |
| OpenBookQA | **47.2%** |
| MATH-500 | **37.0%** |
| MMLU-Pro | **36.5%** |

---

## Installation

Continuum1-9B requires custom modeling code and the Innomium kernel stack. Full setup (Python 3.12, CUDA 13.0, PyTorch 2.10):

### Quick install

```bash
pip install torch transformers safetensors
pip install --no-build-isolation \
  "flash-linear-attention @ git+https://github.com/Innomium/continuum-flash-linear-attention.git"
```

### Recommended environment (CUDA 13.0)

Using [uv](https://github.com/astral-sh/uv):

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
source $HOME/.local/bin/env

uv venv --python python3.12
source .venv/bin/activate

export UV_TORCH_BACKEND=cu130
uv pip install "torch==2.10.0+cu130"

# Innomium GLA / Gated DeltaNet kernels
uv pip install --no-build-isolation \
  "flash-linear-attention @ git+https://github.com/Innomium/continuum-flash-linear-attention.git"

# causal-conv1d (Gated DeltaNet)
PYTAG=$(python -c 'import sys; print(f"cp{sys.version_info.major}{sys.version_info.minor}")')
ARCH=$(uname -m); [ "$ARCH" = x86_64 ] && PLAT=linux_x86_64 || PLAT=linux_aarch64
uv pip install "https://github.com/Dao-AILab/causal-conv1d/releases/download/v1.6.2.post1/causal_conv1d-1.6.2.post1+cu13torch2.10cxx11abiTRUE-${PYTAG}-${PYTAG}-${PLAT}.whl"

# flash-attention (full-attention layers)
uv pip install https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.9.0/flash_attn-2.8.3+cu130torch2.10-cp312-cp312-linux_x86_64.whl
```

| Component | Source |
|-----------|--------|
| **Kernels** | [Innomium/continuum-flash-linear-attention](https://github.com/Innomium/continuum-flash-linear-attention) |
| **Eval tools** | [Innomium/continuum-eval](https://github.com/Innomium/continuum-eval) (optional) |
| **Organization** | [Innomium on Hugging Face](https://huggingface.co/innomium) · [GitHub](https://github.com/Innomium) |

---

## Usage

```bash
pip install torch transformers safetensors
pip install --no-build-isolation \
  "flash-linear-attention @ git+https://github.com/Innomium/continuum-flash-linear-attention.git"
```

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "innomium/Continuum1-9B"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
    trust_remote_code=True,
)

prompt = "Solve step by step: What is the integral of x^2 from 0 to 3?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

---

## Model Files

| File | Description |
|------|-------------|
| `continuum_banner.png` | Model card banner |
| `config.json` | Architecture configuration |
| `generation_config.json` | Generation defaults |
| `configuration_continuum.py` | Custom config (`trust_remote_code`) |
| `modeling_continuum.py` | Model implementation |
| `model-*.safetensors` | BF16 weights (sharded) |
| `model.safetensors.index.json` | Shard index |

---

## About Innomium

[Innomium](https://huggingface.co/innomium) builds production AI — from edge vision (Sentinel, Vantage, Ember) to foundation models like Continuum.

---

## License

Apache 2.0