README.md

---
license: apache-2.0
language:
  - en
tags:
  - text-generation
  - causal-lm
  - pytorch
  - pretrain
  - hybrid
  - gated-deltanet
  - gqa
  - tercet
pipeline_tag: text-generation
library_name: tiny_gdn
datasets:
  - HuggingFaceFW/fineweb-edu
model-index:
  - name: Tercet-base
    results: []
---

<div align="center">

# Tercet-base

### Pretrained base model for the Tercet family (~502M)

[![Model](https://img.shields.io/badge/Model-~502M_params-blue)](.)
[![Stage](https://img.shields.io/badge/Stage-Pretrain_(base)-orange.svg)](.)
[![License](https://img.shields.io/badge/License-Apache_2.0-green.svg)](LICENSE)
[![Architecture](https://img.shields.io/badge/Arch-GDN--2_+_GQA_hybrid-purple.svg)](.)
[![Chat](https://img.shields.io/badge/Chat-Tercet_(SFT)-green.svg)](https://huggingface.co/kerzgrr/Tercet)

*A larger TinyGDN hybrid language model: recurrent GDN-2 memory plus gated GQA*

</div>

---

## What this is

**Tercet-base** is the **pretrained (base) checkpoint** for **Tercet**, the ~502M successor to the Couplet family.

- Scales [`kerzgrr/Couplet-base`](https://huggingface.co/kerzgrr/Couplet-base) from ~268M to ~502M parameters
- Hybrid **Gated DeltaNet-2** recurrent layers + **gated GQA** full-attention layers
- This repo is **pretrain-only** raw text continuation
- **Instruction-tuned / chat (SFT):** [`kerzgrr/Tercet`](https://huggingface.co/kerzgrr/Tercet)

If you want chat behaviour, use [`kerzgrr/Tercet`](https://huggingface.co/kerzgrr/Tercet) — this base model is for continuation / research and will not follow instructions reliably.

---

## Model Architecture

**Pipeline:** `Text Prompt` → `BPE-49K Tokenizer` → `TinyGDN Hybrid Decoder (32L)` → `Next-token Prediction`

### Hybrid block schedule (×32)

Every 4th layer is full attention; the rest are Gated DeltaNet-2:

`GDN2, GDN2, GDN2, GQA, …` (3:1 recurrent-to-attention)

| Component | Details |
|-----------|---------|
| **Gated DeltaNet-2** | Linear-time recurrent memory (`flash-linear-attention`) |
| **Gated GQA** | QK-normalized, partial RoPE, sigmoid output gate (8 Q / 2 KV, head dim 128) |
| **MLP** | SwiGLU |
| **Norm** | Zero-centered RMSNorm |
| **Embeddings** | Tied input / output |

### Technical specifications

| | |
|--|--|
| **Architecture** | TinyGDN hybrid (GDN-2 + GQA) |
| **Parameters** | 501,635,264 deployable |
| **Hidden size** | 1,024 |
| **Intermediate (MLP)** | 2,624 |
| **Layers** | 32 |
| **Attention** | 8 Q heads / 2 KV heads (GQA) |
| **Linear (GDN-2)** | 8 heads × 128 dim |
| **Context (trained)** | 2,048 |
| **Max position embeddings** | 32,768 |
| **Vocabulary** | 49,152 (BPE) |
| **RoPE θ** | 1,000,000 (partial factor 0.5) |
| **Precision (Hub weights)** | bfloat16 EMA |
| **Weight file** | `model.safetensors` (~957 MiB) |

---

## Training (pretrain)

Stopped early when EMA validation loss flattened (~3.004), rather than completing a full packed 10B-token epoch.

| | |
|--|--|
| **Dataset** | [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) (~10.18B packed train tokens) |
| **Tokens (train, with EOS)** | 8,545,893,329 |
| **Sequence length** | 2,048 |
| **Objective** | Next-token prediction |
| **Optimizer** | AdamW — β₁=0.9, β₂=0.95 |
| **Peak LR** | 2 × 10⁻⁴ |
| **Warmup** | 1% of steps |
| **Grad clip** | 1.0 |
| **EMA** | power EMA (max decay 0.9999) — **this Hub file is the EMA weights** |
| **Checkpoint** | optimizer step 16,300 / 19,419 planned (early stop) |
| **Val loss (EMA)** | 3.0044 (ppl 20.17) |

---

## Install

### 1) System requirements

- Python **3.10+**
- **CUDA GPU strongly recommended**
- PyTorch with CUDA matching your driver

### 2) Create an environment

```bash
python -m venv .venv
# Windows
.venv\Scripts\activate
# Linux / macOS
source .venv/bin/activate
```

### 3) Install PyTorch

Pick the build for your platform from https://pytorch.org. Example:

```bash
pip install torch --index-url https://download.pytorch.org/whl/cu124
```

CPU-only:

```bash
pip install torch
```

### 4) Install Python deps

```bash
pip install safetensors tokenizers huggingface_hub
```

**Flash Linear Attention is installed automatically by `inference.py`** on first run (pinned commit + Windows import patches when needed). Git must be on `PATH`.

### 5) Download the inference script

```bash
curl -L -o inference.py https://huggingface.co/kerzgrr/Tercet-base/resolve/main/inference.py

# or Hugging Face CLI
hf download kerzgrr/Tercet-base inference.py --local-dir .
```

The script auto-downloads `model.safetensors`, `config.json`, `tokenizer.json`, and the `tiny_gdn/` package from this repo.

---

## Quick start

**Single prompt (streams tokens):**

```bash
python inference.py --prompt "The history of computing begins"
```

**Interactive REPL:**

```bash
python inference.py
```

**Common options:**

| Flag | Default | Description |
|------|---------|-------------|
| `--prompt` | *(none)* | One-shot continuation; omit for REPL |
| `--temperature` | `0.8` | Sampling temperature |
| `--top-p` | `0.95` | Nucleus sampling |
| `--top-k` | `50` | Top-k (0 disables) |
| `--max-new-tokens` | `256` | Generation length |
| `--repetition-penalty` | `1.08` | Repetition penalty |
| `--context-length` | `2048` | Tokens kept in the window |
| `--seed` | `42` | RNG seed |
| `--device` | `cuda` if available | `cuda` or `cpu` |
| `--no-stream` | off | Print the full completion at once |
| `--no-bos` | off | Do not prepend `<\|begin_of_text\|>` |
| `--local-dir` | *(none)* | Use a local snapshot directory |

---

## Pretrain vs chat

| | **Tercet-base** (this repo) | **Tercet** (SFT) |
|--|--|--|
| Stage | Pretrain | Supervised fine-tune |
| Hub | *this repo* | [`kerzgrr/Tercet`](https://huggingface.co/kerzgrr/Tercet) |
| Prompting | Raw text continuation | Chat / instruction template (ChatML) |
| Stop token | `<\|end_of_text\|>` | Chat end-of-turn tokens |
| Use case | Research, continuation, probing | Assistants, dialogue |

---

## Files

```
kerzgrr/Tercet-base/
  README.md
  inference.py
  requirements.txt
  model.safetensors
  config.json
  tokenizer.json
  tokenizer_config.json
  special_tokens_map.json
  special_token_ids.json
  merges.txt
  vocab.json
  tiny_gdn/
    __init__.py
    config.py
    model.py
```

---

## Limitations

- **Base model**: not instruction-tuned; may ramble or fail at Q&A format
- **Scale**: ~502M parameters — research / edge prototype, not a frontier model
- **Dependency**: requires `flash-linear-attention`; not GGUF / llama.cpp compatible today
- **Context**: trained at 2,048; longer windows are experimental
- **Early stop**: pretrain did not finish a full 10B-token epoch

---

## Model family

| Model | Parameters | Architecture | Stage | Hub |
|-------|------------|--------------|-------|-----|
| **Monostich** | ~100M | LLaMA-style | SFT | [`kerzgrr/Monostich`](https://huggingface.co/kerzgrr/Monostich) |
| **Monostich-2-base** | ~150M | TinyGDN hybrid | Pretrain | [`kerzgrr/Monostich-2-base`](https://huggingface.co/kerzgrr/Monostich-2-base) |
| **Monostich-2** | ~150M | TinyGDN hybrid | SFT | [`kerzgrr/Monostich-2`](https://huggingface.co/kerzgrr/Monostich-2) |
| **Couplet-base** | ~268M | TinyGDN hybrid | Pretrain | [`kerzgrr/Couplet-base`](https://huggingface.co/kerzgrr/Couplet-base) |
| **Couplet** | ~268M | TinyGDN hybrid | SFT | [`kerzgrr/Couplet`](https://huggingface.co/kerzgrr/Couplet) |
| **Tercet-base** | ~502M | TinyGDN hybrid | Pretrain | *this repo* |
| **Tercet** | ~502M | TinyGDN hybrid | SFT | [`kerzgrr/Tercet`](https://huggingface.co/kerzgrr/Tercet) |

---

## Citation

```bibtex
@misc{tercetbase2026,
  title={Tercet-base: A 502M Hybrid GDN-2 + GQA Language Model},
  author={kerzgrr},
  year={2026},
  url={https://huggingface.co/kerzgrr/Tercet-base}
}
```

---

## Acknowledgments

- [flash-linear-attention](https://github.com/fla-org/flash-linear-attention) (Gated DeltaNet-2)
- [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu)
- Couplet family: [`kerzgrr/Couplet`](https://huggingface.co/kerzgrr/Couplet)
- PyTorch SDPA / Hugging Face Hub + tokenizers

---

<div align="center">

*A tercet is a three-line stanza — larger than a couplet, still compact.*

</div>