README.md

---
license: apache-2.0
language:
  - en
tags:
  - text-generation
  - causal-lm
  - pytorch
  - sft
  - instruction-tuned
  - chat
  - hybrid
  - gated-deltanet
  - gqa
  - tercet
pipeline_tag: text-generation
library_name: tiny_gdn
datasets:
  - HuggingFaceTB/smoltalk
  - NousResearch/Hermes-3-Dataset
  - HuggingFaceH4/no_robots
  - HuggingFaceH4/ultrachat_200k
  - allenai/tulu-3-sft-personas-instruction-following
base_model: kerzgrr/Tercet-base
model-index:
  - name: Tercet
    results:
      - task:
          type: text-generation
        dataset:
          name: IFEval
          type: google/IFEval
        metrics:
          - name: Strict prompt-level accuracy
            type: prompt_level_strict_acc
            value: 0.1922365988909427
          - name: Strict instruction-level accuracy
            type: inst_level_strict_acc
            value: 0.328537170263789
          - name: Loose prompt-level accuracy
            type: prompt_level_loose_acc
            value: 0.21256931608133087
          - name: Loose instruction-level accuracy
            type: inst_level_loose_acc
            value: 0.3501199040767386
---

<div align="center">

# Tercet

### Instruction-tuned chat model (~502M) — Tercet family

[![Model](https://img.shields.io/badge/Model-~502M_params-blue)](.)
[![Stage](https://img.shields.io/badge/Stage-SFT_(chat)-green.svg)](.)
[![License](https://img.shields.io/badge/License-Apache_2.0-green.svg)](LICENSE)
[![Base](https://img.shields.io/badge/Base-Tercet--base-orange.svg)](https://huggingface.co/kerzgrr/Tercet-base)

*A ~502M hybrid GDN-2 + GQA model, supervised fine-tuned for chat*

</div>

---

## What this is

**Tercet** is the **supervised fine-tuned (SFT) chat checkpoint** for the Tercet family.

- Base (pretrain): [`kerzgrr/Tercet-base`](https://huggingface.co/kerzgrr/Tercet-base)
- Larger successor to [`kerzgrr/Couplet`](https://huggingface.co/kerzgrr/Couplet)
- Architecture: hybrid **Gated DeltaNet-2** + **gated GQA**

---

## Training

### Pretrain → SFT

| Stage | Details |
|-------|---------|
| **Base** | 8.55B-token FineWeb-Edu pretrain (early stop) → [`Tercet-base`](https://huggingface.co/kerzgrr/Tercet-base) |
| **SFT mix** | [HuggingFaceTB/smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk) + [NousResearch/Hermes-3-Dataset](https://huggingface.co/datasets/NousResearch/Hermes-3-Dataset) + [HuggingFaceH4/no_robots](https://huggingface.co/datasets/HuggingFaceH4/no_robots) + [HuggingFaceH4/ultrachat_200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k) (`train_sft` / `test_sft`) + [allenai/tulu-3-sft-personas-instruction-following](https://huggingface.co/datasets/allenai/tulu-3-sft-personas-instruction-following) |
| **Epochs** | 1 full epoch, full deterministic shuffle |
| **Assistant targets** | 1,190,392,732 |
| **Packed tokens** | 1,664,747,251 |
| **Conversations** | 2,240,172 |
| **Wall time** | 75.4 hours (sum of resumed sessions) |
| **Final step** | optimizer step 12,702 |
| **Weights** | EMA (Hub `model.safetensors` is EMA @ bfloat16) |
| **Seq length** | 8,192 (packed SFT) |
| **Peak LR** | 1 × 10⁻⁴ AdamW, cosine → 10% min |
| **Final val loss (EMA)** | 1.3194 (ppl 3.74) |

### Chat template (ChatML)

```
<|begin_of_text|><|im_start|>system
{system}<|im_end|>
<|im_start|>user
{user}<|im_end|>
<|im_start|>assistant
{assistant}<|im_end|>
```

Generation prompt ends at `<|im_start|>assistant\n`.

---

## Model Architecture

Same TinyGDN hybrid as the base (501,635,264 parameters):

| | |
|--|--|
| **Layers** | 32 (GDN-2 ×3 + GQA every 4th) |
| **Hidden** | 1,024 |
| **MLP** | SwiGLU 2,624 |
| **Attention** | 8 Q / 2 KV, head dim 128, partial RoPE |
| **Linear** | Gated DeltaNet-2, 8 heads × 128 |
| **Vocab** | 49,152 BPE |

---

## IFEval

Official 541-prompt Google IFEval scorer, zero-shot ChatML, greedy decoding (`temperature=0`, `max_new_tokens=1280`):

| Metric | Score |
|--------|------:|
| **Prompt-level strict** | **19.2%** (104 / 541) |
| **Instruction-level strict** | **32.9%** (274 / 834) |
| **Prompt-level loose** | **21.3%** (115 / 541) |
| **Instruction-level loose** | **35.0%** (292 / 834) |

---

## Install & run

```bash
pip install torch safetensors tokenizers huggingface_hub
hf download kerzgrr/Tercet inference.py --local-dir .
python inference.py --prompt "What is the capital of France?"
```

`inference.py` auto-downloads weights/tokenizer/`tiny_gdn/` and **auto-installs** pinned `flash-linear-attention` (Windows applies Hub patches). Git is required on `PATH`.

**Interactive chat:**

```bash
python inference.py
```

| Flag | Default | Description |
|------|---------|-------------|
| `--prompt` | — | One-shot user message |
| `--system` | — | Optional system prompt |
| `--temperature` | `0.7` | Sampling temperature |
| `--top-p` | `0.9` | Nucleus sampling |
| `--top-k` | `50` | Top-k |
| `--max-new-tokens` | `256` | Max generation length |
| `--device` | `cuda` if available | `cuda` / `cpu` |

---

## Limitations

- **Scale**: at ~502M parameters this is a research / edge model, not a frontier system
- **Dependency**: requires `flash-linear-attention`; not GGUF / llama.cpp compatible today

---

## Model family

| Model | Stage | Hub |
|-------|-------|-----|
| Monostich | SFT (~100M LLaMA) | [`kerzgrr/Monostich`](https://huggingface.co/kerzgrr/Monostich) |
| Monostich-2-base | Pretrain (~150M hybrid) | [`kerzgrr/Monostich-2-base`](https://huggingface.co/kerzgrr/Monostich-2-base) |
| Monostich-2 | SFT (~150M hybrid) | [`kerzgrr/Monostich-2`](https://huggingface.co/kerzgrr/Monostich-2) |
| Couplet-base | Pretrain (~268M hybrid) | [`kerzgrr/Couplet-base`](https://huggingface.co/kerzgrr/Couplet-base) |
| Couplet | SFT (~268M hybrid) | [`kerzgrr/Couplet`](https://huggingface.co/kerzgrr/Couplet) |
| Tercet-base | Pretrain (~502M hybrid) | [`kerzgrr/Tercet-base`](https://huggingface.co/kerzgrr/Tercet-base) |
| **Tercet** | **SFT (~502M hybrid)** | **this repo** |

---

## Citation

```bibtex
@misc{tercet2026,
  title={Tercet: A 502M Hybrid GDN-2 + GQA Chat Model},
  author={kerzgrr},
  year={2026},
  url={https://huggingface.co/kerzgrr/Tercet}
}
```

---

## Acknowledgments

- [flash-linear-attention](https://github.com/fla-org/flash-linear-attention) (Gated DeltaNet-2)
- [HuggingFaceTB/smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk)
- [NousResearch/Hermes-3-Dataset](https://huggingface.co/datasets/NousResearch/Hermes-3-Dataset)
- [HuggingFaceH4/no_robots](https://huggingface.co/datasets/HuggingFaceH4/no_robots)
- [HuggingFaceH4/ultrachat_200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k)
- [allenai/tulu-3-sft-personas-instruction-following](https://huggingface.co/datasets/allenai/tulu-3-sft-personas-instruction-following)
- Base: [`kerzgrr/Tercet-base`](https://huggingface.co/kerzgrr/Tercet-base)

---

<div align="center">

*A tercet is a three-line stanza — larger than a couplet, still compact.*

</div>