--- license: apache-2.0 language: - en tags: - text-generation - causal-lm - pytorch - sft - instruction-tuned - chat - hybrid - gated-deltanet - gqa - tercet pipeline_tag: text-generation library_name: tiny_gdn datasets: - HuggingFaceTB/smoltalk - NousResearch/Hermes-3-Dataset - HuggingFaceH4/no_robots - HuggingFaceH4/ultrachat_200k - allenai/tulu-3-sft-personas-instruction-following base_model: kerzgrr/Tercet-base model-index: - name: Tercet results: - task: type: text-generation dataset: name: IFEval type: google/IFEval metrics: - name: Strict prompt-level accuracy type: prompt_level_strict_acc value: 0.1922365988909427 - name: Strict instruction-level accuracy type: inst_level_strict_acc value: 0.328537170263789 - name: Loose prompt-level accuracy type: prompt_level_loose_acc value: 0.21256931608133087 - name: Loose instruction-level accuracy type: inst_level_loose_acc value: 0.3501199040767386 ---
# Tercet ### Instruction-tuned chat model (~502M) — Tercet family [![Model](https://img.shields.io/badge/Model-~502M_params-blue)](.) [![Stage](https://img.shields.io/badge/Stage-SFT_(chat)-green.svg)](.) [![License](https://img.shields.io/badge/License-Apache_2.0-green.svg)](LICENSE) [![Base](https://img.shields.io/badge/Base-Tercet--base-orange.svg)](https://huggingface.co/kerzgrr/Tercet-base) *A ~502M hybrid GDN-2 + GQA model, supervised fine-tuned for chat*
--- ## What this is **Tercet** is the **supervised fine-tuned (SFT) chat checkpoint** for the Tercet family. - Base (pretrain): [`kerzgrr/Tercet-base`](https://huggingface.co/kerzgrr/Tercet-base) - Larger successor to [`kerzgrr/Couplet`](https://huggingface.co/kerzgrr/Couplet) - Architecture: hybrid **Gated DeltaNet-2** + **gated GQA** --- ## Training ### Pretrain → SFT | Stage | Details | |-------|---------| | **Base** | 8.55B-token FineWeb-Edu pretrain (early stop) → [`Tercet-base`](https://huggingface.co/kerzgrr/Tercet-base) | | **SFT mix** | [HuggingFaceTB/smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk) + [NousResearch/Hermes-3-Dataset](https://huggingface.co/datasets/NousResearch/Hermes-3-Dataset) + [HuggingFaceH4/no_robots](https://huggingface.co/datasets/HuggingFaceH4/no_robots) + [HuggingFaceH4/ultrachat_200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k) (`train_sft` / `test_sft`) + [allenai/tulu-3-sft-personas-instruction-following](https://huggingface.co/datasets/allenai/tulu-3-sft-personas-instruction-following) | | **Epochs** | 1 full epoch, full deterministic shuffle | | **Assistant targets** | 1,190,392,732 | | **Packed tokens** | 1,664,747,251 | | **Conversations** | 2,240,172 | | **Wall time** | 75.4 hours (sum of resumed sessions) | | **Final step** | optimizer step 12,702 | | **Weights** | EMA (Hub `model.safetensors` is EMA @ bfloat16) | | **Seq length** | 8,192 (packed SFT) | | **Peak LR** | 1 × 10⁻⁴ AdamW, cosine → 10% min | | **Final val loss (EMA)** | 1.3194 (ppl 3.74) | ### Chat template (ChatML) ``` <|begin_of_text|><|im_start|>system {system}<|im_end|> <|im_start|>user {user}<|im_end|> <|im_start|>assistant {assistant}<|im_end|> ``` Generation prompt ends at `<|im_start|>assistant\n`. --- ## Model Architecture Same TinyGDN hybrid as the base (501,635,264 parameters): | | | |--|--| | **Layers** | 32 (GDN-2 ×3 + GQA every 4th) | | **Hidden** | 1,024 | | **MLP** | SwiGLU 2,624 | | **Attention** | 8 Q / 2 KV, head dim 128, partial RoPE | | **Linear** | Gated DeltaNet-2, 8 heads × 128 | | **Vocab** | 49,152 BPE | --- ## IFEval Official 541-prompt Google IFEval scorer, zero-shot ChatML, greedy decoding (`temperature=0`, `max_new_tokens=1280`): | Metric | Score | |--------|------:| | **Prompt-level strict** | **19.2%** (104 / 541) | | **Instruction-level strict** | **32.9%** (274 / 834) | | **Prompt-level loose** | **21.3%** (115 / 541) | | **Instruction-level loose** | **35.0%** (292 / 834) | --- ## Install & run ```bash pip install torch safetensors tokenizers huggingface_hub hf download kerzgrr/Tercet inference.py --local-dir . python inference.py --prompt "What is the capital of France?" ``` `inference.py` auto-downloads weights/tokenizer/`tiny_gdn/` and **auto-installs** pinned `flash-linear-attention` (Windows applies Hub patches). Git is required on `PATH`. **Interactive chat:** ```bash python inference.py ``` | Flag | Default | Description | |------|---------|-------------| | `--prompt` | — | One-shot user message | | `--system` | — | Optional system prompt | | `--temperature` | `0.7` | Sampling temperature | | `--top-p` | `0.9` | Nucleus sampling | | `--top-k` | `50` | Top-k | | `--max-new-tokens` | `256` | Max generation length | | `--device` | `cuda` if available | `cuda` / `cpu` | --- ## Limitations - **Scale**: at ~502M parameters this is a research / edge model, not a frontier system - **Dependency**: requires `flash-linear-attention`; not GGUF / llama.cpp compatible today --- ## Model family | Model | Stage | Hub | |-------|-------|-----| | Monostich | SFT (~100M LLaMA) | [`kerzgrr/Monostich`](https://huggingface.co/kerzgrr/Monostich) | | Monostich-2-base | Pretrain (~150M hybrid) | [`kerzgrr/Monostich-2-base`](https://huggingface.co/kerzgrr/Monostich-2-base) | | Monostich-2 | SFT (~150M hybrid) | [`kerzgrr/Monostich-2`](https://huggingface.co/kerzgrr/Monostich-2) | | Couplet-base | Pretrain (~268M hybrid) | [`kerzgrr/Couplet-base`](https://huggingface.co/kerzgrr/Couplet-base) | | Couplet | SFT (~268M hybrid) | [`kerzgrr/Couplet`](https://huggingface.co/kerzgrr/Couplet) | | Tercet-base | Pretrain (~502M hybrid) | [`kerzgrr/Tercet-base`](https://huggingface.co/kerzgrr/Tercet-base) | | **Tercet** | **SFT (~502M hybrid)** | **this repo** | --- ## Citation ```bibtex @misc{tercet2026, title={Tercet: A 502M Hybrid GDN-2 + GQA Chat Model}, author={kerzgrr}, year={2026}, url={https://huggingface.co/kerzgrr/Tercet} } ``` --- ## Acknowledgments - [flash-linear-attention](https://github.com/fla-org/flash-linear-attention) (Gated DeltaNet-2) - [HuggingFaceTB/smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk) - [NousResearch/Hermes-3-Dataset](https://huggingface.co/datasets/NousResearch/Hermes-3-Dataset) - [HuggingFaceH4/no_robots](https://huggingface.co/datasets/HuggingFaceH4/no_robots) - [HuggingFaceH4/ultrachat_200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k) - [allenai/tulu-3-sft-personas-instruction-following](https://huggingface.co/datasets/allenai/tulu-3-sft-personas-instruction-following) - Base: [`kerzgrr/Tercet-base`](https://huggingface.co/kerzgrr/Tercet-base) ---
*A tercet is a three-line stanza — larger than a couplet, still compact.*