Tercet / README.md

kerzgrr

Add IFEval scores to model card

05db14a verified 1 day ago

preview code

Raw

History Blame Contribute Delete

6.98 kB

metadata

license: apache-2.0
language:
  - en
tags:
  - text-generation
  - causal-lm
  - pytorch
  - sft
  - instruction-tuned
  - chat
  - hybrid
  - gated-deltanet
  - gqa
  - tercet
pipeline_tag: text-generation
library_name: tiny_gdn
datasets:
  - HuggingFaceTB/smoltalk
  - NousResearch/Hermes-3-Dataset
  - HuggingFaceH4/no_robots
  - HuggingFaceH4/ultrachat_200k
  - allenai/tulu-3-sft-personas-instruction-following
base_model: kerzgrr/Tercet-base
model-index:
  - name: Tercet
    results:
      - task:
          type: text-generation
        dataset:
          name: IFEval
          type: google/IFEval
        metrics:
          - name: Strict prompt-level accuracy
            type: prompt_level_strict_acc
            value: 0.1922365988909427
          - name: Strict instruction-level accuracy
            type: inst_level_strict_acc
            value: 0.328537170263789
          - name: Loose prompt-level accuracy
            type: prompt_level_loose_acc
            value: 0.21256931608133087
          - name: Loose instruction-level accuracy
            type: inst_level_loose_acc
            value: 0.3501199040767386

Tercet

Instruction-tuned chat model (~502M) — Tercet family

A ~502M hybrid GDN-2 + GQA model, supervised fine-tuned for chat

What this is

Tercet is the supervised fine-tuned (SFT) chat checkpoint for the Tercet family.

Base (pretrain): kerzgrr/Tercet-base
Larger successor to kerzgrr/Couplet
Architecture: hybrid Gated DeltaNet-2 + gated GQA

Training

Pretrain → SFT

Stage	Details
Base	8.55B-token FineWeb-Edu pretrain (early stop) → `Tercet-base`
SFT mix	HuggingFaceTB/smoltalk + NousResearch/Hermes-3-Dataset + HuggingFaceH4/no_robots + HuggingFaceH4/ultrachat_200k (`train_sft` / `test_sft`) + allenai/tulu-3-sft-personas-instruction-following
Epochs	1 full epoch, full deterministic shuffle
Assistant targets	1,190,392,732
Packed tokens	1,664,747,251
Conversations	2,240,172
Wall time	75.4 hours (sum of resumed sessions)
Final step	optimizer step 12,702
Weights	EMA (Hub `model.safetensors` is EMA @ bfloat16)
Seq length	8,192 (packed SFT)
Peak LR	1 × 10⁻⁴ AdamW, cosine → 10% min
Final val loss (EMA)	1.3194 (ppl 3.74)

Chat template (ChatML)

<|begin_of_text|><|im_start|>system
{system}<|im_end|>
<|im_start|>user
{user}<|im_end|>
<|im_start|>assistant
{assistant}<|im_end|>

Generation prompt ends at <|im_start|>assistant\n.

Model Architecture

Same TinyGDN hybrid as the base (501,635,264 parameters):


Layers	32 (GDN-2 ×3 + GQA every 4th)
Hidden	1,024
MLP	SwiGLU 2,624
Attention	8 Q / 2 KV, head dim 128, partial RoPE
Linear	Gated DeltaNet-2, 8 heads × 128
Vocab	49,152 BPE

IFEval

Official 541-prompt Google IFEval scorer, zero-shot ChatML, greedy decoding (temperature=0, max_new_tokens=1280):

Metric	Score
Prompt-level strict	19.2% (104 / 541)
Instruction-level strict	32.9% (274 / 834)
Prompt-level loose	21.3% (115 / 541)
Instruction-level loose	35.0% (292 / 834)

Install & run

pip install torch safetensors tokenizers huggingface_hub
hf download kerzgrr/Tercet inference.py --local-dir .
python inference.py --prompt "What is the capital of France?"

inference.py auto-downloads weights/tokenizer/tiny_gdn/ and auto-installs pinned flash-linear-attention (Windows applies Hub patches). Git is required on PATH.

Interactive chat:

python inference.py

Flag	Default	Description
`--prompt`	—	One-shot user message
`--system`	—	Optional system prompt
`--temperature`	`0.7`	Sampling temperature
`--top-p`	`0.9`	Nucleus sampling
`--top-k`	`50`	Top-k
`--max-new-tokens`	`256`	Max generation length
`--device`	`cuda` if available	`cuda` / `cpu`

Limitations

Scale: at ~502M parameters this is a research / edge model, not a frontier system
Dependency: requires flash-linear-attention; not GGUF / llama.cpp compatible today

Model family

Model	Stage	Hub
Monostich	SFT (~100M LLaMA)	`kerzgrr/Monostich`
Monostich-2-base	Pretrain (~150M hybrid)	`kerzgrr/Monostich-2-base`
Monostich-2	SFT (~150M hybrid)	`kerzgrr/Monostich-2`
Couplet-base	Pretrain (~268M hybrid)	`kerzgrr/Couplet-base`
Couplet	SFT (~268M hybrid)	`kerzgrr/Couplet`
Tercet-base	Pretrain (~502M hybrid)	`kerzgrr/Tercet-base`
Tercet	SFT (~502M hybrid)	this repo

Citation

@misc{tercet2026,
  title={Tercet: A 502M Hybrid GDN-2 + GQA Chat Model},
  author={kerzgrr},
  year={2026},
  url={https://huggingface.co/kerzgrr/Tercet}
}

Acknowledgments

flash-linear-attention (Gated DeltaNet-2)
HuggingFaceTB/smoltalk
NousResearch/Hermes-3-Dataset
HuggingFaceH4/no_robots
HuggingFaceH4/ultrachat_200k
allenai/tulu-3-sft-personas-instruction-following
Base: kerzgrr/Tercet-base

A tercet is a three-line stanza — larger than a couplet, still compact.