Monostich-2

Instruction-tuned chat model (~150M) — Monostich-2 family

Second-generation Monostich-class model — hybrid GDN-2 + GQA, chat-tuned

What this is

Monostich-2 is the supervised fine-tuned (SFT) chat checkpoint for the Monostich-2 family.

Base (pretrain): kerzgrr/Monostich-2-base
Successor to kerzgrr/Monostich (~100M LLaMA-style)
Architecture: hybrid Gated DeltaNet-2 + gated GQA

Training

Pretrain → SFT

Stage	Details
Base	FineWeb-Edu pretrain → `Monostich-2-base`
SFT mix	HuggingFaceTB/smoltalk + NousResearch/Hermes-3-Dataset
Epochs	1 full epoch
Wall time	21.70 hours
Final step	optimizer step 10619
Weights	EMA (Hub `model.safetensors` is EMA @ bfloat16)
Seq length	8192 (packed SFT)
Peak LR	1 × 10⁻⁴ AdamW, cosine → 10% min
Final val loss (EMA)	1.532 (ppl ≈ 4.63)

Chat template (ChatML)

<|begin_of_text|><|im_start|>system
{system}<|im_end|>
<|im_start|>user
{user}<|im_end|>
<|im_start|>assistant
{assistant}<|im_end|>

Generation prompt ends at <|im_start|>assistant\n.

Model Architecture

Same TinyGDN hybrid as the base (~149.1M params):


Layers	32 (GDN-2 ×3 + GQA every 4th)
Hidden	512
MLP	SwiGLU 1472
Attention	4 Q / 1 KV, head dim 128, partial RoPE
Linear	Gated DeltaNet-2, 4 heads × 128
Vocab	49,152 BPE

Install & run

pip install torch safetensors tokenizers huggingface_hub
hf download kerzgrr/Monostich-2 inference.py --local-dir .
python inference.py --prompt "What is the capital of France?"

inference.py auto-downloads weights/tokenizer/tiny_gdn/ and auto-installs pinned flash-linear-attention (Windows applies Hub patches). Git required on PATH.

Interactive chat:

python inference.py

Flag	Default	Description
`--prompt`	—	One-shot user message
`--system`	—	Optional system prompt
`--temperature`	`0.7`	Sampling temperature
`--top-p`	`0.9`	Nucleus sampling
`--top-k`	`50`	Top-k
`--max-new-tokens`	`256`	Max generation length
`--device`	`cuda` if available	`cuda` / `cpu`

Limitations

Scale: at ~150M parameters this is a research / edge model, not a frontier system

Model family

Model	Stage	Hub
Monostich	SFT (~100M LLaMA)	`kerzgrr/Monostich`
Monostich-2-base	Pretrain (~150M hybrid)	`kerzgrr/Monostich-2-base`
Monostich-2	SFT (~150M hybrid)	this repo

Citation

@misc{monostich22026,
  title={Monostich-2: A Hybrid GDN-2 + GQA Chat Model},
  author={kerzgrr},
  year={2026},
  url={https://huggingface.co/kerzgrr/Monostich-2}
}