Monostich-2-base

Pretrained base model for the Monostich-2 family (~150M)

Second-generation Monostich-class model with a renewed hybrid architecture

What this is

Monostich-2-base is the pretrained (base) checkpoint for Monostich-2, my second Monostich-class language model.

Successor to kerzgrr/Monostich (Monostich-1, ~100M LLaMA-style)
Renewed architecture: hybrid Gated DeltaNet-2 recurrent layers + gated GQA full-attention layers (not a plain LLaMA stack)
This repo is pretrain-only raw text continuation
Instruction-tuned / chat (SFT): kerzgrr/Monostich-2

If you want chat behaviour, use kerzgrr/Monostich-2 — this base model is for continuation / research and will not follow instructions reliably.

Model Architecture

Pipeline: Text Prompt → BPE-49K Tokenizer → TinyGDN Hybrid Decoder (32L) → Next-token Prediction

Hybrid block schedule (×32)

Every 4th layer is full attention; the rest are Gated DeltaNet-2:

GDN2, GDN2, GDN2, GQA, … (3:1 recurrent-to-attention)

Component	Details
Gated DeltaNet-2	Linear-time recurrent memory (`flash-linear-attention`)
Gated GQA	QK-normalized, partial RoPE, sigmoid output gate (4 Q / 1 KV, head dim 128)
MLP	SwiGLU
Norm	Zero-centered RMSNorm
Embeddings	Tied input / output

Technical specifications


Architecture	TinyGDN hybrid (GDN-2 + GQA)
Parameters	~149.1M deployable
Hidden size	512
Intermediate (MLP)	1,472
Layers	32
Attention	4 Q heads / 1 KV head (GQA)
Linear (GDN-2)	4 heads × 128 dim
Context (trained)	2,048
Max position embeddings	32,768
Vocabulary	49,152 (BPE)
RoPE θ	1,000,000 (partial factor 0.5)
Precision (Hub weights)	bfloat16 EMA
Weight file	`model.safetensors` (~298 MiB)

Training (pretrain)


Dataset	FineWeb-Edu
Tokens (train, with EOS)	~1.46B
Sequence length	2,048
Objective	Next-token prediction
Optimizer	AdamW — β₁=0.9, β₂=0.95
Peak LR	4 × 10⁻⁴
Warmup	1% of steps
Grad clip	1.0
EMA	power EMA (max decay 0.9999) — this Hub file is the EMA weights
Checkpoint	optimizer step 1500
Val loss (EMA)	~3.70 (ppl ~40.4)

Install

1) System requirements

Python 3.10+
CUDA GPU strongly recommended (CPU works but is slow; GDN-2 kernels expect GPU for best path)
PyTorch with CUDA matching your driver

2) Create an environment

python -m venv .venv
# Windows
.venv\Scripts\activate
# Linux / macOS
source .venv/bin/activate

3) Install PyTorch

Pick the build for your platform from https://pytorch.org. Example (CUDA 12.4 wheels):

pip install torch --index-url https://download.pytorch.org/whl/cu124

CPU-only:

pip install torch

4) Install Python deps

pip install safetensors tokenizers huggingface_hub

Flash Linear Attention is installed automatically by inference.py on first run (pinned commit + Windows import patches when needed). Git must be on PATH.

Manual install is optional; see windows_fla_patches/ if you want to vendor FLA yourself.

5) Download the inference script

# wget / curl
curl -L -o inference.py https://huggingface.co/kerzgrr/Monostich-2-base/resolve/main/inference.py

# or Hugging Face CLI
hf download kerzgrr/Monostich-2-base inference.py --local-dir .

The script auto-downloads model.safetensors, config.json, tokenizer.json, and the tiny_gdn/ package from this repo (cached after the first run). It also auto-installs flash-linear-attention if missing.

Quick start

Single prompt (streams tokens):

python inference.py --prompt "The history of computing begins"

Interactive REPL:

python inference.py

Common options:

Flag	Default	Description
`--prompt`	(none)	One-shot continuation; omit for REPL
`--temperature`	`0.8`	Sampling temperature
`--top-p`	`0.95`	Nucleus sampling
`--top-k`	`50`	Top-k (0 disables)
`--max-new-tokens`	`256`	Generation length
`--repetition-penalty`	`1.08`	Repetition penalty
`--context-length`	`2048`	Tokens kept in the window
`--seed`	`42`	RNG seed
`--device`	`cuda` if available	`cuda` or `cpu`
`--no-stream`	off	Print the full completion at once
`--no-bos`	off	Do not prepend `<\|begin_of_text\|>`
`--local-dir`	(none)	Use a local snapshot directory

Example:

python inference.py \
  --prompt "In a distant galaxy," \
  --temperature 0.9 \
  --top-p 0.95 \
  --max-new-tokens 200 \
  --seed 1234

Pretrain vs chat

	Monostich-2-base (this repo)	Monostich-2 (SFT)
Stage	Pretrain	Supervised fine-tune
Hub	this repo	`kerzgrr/Monostich-2`
Prompting	Raw text continuation	Chat / instruction template (ChatML)
Stop token	`<\|end_of_text\|>`	Chat end-of-turn tokens
Use case	Research, continuation, probing	Assistants, dialogue

Files

kerzgrr/Monostich-2-base/
  README.md                 # This model card
  inference.py              # Terminal inference (streaming)
  requirements.txt          # Pip helper list
  model.safetensors         # EMA weights (bfloat16)
  config.json               # TinyGDN architecture config
  tokenizer.json            # BPE tokenizer
  tokenizer_config.json
  special_tokens_map.json
  special_token_ids.json
  merges.txt
  vocab.json
  tiny_gdn/                 # Minimal model package for inference
    __init__.py
    config.py
    model.py

Limitations

Base model: not instruction-tuned; may ramble, ignore “user” framing, or fail at Q&A format
Scale: ~150M parameters — research / edge prototype, not a frontier model
Dependency: requires flash-linear-attention (GDN-2); not GGUF / llama.cpp compatible today
Context: trained at 2,048; longer windows are experimental

Model family

Model	Parameters	Architecture	Stage	Hub
Monostich	~100M	LLaMA-style	SFT	`kerzgrr/Monostich`
Monostich-2-base	~150M	TinyGDN hybrid	Pretrain	this repo
Monostich-2	~150M	TinyGDN hybrid	SFT	`kerzgrr/Monostich-2`

Citation

@misc{monostich2base2026,
  title={Monostich-2-base: A Hybrid GDN-2 + GQA Language Model},
  author={kerzgrr},
  year={2026},
  url={https://huggingface.co/kerzgrr/Monostich-2-base}
}