ChonkyKedi-nano-base

ChonkyKedi-nano-base is a small dense Llama-style language model trained from scratch. It has 25.2M parameters (12.6M excluding the embedding table) and was pretrained on a 1.00B-token mix (FineWeb-Edu, Cosmopedia v2 (SmolLM-Corpus), FineMath 4+, SmolTalk, Wikipedia (English, 2023-11-01), github-code-clean).

This is the base model, before any instruction tuning. It continues text; it doesn't follow instructions. It's an educational and research artifact: it writes fluent, on-topic English but knows few facts and confidently makes things up. Don't use it for anything that matters.

The ChonkyKedi family

ChonkyKedi models share one architecture, tokenizer and data pipeline, and differ only in scale. Each is trained on about 80 tokens per non-embedding parameter. The table is generated from the current presets.

Scale	Hidden	Layers	Heads (query / KV)	FFN	Total	Non-embedding
pico	256	6	4 / 2	704	12.8M	4.4M
nano ← this model	384	8	6 / 2	1024	25.2M	12.6M
micro	512	10	8 / 2	1408	45.0M	28.2M
mini	640	12	10 / 2	1728	72.6M	51.6M
small	768	18	12 / 4	2048	138.4M	113.3M
base	1024	24	16 / 4	2752	299.4M	265.9M
medium	2048	24	16 / 4	5504	1,130.5M	1,063.4M
large	3072	28	24 / 8	8192	2,919.4M	2,818.8M

Architecture

A dense, Llama-style decoder-only transformer (llama_gqa).

Component	This model
Parameters	25.17M total, 12.59M excluding the embedding table
Layers / hidden size	8 / 384, pre-norm (RMSNorm, eps 1e-06)
Attention	Grouped-query attention: 6 query heads sharing 2 key/value heads, head dim 64, QK-norm
Positions	RoPE (θ = 10000); YaRN scaling available for longer contexts
Feed-forward	SwiGLU, 1024 hidden
Embeddings	Input embedding and output head tied
Output	Logit soft-cap 30 (cap · tanh(logits / cap))
Context	1,024 tokens

Training

Data	1.00B tokens, one pass (~79 tokens per non-embedding parameter), sources interleaved throughout (mix below)
Tokenizer	Byte-level BPE, 32,768 tokens, with chat, reasoning and tool-call special tokens
Steps	61,035 × 16,384 tokens (batch 4 × 1,024 × 4 accumulation)
Optimizer	AdamW (β = 0.9, 0.95, ε = 1e-8), weight decay 0.1 (not on norms), gradient clipping 1.0
Learning rate	2.5e-3 peak; WSD: 1,831 warm-up steps, 50,049 constant, then linear decay to 10% of peak over the last 9,155
Precision	bf16 autocast
Compute	~7.6e16 FLOPs (6 × non-embedding parameters × tokens)
Hardware	One AMD Radeon 890M (laptop iGPU, ROCm on Windows): 100.3 hours at ~3K tokens/s

Source	What it is	Tokens	Share
FineWeb-Edu sample/100BT	educational web pages	600M	60.0%
Cosmopedia v2 (SmolLM-Corpus)	synthetic textbooks and stories	150M	15.0%
FineMath 4+	maths web pages	100M	10.0%
SmolTalk	chat conversations	50M	5.0%
Wikipedia (English, 2023-11-01)	encyclopedia articles	50M	5.0%
github-code-clean	source code	50M	5.0%

79 documents were dropped by decontamination in total.

Every training document was checked against every item of the five evaluation sets below (10-gram overlap, plus exact matches of short answers) and dropped on a hit. The validation set comes from shards the training stream never reads.

Evaluation

Metric	Value
Validation loss (nats/token, held-out mix)	3.676
Perplexity	39.5

Validation loss over training: 10.48 (0.00B tok) → 4.16 (0.15B tok) → 4.11 (0.29B tok) → 4.07 (0.43B tok) → 4.05 (0.57B tok) → 4.03 (0.72B tok) → 4.03 (0.86B tok) → 3.68 (1.00B tok)

Zero-shot benchmarks (main.py --mode bench)

Task	acc_norm	acc	Items	Split	Chance
BLiMP	67.7% ± 0.2%	71.8%	67,000	all 67 phenomena	50%
SciQ	62.7% ± 1.1%	64.5%	2,000	validation + test	25%
ARC-Easy	36.8% ± 0.9%	40.4%	2,946	validation + test	25%
PIQA	56.9% ± 1.2%	57.0%	1,838	validation	50%
HellaSwag	26.7% ± 0.4%	26.7%	10,042	validation	25%

Each answer choice is scored by the log-probability of its text after the prompt (BLiMP: the whole sentence of each minimal pair). acc_norm divides by the answer's length in UTF-8 bytes; acc doesn't. Every item of each set is scored, not a sample; ± is one standard error.

Compared with other small models

	This model	ChonkyKedi-mini-base	Pythia-70M	GPT-2	Supra-50M
Parameters (total)	25.2M	72.6M	70.4M	124.4M	51.8M
Non-embedding parameters	12.6M	51.6M	18.9M	85.1M	35.4M
Training tokens	1.00B	4.13B	300B (the Pile)	undisclosed (WebText)	20B (FineWeb-Edu)
BLiMP (acc_norm)	67.7%	74.5%	–	–	–
SciQ (acc_norm)	62.7%	67.8%	55.2%	53.2%	77.2%
ARC-Easy (acc_norm)	36.8%	43.6%	35.0%	42.0%	52.2%
PIQA (acc_norm)	56.9%	60.6%	–	–	–
HellaSwag (acc_norm)	26.7%	29.8%	–	29.5%	31.8%
Validation loss (held-out mix)	3.676	3.129	–	–	–

ChonkyKedi columns are measured with this repo's evaluator on the checkpoints named; validation losses are only comparable between models that share a tokenizer and validation set. The other columns are published figures: Pythia-70M from EleutherAI's evaluations as quoted on the Wisp-15M card, GPT-2 and Supra-50M from the Supra-50M-Base card. Their prompts, splits and length normalisation differ from ours, so treat gaps of a few points as noise.

What it can and can't do

This is the smallest ChonkyKedi: a dense 25.2M-parameter model (12.6M excluding the embedding table) trained from scratch on a laptop's integrated GPU, reaching a validation loss of 3.676 after 1.00B tokens. It writes grammatical, on-topic English sentences and simple children's stories, but it has almost no world knowledge and it repeats itself heavily. It's a research and educational artifact, and a baseline for the larger ChonkyKedi models (mini, at 2.9× the parameters and 4.1× the data, is better on every benchmark below), not a source of information.

What it does

• ✅ Writes grammatical sentences that stay on the prompt's topic. Grammar is its strongest skill: 67.7% on BLiMP across all 67,000 minimal pairs.
• ✅ Tells simple children's stories in the style of its synthetic-textbook data: "Once upon a time, a little cat" → "named Lily was playing with her friends… One day, while playing outside, Lily noticed something strange." This is the most consistently good output it has.
• ✅ Gets the rough shape of a few very common topics: "The heart pumps" → "blood, providing oxygen and food for the body" (one of four tries); "Photosynthesis is" → "a process of converting the energy from the sun into energy"; "Albert Einstein was" → "a brilliant scientist who was known for his groundbreaking discoveries in the field of physics".
• ✅ Knows the chat format. About 5% of its pretraining data was SmolTalk conversations, so given <|user|>…<|assistant|> it replies in the assistant's voice, often ending with <|eos|>, as a base model rather than an instruction-tuned one.
• ✅ Zero-shot benchmarks (acc_norm, every item scored): SciQ 62.7%, ARC-Easy 36.8%, PIQA 56.9%, HellaSwag 26.7% (close to its 25% chance level). That's above Pythia-70M on SciQ and ARC-Easy and above GPT-2 on SciQ, from 1.00B training tokens, but below GPT-2 on ARC-Easy and HellaSwag; prompts and normalisation differ between published numbers and ours.

What it can't do

• ❌ Facts. It got none of the factual prompts right in four tries each (greedy plus three seeds): "The capital of France is" → "the capital of France", "the state of Paris"; "Water boils at" → "the bottom of the pond"; "The largest planet in the solar system is" → "the Sun"; "The chemical symbol for gold is" → "the symbol for gold"; "The French Revolution began in" → "1776", "1849"; "Albert Einstein was" → "born in 1907… a student at the University of Michigan". Treat every name, date and number as made up.
• ❌ Arithmetic. "2 + 2 =" → 2, 90; "7 + 6 =" → 12, 109, 6, 90. A train going 60 miles in 2 hours does "100 miles per hour" or "90 miles per hour".
• ❌ Code. It writes Python-shaped lines but nothing that works: def fibonacci(n): → if n < 0: return fibonacci(n), or the definition repeated.
• ❌ Instructions. No instruction tuning. Asked for a haiku about the sea, it writes a prose sentence ("The sea is a lovely sea, and the sea is a vast ocean."); asked for the largest planet, it says the Sun, the Kuiper Belt or the Solar System.
• ❌ Causal reasoning and history. "The main causes of the First World War were" → "the loss of supplies, the loss of land, and the loss of the land" or a 1949 battle.
• ❌ Coherence beyond a sentence or two. It restates the prompt ("The capital of France is the capital of France"), repeats words within a sentence ("vibrant, and vibrant"), and drifts off topic within a paragraph.
• ❌ Long context. It was trained on 1,024-token sequences; nothing longer was tested.

Decoding

Greedy decoding loops almost immediately ("The symbol for gold is the symbol for gold. The symbol for gold is the symbol for gold…"; "The water is in the water, and the water is in the water…"). Use sampling with temperature ≈ 0.7 and top-k ≈ 40, the defaults of ChonkyKediModel.generate, and expect to try several seeds.

Usage

PyTorch (needs the chonkykedi package from the training code):

import torch
from chonkykedi.models.model import ChonkyKediModel
from chonkykedi.tokenizer.tokenizer import ChonkyTokenizer

model = ChonkyKediModel.from_pretrained(".")  # this folder: config.json + model.safetensors
tok = ChonkyTokenizer.load("tokenizer.json")
ids = torch.tensor([tok.encode("Photosynthesis is", add_bos=True)])
out = model.generate(ids, max_new_tokens=60, temperature=0.7, top_k=40)
print(tok.decode(out[0].tolist(), skip_special_tokens=True))

Files

File	Contents
model.safetensors	PyTorch weights, chonkykedi parameter names; the output head is tied to tok_embeddings.weight and not stored
config.json	ModelConfig for chonkykedi
tokenizer.json	Byte-level BPE tokenizer (Hugging Face tokenizers format)
chonky_kedi.jpg	The ChonkyKedi logo

Acknowledgements

• Pretraining data: FineWeb-Edu (ODC-BY 1.0).
• Pretraining data: Cosmopedia v2 (ODC-BY 1.0).
• Pretraining data: FineMath 4+ (ODC-BY 1.0).
• Pretraining data: SmolTalk (Apache 2.0 for its new subsets, with the datasets it includes under their own licenses).
• Pretraining data: Wikipedia (CC BY-SA 3.0 and GFDL).
• Pretraining data: github-code-clean (Apache 2.0, with each file under its own open-source license).

Card generated 2026-11-02 by prepare_release.py from llama_gqa_nano_1b.pt.