Zenyx-100M-Base

A 151M-parameter dense decoder-only language model trained from scratch on web, code and math text (the "100M" in the name is the original plan; the trained model is 151M).

Status: pre-training in progress. All numbers below come from mid-training checkpoints (about 1B of a planned 10B tokens) and will be updated. This is a base model: not instruction-tuned, no safety tuning, and it makes up facts.

At a glance

Parameters	151.0M
Architecture	18-layer Llama-style decoder, grouped-query attention, SwiGLU
Context length	2,048 tokens
Vocabulary	49,105 tokens (byte-level BPE, 248 special tokens)
Training data	web, code, math, arithmetic reasoning (18 sources)
Training hardware	2 × Tesla T4, fp16
Training plan	10B tokens (about 1B done at the time of writing)
Decode speed	373 tokens/s single stream on one T4
License	Apache-2.0

Quick results

Measured at step 1,810 (0.95B tokens) unless noted. Base model, no fine-tuning.

Benchmark	Result
Zenyx-Fact1000 (4-way factual test, chance 25%)	50.6% (95% CI 47.5–53.7)
Fact1000 without the arithmetic items	53.6%
Held-out bits per byte (lower is better)	0.944
Chat-format arithmetic (120 problems)	95 / 120 correct
Decode throughput, one T4 (step 1,900)	373 tokens/s, batch 1

Details and caveats are in Evaluation.

Model details

tokens → embedding (tied with the output head)
       → 18 × [ RMSNorm → grouped-query attention → add ]
               [ RMSNorm → SwiGLU MLP             → add ]
       → RMSNorm → output head

Component	Setting
Width / depth	d_model 768, 18 blocks
Attention	12 query heads × 64, 4 key/value heads (grouped-query), causal, per-head QK-norm
Positions	RoPE, θ = 500,000
MLP	SwiGLU, hidden size 2,048
Normalisation	RMSNorm, pre-norm
Embeddings	tied input/output, 49,152 rows
Bias / dropout	none
Precision	fp16 sub-layers with an fp32 residual stream (the T4 has no bf16)
Init	normal(0, 0.02); residual output projections scaled by 1/√(2·layers)
KV cache	18,432 bytes per token in fp16 (37.7 MB for a full 2,048-token sequence)

Why only grouped-query attention? Sliding-window, gated linear attention (GLA), plain linear attention and Gated DeltaNet-2 were benchmarked as replacements on a T4 at 2k context. None beat flash-attention GQA on speed, so every layer is plain GQA. Quality of a hybrid at equal compute was not trained or compared.

Tokenizer

Byte-level BPE with a SuperBPE-style second stage (merges may cross spaces; digits stay single tokens). Compression on fresh text from a 16-source mix, in bytes per token (higher means fewer tokens for the same text):

Tokenizer	Bytes / token
Zenyx (this model)	4.56
Llama-3.2 (128k vocabulary)	4.17
Qwen3 (151k vocabulary)	4.03
Same BPE without the superword stage	3.94

File: tokenizer_candidates_v3/F-49152-f70/tokenizer.json.

Training

Optimiser	Muon on hidden matrices (lr 0.012), AdamW on embeddings and norms
Schedule	warmup → constant → linear decay over the last 2B of a 10B-token plan
Batch	524,288 tokens per step
Throughput	about 23.5k tokens/s on 2 × T4
Kernels	flash-attention for Turing, compiled blocks, fused cross-entropy

Data mixture (share of training tokens)

The decay phase is the end of base pre-training, not SFT.

Source	Stable phase	Decay phase
Code: StarCoderData (Python, Java, JavaScript, HTML, CSS, Markdown, TeX) + Nemotron code concepts	35%	35%
FineWeb-Edu	22%	16%
Nemotron-CC-Math (4+)	10.5%	13%
Arithmetic reasoning (SagheerLab/Arithmetic-Reasoning)	8%	4%
FineMath-4+	6%	6.5%
InfiWebMath-4+	5%	5.5%
Wikipedia (English)	5%	4%
Ultra-FineWeb (multi-style)	4%	3%
Nemotron fact-seeking	2%	3%
Nemotron SFT-math	1%	8.5%
SmolTalk (chat, with varied system prompts)	1.5%	1.5%

Evaluation

Knowledge and language modelling (step 1,810)

Metric	Value
Zenyx-Fact1000, all 1,000 items	50.6% (506/1000)
Zenyx-Fact1000, without 174 arithmetic items	53.6% (443/826)
Prior-only control (neutral context)	29.4%
Bits per byte, all held-out sources	0.944

Held-out source	Bits per byte
FineWeb-Edu	1.200
Wikipedia	1.244
Python	0.693
FineMath-4+	1.035
Nemotron-CC-Math	0.739
Nemotron fact-seeking	0.985

Zenyx-Fact1000 is a frozen, hash-checked 1,000-item four-way test written by the author, so treat it as a progress measure, not an independent benchmark.

Arithmetic in the model's own training format (step 1,810)

Prompt: What is a op b? → the model writes a <think> block and an answer. 20 problems per row, so expect about ±10 points of noise.

Problem type	Correct
2-digit addition	19 / 20
3-digit addition	19 / 20
2-digit subtraction	20 / 20
3-digit subtraction	19 / 20
2-digit × 1-digit	14 / 20
2-digit × 2-digit	4 / 20
Total	95 / 120

Inference speed (one T4, fp16, step 1,900)

CUDA-graph decode step with torch.compile. Batch columns are totals across all streams; short context.

Batch	Eager	CUDA graph	Graph + compile
1	41 tok/s	186 tok/s	373 tok/s
8	330 tok/s	1,399 tok/s	2,668 tok/s
32	1,286 tok/s	4,475 tok/s	7,040 tok/s
64	2,541 tok/s	7,038 tok/s	9,821 tok/s

A full 2,048-token generation with sampling and a repetition penalty runs at 248–259 tokens/s.

Long generation (step 1,900)

Decoding	Behaviour
Greedy, no repetition penalty	repeats itself within about 30 words
Greedy, penalty 1.2–1.3	loops delayed, or ends cleanly with the end-of-text token
Sampling (temperature 0.8, top-k 50), penalty 1.1–1.2	no repetition loops

A 2,048-token window holds about 1,000–1,900 words. Text is grammatical, but after roughly 100 words it drifts off topic and invents facts. Suggested settings for now: sampling with a repetition penalty of 1.1–1.2.

Intended use and limitations

• Use: research, and as a starting point for fine-tuning.
• Not an assistant: it ignores system prompts and may continue a prompt with an unrelated document.
• Facts: expect errors and invented details.
• Language: English only; no multilingual evaluation.
• Safety: not evaluated for bias or harmful content.
• Evaluation: single training run, mid-training checkpoint, small test sets.

Repository contents

Path	Contents
checkpoints/<run>/step_N/	model weights (model.pt), optimiser state, training state
tokenizer_candidates_v3/	tokenizers, including the one used
wheels/	flash-attention build used for training
runs/, data/	training manifests

The model code is not published yet.