license: apache-2.0
pipeline_tag: text-generation
tags:
- base-model
language:
- en
Prochlo-190M-Base
The smallest and first model from CyanoAI — a 190.5M-parameter dense English base LM. Named after Prochlorococcus, the tiniest and most oxygen-producing photosynthetic organism on Earth: small, dense, and outproducing everything its size.
Model family
| Repo | Stage | Status |
|---|---|---|
CyanoAI/Prochlo-190M-Base |
Pre-trained base | this repo |
CyanoAI/Prochlo-190M-SFT |
Supervised fine-tune | available |
CyanoAI/Prochlo-190M |
Preference-aligned (DPO + RLAIF) | in training |
All variants share the same architecture, tokenizer, and vocabulary, so they are drop-in interchangeable.
Model at a glance
| Parameters | 190.5M dense (untied embeddings) |
| Architecture | LLaMA-style decoder: RMSNorm, RoPE (θ=1e5), SwiGLU, no biases, per-head QK RMSNorm |
| Shape | 12 layers × dim 768, 12 heads, hidden 3072 |
| Vocab | 50,261 (GPT-2 BPE + 4 control tokens) |
| Context | 1024 |
| Precision | FP16 |
| Format | Safetensors, loads with Qwen3ForCausalLM (architecture-compatible; trained from scratch by CyanoAI, not derived from Qwen) |
Pretraining
| Tokens | ~7.7B |
| Steps | 300,000 |
| Optimizer | Muon (2D matrices) + AdamW |
| Schedule | lr 1e-3, warmup 64, linear decay |
| Final train loss | 1.7947 (ppl ≈ 6.0) |
| Final eval loss | 1.6205 (ppl ≈ 5.06, still descending) |
Training corpus: a curated ~826GB English mixture (web / books / code / reference), stream-packed with the tokenizer jointly shipped in this repo.
Intended use
This is a base model — it speaks fluent Wikipedia-style prose but has not been aligned. For chat or assistant behaviour, start from the SFT or aligned variants.
Load
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("CyanoAI/Prochlo-190M-Base")
model = AutoModelForCausalLM.from_pretrained("CyanoAI/Prochlo-190M-Base", dtype="float16")
Evaluation
Zero-shot on wikitext (JSON subset, ~500k tokens): ppl ≈ 109.7.
Not directly comparable to GPT-2-series numbers — different training distribution.
License
Apache 2.0