| --- |
| license: apache-2.0 |
| pipeline_tag: text-generation |
| tags: |
| - base-model |
| language: |
| - en |
| --- |
| |
| # Prochlo-190M-Base |
|
|
| The smallest and first model from CyanoAI — a 190.5M-parameter dense English base LM. |
| Named after *Prochlorococcus*, the tiniest and most oxygen-producing photosynthetic organism on Earth: small, dense, and outproducing everything its size. |
|
|
| ## Model family |
|
|
| | Repo | Stage | Status | |
| |---|---|---| |
| | [`CyanoAI/Prochlo-190M-Base`](https://huggingface.co/CyanoAI/Prochlo-190M-Base) | Pre-trained base | this repo | |
| | [`CyanoAI/Prochlo-190M-SFT`](https://huggingface.co/CyanoAI/Prochlo-190M-SFT) | Supervised fine-tune | available | |
| | `CyanoAI/Prochlo-190M` | Preference-aligned (DPO + RLAIF) | in training | |
|
|
| All variants share the same architecture, tokenizer, and vocabulary, so they are drop-in interchangeable. |
|
|
| ## Model at a glance |
|
|
| | | | |
| |---|---| |
| | Parameters | 190.5M dense (untied embeddings) | |
| | Architecture | LLaMA-style decoder: RMSNorm, RoPE (θ=1e5), SwiGLU, no biases, per-head QK RMSNorm | |
| | Shape | 12 layers × dim 768, 12 heads, hidden 3072 | |
| | Vocab | 50,261 (GPT-2 BPE + 4 control tokens) | |
| | Context | 1024 | |
| | Precision | FP16 | |
| | Format | Safetensors, loads with `Qwen3ForCausalLM` (architecture-compatible; trained from scratch by CyanoAI, not derived from Qwen) | |
|
|
| ## Pretraining |
|
|
| | | | |
| |---|---| |
| | Tokens | ~7.7B | |
| | Steps | 300,000 | |
| | Optimizer | Muon (2D matrices) + AdamW | |
| | Schedule | lr 1e-3, warmup 64, linear decay | |
| | Final train loss | 1.7947 (ppl ≈ 6.0) | |
| | Final eval loss | 1.6205 (ppl ≈ 5.06, still descending) | |
|
|
| Training corpus: a curated ~826GB English mixture (web / books / code / reference), stream-packed with the tokenizer jointly shipped in this repo. |
|
|
| ## Intended use |
|
|
| This is a **base model** — it speaks fluent Wikipedia-style prose but has not been aligned. |
| For chat or assistant behaviour, start from the [SFT](https://huggingface.co/CyanoAI/Prochlo-190M-SFT) or aligned variants. |
|
|
| ## Load |
|
|
| ```python |
| from transformers import AutoModelForCausalLM, AutoTokenizer |
| |
| tok = AutoTokenizer.from_pretrained("CyanoAI/Prochlo-190M-Base") |
| model = AutoModelForCausalLM.from_pretrained("CyanoAI/Prochlo-190M-Base", dtype="float16") |
| ``` |
|
|
| ## Evaluation |
|
|
| Zero-shot on `wikitext` (JSON subset, ~500k tokens): ppl ≈ 109.7. |
| Not directly comparable to GPT-2-series numbers — different training distribution. |
|
|
| ## License |
|
|
| Apache 2.0 |
