---
license: apache-2.0
pipeline_tag: text-generation
tags:
- base-model
language:
- en
---

# Prochlo-190M-Base

The smallest and first model from CyanoAI — a 190.5M-parameter dense English base LM.
Named after *Prochlorococcus*, the tiniest and most oxygen-producing photosynthetic organism on Earth: small, dense, and outproducing everything its size.

## Model family

| Repo | Stage | Status |
|---|---|---|
| [`CyanoAI/Prochlo-190M-Base`](https://huggingface.co/CyanoAI/Prochlo-190M-Base) | Pre-trained base | this repo |
| [`CyanoAI/Prochlo-190M-SFT`](https://huggingface.co/CyanoAI/Prochlo-190M-SFT) | Supervised fine-tune | available |
| `CyanoAI/Prochlo-190M` | Preference-aligned (DPO + RLAIF) | in training |

All variants share the same architecture, tokenizer, and vocabulary, so they are drop-in interchangeable.

## Model at a glance

| | |
|---|---|
| Parameters | 190.5M dense (untied embeddings) |
| Architecture | LLaMA-style decoder: RMSNorm, RoPE (θ=1e5), SwiGLU, no biases, per-head QK RMSNorm |
| Shape | 12 layers × dim 768, 12 heads, hidden 3072 |
| Vocab | 50,261 (GPT-2 BPE + 4 control tokens) |
| Context | 1024 |
| Precision | FP16 |
| Format | Safetensors, loads with `Qwen3ForCausalLM` (architecture-compatible; trained from scratch by CyanoAI, not derived from Qwen) |

## Pretraining

| | |
|---|---|
| Tokens | ~7.7B |
| Steps | 300,000 |
| Optimizer | Muon (2D matrices) + AdamW |
| Schedule | lr 1e-3, warmup 64, linear decay |
| Final train loss | 1.7947 (ppl ≈ 6.0) |
| Final eval loss | 1.6205 (ppl ≈ 5.06, still descending) |

Training corpus: a curated ~826GB English mixture (web / books / code / reference), stream-packed with the tokenizer jointly shipped in this repo.

## Intended use

This is a **base model** — it speaks fluent Wikipedia-style prose but has not been aligned.
For chat or assistant behaviour, start from the [SFT](https://huggingface.co/CyanoAI/Prochlo-190M-SFT) or aligned variants.

## Load

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("CyanoAI/Prochlo-190M-Base")
model = AutoModelForCausalLM.from_pretrained("CyanoAI/Prochlo-190M-Base", dtype="float16")
```

## Evaluation

Zero-shot on `wikitext` (JSON subset, ~500k tokens): ppl ≈ 109.7.
Not directly comparable to GPT-2-series numbers — different training distribution.

## License

Apache 2.0