stefaniancu/rost-d6-base

the pretrained base model, depth d6, checkpoint step 000708.

Trained from scratch on a bilingual Romanian/English mixture. This is a research model at GPT-2 scale, not a product.

architecture	nanochat GPT, d6, 2048 context
tokenizer	stefaniancu/rost-tok-bilingual, 32,768 vocab, bilingual
pretraining mixture	30% Romanian / 70% English, interleaved at document level
Romanian source	FineWeb2-ro, educational score >= 3, diacritic-normalized
English source	ClimbMix
stage	base

The tokenizer is included under tokenizer/ so this repo is self-contained.

Licence

CC-BY-NC-4.0, non-commercial. Both halves of the training data carry non-commercial terms: ClimbMix is CC-BY-NC-4.0 ("for research and development only"), and the OpenLLM-Ro SFT datasets are CC-BY-NC-4.0. The model inherits that. Note that the ClimbMix mirror used is tagged MIT, which does not override the upstream terms.

Known characteristics

The Romanian pretraining data is roughly one third health content by character count (measured across three shards of the source), with history, geography, finance and education each near 10%. Entertainment, gaming and software are each under 1%. Expect fluency to be uneven across domains accordingly.

Reproduce

bash runs/speedrun_ro.sh

Source and full experiment log: https://github.com/StefanIancu/rost