Upload folder using huggingface_hub

README.md

@@ -0,0 +1,69 @@
+---
+language: [ro, en]
+license: cc-by-nc-4.0
+tags: [romanian, bilingual, nanochat, pretrained]
+---
+
+# stefaniancu/rost-d24-sft
+
+the instruction-tuned chat model, depth d24, checkpoint step 000387.
+
+Trained from scratch on a bilingual Romanian/English mixture — tokenizer
+included, so the Romanian tokenization is purpose-built rather than inherited
+from an English model.
+
+| | |
+|---|---|
+| architecture | nanochat GPT, d24, 4,096 context |
+| tokenizer | [stefaniancu/rost-tok-bilingual](https://huggingface.co/stefaniancu/rost-tok-bilingual), 32,768 vocab, bilingual |
+| pretraining mixture | 30% Romanian / 70% English, interleaved at document level |
+| Romanian source | FineWeb2-ro, educational score >= 3, diacritic-normalized |
+| English source | ClimbMix |
+| stage | sft |
+
+The tokenizer is included under `tokenizer/` so this repo is self-contained.
+
+## Running it
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model = AutoModelForCausalLM.from_pretrained("stefaniancu/rost-d24-sft", trust_remote_code=True)
+tokenizer = AutoTokenizer.from_pretrained("stefaniancu/rost-d24-sft")
+```
+
+`trust_remote_code` is required because RoST is a custom architecture, not a
+Llama variant: the modelling code ships in this repo (`modeling_rost.py`) and
+is a transcription of the training implementation, verified to reproduce its
+logits exactly.
+
+The original training checkpoint (`model_*.pt` + `meta_*.json`) is included too,
+for use with [the rost repository](https://github.com/StefanIancu/rost).
+
+**GGUF / llama.cpp is not available yet.** llama.cpp compiles architectures in
+rather than loading them dynamically, so Ollama and LM Studio support needs the
+architecture implemented and upstreamed there.
+
+## Licence
+
+**CC-BY-NC-4.0, non-commercial.** Both halves of the training data carry
+non-commercial terms: ClimbMix is CC-BY-NC-4.0 ("for research and development
+only"), and the OpenLLM-Ro SFT datasets are CC-BY-NC-4.0. The model inherits
+that. Note that the ClimbMix mirror used is tagged MIT, which does not override
+the upstream terms.
+
+## Known characteristics
+
+The Romanian pretraining data is roughly **one third health content** by
+character count (measured across three shards of the source), with history,
+geography, finance and education each near 10%. Entertainment, gaming and
+software are each under 1%. Expect fluency to be uneven across domains
+accordingly.
+
+## Reproduce
+
+```bash
+bash runs/speedrun_ro.sh
+```
+
+Source and full experiment log: https://github.com/StefanIancu/rost