README.md

---
language:
- ro
- en
license: cc-by-nc-4.0
library_name: transformers
pipeline_tag: text-generation
tags:
- romanian
- bilingual
- nanochat
- text-generation
---

# rostlabs/rost-1b-base

rost is a bilingual Romanian/English language model trained from scratch, with a
purpose-built Romanian tokenizer rather than one inherited from an English model.
This repository holds the pretrained base model.

## Model overview

| | |
|---|---|
| parameters | 1.384B total (24 layers, 1,536 hidden, 12 heads) |
| context length | **4,096 tokens** |
| vocabulary | 32,768, bilingual -- [rostlabs/rost-tok-bilingual](https://huggingface.co/rostlabs/rost-tok-bilingual) |
| position encoding | RoPE, theta 100,000 |
| attention | sliding-window pattern SSSL; 18 of 24 layers see a quarter context |
| precision on disk | float32 safetensors. Load as bfloat16 -- that is what it trained in |
| stage | base, checkpoint step 011136 |
| languages | Romanian (primary), English |

Not a Llama or Mistral derivative: the architecture is its own, so the modelling
code ships in this repository and `trust_remote_code=True` is required.

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "rostlabs/rost-1b-base", trust_remote_code=True, dtype=torch.bfloat16).to("cuda").eval()
tokenizer = AutoTokenizer.from_pretrained("rostlabs/rost-1b-base")

messages = [{"role": "user", "content": "Care este capitala Romaniei?"}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True)
out = model.generate(inputs["input_ids"].to("cuda"), max_new_tokens=128)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

The weights are stored in bfloat16, the precision the model was trained in and the
precision the forward pass casts to regardless. They were published as float32 at first,
because release verification compares logits exactly and that is only meaningful in float32 —
but that made the files twice the size for no information: 47% of the parameters were already
bfloat16-exact, and under bfloat16 the two files produce bitwise identical logits. Loading in
float32 is still possible and shifts the top-8 probabilities by up to 1.7e-03, a mode the
model was never trained in.

Measured on a single RTX 5070 (12 GB, SDPA path): **~91 tokens/second**, ~3.3 GB
resident.

### Recommended sampling parameters

`generation_config.json` carries these, so `model.generate()` uses them without being
asked:

```json
{
  "do_sample": true,
  "temperature": 0.8,
  "top_k": 50,
  "repetition_penalty": 1.1,
  "eos_token_id": [32759, 32763]
}
```

The repetition penalty is not decoration. Measured over 198 generations of 400 tokens,
on prompts chosen to provoke the failure:

| decoding | replies that looped | worst repeated 6-gram |
|---|---:|---:|
| greedy | 50% | 44x |
| temperature 0.2, no penalty | 44% | 44x |
| temperature 0.6, no penalty | 22% | 10x |
| **temperature 0.6, penalty 1.1** | **0%** | **2x** |

A penalty of 1.1 also raises the share of replies that end by emitting their stop token,
rather than running out of budget, from 50% to 83%. Stronger settings suppress repetition
further and cost accuracy: at 1.3 the model stopped looping and began inventing
etymologies, so 1.1 is the mildest setting that works and that is why it is the default.

Before this revision the repository set no `eos_token_id` in any file, so `generate()`
had no stop condition and ran to `max_new_tokens` on every call.

### Running it with llama.cpp

No GGUF files are published for this base checkpoint; the quantisations at
[rostlabs/rost-1b-instruct-GGUF](https://huggingface.co/rostlabs/rost-1b-instruct-GGUF) are of
the instruct model. The same conversion produces them from these weights.

Either way the file needs a llama.cpp with this architecture compiled in, because llama.cpp
compiles architectures in rather than loading them dynamically. A fork with it applied is at
[rostlabs/llama.cpp](https://github.com/rostlabs/llama.cpp), whose `master` is upstream
master plus one commit.

### Not available

**Ollama and LM Studio.** Both bundle their own llama.cpp, so neither will read those files
until this architecture is merged upstream. That is prepared but not submitted.

**Tool and function calling.** The instruct checkpoint has had conversational
fine-tuning only. No agent fine-tuning has been applied, so the tool tokens are
not reachable under any prompt framing. Do not build an agent on this checkpoint
yet.

## Training

| | |
|---|---|
| tokens | 11.68B, single pass -- no data was repeated |
| optimizer steps | 11,136 at a 1,048,576-token batch |
| hardware | 8x H100 80GB, ~3.6 hours |
| schedule | warmup-stable-decay, decay over the final 30% |
| precision | bfloat16 with FP8 matmuls |

Trained in two phases, which is the substance of the recipe rather than a detail:

| | Romanian | English (ClimbMix) | DQA | code |
|---|---:|---:|---:|---:|
| phase 1, steps 0-7,795 | 30% | 60% | 5% | 5% |
| phase 2, steps 7,795-11,136 | **55%** | 35% | 10% | -- |

The second phase raises Romanian while the learning rate decays, so the model
finishes on a Romanian-heavy diet. A d6 rehearsal measured Romanian improving
roughly 8x faster than English across that phase.

### Data

| source | licence |
|---|---|
| Romanian: FineWeb2-ro, educational score >= 3, diacritic-normalised | ODC-BY |
| English: Nemotron ClimbMix | CC-BY-NC-4.0 |
| High-quality QA: Nemotron-CC-v2.1 HQ-DQA | gated |
| Code: Nemotron-CC-Code-v1 (phase 1 only) | gated |
| Chat fine-tuning: OpenLLM-Ro sets | CC-BY-NC-4.0 |

## Evaluation

**OpenLLM-Ro suite**, base checkpoint, 400 rows per task, **zero-shot**, scored by
likelihood over the options:

| task | accuracy | normalised | chance |
|---|---:|---:|---:|
| ro_hellaswag | 31.25 | **38.50** | 25.0 |
| ro_truthfulqa (MC1) | 21.25 | **34.00** | 14.3 |
| ro_arc_challenge | 24.50 | 30.25 | 28.6 |
| ro_mmlu | 28.50 | 29.75 | 25.0 |
| ro_winogrande | 52.25 | 52.25 | 50.0 |
| **mean (normalised)** | | **36.95** | 28.6 |

**Read these against the chance column, not on their own.** The model is clearly
above chance on sentence completion and truthfulness, and close to chance on
`ro_arc_challenge` and `ro_winogrande`. At 400 rows a task, differences under about
5 points are inside the noise.

**These are not comparable to the OpenLLM-Ro leaderboard.** Published figures there
average each task over several few-shot settings; these are zero-shot, which
understates them. For scale rather than ranking: RoLlama2-7b-Base reports a 42.05
four-task accuracy average against this model's 34.12 -- from a model 5x larger
trained on far more data.

**English suite**, base checkpoint, EleutherAI lm-evaluation-harness **0.4.12**,
zero-shot except where marked, full test sets, scored by likelihood over the
options -- the standard method for base models, so these are comparable to
published tables produced at the same settings:

| task | accuracy | normalised | chance |
|---|---:|---:|---:|
| sciq | **77.8** | 73.3 | 25.0 |
| hellaswag | 37.7 | **46.5** | 25.0 |
| piqa | 61.6 | **61.6** | 50.0 |
| boolq | **58.1** | -- | ~50 |
| arc_easy | **43.0** | 41.8 | 25.0 |
| lambada_openai | 34.5 | -- | -- |
| winogrande | 52.5 | -- | 50.0 |
| arc_challenge | 23.9 | 27.9 | 25.0 |
| openbookqa | 16.6 | 26.0 | 25.0 |
| mmlu (5-shot) | 26.7 | -- | 25.0 |

The same reading rule applies: judge against the chance column. Science QA,
sentence completion and physical common sense are clearly learned; winogrande,
arc_challenge, openbookqa and mmlu are at or near chance; and lambada is the weakest
skill, consistent with English being the minority share of an 11.68B-token
budget. For scale rather than ranking: on the completion tasks this sits
between Pythia-410M and Pythia-1B -- models that saw roughly 40x more English
text.

Also measured: CORE metric 0.2450 on the base model (0.1608 at step 2,000), and
0.3021 validation bits-per-byte for the instruct checkpoint against 0.5113 for the
base model it started from.

## Intended use

Research on Romanian language modelling, Romanian text generation and completion,
and as a base for further fine-tuning. It is small enough to run on a consumer GPU,
which is the point.

**Out of scope:** anything requiring factual reliability, agent or tool use,
long-context work beyond 4,096 tokens, commercial deployment (see the licence),
and any decision affecting a person's rights, health, safety or finances.

## Limitations

- **It confabulates confidently.** Asked about Bucharest it correctly names the
  capital and then places it in the wrong county. Specifics need checking.
- **It repeats.** Restating a sentence with the clauses swapped is a common
  failure, inherited from the base model and only partly removed by fine-tuning.
  Decode with the shipped `repetition_penalty` of 1.1; with penalties off and a low
  temperature it will repeat one sentence until it runs out of tokens.
- **Reasoning is near chance.** See `ro_arc_challenge` and `ro_winogrande` above.
- **4,096 tokens** of context, well short of contemporary models.
- **Domain skew.** The Romanian pretraining data is roughly one third health
  content by character count, with history, geography, finance and education each
  near 10%, and entertainment, gaming and software each under 1%. Fluency is uneven
  accordingly.
- **The Romanian corpus is internally duplicated**, which inflates absolute
  Romanian bits-per-byte by around 0.09. Comparisons on fixed sets are unaffected.
- **No safety tuning of any kind** has been applied.

## Licence

**CC-BY-NC-4.0, non-commercial.** Both halves of the training data carry
non-commercial terms -- ClimbMix is CC-BY-NC-4.0 ("for research and development
only") and the OpenLLM-Ro fine-tuning sets are CC-BY-NC-4.0 -- and the model
inherits them. The ClimbMix mirror used is tagged MIT, which does not override the
upstream terms. The Romanian data is ODC-BY and requires attribution to FineWeb2.

## Citation

```bibtex
@misc{rost2026,
  title  = {rost: a bilingual Romanian-English language model trained from scratch},
  author = {Iancu, Stefan},
  year   = {2026},
  url    = {https://huggingface.co/rostlabs/rost-1b-base}
}
```

The original training checkpoint (`model_*.pt`, `meta_*.json`) ships alongside the
safetensors, for use with the training code.