OpenEuroLLM Prelude-base 9B (WIP Research Artifact)

This repository contains intermediate training checkpoints for OpenEuroLLM Prelude-base, an experimental 9B dense base model currently training on Leonardo (CINECA). See the OpenEuroLLM progress tracker for live status across all OpenEuroLLM training runs. It is being trained on 10T tokens (~80% English + code + math, ~20% multilingual across the 36 initial OpenEuroLLM target languages, represented as 41 language/script codes in the data mix, since some languages are split into macrolanguage + specific-variant codes, e.g. Latvian/Standard Latvian/Latgalian, Estonian/Standard Estonian, Albanian/Tosk Albanian, Norwegian Bokmål/Nynorsk). Full dataset composition, sources, and the per-language token breakdown are documented in the training-data-collection repo.

Checkpoints are published as per-iteration branches. Each branch (e.g. iter_0124800) is a complete, self-contained HuggingFace checkpoint: config.json, sharded model-*.safetensors (5GB shards), tokenizer files, and a validation.json with example generations (see Sanity-check generations below). main itself is intentionally empty and reserved for the final checkpoint.

Note on training phase: these are checkpoints from the "stable" (constant max-LR) phase of the WSD schedule and are not cooled down. Only a checkpoint at/after iteration ~953,600 would reflect the cooldown phase; everything currently published predates that point.

Model Details

Dense, untied input/output embeddings, closely follows the Qwen3 architecture.

Layers	36
Hidden size	4096
FFN hidden size	12288 (dense)
Attention heads	32
KV groups (GQA)	8
Head dim (kv_channels)	128
Position embedding	RoPE, base 100000
Normalization	RMSNorm, eps 1e-5, with QK-norm
Activation	SwiGLU
Sequence length	4096
Vocab size	262,145 (256k tokenizer, padded to 262,272 for TP efficiency)
Dtype	bf16

num tensors        : 399
dtypes             : ['BF16']
embed_tokens params: 1,074,266,112
lm_head params     : 1,074,266,112   (untied from embeddings)
transformer body   : 6,946,075,648
TOTAL params       : 9,094,607,872   (~9.095 B)
weight bytes (hdr) : 18,189,215,744  (~18.19 GB)

• Developed by: OpenEuroLLM
• Model type: dense, autoregressive decoder-only Transformer
• Languages: 36 initial OpenEuroLLM target languages (41 language/script codes in the data mix, see note above); per-language token breakdown
• License: Apache 2.0
• Training repository: oellm-autoexp, baby_9b_dense experiment
• Architecture config: qwen3_dense_9b.yaml

How to Get Started

Load a specific training checkpoint via the revision argument (branch name = Megatron iteration, e.g. iter_0124800):

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

repo = "openeurollm/prelude-base"
revision = "iter_0124800"

model = AutoModelForCausalLM.from_pretrained(
    repo, revision=revision, dtype=torch.bfloat16, device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(repo, revision=revision)

prompts = [
    "Hola, ¿qué tal?",
    "Bonjour, comment ça va ?",
    "Guten Tag, wie geht es dir?",
    "Ciao, come stai?",
    "Hei, mitä kuuluu?",
]
generate_kwargs = dict(max_new_tokens=50, do_sample=True, temperature=0.7, top_p=0.9, top_k=50, repetition_penalty=1.3, no_repeat_ngram_size=3)

for prompt in prompts:
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    outputs = model.generate(**inputs, **generate_kwargs)
    print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Prelude-base was trained without a <bos> token: documents are packed back to back, each ending in <eos>. The tokenizers on all branches therefore do not add <bos> (add_bos_token: false). Copies of a branch downloaded before 2 November 2026 still add it, so pass add_special_tokens=False when you tokenize with such a copy.

See the repository's branch list for all available checkpoints.

Sanity-check generations

Every checkpoint's branch includes a validation.json with greedy-decoded completions for a fixed battery of prompts, generated at conversion time as a quick "does this checkpoint produce coherent text" spot-check:

• 4 canonical incomplete-sentence prompts, covering general knowledge ("The capital of France is..."), common sense ("The opposite of hot is..."), science ("Water boils at a temperature of..."), and creative/narrative ("Once upon a time...")
• Each of the 4 is localized into all 36 target languages (144 generations total), using each language's actual native conventions where relevant (e.g. its own fairy-tale opening phrase for the narrative prompt) rather than literal word-for-word translation. English is simply one of the 36 rows, not a separate set

These completions were generated with <bos> in front of each prompt. A rerun on the final checkpoint without <bos> gave completions of the same quality.

This is a lightweight smoke test, not a benchmark. See Evaluation for actual downstream results.

Training Details

Global batch size	2048
Micro batch size	4
Sequence length	4096
Tokens / iteration	~8.39M
Total training iterations	1,192,093 (~10T tokens)
LR schedule	WSD (warmup-stable-decay), linear decay
Peak LR	3.5e-4
Warmup iterations	4,000
Cooldown iterations	238,476 (final 20% of training)
Optimizer	Adam (β1=0.9, β2=0.95, eps=1e-8), distributed optimizer
Weight decay	0.1
Gradient clipping	1.0
Parallelism	TP=4, PP=1, CP=1, sequence parallel
Checkpoint format	Megatron torch_dist, saved every 2,400 iterations
Hardware	256 nodes / 1024 GPUs, Leonardo (CINECA)

Training logs (loss curves, throughput, etc.) are tracked on Weights & Biases.

Tokens trained at a given checkpoint

Each iteration consumes exactly one global batch, so the number of tokens seen at a given branch's iteration can be computed directly from its name:

tokens(iteration) = iteration × global_batch_size × sequence_length
                   = iteration × 2048 × 4096
                   = iteration × 8,388,608

For example, iter_0124800 corresponds to 124,800 × 8,388,608 ≈ 1.05T tokens.

Evaluation

Downstream evaluation is ongoing, see this issue.

Uses

These are intermediate, non-cooled-down research checkpoints, intended for studying training dynamics (loss/capability progression, multilingual acquisition, etc.), not for production or downstream deployment. No safety alignment or instruction tuning has been applied.