Boldt-Random-350M-100BT

Boldt-Random-350M-100BT is a German base language model in the 350M parameter class, trained from scratch on the unfiltered Random baseline data for 100B tokens.

This is a research checkpoint from the experiments in our paper Repetition over Diversity (EMNLP 2026). It is released for reproducibility and for studying how different data-filtering choices affect pre-training.

Training Data

A random 100B-token sample of the German split of FineWeb2 (FW2-DE), with no quality filtering beyond FineWeb2's own pipeline. The model sees every document once (1 epoch). It is the Random baseline in the paper.

Model Architecture

• Architecture: Llama-style decoder-only transformer, 24 layers, hidden size 1024, FFN size 4096, 16 attention heads
• Context Window: 2048 tokens
• Tokenizer: 32k-vocabulary BPE trained on the model's own training subset. Always load the tokenizer from this repository.

Usage

Note: This is a base language model, not an instruction-tuned model. It is not optimized for chat or instruction following. Use plain text completion rather than chat templates.

from transformers import AutoTokenizer, AutoModelForCausalLM
model_name = "Boldt/Boldt-Random-350M-100BT"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

text = "Berlin ist eine Stadt, wo"
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Citation

@inproceedings{aynetdinov-etal-2026-repetition,
    title = "Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient {G}erman Language Modeling",
    author = "Aynetdinov, Ansar and Haller, Patrick and Akbik, Alan",
    editor = "Lee, Hung-Yi and Dev, Sunipa and Li, Jessy and Elliott, Desmond",
    booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing",
    month = oct,
    year = "2026",
    address = "Budapest, Hungary",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.emnlp-main.1473/",
    doi = "10.18653/2026.emnlp-main.1473",
    pages = "33690--33710"
}