Boldt-Random-350M-200BT

Boldt-Random-350M-200BT is a German base language model in the 350M parameter class, trained from scratch on the unfiltered Random baseline data for 200B tokens.

This is a research checkpoint from the experiments in our paper Repetition over Diversity (EMNLP 2026). It is released for reproducibility and for studying how different data-filtering choices affect pre-training.

Training Data

A random 200B-token sample of the German split of FineWeb2 (FW2-DE), seen once (1 epoch). This extends the 100B Random baseline with new, unseen documents. It tests whether more unique data can make up for the lack of quality filtering.

Model Architecture

• Architecture: Llama-style decoder-only transformer, 24 layers, hidden size 1024, FFN size 4096, 16 attention heads
• Context Window: 2048 tokens
• Tokenizer: 32k-vocabulary BPE trained on the model's own training subset. Always load the tokenizer from this repository.

Usage

Note: This is a base language model, not an instruction-tuned model. It is not optimized for chat or instruction following. Use plain text completion rather than chat templates.

from transformers import AutoTokenizer, AutoModelForCausalLM
model_name = "Boldt/Boldt-Random-350M-200BT"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

text = "Berlin ist eine Stadt, wo"
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Citation

@inproceedings{aynetdinov-etal-2026-repetition,
    title = "Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient {G}erman Language Modeling",
    author = "Aynetdinov, Ansar and Haller, Patrick and Akbik, Alan",
    editor = "Lee, Hung-Yi and Dev, Sunipa and Li, Jessy and Elliott, Desmond",
    booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing",
    month = oct,
    year = "2026",
    address = "Budapest, Hungary",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.emnlp-main.1473/",
    doi = "10.18653/2026.emnlp-main.1473",
    pages = "33690--33710"
}