Boldt-DC-1B-100BT

Boldt-DC-1B-100BT is a German base language model with 1B parameters, trained from scratch on the Dense-Core subset for 100B tokens.

This is a research checkpoint from the experiments in our paper Repetition over Diversity (EMNLP 2026). It is released for reproducibility and for studying how different data-filtering choices affect pre-training.

Training Data

The 28B-token Dense-Core subset (Boldt/fineweb2-de-dc), repeated for 3.6 epochs to reach 100B tokens. Boldt-DC-1B is the same setup trained for 200B tokens (7.2 epochs).

Model Architecture

• Architecture: Llama-style decoder-only transformer, 16 layers, hidden size 2048, FFN size 8192, 16 attention heads
• Context Window: 2048 tokens
• Tokenizer: 32k-vocabulary BPE trained on the model's own training subset. Always load the tokenizer from this repository.

Usage

Note: This is a base language model, not an instruction-tuned model. It is not optimized for chat or instruction following. Use plain text completion rather than chat templates.

from transformers import AutoTokenizer, AutoModelForCausalLM
model_name = "Boldt/Boldt-DC-1B-100BT"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

text = "Berlin ist eine Stadt, wo"
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=64)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Citation

@inproceedings{aynetdinov-etal-2026-repetition,
    title = "Repetition over Diversity: High-Signal Data Filtering for Sample-Efficient {G}erman Language Modeling",
    author = "Aynetdinov, Ansar and Haller, Patrick and Akbik, Alan",
    editor = "Lee, Hung-Yi and Dev, Sunipa and Li, Jessy and Elliott, Desmond",
    booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing",
    month = oct,
    year = "2026",
    address = "Budapest, Hungary",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.emnlp-main.1473/",
    doi = "10.18653/2026.emnlp-main.1473",
    pages = "33690--33710"
}