---
license: mit
language:
- de
base_model:
- TUM/GottBERT_filtered_base_best
---

# GeistBERT
GeistBERT is a **German language model** trained on a **for the most part deduplicated corpus** including **OSCAR23, OPUS, and MC4**. It builds on **GottBERT** while introducing **Whole Word Masking (WWM)** to improve contextual language representation. The model achieves **state-of-the-art (SOTA) performance** on multiple German NLP benchmarks.

GeistBERT comes in **three versions**:
- GeistBERT (Standard, this repo)
- [GeistBERT-Nyströmformer](https://huggingface.co/GeistBERT/GeistBERT_base_nystromformer) (Efficient self-attention)
- [GeistBERT-Longformer](https://huggingface.co/GeistBERT/GeistBERT_base_longformer) (Extended context length)

## Training Data
GeistBERT was trained on a **diverse German corpus** combining:
- **OSCAR23, OPUS, and MC4** (for the most part deduplicated)
- **German Wikipedia**
- **OpenLegalData**
- **Europarl, EUbookshop, ECB, and EuroPat**
- **OpenSubtitles and TildeMODEL**

The dataset amounts to **approximately 1.3T tokens**, shuffled for improved variance.

## Training Procedure
### Hardware
- Training was conducted on **multiple GPUs**, including **NVIDIA RTX 3090 (24GB VRAM)**.
- **Gradient accumulation** was used for **Longformer**, requiring **more VRAM** compared to Nyströmformer and RoBERTa, which fit on a single RTX 3090.

### Hyperparameters
| Parameter          | Value                  |
|--------------------|------------------------|
| **Model Architecture** | RoBERTa (Base)      |
| **Batch Size**     | 8,000                  |
| **Training Steps** | 100k                   |
| **Weight Initialization** | [GottBERT filtered base](https://huggingface.co/TUM/GottBERT_filtered_base_best) |
| **Warmup Iterations** | 10k                  |
| **Peak Learning Rate** | 0.0007              |
| **Learning Rate Decay** | Polynomial to zero |

## Performance
GeistBERT achieves **SOTA results** on multiple tasks:
- **NER**: CoNLL 2003, GermEval 2014
- **Text Classification**: GermEval 2018 (coarse & fine), 10kGNAD
- **NLI**: German subset of XNLI

Mertics:
- **NER and Text Classification**: F1 Score
- **NLI**: Accuracy


Details:
- **bold** values indicate the best performing model within one architecure (base, large), <ins>undescored</ins> values the second best.

| Model                               | Accuracy NLI | GermEval\_14 F1 | CoNLL F1 | Coarse F1 | Fine F1 | 10kGNAD F1 |
|-------------------------------------|--------------|----------------|----------|-----------|---------|------------|
| [GeistBERT](https://huggingface.co/GeistBERT/GeistBERT_base)                | **82.67**   | **88.47**   | _86.17_  | _79.67_  | 66.42   | **90.89**  |
| [GeistBERT-Nyströmformer](https://huggingface.co/GeistBERT/GeistBERT_base_nystromformer) | 82.50       | 88.23          | 85.76    | 79.17     | **78.57** | 90.33      |
| [GeistBERT-Longformer](https://huggingface.co/GeistBERT/GeistBERT_base_longformer) | _82.51_     | _88.45_        | **86.71** | **80.56** | _66.76_ | 90.32      |
| [GottBERT_base_best](https://huggingface.co/TUM/GottBERT_base_best)                | 80.82       | 87.55          | 85.93  | 78.17     | 53.30   | 89.64      |
| [GottBERT_base_last](https://huggingface.co/TUM/GottBERT_base_last)                | 81.04       | 87.48          | 85.61    | 78.18   | 53.92 | 90.27  |
| [GottBERT_filtered_base_best](https://huggingface.co/TUM/GottBERT_filtered_base_best)         | 80.56       | 87.57 | 86.14 | 78.65 | 52.82   | 89.79      |
| [GottBERT_filtered_base_last](https://huggingface.co/TUM/GottBERT_filtered_base_last)         | 80.74       | 87.59      | 85.66    | 78.08     | 52.39   | 89.92      |
| GELECTRA_base                   | 81.70   | 86.91          | 85.37    | 77.26     | 50.07   | 89.02      |
| GBERT_base                        | 80.06       | 87.24          | 85.16    | 77.37     | 51.51   | 90.30  |
| dbmdzBERT                          | 68.12       | 86.82          | 85.15    | 77.46     | 52.07   | _90.34_  |
| GermanBERT                        | 78.16       | 86.53          | 83.87    | 74.81     | 47.78   | 90.18      |
| XLM-R_base                        | 79.76       | 86.14          | 84.46    | 77.13     | 50.54   | 89.81      |
| mBERT                              | 77.03       | 86.67          | 83.18    | 73.54     | 48.32   | 88.90      |
| [GottBERT_large](https://huggingface.co/TUM/GottBERT_large)                | 82.46       | 88.20          | _86.78_  | 79.40     | 54.61   | 90.24      |
| [GottBERT_filtered_large_best](https://huggingface.co/TUM/GottBERT_filtered_large_best)     | 83.31       | 88.13          | 86.30    | 79.32     | 54.70   | 90.31      |
| [GottBERT_filtered_large_last](https://huggingface.co/TUM/GottBERT_filtered_large_last)     | 82.79       | _88.27_ | 86.28    | 78.96     | 54.72   | 90.17      |
| GELECTRA_large                | **86.33**   | _88.72_ | _86.78_  | **81.28** | _56.17_ | **90.97**  |
| GBERT_large                      | _84.21_     | _88.72_        | **87.19** | _80.84_   | **57.37** | _90.74_   |
| XLM-R_large                      | 84.07       | **88.83**      | 86.54    | 79.05     | 55.06   | 90.17      |


## Intended Use
This model is designed for **German NLP tasks**, including:
- **Text classification**
- **Named Entity Recognition (NER)**
- **Machine Translation Pre-training**
- **Document Understanding**

## Limitations
- Trained on **unfiltered data**, meaning some **redundant or lower-quality samples** may be present.
- Longformer **requires more VRAM**, making it less accessible for smaller GPU setups.
- While deduplication was applied to **specific subcorpora**, the full corpus **was not manually curated**.

## Fairseq Checkpoints
Get the fairseq checkpoints [here](https://drive.proton.me/urls/P83GCPNM40#2f0f87XEIrQP).