README.md

---
library_name: transformers
license: cc0-1.0
datasets:
- deutsche-telekom/Ger-RAG-eval
tags:
- tokenization
language:
- de
---

# Small German Tokenizer

This is a small public domain-like tokenizer optimized for German.

## Special Tokens

- End-of-Sequence token: `[EOS]`
- Padding token: `[PAD]`

## Training

This tokenizer was trained on the `context` column of the configs `task1` and `task4` in [deutsche-telekom/Ger-RAG-eval](https://huggingface.co/datasets/deutsche-telekom/Ger-RAG-eval).

## Limitations

Due to its small corpus, this tokenizer may split words into smaller pieces. Also, some uncommon special tokens aren't present, you'll have to add them manually if needed.