File size: 1,321 Bytes
31b7c1c 567a7d4 b2cebd5 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | ---
title: README
emoji: 📈
colorFrom: red
colorTo: red
sdk: static
pinned: false
---
# 🇩🇪 German Tokenizer Benchmark
A curated collection of German datasets for comprehensive tokenizer evaluation across diverse domains and text types.
This organization hosts datasets used by the [GerTokEval](https://github.com/stefan-it/german-tokenizer-benchmark) framework to evaluate German tokenizers with standardized metrics and fairness analysis.
## 🔎 Datasets
The following datasets are currently supported in the main framework:
* [GermEval 2014](https://huggingface.co/datasets/german-tokenizer-benchmark/germeval14)
* [BIOfid](https://huggingface.co/datasets/german-tokenizer-benchmark/biofid)
* [CO-Fun](https://huggingface.co/datasets/german-tokenizer-benchmark/co-funer)
* [German LER](https://huggingface.co/datasets/german-tokenizer-benchmark/german-ler)
* [DFKI MobIE](https://huggingface.co/datasets/german-tokenizer-benchmark/mobie)
* [UD HDT](https://huggingface.co/datasets/german-tokenizer-benchmark/ud-hdt)
The main goal for choosing these datasets is to evaluate tokenizers on a broad range of domains.
# ❤️ Acknowledgements
Many thanks to [Clara Meister](https://github.com/cimeister) for releasing the amazing [TokEval framework](https://github.com/cimeister/tokenizer-analysis-suite)! |