| title: README | |
| emoji: 📈 | |
| colorFrom: red | |
| colorTo: red | |
| sdk: static | |
| pinned: false | |
| # 🇩🇪 German Tokenizer Benchmark | |
| A curated collection of German datasets for comprehensive tokenizer evaluation across diverse domains and text types. | |
| This organization hosts datasets used by the [GerTokEval](https://github.com/stefan-it/german-tokenizer-benchmark) framework to evaluate German tokenizers with standardized metrics and fairness analysis. | |
| ## 🔎 Datasets | |
| The following datasets are currently supported in the main framework: | |
| * [GermEval 2014](https://huggingface.co/datasets/german-tokenizer-benchmark/germeval14) | |
| * [BIOfid](https://huggingface.co/datasets/german-tokenizer-benchmark/biofid) | |
| * [CO-Fun](https://huggingface.co/datasets/german-tokenizer-benchmark/co-funer) | |
| * [German LER](https://huggingface.co/datasets/german-tokenizer-benchmark/german-ler) | |
| * [DFKI MobIE](https://huggingface.co/datasets/german-tokenizer-benchmark/mobie) | |
| * [UD HDT](https://huggingface.co/datasets/german-tokenizer-benchmark/ud-hdt) | |
| The main goal for choosing these datasets is to evaluate tokenizers on a broad range of domains. | |
| # ❤️ Acknowledgements | |
| Many thanks to [Clara Meister](https://github.com/cimeister) for releasing the amazing [TokEval framework](https://github.com/cimeister/tokenizer-analysis-suite)! |