metadata
datasets:
- togethercomputer/RedPajama-Data-V2
language:
- de
library_name: transformers
license: other
pipeline_tag: feature-extraction
tags:
- masked-lm
- long-context
- modernbert
ModernGBERT 1B
This is a German ModernBERT 1B language model trained from scratch using the ModernBERT codebase and the same German portion of RedPajama V2 as our LLäMmlein family. Find more details in our preprint!
Usage
from transformers import AutoModel, AutoTokenizer
model = AutoModel.from_pretrained("LSX-UniWue/ModernGBERT_1B")
tokenizer = AutoTokenizer.from_pretrained("LSX-UniWue/ModernGBERT_1B")
Performance
We evaluated our model on the SuperGLEBer benchmark.