metadata
language:
- de
license: mit
library_name: transformers
pipeline_tag: text-generation
tags:
- t5
- german
- wechsel
- cross-lingual
datasets:
- unpaywall-scientific
DE-T5-Sci-Transfer-Init
WECHSEL-initialized checkpoint: English EN-T5-Sci weights + German tokenizer (GermanT5/t5-efficient-gc4-german-base-nl36) aligned using WECHSEL (static embeddings + bilingual dictionary). No additional German training after transfer. Folder includes transfer_metadata.pt with alignment diagnostics.
Model Details
- Embedding init: Orthogonal Procrustes map (fastText n-gram embeddings) + temperature-weighted mixtures (k-nearest neighbors)
- Special tokens:
<extra_id_0..99>aligned, sentinel behavior preserved - Tokenizer: GermanT5 SentencePiece (files bundled here)
Evaluation (Global-MMLU, zero-shot)
| Metric | EN | DE |
|---|---|---|
| Overall accuracy | 0.2434 | 0.2463 |
| Humanities | 0.2485 | 0.2559 |
| STEM | 0.2391 | 0.2445 |
| Social Sciences | 0.2317 | 0.2307 |
| Other | 0.2517 | 0.2491 |
This demonstrates immediate cross-lingual transfer without any German gradient steps.
Intended Use
Starting point for German continued pretraining or fine-tuning where English scientific knowledge should be retained but a German tokenizer is required.
Limitations
- No German data exposure beyond embedding alignment; you should run additional continued pretraining (see next model) for best performance.
- Still limited to 512-token context.