|
|
--- |
|
|
language: |
|
|
- de |
|
|
license: mit |
|
|
library_name: transformers |
|
|
pipeline_tag: text-generation |
|
|
tags: |
|
|
- t5 |
|
|
- german |
|
|
- wechsel |
|
|
- cross-lingual |
|
|
datasets: |
|
|
- unpaywall-scientific |
|
|
--- |
|
|
|
|
|
# DE-T5-Sci-Transfer-Init |
|
|
|
|
|
WECHSEL-initialized checkpoint: English EN-T5-Sci weights + German tokenizer (`GermanT5/t5-efficient-gc4-german-base-nl36`) aligned using WECHSEL (static embeddings + bilingual dictionary). **No additional German training** after transfer. Folder includes `transfer_metadata.pt` with alignment diagnostics. |
|
|
|
|
|
## Model Details |
|
|
- Embedding init: Orthogonal Procrustes map (fastText n-gram embeddings) + temperature-weighted mixtures (k-nearest neighbors) |
|
|
- Special tokens: `<extra_id_0..99>` aligned, sentinel behavior preserved |
|
|
- Tokenizer: GermanT5 SentencePiece (files bundled here) |
|
|
|
|
|
## Evaluation (Global-MMLU, zero-shot) |
|
|
| Metric | EN | DE | |
|
|
| --- | --- | --- | |
|
|
| Overall accuracy | 0.2434 | 0.2463 | |
|
|
| Humanities | 0.2485 | 0.2559 | |
|
|
| STEM | 0.2391 | 0.2445 | |
|
|
| Social Sciences | 0.2317 | 0.2307 | |
|
|
| Other | 0.2517 | 0.2491 | |
|
|
|
|
|
This demonstrates immediate cross-lingual transfer without any German gradient steps. |
|
|
|
|
|
## Intended Use |
|
|
Starting point for German continued pretraining or fine-tuning where English scientific knowledge should be retained but a German tokenizer is required. |
|
|
|
|
|
## Limitations |
|
|
- No German data exposure beyond embedding alignment; you should run additional continued pretraining (see next model) for best performance. |
|
|
- Still limited to 512-token context. |
|
|
|
