---
language:
- de
license: mit
library_name: transformers
pipeline_tag: text-generation
tags:
- t5
- german
- scientific
- wechsel
datasets:
- unpaywall-scientific
---

# DE-T5-Sci-Transfer-15k

Final German scientific model: WECHSEL-initialized EN-T5-Sci → continued for 15 000 steps on German scientific data (same regimen as DE-T5-Base-15k). Checkpoint: `cross_lingual_transfer/logs/train/.../step-step=015000.ckpt`.

## Model Details
- Base: EN-T5-Sci weights, German tokenizer
- Optimizer: Adafactor, lr=1e-3, inverse sqrt schedule, warmup 1.5k, grad clip 1.0
- Effective batch: 48 (per-GPU 48, grad accumulation 1)
- Objective: Span corruption (15 % noise, mean span 3)

## Training Data
German subset of the Unpaywall-derived scientific corpus (sliding windows 512 tokens, 50 % overlap). Same cleaning pipeline as the English run.

## Evaluation (Global-MMLU, zero-shot)
| Metric | EN | DE |
| --- | --- | --- |
| Overall accuracy | **0.2738** | **0.2700** |
| Humanities | 0.2559 | 0.2536 |
| STEM | 0.2867 | 0.2851 |
| Social Sciences | 0.3058 | 0.3055 |
| Other | 0.2562 | 0.2443 |

This is the best-performing German checkpoint across both languages in the final evaluation (`evaluation_results/scientific_crosslingual_transfer_eval_full_15k`).

## Intended Use
Zero-shot QA in German/English scientific domains, or as a strong starting point for German task-specific fine-tuning (NER, relation extraction, etc.).

## Limitations
- Still inherits T5-base context length/parameter budget.
- Evaluated only on Global-MMLU; downstream fine-tuning recommended for specialized tasks.
- Training corpus is domain-specific (scientific); may underperform on casual text.

## Citation
Please cite the thesis (Nikolas Rauscher, 2025) and the WECHSEL paper (Minixhofer et al. 2022).