|
|
--- |
|
|
language: |
|
|
- de |
|
|
license: mit |
|
|
library_name: transformers |
|
|
pipeline_tag: text-generation |
|
|
tags: |
|
|
- t5 |
|
|
- german |
|
|
- scientific |
|
|
- wechsel |
|
|
datasets: |
|
|
- unpaywall-scientific |
|
|
--- |
|
|
|
|
|
# DE-T5-Sci-Transfer-15k |
|
|
|
|
|
Final German scientific model: WECHSEL-initialized EN-T5-Sci → continued for 15 000 steps on German scientific data (same regimen as DE-T5-Base-15k). Checkpoint: `cross_lingual_transfer/logs/train/.../step-step=015000.ckpt`. |
|
|
|
|
|
## Model Details |
|
|
- Base: EN-T5-Sci weights, German tokenizer |
|
|
- Optimizer: Adafactor, lr=1e-3, inverse sqrt schedule, warmup 1.5k, grad clip 1.0 |
|
|
- Effective batch: 48 (per-GPU 48, grad accumulation 1) |
|
|
- Objective: Span corruption (15 % noise, mean span 3) |
|
|
|
|
|
## Training Data |
|
|
German subset of the Unpaywall-derived scientific corpus (sliding windows 512 tokens, 50 % overlap). Same cleaning pipeline as the English run. |
|
|
|
|
|
## Evaluation (Global-MMLU, zero-shot) |
|
|
| Metric | EN | DE | |
|
|
| --- | --- | --- | |
|
|
| Overall accuracy | **0.2738** | **0.2700** | |
|
|
| Humanities | 0.2559 | 0.2536 | |
|
|
| STEM | 0.2867 | 0.2851 | |
|
|
| Social Sciences | 0.3058 | 0.3055 | |
|
|
| Other | 0.2562 | 0.2443 | |
|
|
|
|
|
This is the best-performing German checkpoint across both languages in the final evaluation (`evaluation_results/scientific_crosslingual_transfer_eval_full_15k`). |
|
|
|
|
|
## Intended Use |
|
|
Zero-shot QA in German/English scientific domains, or as a strong starting point for German task-specific fine-tuning (NER, relation extraction, etc.). |
|
|
|
|
|
## Limitations |
|
|
- Still inherits T5-base context length/parameter budget. |
|
|
- Evaluated only on Global-MMLU; downstream fine-tuning recommended for specialized tasks. |
|
|
- Training corpus is domain-specific (scientific); may underperform on casual text. |
|
|
|
|
|
## Citation |
|
|
Please cite the thesis (Nikolas Rauscher, 2025) and the WECHSEL paper (Minixhofer et al. 2022). |
|
|
|
