Create README.md

README.md

@@ -0,0 +1,49 @@
+---
+language:
+- de
+license: mit
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- t5
+- german
+- scientific
+- wechsel
+datasets:
+- unpaywall-scientific
+---
+
+# DE-T5-Sci-Transfer-15k
+
+Final German scientific model: WECHSEL-initialized EN-T5-Sci → continued for 15 000 steps on German scientific data (same regimen as DE-T5-Base-15k). Checkpoint: `cross_lingual_transfer/logs/train/.../step-step=015000.ckpt`.
+
+## Model Details
+- Base: EN-T5-Sci weights, German tokenizer
+- Optimizer: Adafactor, lr=1e-3, inverse sqrt schedule, warmup 1.5k, grad clip 1.0
+- Effective batch: 48 (per-GPU 48, grad accumulation 1)
+- Objective: Span corruption (15 % noise, mean span 3)
+
+## Training Data
+German subset of the Unpaywall-derived scientific corpus (sliding windows 512 tokens, 50 % overlap). Same cleaning pipeline as the English run.
+
+## Evaluation (Global-MMLU, zero-shot)
+| Metric | EN | DE |
+| --- | --- | --- |
+| Overall accuracy | **0.2738** | **0.2700** |
+| Humanities | 0.2559 | 0.2536 |
+| STEM | 0.2867 | 0.2851 |
+| Social Sciences | 0.3058 | 0.3055 |
+| Other | 0.2562 | 0.2443 |
+
+This is the best-performing German checkpoint across both languages in the final evaluation (`evaluation_results/scientific_crosslingual_transfer_eval_full_15k`).
+
+## Intended Use
+Zero-shot QA in German/English scientific domains, or as a strong starting point for German task-specific fine-tuning (NER, relation extraction, etc.).
+
+## Limitations
+- Still inherits T5-base context length/parameter budget.
+- Evaluated only on Global-MMLU; downstream fine-tuning recommended for specialized tasks.
+- Training corpus is domain-specific (scientific); may underperform on casual text.
+
+## Citation
+Please cite the thesis (Nikolas Rauscher, 2025) and the WECHSEL paper (Minixhofer et al. 2022).