| | --- |
| | license: mit |
| | --- |
| | |
| | # umT5 Small |
| |
|
| | The UMT5 model was proposed in [UniMax: Fairer and More Effective Language Sampling for Large-Scale Multilingual Pretraining](https://openreview.net/forum?id=kXwdL1cWOAi) |
| | by Hyung Won Chung, Xavier Garcia, Adam Roberts, Yi Tay, Orhan Firat, Sharan Narang, Noah Constant. |
| |
|
| | The abstract from the paper is the following: |
| |
|
| | *Pretrained multilingual large language models have typically used heuristic temperature-based sampling to balance |
| | between different languages. However previous work has not systematically evaluated the efficacy of different |
| | pretraining language distributions across model scales. In this paper, we propose a new sampling method, UniMax, |
| | that delivers more uniform coverage of head languages while mitigating overfitting on tail languages by explicitly |
| | capping the number of repeats over each language's corpus. We perform an extensive series of ablations testing a |
| | range of sampling strategies on a suite of multilingual benchmarks, while varying model scale. We find that UniMax |
| | outperforms standard temperature-based sampling, and the benefits persist as scale increases. As part of our |
| | contribution, we release: (i) an improved and refreshed mC4 multilingual corpus consisting of 29 trillion characters |
| | across 107 languages, and (ii) a suite of pretrained umT5 model checkpoints trained with UniMax sampling.* |
| |
|
| | # Integration into Transformers |
| |
|
| | Overview of umT5 model integration: |
| |
|
| | * Transformers Integration is on-going, see this awesome [PR](https://github.com/huggingface/transformers/pull/22626) by @agemagician! |
| | * Conversion script (umT5X checkpoints to FLAX) is [here](https://gist.github.com/stefan-it/5d6a4ec89e7ad97181983881434cb4eb). |
