| | --- |
| | language: |
| | - multilingual |
| | - af |
| | - am |
| | - ar |
| | - as |
| | - az |
| | - be |
| | - bg |
| | - bn |
| | - br |
| | - bs |
| | - ca |
| | - cs |
| | - cy |
| | - da |
| | - de |
| | - el |
| | - en |
| | - eo |
| | - es |
| | - et |
| | - eu |
| | - fa |
| | - fi |
| | - fr |
| | - fy |
| | - ga |
| | - gd |
| | - gl |
| | - gu |
| | - ha |
| | - he |
| | - hi |
| | - hr |
| | - hu |
| | - hy |
| | - id |
| | - is |
| | - it |
| | - ja |
| | - jv |
| | - ka |
| | - kk |
| | - km |
| | - kn |
| | - ko |
| | - ku |
| | - ky |
| | - la |
| | - lo |
| | - lt |
| | - lv |
| | - mg |
| | - mk |
| | - ml |
| | - mn |
| | - mr |
| | - ms |
| | - my |
| | - ne |
| | - nl |
| | - no |
| | - om |
| | - or |
| | - pa |
| | - pl |
| | - ps |
| | - pt |
| | - ro |
| | - ru |
| | - sa |
| | - sd |
| | - si |
| | - sk |
| | - sl |
| | - so |
| | - sq |
| | - sr |
| | - su |
| | - sv |
| | - sw |
| | - ta |
| | - te |
| | - th |
| | - tl |
| | - tr |
| | - ug |
| | - uk |
| | - ur |
| | - uz |
| | - vi |
| | - xh |
| | - yi |
| | - zh |
| | license: mit |
| | --- |
| | |
| | # XLM-V (Base-sized model) |
| |
|
| | XLM-V is multilingual language model with a one million token vocabulary trained on 2.5TB of data from Common Crawl (same as XLM-R). |
| | It was introduced in the [XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models](https://arxiv.org/abs/2301.10472) |
| | paper by Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Naman Goyal, Marjan Ghazvininejad, Luke Zettlemoyer and Madian Khabsa. |
| |
|
| | **Disclaimer**: The team releasing XLM-V did not write a model card for this model so this model card has been written by the Hugging Face team. [This repository](https://github.com/stefan-it/xlm-v-experiments) documents all necessary integeration steps. |
| |
|
| | ## Model description |
| |
|
| | From the abstract of the XLM-V paper: |
| |
|
| | > Large multilingual language models typically rely on a single vocabulary shared across 100+ languages. |
| | > As these models have increased in parameter count and depth, vocabulary size has remained largely unchanged. |
| | > This vocabulary bottleneck limits the representational capabilities of multilingual models like XLM-R. |
| | > In this paper, we introduce a new approach for scaling to very large multilingual vocabularies by |
| | > de-emphasizing token sharing between languages with little lexical overlap and assigning vocabulary capacity |
| | > to achieve sufficient coverage for each individual language. Tokenizations using our vocabulary are typically |
| | > more semantically meaningful and shorter compared to XLM-R. Leveraging this improved vocabulary, we train XLM-V, |
| | > a multilingual language model with a one million token vocabulary. XLM-V outperforms XLM-R on every task we |
| | > tested on ranging from natural language inference (XNLI), question answering (MLQA, XQuAD, TyDiQA), and |
| | > named entity recognition (WikiAnn) to low-resource tasks (Americas NLI, MasakhaNER). |
| |
|
| | ## Usage |
| |
|
| | You can use this model directly with a pipeline for masked language modeling: |
| |
|
| | ```python |
| | >>> from transformers import pipeline |
| | >>> unmasker = pipeline('fill-mask', model='facebook/xlm-v-base') |
| | >>> unmasker("Paris is the <mask> of France.") |
| | |
| | [{'score': 0.9286897778511047, |
| | 'token': 133852, |
| | 'token_str': 'capital', |
| | 'sequence': 'Paris is the capital of France.'}, |
| | {'score': 0.018073994666337967, |
| | 'token': 46562, |
| | 'token_str': 'Capital', |
| | 'sequence': 'Paris is the Capital of France.'}, |
| | {'score': 0.013238662853837013, |
| | 'token': 8696, |
| | 'token_str': 'centre', |
| | 'sequence': 'Paris is the centre of France.'}, |
| | {'score': 0.010450296103954315, |
| | 'token': 550136, |
| | 'token_str': 'heart', |
| | 'sequence': 'Paris is the heart of France.'}, |
| | {'score': 0.005028395913541317, |
| | 'token': 60041, |
| | 'token_str': 'center', |
| | 'sequence': 'Paris is the center of France.'}] |
| | ``` |
| |
|
| | ## Bias, Risks, and Limitations |
| |
|
| | Please refer to the model card of [XLM-R](https://huggingface.co/xlm-roberta-base), because XLM-V has a similar architecture |
| | and has been trained on similar training data. |
| |
|
| | ### BibTeX entry and citation info |
| |
|
| | ```bibtex |
| | @ARTICLE{2023arXiv230110472L, |
| | author = {{Liang}, Davis and {Gonen}, Hila and {Mao}, Yuning and {Hou}, Rui and {Goyal}, Naman and {Ghazvininejad}, Marjan and {Zettlemoyer}, Luke and {Khabsa}, Madian}, |
| | title = "{XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models}", |
| | journal = {arXiv e-prints}, |
| | keywords = {Computer Science - Computation and Language, Computer Science - Machine Learning}, |
| | year = 2023, |
| | month = jan, |
| | eid = {arXiv:2301.10472}, |
| | pages = {arXiv:2301.10472}, |
| | doi = {10.48550/arXiv.2301.10472}, |
| | archivePrefix = {arXiv}, |
| | eprint = {2301.10472}, |
| | primaryClass = {cs.CL}, |
| | adsurl = {https://ui.adsabs.harvard.edu/abs/2023arXiv230110472L}, |
| | adsnote = {Provided by the SAO/NASA Astrophysics Data System} |
| | } |
| | ``` |
