| | --- |
| | pretty_name: mC4 |
| | annotations_creators: |
| | - no-annotation |
| | language_creators: |
| | - found |
| | languages: |
| | - af |
| | - am |
| | - ar |
| | - az |
| | - be |
| | - bg |
| | - bg-Latn |
| | - bn |
| | - ca |
| | - ceb |
| | - co |
| | - cs |
| | - cy |
| | - da |
| | - de |
| | - el |
| | - el-Latn |
| | - en |
| | - eo |
| | - es |
| | - et |
| | - eu |
| | - fa |
| | - fi |
| | - fil |
| | - fr |
| | - fy |
| | - ga |
| | - gd |
| | - gl |
| | - gu |
| | - ha |
| | - haw |
| | - hi |
| | - hi-Latn |
| | - hmn |
| | - ht |
| | - hu |
| | - hy |
| | - id |
| | - ig |
| | - is |
| | - it |
| | - iw |
| | - ja |
| | - ja-Latn |
| | - jv |
| | - ka |
| | - kk |
| | - km |
| | - kn |
| | - ko |
| | - ku |
| | - ky |
| | - la |
| | - lb |
| | - lo |
| | - lt |
| | - lv |
| | - mg |
| | - mi |
| | - mk |
| | - ml |
| | - mn |
| | - mr |
| | - ms |
| | - mt |
| | - my |
| | - ne |
| | - nl |
| | - "no" |
| | - ny |
| | - pa |
| | - pl |
| | - ps |
| | - pt |
| | - ro |
| | - ru |
| | - ru-Latn |
| | - sd |
| | - si |
| | - sk |
| | - sl |
| | - sm |
| | - sn |
| | - so |
| | - sq |
| | - sr |
| | - st |
| | - su |
| | - sv |
| | - sw |
| | - ta |
| | - te |
| | - tg |
| | - th |
| | - tr |
| | - uk |
| | - und |
| | - ur |
| | - uz |
| | - vi |
| | - xh |
| | - yi |
| | - yo |
| | - zh |
| | - zh-Latn |
| | - zu |
| | licenses: |
| | - odc-by-1.0 |
| | multilinguality: |
| | - multilingual |
| | size_categories: |
| | - n<1K |
| | - 1K<n<10K |
| | - 10K<n<100K |
| | - 100K<n<1M |
| | - 1M<n<10M |
| | - 10M<n<100M |
| | - 100M<n<1B |
| | - 1B<n<10B |
| | source_datasets: |
| | - original |
| | task_categories: |
| | - sequence-modeling |
| | task_ids: |
| | - language-modeling |
| | paperswithcode_id: mc4 |
| | --- |
| | |
| | # Dataset Card for mC4 |
| |
|
| | ## Table of Contents |
| |
|
| | - [Dataset Card for mC4](#dataset-card-for-mc4) |
| | - [Table of Contents](#table-of-contents) |
| | - [Dataset Description](#dataset-description) |
| | - [Dataset Summary](#dataset-summary) |
| | - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards) |
| | - [Languages](#languages) |
| | - [Dataset Structure](#dataset-structure) |
| | - [Data Instances](#data-instances) |
| | - [Data Fields](#data-fields) |
| | - [Data Splits](#data-splits) |
| | - [Dataset Creation](#dataset-creation) |
| | - [Curation Rationale](#curation-rationale) |
| | - [Source Data](#source-data) |
| | - [Initial Data Collection and Normalization](#initial-data-collection-and-normalization) |
| | - [Who are the source language producers?](#who-are-the-source-language-producers) |
| | - [Annotations](#annotations) |
| | - [Annotation process](#annotation-process) |
| | - [Who are the annotators?](#who-are-the-annotators) |
| | - [Personal and Sensitive Information](#personal-and-sensitive-information) |
| | - [Considerations for Using the Data](#considerations-for-using-the-data) |
| | - [Social Impact of Dataset](#social-impact-of-dataset) |
| | - [Discussion of Biases](#discussion-of-biases) |
| | - [Other Known Limitations](#other-known-limitations) |
| | - [Additional Information](#additional-information) |
| | - [Dataset Curators](#dataset-curators) |
| | - [Licensing Information](#licensing-information) |
| | - [Citation Information](#citation-information) |
| | - [Contributions](#contributions) |
| |
|
| | ## Dataset Description |
| |
|
| | - **Homepage:** https://huggingface.co/datasets/allenai/c4 |
| | - **Paper:** https://arxiv.org/abs/1910.10683 |
| |
|
| | ### Dataset Summary |
| |
|
| | A multilingual colossal, cleaned version of Common Crawl's web crawl corpus. Based on Common Crawl dataset: "https://commoncrawl.org". |
| |
|
| | This is the version prepared by AllenAI, hosted at this address: https://huggingface.co/datasets/allenai/c4 |
| |
|
| | 108 languages are available and are reported in the table below. |
| |
|
| | Note that the languages that end with "-Latn" are simply romanized variants, i.e. written using the Latin script. |
| |
|
| | | language code | language name | |
| | |:----------------|:---------------------| |
| | | af | Afrikaans | |
| | | am | Amharic | |
| | | ar | Arabic | |
| | | az | Azerbaijani | |
| | | be | Belarusian | |
| | | bg | Bulgarian | |
| | | bg-Latn | Bulgarian (Latin) | |
| | | bn | Bangla | |
| | | ca | Catalan | |
| | | ceb | Cebuano | |
| | | co | Corsican | |
| | | cs | Czech | |
| | | cy | Welsh | |
| | | da | Danish | |
| | | de | German | |
| | | el | Greek | |
| | | el-Latn | Greek (Latin) | |
| | | en | English | |
| | | eo | Esperanto | |
| | | es | Spanish | |
| | | et | Estonian | |
| | | eu | Basque | |
| | | fa | Persian | |
| | | fi | Finnish | |
| | | fil | Filipino | |
| | | fr | French | |
| | | fy | Western Frisian | |
| | | ga | Irish | |
| | | gd | Scottish Gaelic | |
| | | gl | Galician | |
| | | gu | Gujarati | |
| | | ha | Hausa | |
| | | haw | Hawaiian | |
| | | hi | Hindi | |
| | | hi-Latn | Hindi (Latin script) | |
| | | hmn | Hmong, Mong | |
| | | ht | Haitian | |
| | | hu | Hungarian | |
| | | hy | Armenian | |
| | | id | Indonesian | |
| | | ig | Igbo | |
| | | is | Icelandic | |
| | | it | Italian | |
| | | iw | former Hebrew | |
| | | ja | Japanese | |
| | | ja-Latn | Japanese (Latin) | |
| | | jv | Javanese | |
| | | ka | Georgian | |
| | | kk | Kazakh | |
| | | km | Khmer | |
| | | kn | Kannada | |
| | | ko | Korean | |
| | | ku | Kurdish | |
| | | ky | Kyrgyz | |
| | | la | Latin | |
| | | lb | Luxembourgish | |
| | | lo | Lao | |
| | | lt | Lithuanian | |
| | | lv | Latvian | |
| | | mg | Malagasy | |
| | | mi | Maori | |
| | | mk | Macedonian | |
| | | ml | Malayalam | |
| | | mn | Mongolian | |
| | | mr | Marathi | |
| | | ms | Malay | |
| | | mt | Maltese | |
| | | my | Burmese | |
| | | ne | Nepali | |
| | | nl | Dutch | |
| | | no | Norwegian | |
| | | ny | Nyanja | |
| | | pa | Punjabi | |
| | | pl | Polish | |
| | | ps | Pashto | |
| | | pt | Portuguese | |
| | | ro | Romanian | |
| | | ru | Russian | |
| | | ru-Latn | Russian (Latin) | |
| | | sd | Sindhi | |
| | | si | Sinhala | |
| | | sk | Slovak | |
| | | sl | Slovenian | |
| | | sm | San Marino | |
| | | sn | Shona | |
| | | so | Somali | |
| | | sq | Albanian | |
| | | sr | Serbian | |
| | | st | Southern Sotho | |
| | | su | Sundanese | |
| | | sv | Swedish | |
| | | sw | Swahili | |
| | | ta | Tamil | |
| | | te | Telugu | |
| | | tg | Tajik | |
| | | th | Thai | |
| | | tr | Turkish | |
| | | uk | Ukrainian | |
| | | und | Unknown language | |
| | | ur | Urdu | |
| | | uz | Uzbek | |
| | | vi | Vietnamese | |
| | | xh | Xhosa | |
| | | yi | Yiddish | |
| | | yo | Yoruba | |
| | | zh | Chinese | |
| | | zh-Latn | Chinese (Latin) | |
| | | zu | Zulu | |
| |
|
| | You can load the mC4 subset of any language like this: |
| |
|
| | ```python |
| | from datasets import load_dataset |
| | |
| | en_mc4 = load_dataset("mc4", "en") |
| | ``` |
| |
|
| | And if you can even specify a list of languages: |
| |
|
| | ```python |
| | from datasets import load_dataset |
| | |
| | mc4_subset_with_five_languages = load_dataset("mc4", languages=["en", "fr", "es", "de", "zh"]) |
| | ``` |
| |
|
| | ### Supported Tasks and Leaderboards |
| |
|
| | mC4 is mainly intended to pretrain language models and word representations. |
| |
|
| | ### Languages |
| |
|
| | The dataset supports 108 languages. |
| |
|
| | ## Dataset Structure |
| |
|
| | ### Data Instances |
| |
|
| | An example form the `en` config is: |
| |
|
| | ``` |
| | {'timestamp': '2018-06-24T01:32:39Z', |
| | 'text': 'Farm Resources in Plumas County\nShow Beginning Farmer Organizations & Professionals (304)\nThere are 304 resources serving Plumas County in the following categories:\nMap of Beginning Farmer Organizations & Professionals serving Plumas County\nVictoria Fisher - Office Manager - Loyalton, CA\nAmy Lynn Rasband - UCCE Plumas-Sierra Administrative Assistant II - Quincy , CA\nShow Farm Income Opportunities Organizations & Professionals (353)\nThere are 353 resources serving Plumas County in the following categories:\nFarm Ranch And Forest Retailers (18)\nMap of Farm Income Opportunities Organizations & Professionals serving Plumas County\nWarner Valley Wildlife Area - Plumas County\nShow Farm Resources Organizations & Professionals (297)\nThere are 297 resources serving Plumas County in the following categories:\nMap of Farm Resources Organizations & Professionals serving Plumas County\nThere are 57 resources serving Plumas County in the following categories:\nMap of Organic Certification Organizations & Professionals serving Plumas County', |
| | 'url': 'http://www.californialandcan.org/Plumas/Farm-Resources/'} |
| | ``` |
| |
|
| | ### Data Fields |
| |
|
| | The data have several fields: |
| |
|
| | - `url`: url of the source as a string |
| | - `text`: text content as a string |
| | - `timestamp`: timestamp as a string |
| |
|
| | ### Data Splits |
| |
|
| | To build mC4, the authors used [CLD3](https://github.com/google/cld3) to identify over 100 languages. The resulting mC4 subsets for each language are reported in this table: |
| |
|
| | | config | train | validation | |
| | |:---------|:--------|:-------------| |
| | | af | ? | ? | |
| | | am | ? | ? | |
| | | ar | ? | ? | |
| | | az | ? | ? | |
| | | be | ? | ? | |
| | | bg | ? | ? | |
| | | bg-Latn | ? | ? | |
| | | bn | ? | ? | |
| | | ca | ? | ? | |
| | | ceb | ? | ? | |
| | | co | ? | ? | |
| | | cs | ? | ? | |
| | | cy | ? | ? | |
| | | da | ? | ? | |
| | | de | ? | ? | |
| | | el | ? | ? | |
| | | el-Latn | ? | ? | |
| | | en | ? | ? | |
| | | eo | ? | ? | |
| | | es | ? | ? | |
| | | et | ? | ? | |
| | | eu | ? | ? | |
| | | fa | ? | ? | |
| | | fi | ? | ? | |
| | | fil | ? | ? | |
| | | fr | ? | ? | |
| | | fy | ? | ? | |
| | | ga | ? | ? | |
| | | gd | ? | ? | |
| | | gl | ? | ? | |
| | | gu | ? | ? | |
| | | ha | ? | ? | |
| | | haw | ? | ? | |
| | | hi | ? | ? | |
| | | hi-Latn | ? | ? | |
| | | hmn | ? | ? | |
| | | ht | ? | ? | |
| | | hu | ? | ? | |
| | | hy | ? | ? | |
| | | id | ? | ? | |
| | | ig | ? | ? | |
| | | is | ? | ? | |
| | | it | ? | ? | |
| | | iw | ? | ? | |
| | | ja | ? | ? | |
| | | ja-Latn | ? | ? | |
| | | jv | ? | ? | |
| | | ka | ? | ? | |
| | | kk | ? | ? | |
| | | km | ? | ? | |
| | | kn | ? | ? | |
| | | ko | ? | ? | |
| | | ku | ? | ? | |
| | | ky | ? | ? | |
| | | la | ? | ? | |
| | | lb | ? | ? | |
| | | lo | ? | ? | |
| | | lt | ? | ? | |
| | | lv | ? | ? | |
| | | mg | ? | ? | |
| | | mi | ? | ? | |
| | | mk | ? | ? | |
| | | ml | ? | ? | |
| | | mn | ? | ? | |
| | | mr | ? | ? | |
| | | ms | ? | ? | |
| | | mt | ? | ? | |
| | | my | ? | ? | |
| | | ne | ? | ? | |
| | | nl | ? | ? | |
| | | no | ? | ? | |
| | | ny | ? | ? | |
| | | pa | ? | ? | |
| | | pl | ? | ? | |
| | | ps | ? | ? | |
| | | pt | ? | ? | |
| | | ro | ? | ? | |
| | | ru | ? | ? | |
| | | ru-Latn | ? | ? | |
| | | sd | ? | ? | |
| | | si | ? | ? | |
| | | sk | ? | ? | |
| | | sl | ? | ? | |
| | | sm | ? | ? | |
| | | sn | ? | ? | |
| | | so | ? | ? | |
| | | sq | ? | ? | |
| | | sr | ? | ? | |
| | | st | ? | ? | |
| | | su | ? | ? | |
| | | sv | ? | ? | |
| | | sw | ? | ? | |
| | | ta | ? | ? | |
| | | te | ? | ? | |
| | | tg | ? | ? | |
| | | th | ? | ? | |
| | | tr | ? | ? | |
| | | uk | ? | ? | |
| | | und | ? | ? | |
| | | ur | ? | ? | |
| | | uz | ? | ? | |
| | | vi | ? | ? | |
| | | xh | ? | ? | |
| | | yi | ? | ? | |
| | | yo | ? | ? | |
| | | zh | ? | ? | |
| | | zh-Latn | ? | ? | |
| | | zu | ? | ? | |
| |
|
| | ## Dataset Creation |
| |
|
| | ### Curation Rationale |
| |
|
| | [More Information Needed] |
| |
|
| | ### Source Data |
| |
|
| | #### Initial Data Collection and Normalization |
| |
|
| | [More Information Needed] |
| |
|
| | #### Who are the source language producers? |
| |
|
| | [More Information Needed] |
| |
|
| | ### Annotations |
| |
|
| | #### Annotation process |
| |
|
| | [More Information Needed] |
| |
|
| | #### Who are the annotators? |
| |
|
| | [More Information Needed] |
| |
|
| | ### Personal and Sensitive Information |
| |
|
| | [More Information Needed] |
| |
|
| | ## Considerations for Using the Data |
| |
|
| | ### Social Impact of Dataset |
| |
|
| | [More Information Needed] |
| |
|
| | ### Discussion of Biases |
| |
|
| | [More Information Needed] |
| |
|
| | ### Other Known Limitations |
| |
|
| | [More Information Needed] |
| |
|
| | ## Additional Information |
| |
|
| | ### Dataset Curators |
| |
|
| | [More Information Needed] |
| |
|
| | ### Licensing Information |
| |
|
| | AllenAI are releasing this dataset under the terms of ODC-BY. By using this, you are also bound by the Common Crawl terms of use in respect of the content contained in the dataset. |
| |
|
| | ### Citation Information |
| |
|
| | ``` |
| | @article{2019t5, |
| | author = {Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu}, |
| | title = {Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer}, |
| | journal = {arXiv e-prints}, |
| | year = {2019}, |
| | archivePrefix = {arXiv}, |
| | eprint = {1910.10683}, |
| | } |
| | ``` |
| |
|
| | ### Contributions |
| |
|
| | Thanks to [@dirkgr](https://github.com/dirkgr) and [@lhoestq](https://github.com/lhoestq) for adding this dataset. |
| |
|
