Add files using upload-large-folder tool

hf_cache/hub/CACHEDIR.TAG

@@ -0,0 +1,4 @@
+Signature: 8a477f597d28d172789f06886806bc55
+# This file is a cache directory tag created by huggingface_hub.
+# For information about cache directory tags, see:
+#	https://bford.info/cachedir/

hf_cache/hub/datasets--EdinburghNLP--xsum/blobs/fee00d9f711981e883d7a06af05d4ee18b7fe5d9

@@ -0,0 +1,222 @@
+---
+annotations_creators:
+- found
+language_creators:
+- found
+language:
+- en
+license:
+- unknown
+multilinguality:
+- monolingual
+pretty_name: Extreme Summarization (XSum)
+paperswithcode_id: xsum
+size_categories:
+- 100K<n<1M
+source_datasets:
+- original
+task_categories:
+- summarization
+task_ids:
+- news-articles-summarization
+dataset_info:
+  features:
+  - name: document
+    dtype: string
+  - name: summary
+    dtype: string
+  - name: id
+    dtype: string
+  splits:
+  - name: train
+    num_bytes: 479206363
+    num_examples: 204045
+  - name: validation
+    num_bytes: 26292877
+    num_examples: 11332
+  - name: test
+    num_bytes: 26756141
+    num_examples: 11334
+  download_size: 332791351
+  dataset_size: 532255381
+configs:
+- config_name: default
+  data_files:
+  - split: train
+    path: data/train-*
+  - split: validation
+    path: data/validation-*
+  - split: test
+    path: data/test-*
+train-eval-index:
+- config: default
+  task: summarization
+  task_id: summarization
+  splits:
+    train_split: train
+    eval_split: test
+  col_mapping:
+    document: text
+    summary: target
+  metrics:
+  - type: rouge
+    name: Rouge
+---
+
+# Dataset Card for "xsum"
+
+## Table of Contents
+- [Dataset Description](#dataset-description)
+  - [Dataset Summary](#dataset-summary)
+  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
+  - [Languages](#languages)
+- [Dataset Structure](#dataset-structure)
+  - [Data Instances](#data-instances)
+  - [Data Fields](#data-fields)
+  - [Data Splits](#data-splits)
+- [Dataset Creation](#dataset-creation)
+  - [Curation Rationale](#curation-rationale)
+  - [Source Data](#source-data)
+  - [Annotations](#annotations)
+  - [Personal and Sensitive Information](#personal-and-sensitive-information)
+- [Considerations for Using the Data](#considerations-for-using-the-data)
+  - [Social Impact of Dataset](#social-impact-of-dataset)
+  - [Discussion of Biases](#discussion-of-biases)
+  - [Other Known Limitations](#other-known-limitations)
+- [Additional Information](#additional-information)
+  - [Dataset Curators](#dataset-curators)
+  - [Licensing Information](#licensing-information)
+  - [Citation Information](#citation-information)
+  - [Contributions](#contributions)
+
+## Dataset Description
+
+- **Homepage:**
+- **Repository:** https://github.com/EdinburghNLP/XSum
+- **Paper:** [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://arxiv.org/abs/1808.08745)
+- **Point of Contact:** [Shashi Narayan](mailto:shashi.narayan@ed.ac.uk)
+- **Size of downloaded dataset files:** 257.30 MB
+- **Size of the generated dataset:** 532.26 MB
+- **Total amount of disk used:** 789.56 MB
+
+### Dataset Summary
+
+Extreme Summarization (XSum) Dataset.
+
+There are three features:
+  - document: Input news article.
+  - summary: One sentence summary of the article.
+  - id: BBC ID of the article.
+
+### Supported Tasks and Leaderboards
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Languages
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Dataset Structure
+
+### Data Instances
+
+#### default
+
+- **Size of downloaded dataset files:** 257.30 MB
+- **Size of the generated dataset:** 532.26 MB
+- **Total amount of disk used:** 789.56 MB
+
+An example of 'validation' looks as follows.
+```
+{
+    "document": "some-body",
+    "id": "29750031",
+    "summary": "some-sentence"
+}
+```
+
+### Data Fields
+
+The data fields are the same among all splits.
+
+#### default
+- `document`: a `string` feature.
+- `summary`: a `string` feature.
+- `id`: a `string` feature.
+
+### Data Splits
+
+| name  |train |validation|test |
+|-------|-----:|---------:|----:|
+|default|204045|     11332|11334|
+
+## Dataset Creation
+
+### Curation Rationale
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Source Data
+
+#### Initial Data Collection and Normalization
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+#### Who are the source language producers?
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Annotations
+
+#### Annotation process
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+#### Who are the annotators?
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Personal and Sensitive Information
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Considerations for Using the Data
+
+### Social Impact of Dataset
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Discussion of Biases
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Other Known Limitations
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Additional Information
+
+### Dataset Curators
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Licensing Information
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Citation Information
+
+```
+@article{Narayan2018DontGM,
+  title={Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization},
+  author={Shashi Narayan and Shay B. Cohen and Mirella Lapata},
+  journal={ArXiv},
+  year={2018},
+  volume={abs/1808.08745}
+}
+```
+
+
+### Contributions
+
+Thanks to [@thomwolf](https://github.com/thomwolf), [@lewtun](https://github.com/lewtun), [@mariamabarham](https://github.com/mariamabarham), [@jbragg](https://github.com/jbragg), [@lhoestq](https://github.com/lhoestq), [@patrickvonplaten](https://github.com/patrickvonplaten) for adding this dataset.

hf_cache/hub/datasets--EdinburghNLP--xsum/refs/main

@@ -0,0 +1 @@
+7d4d486c2f8ef850b1a11aead99b894ff3dd7da9

hf_cache/hub/datasets--EdinburghNLP--xsum/snapshots/7d4d486c2f8ef850b1a11aead99b894ff3dd7da9/README.md

@@ -0,0 +1,222 @@
+---
+annotations_creators:
+- found
+language_creators:
+- found
+language:
+- en
+license:
+- unknown
+multilinguality:
+- monolingual
+pretty_name: Extreme Summarization (XSum)
+paperswithcode_id: xsum
+size_categories:
+- 100K<n<1M
+source_datasets:
+- original
+task_categories:
+- summarization
+task_ids:
+- news-articles-summarization
+dataset_info:
+  features:
+  - name: document
+    dtype: string
+  - name: summary
+    dtype: string
+  - name: id
+    dtype: string
+  splits:
+  - name: train
+    num_bytes: 479206363
+    num_examples: 204045
+  - name: validation
+    num_bytes: 26292877
+    num_examples: 11332
+  - name: test
+    num_bytes: 26756141
+    num_examples: 11334
+  download_size: 332791351
+  dataset_size: 532255381
+configs:
+- config_name: default
+  data_files:
+  - split: train
+    path: data/train-*
+  - split: validation
+    path: data/validation-*
+  - split: test
+    path: data/test-*
+train-eval-index:
+- config: default
+  task: summarization
+  task_id: summarization
+  splits:
+    train_split: train
+    eval_split: test
+  col_mapping:
+    document: text
+    summary: target
+  metrics:
+  - type: rouge
+    name: Rouge
+---
+
+# Dataset Card for "xsum"
+
+## Table of Contents
+- [Dataset Description](#dataset-description)
+  - [Dataset Summary](#dataset-summary)
+  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
+  - [Languages](#languages)
+- [Dataset Structure](#dataset-structure)
+  - [Data Instances](#data-instances)
+  - [Data Fields](#data-fields)
+  - [Data Splits](#data-splits)
+- [Dataset Creation](#dataset-creation)
+  - [Curation Rationale](#curation-rationale)
+  - [Source Data](#source-data)
+  - [Annotations](#annotations)
+  - [Personal and Sensitive Information](#personal-and-sensitive-information)
+- [Considerations for Using the Data](#considerations-for-using-the-data)
+  - [Social Impact of Dataset](#social-impact-of-dataset)
+  - [Discussion of Biases](#discussion-of-biases)
+  - [Other Known Limitations](#other-known-limitations)
+- [Additional Information](#additional-information)
+  - [Dataset Curators](#dataset-curators)
+  - [Licensing Information](#licensing-information)
+  - [Citation Information](#citation-information)
+  - [Contributions](#contributions)
+
+## Dataset Description
+
+- **Homepage:**
+- **Repository:** https://github.com/EdinburghNLP/XSum
+- **Paper:** [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://arxiv.org/abs/1808.08745)
+- **Point of Contact:** [Shashi Narayan](mailto:shashi.narayan@ed.ac.uk)
+- **Size of downloaded dataset files:** 257.30 MB
+- **Size of the generated dataset:** 532.26 MB
+- **Total amount of disk used:** 789.56 MB
+
+### Dataset Summary
+
+Extreme Summarization (XSum) Dataset.
+
+There are three features:
+  - document: Input news article.
+  - summary: One sentence summary of the article.
+  - id: BBC ID of the article.
+
+### Supported Tasks and Leaderboards
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Languages
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Dataset Structure
+
+### Data Instances
+
+#### default
+
+- **Size of downloaded dataset files:** 257.30 MB
+- **Size of the generated dataset:** 532.26 MB
+- **Total amount of disk used:** 789.56 MB
+
+An example of 'validation' looks as follows.
+```
+{
+    "document": "some-body",
+    "id": "29750031",
+    "summary": "some-sentence"
+}
+```
+
+### Data Fields
+
+The data fields are the same among all splits.
+
+#### default
+- `document`: a `string` feature.
+- `summary`: a `string` feature.
+- `id`: a `string` feature.
+
+### Data Splits
+
+| name  |train |validation|test |
+|-------|-----:|---------:|----:|
+|default|204045|     11332|11334|
+
+## Dataset Creation
+
+### Curation Rationale
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Source Data
+
+#### Initial Data Collection and Normalization
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+#### Who are the source language producers?
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Annotations
+
+#### Annotation process
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+#### Who are the annotators?
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Personal and Sensitive Information
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Considerations for Using the Data
+
+### Social Impact of Dataset
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Discussion of Biases
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Other Known Limitations
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Additional Information
+
+### Dataset Curators
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Licensing Information
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Citation Information
+
+```
+@article{Narayan2018DontGM,
+  title={Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization},
+  author={Shashi Narayan and Shay B. Cohen and Mirella Lapata},
+  journal={ArXiv},
+  year={2018},
+  volume={abs/1808.08745}
+}
+```
+
+
+### Contributions
+
+Thanks to [@thomwolf](https://github.com/thomwolf), [@lewtun](https://github.com/lewtun), [@mariamabarham](https://github.com/mariamabarham), [@jbragg](https://github.com/jbragg), [@lhoestq](https://github.com/lhoestq), [@patrickvonplaten](https://github.com/patrickvonplaten) for adding this dataset.

hf_cache/hub/datasets--HuggingFaceFW--fineweb-2/refs/main

@@ -0,0 +1 @@
+af9c13333eb981300149d5ca60a8e9d659b276b9

hf_cache/hub/datasets--HuggingFaceFW--fineweb-edu/blobs/16e64d907529c7f742578a2fee4b72039d64211d

@@ -0,0 +1,651 @@
+---
+license: odc-by
+task_categories:
+  - text-generation
+language:
+  - en
+pretty_name: FineWeb-Edu
+size_categories:
+  - n>1T
+configs:
+  - config_name: default
+    data_files:
+      - split: train
+        path: data/*/*
+    features:
+    - name: text
+      dtype: string
+    - name: id
+      dtype: string
+    - name: dump
+      dtype: string
+    - name: url
+      dtype: string
+    - name: date
+      dtype: string
+    - name: file_path
+      dtype: string
+    - name: language
+      dtype: string
+    - name: language_score
+      dtype: float64
+    - name: token_count
+      dtype: int64
+    - name: score
+      dtype: float64
+    - name: int_score
+      dtype: int64
+  - config_name: sample-10BT
+    data_files:
+      - split: train
+        path: sample/10BT/*
+  - config_name: sample-100BT
+    data_files:
+      - split: train
+        path: sample/100BT/*
+  - config_name: sample-350BT
+    data_files:
+      - split: train
+        path: sample/350BT/*
+  - config_name: CC-MAIN-2025-05
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2025-05/*
+  - config_name: CC-MAIN-2025-08
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2025-08/*
+  - config_name: CC-MAIN-2025-13
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2025-13/*
+  - config_name: CC-MAIN-2025-18
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2025-18/*
+  - config_name: CC-MAIN-2025-21
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2025-21/*
+  - config_name: CC-MAIN-2025-26
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2025-26/*
+  - config_name: CC-MAIN-2024-51
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2024-51/*
+  - config_name: CC-MAIN-2024-46
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2024-46/*
+  - config_name: CC-MAIN-2024-42
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2024-42/*
+  - config_name: CC-MAIN-2024-38
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2024-38/*
+  - config_name: CC-MAIN-2024-33
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2024-33/*
+  - config_name: CC-MAIN-2024-30
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2024-30/*
+  - config_name: CC-MAIN-2024-26
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2024-26/*
+  - config_name: CC-MAIN-2024-22
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2024-22/*
+  - config_name: CC-MAIN-2024-18
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2024-18/*
+  - config_name: CC-MAIN-2024-10
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2024-10/*
+  - config_name: CC-MAIN-2023-50
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2023-50/*
+  - config_name: CC-MAIN-2023-40
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2023-40/*
+  - config_name: CC-MAIN-2023-23
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2023-23/*
+  - config_name: CC-MAIN-2023-14
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2023-14/*
+  - config_name: CC-MAIN-2023-06
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2023-06/*
+  - config_name: CC-MAIN-2022-49
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2022-49/*
+  - config_name: CC-MAIN-2022-40
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2022-40/*
+  - config_name: CC-MAIN-2022-33
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2022-33/*
+  - config_name: CC-MAIN-2022-27
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2022-27/*
+  - config_name: CC-MAIN-2022-21
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2022-21/*
+  - config_name: CC-MAIN-2022-05
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2022-05/*
+  - config_name: CC-MAIN-2021-49
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2021-49/*
+  - config_name: CC-MAIN-2021-43
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2021-43/*
+  - config_name: CC-MAIN-2021-39
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2021-39/*
+  - config_name: CC-MAIN-2021-31
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2021-31/*
+  - config_name: CC-MAIN-2021-25
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2021-25/*
+  - config_name: CC-MAIN-2021-21
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2021-21/*
+  - config_name: CC-MAIN-2021-17
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2021-17/*
+  - config_name: CC-MAIN-2021-10
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2021-10/*
+  - config_name: CC-MAIN-2021-04
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2021-04/*
+  - config_name: CC-MAIN-2020-50
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2020-50/*
+  - config_name: CC-MAIN-2020-45
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2020-45/*
+  - config_name: CC-MAIN-2020-40
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2020-40/*
+  - config_name: CC-MAIN-2020-34
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2020-34/*
+  - config_name: CC-MAIN-2020-29
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2020-29/*
+  - config_name: CC-MAIN-2020-24
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2020-24/*
+  - config_name: CC-MAIN-2020-16
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2020-16/*
+  - config_name: CC-MAIN-2020-10
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2020-10/*
+  - config_name: CC-MAIN-2020-05
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2020-05/*
+  - config_name: CC-MAIN-2019-51
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2019-51/*
+  - config_name: CC-MAIN-2019-47
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2019-47/*
+  - config_name: CC-MAIN-2019-43
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2019-43/*
+  - config_name: CC-MAIN-2019-39
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2019-39/*
+  - config_name: CC-MAIN-2019-35
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2019-35/*
+  - config_name: CC-MAIN-2019-30
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2019-30/*
+  - config_name: CC-MAIN-2019-26
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2019-26/*
+  - config_name: CC-MAIN-2019-22
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2019-22/*
+  - config_name: CC-MAIN-2019-18
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2019-18/*
+  - config_name: CC-MAIN-2019-13
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2019-13/*
+  - config_name: CC-MAIN-2019-09
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2019-09/*
+  - config_name: CC-MAIN-2019-04
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2019-04/*
+  - config_name: CC-MAIN-2018-51
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2018-51/*
+  - config_name: CC-MAIN-2018-47
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2018-47/*
+  - config_name: CC-MAIN-2018-43
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2018-43/*
+  - config_name: CC-MAIN-2018-39
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2018-39/*
+  - config_name: CC-MAIN-2018-34
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2018-34/*
+  - config_name: CC-MAIN-2018-30
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2018-30/*
+  - config_name: CC-MAIN-2018-26
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2018-26/*
+  - config_name: CC-MAIN-2018-22
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2018-22/*
+  - config_name: CC-MAIN-2018-17
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2018-17/*
+  - config_name: CC-MAIN-2018-13
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2018-13/*
+  - config_name: CC-MAIN-2018-09
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2018-09/*
+  - config_name: CC-MAIN-2018-05
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2018-05/*
+  - config_name: CC-MAIN-2017-51
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2017-51/*
+  - config_name: CC-MAIN-2017-47
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2017-47/*
+  - config_name: CC-MAIN-2017-43
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2017-43/*
+  - config_name: CC-MAIN-2017-39
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2017-39/*
+  - config_name: CC-MAIN-2017-34
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2017-34/*
+  - config_name: CC-MAIN-2017-30
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2017-30/*
+  - config_name: CC-MAIN-2017-26
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2017-26/*
+  - config_name: CC-MAIN-2017-22
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2017-22/*
+  - config_name: CC-MAIN-2017-17
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2017-17/*
+  - config_name: CC-MAIN-2017-13
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2017-13/*
+  - config_name: CC-MAIN-2017-09
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2017-09/*
+  - config_name: CC-MAIN-2017-04
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2017-04/*
+  - config_name: CC-MAIN-2016-50
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2016-50/*
+  - config_name: CC-MAIN-2016-44
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2016-44/*
+  - config_name: CC-MAIN-2016-40
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2016-40/*
+  - config_name: CC-MAIN-2016-36
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2016-36/*
+  - config_name: CC-MAIN-2016-30
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2016-30/*
+  - config_name: CC-MAIN-2016-26
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2016-26/*
+  - config_name: CC-MAIN-2016-22
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2016-22/*
+  - config_name: CC-MAIN-2016-18
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2016-18/*
+  - config_name: CC-MAIN-2016-07
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2016-07/*
+  - config_name: CC-MAIN-2015-48
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2015-48/*
+  - config_name: CC-MAIN-2015-40
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2015-40/*
+  - config_name: CC-MAIN-2015-35
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2015-35/*
+  - config_name: CC-MAIN-2015-32
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2015-32/*
+  - config_name: CC-MAIN-2015-27
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2015-27/*
+  - config_name: CC-MAIN-2015-22
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2015-22/*
+  - config_name: CC-MAIN-2015-18
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2015-18/*
+  - config_name: CC-MAIN-2015-14
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2015-14/*
+  - config_name: CC-MAIN-2015-11
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2015-11/*
+  - config_name: CC-MAIN-2015-06
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2015-06/*
+  - config_name: CC-MAIN-2014-52
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2014-52/*
+  - config_name: CC-MAIN-2014-49
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2014-49/*
+  - config_name: CC-MAIN-2014-42
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2014-42/*
+  - config_name: CC-MAIN-2014-41
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2014-41/*
+  - config_name: CC-MAIN-2014-35
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2014-35/*
+  - config_name: CC-MAIN-2014-23
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2014-23/*
+  - config_name: CC-MAIN-2014-15
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2014-15/*
+  - config_name: CC-MAIN-2014-10
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2014-10/*
+  - config_name: CC-MAIN-2013-48
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2013-48/*
+  - config_name: CC-MAIN-2013-20
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2013-20/*
+---
+
+# 📚 FineWeb-Edu 
+<center>
+    <img src="https://cdn-uploads.huggingface.co/production/uploads/61c141342aac764ce1654e43/wwRnEQydH9qdRtFofIE-A.png" alt="FineWeb-Edu: The finest collection of educational content the web has to offer">
+</center>
+
+> 1.3 trillion tokens of the finest educational data the 🌐 web has to offer
+
+**Paper:** https://arxiv.org/abs/2406.17557
+
+## What is it?
+
+📚 FineWeb-Edu  dataset consists of **1.3T tokens**  and  **5.4T tokens** ([FineWeb-Edu-score-2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu-score-2)) of educational web pages filtered from 🍷 FineWeb dataset. This is the 1.3 trillion version.
+
+To enhance FineWeb's quality, we developed an [educational quality classifier](https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier) using annotations generated by LLama3-70B-Instruct. We then used this classifier to retain only the most educational web pages. FineWeb-Edu outperforms FineWeb on popular benchmarks and shows the power of classifiers trained on synthetic data. 
+
+The [Dataset Curation](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu#dataset-curation) section details the process for creating the dataset.
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/61c141342aac764ce1654e43/QqXOM8h_ZjjhuCv71xmV7.png)
+
+You can find a deduplicated version of FineWeb-edu in [SmolLM-Corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus). We find that the deduplication of this dataset doesn't have any impact on model performance in our ablation setup (1.8B trained on 350B tokens).
+
+## What is being released?
+
+Along with the dataset, which includes all filtered CommonCrawl dumps since 2013, we also release the educational classifier used for the filtering as well as the code for training it and running inference at: https://github.com/huggingface/cosmopedia/tree/main/classification 
+
+## Changelog
+_Previous versions remain available in the branch `version name`._
+
+- **v1.4.0 (11-07-2025):** Added 6 new snapshots: `CC-MAIN-2025-05`, `CC-MAIN-2025-08`, `CC-MAIN-2025-13`, `CC-MAIN-2025-18`, `CC-MAIN-2025-21`, and `CC-MAIN-2025-26` (January to June 2025)
+- **v1.3.0 (31-01-2025):** Fixed an issue with some dumps where some documents hadn't been processed: `CC-MAIN-2024-10`, `CC-MAIN-2024-18`, `CC-MAIN-2024-22`, `CC-MAIN-2024-26`, `CC-MAIN-2024-30`, `CC-MAIN-2024-33`, `CC-MAIN-2024-38`, `CC-MAIN-2024-42`, `CC-MAIN-2024-46` -- they now contain more data (~35B additional tokens).
+- **v1.2.0 (03-01-2025):** Added 9 new snapshots: `CC-MAIN-2024-18`, `CC-MAIN-2024-22`, `CC-MAIN-2024-26`, `CC-MAIN-2024-30`, `CC-MAIN-2024-33`, `CC-MAIN-2024-38`, `CC-MAIN-2024-42`, `CC-MAIN-2024-46`, `CC-MAIN-2024-51`, covering April to December 2024.
+- **v1.0.0 (02-06-2024):** Initial version
+
+
+## How to load the dataset
+Similarily to FineWeb, You can load the full dataset or a specific crawl/dump. Dumps have the format `CC-MAIN-(year)-(week number)`.
+
+### (Smaller) sample versions
+Along with config `default` (all the data), and the configs for each individual dump, you can also download the following configs:
+- `sample-350BT`: a subset randomly sampled from the whole dataset of around 350B gpt2 tokens 
+- `sample-100BT`: a subset randomly sampled from the whole dataset of around 100B gpt2 tokens 
+- `sample-10BT`: a subset randomly sampled from the whole dataset of around 10B gpt2 tokens 
+
+`sample-10BT` was sampled from `sample-100BT` which in turn was sampled from `sample-350BT`.
+
+### Using 🏭 [`datatrove`](https://github.com/huggingface/datatrove/)
+
+```python
+from datatrove.pipeline.readers import ParquetReader
+
+# limit determines how many documents will be streamed (remove for all)
+data_reader = ParquetReader("hf://datasets/HuggingFaceFW/fineweb-edu", glob_pattern="data/*/*.parquet", limit=1000)
+# or to fetch a specific dump CC-MAIN-2024-10,  eplace "CC-MAIN-2024-10" with "sample/100BT" to use the 100BT sample
+data_reader = ParquetReader("hf://datasets/HuggingFaceFW/fineweb-edu/CC-MAIN-2024-10", limit=1000) 
+for document in data_reader():
+    # do something with document
+    print(document)
+
+###############################    
+# OR for a processing pipeline:
+###############################
+
+from datatrove.executor import LocalPipelineExecutor
+from datatrove.pipeline.readers import ParquetReader
+from datatrove.pipeline.filters import LambdaFilter
+from datatrove.pipeline.writers import JsonlWriter
+
+pipeline_exec = LocalPipelineExecutor(
+    pipeline=[
+        # replace "CC-MAIN-2024-10" with "sample/100BT" to use the 100BT sample
+        ParquetReader("hf://datasets/HuggingFaceFW/fineweb-edu/CC-MAIN-2024-10", limit=1000),
+        LambdaFilter(lambda doc: "hugging" in doc.text),
+        JsonlWriter("some-output-path")
+    ],
+    tasks=10
+)
+pipeline_exec.run()
+```
+
+### Using `datasets`
+
+```python
+from datasets import load_dataset
+# use name="sample-10BT" to use the 10BT sample
+fw = load_dataset("HuggingFaceFW/fineweb-edu", name="CC-MAIN-2024-10", split="train", streaming=True)
+```
+
+## Dataset curation
+A new approach has recently emerged for filtering LLM training datasets: using synthetic data to develop classifiers for identifying educational content. This technique was used in the trainings of [LLama3](https://ai.meta.com/blog/meta-llama-3-meta-ai-responsibility/) and [Phi3](https://arxiv.org/abs/2404.14219), but its large-scale impact on web data filtering hasn't been fully explored or published.
+
+The highly popular Phi3 models were trained on 3.3 and 4.8 trillion tokens, with the paper stating: “Our training data consists of heavily filtered publicly available web data (according to the 'educational level') from various open internet sources, as well as synthetic LLM-generated data". Similarly, the LLama3 blog post notes: “We found that previous generations of Llama are good at identifying high-quality data, so we used Llama 2 to help build the text-quality classifiers that are powering Llama 3.” However these classifiers and filtered datasets are not publicly available. To enhance FineWeb's quality, we developed an educational quality classifier using annotations generated by [LLama3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) to create FineWeb-Edu.
+
+### Annotation
+We used [Llama3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) to score 500k FineWeb samples for their educational quality on a scale from 0 to 5.
+
+We explored various prompts and found that the additive scale by [Yuan et al.](https://arxiv.org/pdf/2401.10020) worked best. To avoid the LLM favoring highly technical pages like arXiv abstracts and submissions, we focused on grade-school and middle-school level knowledge. By setting a threshold of 3 (on a scale of 0 to 5) during the filtering process, we were able to also retain some high-level educational pages. The final prompt can be found [here](https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier/blob/main/utils/prompt.txt).
+
+We also experimented with different LLMs: Llama3-70B-Instruct, Mixtral-8x-7B-Instruct, and Mixtral-8x22B-Instruct. Llama 3 and Mixtral-8x22B produced similar scores, while Mixtral-8x7B tended to be more generous, not fully adhering to the score scale. Verga et al. suggest using multiple LLMs as juries. We tried averaging the scores from the three models, but this shifted the distribution to the right due to the higher scores from Mixtral-8x7B. Training on a dataset filtered with a classifier using jury annotations performed worse than using a classifier based on Llama3 annotations. We hypothesize that the jury-based approach retains more low-quality samples.
+
+### Classifier training
+We fine-tuned a Bert-like regression model using these annotations, based on [Snowflake-arctic-embed](https://huggingface.co/Snowflake/snowflake-arctic-embed-m). When converted to a binary classification  using a score of 3 as a threshold for keeping and removing files, the model achieved an F1 score of 82%. The classification of FineWeb 15T tokens took 6k H100 GPU hours.
+
+The classifier is available at: [HuggingFaceFW/fineweb-edu-classifier/](https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier/)
+
+### Filtering and results
+**Note**: You can find more details about the ablations and results in the FineWeb [blog post](https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1).
+
+We investigated the impact of using different thresholds for the filtering and found that threshold 3 gave the best overall results. Although using a threshold higher than 3 improves performance on knowledge and reasoning intensive benchmarks, it significantly degrades performance on HellaSwag and PIQA.
+
+We then built 📚 FineWeb-Edu by filtering out samples with scores lower than 3. This removed 92% of the dataset, leaving us with 1.3T educational tokens. Our ablation demonstrated that this refined dataset surpasses 🍷 FineWeb and all other open web datasets, with remarkable improvements on educational benchmarks such as MMLU, ARC, and OpenBookQA. The plot below compares FineWeb-Edu to other web datasets:
+ 
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/61c141342aac764ce1654e43/hJlyTgDzZpYuxO9LUm0PF.png)
+
+To retain more tokens, we also experimented with a less strict threshold of 2 instead of 3. While being less performant than using threshold 3, it still outperformed FineWeb and it preserved 5.4T tokens. We release these two dataset as [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) and [FineWeb-Edu-score-2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu-score-2) along with the [classifier](https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier).
+
+You will find all the ablation models in [this collection](https://huggingface.co/collections/HuggingFaceFW/ablation-models-662457b0d213e8c14fe47f32). The FineWeb-Edu ablation model (trained on 350B tokens) is available at [https://huggingface.co/HuggingFaceFW/ablation-model-fineweb-edu](https://huggingface.co/HuggingFaceFW/ablation-model-fineweb-edu).
+
+## Considerations for Using the Data
+This section is copied from the parent dataset: [FineWeb](https://huggingface.co/datasets/HuggingFaceFW/fineweb).
+
+### Social Impact of Dataset
+
+With the release of this dataset we aim to make model training more accessible to the machine learning community at large. 
+
+While multiple open-weights models with strong performance have been publicly released in the past, more often than not these releases are not accompanied by the corresponding training dataset. This is unfortunate as the dataset specificities and characteristics have been demonstrated to have a very large impact and role in the performances of the models. As the creation of a high quality training dataset is a fundamental requirement to training an LLM capable of excelling at downstream tasks, with 🍷 FineWeb we (a) not only make the dataset creation process more transparent, by sharing our entire processing setup including the codebase used, we also (b) help alleviate the costs of dataset curation, both in time and in compute, for model creators by publicly releasing our dataset with the community.
+
+### Discussion of Biases
+
+Efforts were made to minimize the amount of NSFW and toxic content present in the dataset by employing filtering on the URL level. However, there are still a significant number of documents present in the final dataset that could be considered toxic or contain harmful content. As 🍷 FineWeb was sourced from the web as a whole, any harmful biases typically present in it may be reproduced on our dataset.
+
+We deliberately avoided using machine learning filtering methods that define text quality based on the similarity to a “gold” source such as wikipedia or toxicity classifiers as these methods have been known to [disproportionately remove content in specific dialects](https://aclanthology.org/D16-1120/) and [overclassify as toxic text related to specific social identities](https://arxiv.org/pdf/2109.07445.pdf), respectively.
+
+### Other Known Limitations
+
+As a consequence of some of the filtering steps applied, it is likely that code content is not prevalent in our dataset. If you are training a model that should also perform code tasks, we recommend you use 🍷 FineWeb with a code dataset, such as [The Stack v2](https://huggingface.co/datasets/bigcode/the-stack-v2). You should also probably consider complementing 🍷 FineWeb with specialized curated sources (such as Wikipedia, for example) as they will likely have better formatting than the wikipedia content included in 🍷 FineWeb (we did not tailor the processing to individual websites).
+
+## Additional Information
+
+### Licensing Information
+
+The dataset is released under the **Open Data Commons Attribution License (ODC-By) v1.0** [license](https://opendatacommons.org/licenses/by/1-0/). The use of this dataset is also subject to [CommonCrawl's Terms of Use](https://commoncrawl.org/terms-of-use).
+
+### Future work
+
+We plan to work on better educational classifier to improve the quality of FineWeb-Edu.
+
+### Citation Information
+
+You can cite our paper https://arxiv.org/abs/2406.17557 or this dataset:
+
+```
+@misc{lozhkov2024fineweb-edu,
+    author       = { Lozhkov, Anton and Ben Allal, Loubna and von Werra, Leandro and Wolf, Thomas },  
+    title        = { FineWeb-Edu: the Finest Collection of Educational Content }, 
+    year         = 2024,  
+    url          = { https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu },  
+    doi          = { 10.57967/hf/2497 },
+	publisher    = { Hugging Face }
+}
+```

hf_cache/hub/datasets--HuggingFaceFW--fineweb-edu/refs/main

@@ -0,0 +1 @@
+87f09149ef4734204d70ed1d046ddc9ca3f2b8f9

hf_cache/hub/datasets--HuggingFaceFW--fineweb-edu/snapshots/87f09149ef4734204d70ed1d046ddc9ca3f2b8f9/README.md

@@ -0,0 +1,651 @@
+---
+license: odc-by
+task_categories:
+  - text-generation
+language:
+  - en
+pretty_name: FineWeb-Edu
+size_categories:
+  - n>1T
+configs:
+  - config_name: default
+    data_files:
+      - split: train
+        path: data/*/*
+    features:
+    - name: text
+      dtype: string
+    - name: id
+      dtype: string
+    - name: dump
+      dtype: string
+    - name: url
+      dtype: string
+    - name: date
+      dtype: string
+    - name: file_path
+      dtype: string
+    - name: language
+      dtype: string
+    - name: language_score
+      dtype: float64
+    - name: token_count
+      dtype: int64
+    - name: score
+      dtype: float64
+    - name: int_score
+      dtype: int64
+  - config_name: sample-10BT
+    data_files:
+      - split: train
+        path: sample/10BT/*
+  - config_name: sample-100BT
+    data_files:
+      - split: train
+        path: sample/100BT/*
+  - config_name: sample-350BT
+    data_files:
+      - split: train
+        path: sample/350BT/*
+  - config_name: CC-MAIN-2025-05
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2025-05/*
+  - config_name: CC-MAIN-2025-08
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2025-08/*
+  - config_name: CC-MAIN-2025-13
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2025-13/*
+  - config_name: CC-MAIN-2025-18
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2025-18/*
+  - config_name: CC-MAIN-2025-21
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2025-21/*
+  - config_name: CC-MAIN-2025-26
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2025-26/*
+  - config_name: CC-MAIN-2024-51
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2024-51/*
+  - config_name: CC-MAIN-2024-46
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2024-46/*
+  - config_name: CC-MAIN-2024-42
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2024-42/*
+  - config_name: CC-MAIN-2024-38
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2024-38/*
+  - config_name: CC-MAIN-2024-33
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2024-33/*
+  - config_name: CC-MAIN-2024-30
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2024-30/*
+  - config_name: CC-MAIN-2024-26
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2024-26/*
+  - config_name: CC-MAIN-2024-22
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2024-22/*
+  - config_name: CC-MAIN-2024-18
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2024-18/*
+  - config_name: CC-MAIN-2024-10
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2024-10/*
+  - config_name: CC-MAIN-2023-50
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2023-50/*
+  - config_name: CC-MAIN-2023-40
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2023-40/*
+  - config_name: CC-MAIN-2023-23
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2023-23/*
+  - config_name: CC-MAIN-2023-14
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2023-14/*
+  - config_name: CC-MAIN-2023-06
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2023-06/*
+  - config_name: CC-MAIN-2022-49
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2022-49/*
+  - config_name: CC-MAIN-2022-40
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2022-40/*
+  - config_name: CC-MAIN-2022-33
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2022-33/*
+  - config_name: CC-MAIN-2022-27
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2022-27/*
+  - config_name: CC-MAIN-2022-21
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2022-21/*
+  - config_name: CC-MAIN-2022-05
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2022-05/*
+  - config_name: CC-MAIN-2021-49
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2021-49/*
+  - config_name: CC-MAIN-2021-43
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2021-43/*
+  - config_name: CC-MAIN-2021-39
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2021-39/*
+  - config_name: CC-MAIN-2021-31
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2021-31/*
+  - config_name: CC-MAIN-2021-25
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2021-25/*
+  - config_name: CC-MAIN-2021-21
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2021-21/*
+  - config_name: CC-MAIN-2021-17
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2021-17/*
+  - config_name: CC-MAIN-2021-10
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2021-10/*
+  - config_name: CC-MAIN-2021-04
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2021-04/*
+  - config_name: CC-MAIN-2020-50
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2020-50/*
+  - config_name: CC-MAIN-2020-45
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2020-45/*
+  - config_name: CC-MAIN-2020-40
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2020-40/*
+  - config_name: CC-MAIN-2020-34
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2020-34/*
+  - config_name: CC-MAIN-2020-29
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2020-29/*
+  - config_name: CC-MAIN-2020-24
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2020-24/*
+  - config_name: CC-MAIN-2020-16
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2020-16/*
+  - config_name: CC-MAIN-2020-10
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2020-10/*
+  - config_name: CC-MAIN-2020-05
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2020-05/*
+  - config_name: CC-MAIN-2019-51
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2019-51/*
+  - config_name: CC-MAIN-2019-47
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2019-47/*
+  - config_name: CC-MAIN-2019-43
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2019-43/*
+  - config_name: CC-MAIN-2019-39
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2019-39/*
+  - config_name: CC-MAIN-2019-35
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2019-35/*
+  - config_name: CC-MAIN-2019-30
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2019-30/*
+  - config_name: CC-MAIN-2019-26
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2019-26/*
+  - config_name: CC-MAIN-2019-22
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2019-22/*
+  - config_name: CC-MAIN-2019-18
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2019-18/*
+  - config_name: CC-MAIN-2019-13
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2019-13/*
+  - config_name: CC-MAIN-2019-09
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2019-09/*
+  - config_name: CC-MAIN-2019-04
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2019-04/*
+  - config_name: CC-MAIN-2018-51
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2018-51/*
+  - config_name: CC-MAIN-2018-47
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2018-47/*
+  - config_name: CC-MAIN-2018-43
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2018-43/*
+  - config_name: CC-MAIN-2018-39
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2018-39/*
+  - config_name: CC-MAIN-2018-34
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2018-34/*
+  - config_name: CC-MAIN-2018-30
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2018-30/*
+  - config_name: CC-MAIN-2018-26
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2018-26/*
+  - config_name: CC-MAIN-2018-22
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2018-22/*
+  - config_name: CC-MAIN-2018-17
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2018-17/*
+  - config_name: CC-MAIN-2018-13
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2018-13/*
+  - config_name: CC-MAIN-2018-09
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2018-09/*
+  - config_name: CC-MAIN-2018-05
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2018-05/*
+  - config_name: CC-MAIN-2017-51
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2017-51/*
+  - config_name: CC-MAIN-2017-47
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2017-47/*
+  - config_name: CC-MAIN-2017-43
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2017-43/*
+  - config_name: CC-MAIN-2017-39
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2017-39/*
+  - config_name: CC-MAIN-2017-34
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2017-34/*
+  - config_name: CC-MAIN-2017-30
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2017-30/*
+  - config_name: CC-MAIN-2017-26
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2017-26/*
+  - config_name: CC-MAIN-2017-22
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2017-22/*
+  - config_name: CC-MAIN-2017-17
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2017-17/*
+  - config_name: CC-MAIN-2017-13
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2017-13/*
+  - config_name: CC-MAIN-2017-09
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2017-09/*
+  - config_name: CC-MAIN-2017-04
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2017-04/*
+  - config_name: CC-MAIN-2016-50
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2016-50/*
+  - config_name: CC-MAIN-2016-44
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2016-44/*
+  - config_name: CC-MAIN-2016-40
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2016-40/*
+  - config_name: CC-MAIN-2016-36
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2016-36/*
+  - config_name: CC-MAIN-2016-30
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2016-30/*
+  - config_name: CC-MAIN-2016-26
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2016-26/*
+  - config_name: CC-MAIN-2016-22
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2016-22/*
+  - config_name: CC-MAIN-2016-18
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2016-18/*
+  - config_name: CC-MAIN-2016-07
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2016-07/*
+  - config_name: CC-MAIN-2015-48
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2015-48/*
+  - config_name: CC-MAIN-2015-40
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2015-40/*
+  - config_name: CC-MAIN-2015-35
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2015-35/*
+  - config_name: CC-MAIN-2015-32
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2015-32/*
+  - config_name: CC-MAIN-2015-27
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2015-27/*
+  - config_name: CC-MAIN-2015-22
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2015-22/*
+  - config_name: CC-MAIN-2015-18
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2015-18/*
+  - config_name: CC-MAIN-2015-14
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2015-14/*
+  - config_name: CC-MAIN-2015-11
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2015-11/*
+  - config_name: CC-MAIN-2015-06
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2015-06/*
+  - config_name: CC-MAIN-2014-52
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2014-52/*
+  - config_name: CC-MAIN-2014-49
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2014-49/*
+  - config_name: CC-MAIN-2014-42
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2014-42/*
+  - config_name: CC-MAIN-2014-41
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2014-41/*
+  - config_name: CC-MAIN-2014-35
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2014-35/*
+  - config_name: CC-MAIN-2014-23
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2014-23/*
+  - config_name: CC-MAIN-2014-15
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2014-15/*
+  - config_name: CC-MAIN-2014-10
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2014-10/*
+  - config_name: CC-MAIN-2013-48
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2013-48/*
+  - config_name: CC-MAIN-2013-20
+    data_files:
+      - split: train
+        path: data/CC-MAIN-2013-20/*
+---
+
+# 📚 FineWeb-Edu 
+<center>
+    <img src="https://cdn-uploads.huggingface.co/production/uploads/61c141342aac764ce1654e43/wwRnEQydH9qdRtFofIE-A.png" alt="FineWeb-Edu: The finest collection of educational content the web has to offer">
+</center>
+
+> 1.3 trillion tokens of the finest educational data the 🌐 web has to offer
+
+**Paper:** https://arxiv.org/abs/2406.17557
+
+## What is it?
+
+📚 FineWeb-Edu  dataset consists of **1.3T tokens**  and  **5.4T tokens** ([FineWeb-Edu-score-2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu-score-2)) of educational web pages filtered from 🍷 FineWeb dataset. This is the 1.3 trillion version.
+
+To enhance FineWeb's quality, we developed an [educational quality classifier](https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier) using annotations generated by LLama3-70B-Instruct. We then used this classifier to retain only the most educational web pages. FineWeb-Edu outperforms FineWeb on popular benchmarks and shows the power of classifiers trained on synthetic data. 
+
+The [Dataset Curation](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu#dataset-curation) section details the process for creating the dataset.
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/61c141342aac764ce1654e43/QqXOM8h_ZjjhuCv71xmV7.png)
+
+You can find a deduplicated version of FineWeb-edu in [SmolLM-Corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus). We find that the deduplication of this dataset doesn't have any impact on model performance in our ablation setup (1.8B trained on 350B tokens).
+
+## What is being released?
+
+Along with the dataset, which includes all filtered CommonCrawl dumps since 2013, we also release the educational classifier used for the filtering as well as the code for training it and running inference at: https://github.com/huggingface/cosmopedia/tree/main/classification 
+
+## Changelog
+_Previous versions remain available in the branch `version name`._
+
+- **v1.4.0 (11-07-2025):** Added 6 new snapshots: `CC-MAIN-2025-05`, `CC-MAIN-2025-08`, `CC-MAIN-2025-13`, `CC-MAIN-2025-18`, `CC-MAIN-2025-21`, and `CC-MAIN-2025-26` (January to June 2025)
+- **v1.3.0 (31-01-2025):** Fixed an issue with some dumps where some documents hadn't been processed: `CC-MAIN-2024-10`, `CC-MAIN-2024-18`, `CC-MAIN-2024-22`, `CC-MAIN-2024-26`, `CC-MAIN-2024-30`, `CC-MAIN-2024-33`, `CC-MAIN-2024-38`, `CC-MAIN-2024-42`, `CC-MAIN-2024-46` -- they now contain more data (~35B additional tokens).
+- **v1.2.0 (03-01-2025):** Added 9 new snapshots: `CC-MAIN-2024-18`, `CC-MAIN-2024-22`, `CC-MAIN-2024-26`, `CC-MAIN-2024-30`, `CC-MAIN-2024-33`, `CC-MAIN-2024-38`, `CC-MAIN-2024-42`, `CC-MAIN-2024-46`, `CC-MAIN-2024-51`, covering April to December 2024.
+- **v1.0.0 (02-06-2024):** Initial version
+
+
+## How to load the dataset
+Similarily to FineWeb, You can load the full dataset or a specific crawl/dump. Dumps have the format `CC-MAIN-(year)-(week number)`.
+
+### (Smaller) sample versions
+Along with config `default` (all the data), and the configs for each individual dump, you can also download the following configs:
+- `sample-350BT`: a subset randomly sampled from the whole dataset of around 350B gpt2 tokens 
+- `sample-100BT`: a subset randomly sampled from the whole dataset of around 100B gpt2 tokens 
+- `sample-10BT`: a subset randomly sampled from the whole dataset of around 10B gpt2 tokens 
+
+`sample-10BT` was sampled from `sample-100BT` which in turn was sampled from `sample-350BT`.
+
+### Using 🏭 [`datatrove`](https://github.com/huggingface/datatrove/)
+
+```python
+from datatrove.pipeline.readers import ParquetReader
+
+# limit determines how many documents will be streamed (remove for all)
+data_reader = ParquetReader("hf://datasets/HuggingFaceFW/fineweb-edu", glob_pattern="data/*/*.parquet", limit=1000)
+# or to fetch a specific dump CC-MAIN-2024-10,  eplace "CC-MAIN-2024-10" with "sample/100BT" to use the 100BT sample
+data_reader = ParquetReader("hf://datasets/HuggingFaceFW/fineweb-edu/CC-MAIN-2024-10", limit=1000) 
+for document in data_reader():
+    # do something with document
+    print(document)
+
+###############################    
+# OR for a processing pipeline:
+###############################
+
+from datatrove.executor import LocalPipelineExecutor
+from datatrove.pipeline.readers import ParquetReader
+from datatrove.pipeline.filters import LambdaFilter
+from datatrove.pipeline.writers import JsonlWriter
+
+pipeline_exec = LocalPipelineExecutor(
+    pipeline=[
+        # replace "CC-MAIN-2024-10" with "sample/100BT" to use the 100BT sample
+        ParquetReader("hf://datasets/HuggingFaceFW/fineweb-edu/CC-MAIN-2024-10", limit=1000),
+        LambdaFilter(lambda doc: "hugging" in doc.text),
+        JsonlWriter("some-output-path")
+    ],
+    tasks=10
+)
+pipeline_exec.run()
+```
+
+### Using `datasets`
+
+```python
+from datasets import load_dataset
+# use name="sample-10BT" to use the 10BT sample
+fw = load_dataset("HuggingFaceFW/fineweb-edu", name="CC-MAIN-2024-10", split="train", streaming=True)
+```
+
+## Dataset curation
+A new approach has recently emerged for filtering LLM training datasets: using synthetic data to develop classifiers for identifying educational content. This technique was used in the trainings of [LLama3](https://ai.meta.com/blog/meta-llama-3-meta-ai-responsibility/) and [Phi3](https://arxiv.org/abs/2404.14219), but its large-scale impact on web data filtering hasn't been fully explored or published.
+
+The highly popular Phi3 models were trained on 3.3 and 4.8 trillion tokens, with the paper stating: “Our training data consists of heavily filtered publicly available web data (according to the 'educational level') from various open internet sources, as well as synthetic LLM-generated data". Similarly, the LLama3 blog post notes: “We found that previous generations of Llama are good at identifying high-quality data, so we used Llama 2 to help build the text-quality classifiers that are powering Llama 3.” However these classifiers and filtered datasets are not publicly available. To enhance FineWeb's quality, we developed an educational quality classifier using annotations generated by [LLama3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) to create FineWeb-Edu.
+
+### Annotation
+We used [Llama3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) to score 500k FineWeb samples for their educational quality on a scale from 0 to 5.
+
+We explored various prompts and found that the additive scale by [Yuan et al.](https://arxiv.org/pdf/2401.10020) worked best. To avoid the LLM favoring highly technical pages like arXiv abstracts and submissions, we focused on grade-school and middle-school level knowledge. By setting a threshold of 3 (on a scale of 0 to 5) during the filtering process, we were able to also retain some high-level educational pages. The final prompt can be found [here](https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier/blob/main/utils/prompt.txt).
+
+We also experimented with different LLMs: Llama3-70B-Instruct, Mixtral-8x-7B-Instruct, and Mixtral-8x22B-Instruct. Llama 3 and Mixtral-8x22B produced similar scores, while Mixtral-8x7B tended to be more generous, not fully adhering to the score scale. Verga et al. suggest using multiple LLMs as juries. We tried averaging the scores from the three models, but this shifted the distribution to the right due to the higher scores from Mixtral-8x7B. Training on a dataset filtered with a classifier using jury annotations performed worse than using a classifier based on Llama3 annotations. We hypothesize that the jury-based approach retains more low-quality samples.
+
+### Classifier training
+We fine-tuned a Bert-like regression model using these annotations, based on [Snowflake-arctic-embed](https://huggingface.co/Snowflake/snowflake-arctic-embed-m). When converted to a binary classification  using a score of 3 as a threshold for keeping and removing files, the model achieved an F1 score of 82%. The classification of FineWeb 15T tokens took 6k H100 GPU hours.
+
+The classifier is available at: [HuggingFaceFW/fineweb-edu-classifier/](https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier/)
+
+### Filtering and results
+**Note**: You can find more details about the ablations and results in the FineWeb [blog post](https://huggingface.co/spaces/HuggingFaceFW/blogpost-fineweb-v1).
+
+We investigated the impact of using different thresholds for the filtering and found that threshold 3 gave the best overall results. Although using a threshold higher than 3 improves performance on knowledge and reasoning intensive benchmarks, it significantly degrades performance on HellaSwag and PIQA.
+
+We then built 📚 FineWeb-Edu by filtering out samples with scores lower than 3. This removed 92% of the dataset, leaving us with 1.3T educational tokens. Our ablation demonstrated that this refined dataset surpasses 🍷 FineWeb and all other open web datasets, with remarkable improvements on educational benchmarks such as MMLU, ARC, and OpenBookQA. The plot below compares FineWeb-Edu to other web datasets:
+ 
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/61c141342aac764ce1654e43/hJlyTgDzZpYuxO9LUm0PF.png)
+
+To retain more tokens, we also experimented with a less strict threshold of 2 instead of 3. While being less performant than using threshold 3, it still outperformed FineWeb and it preserved 5.4T tokens. We release these two dataset as [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) and [FineWeb-Edu-score-2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu-score-2) along with the [classifier](https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier).
+
+You will find all the ablation models in [this collection](https://huggingface.co/collections/HuggingFaceFW/ablation-models-662457b0d213e8c14fe47f32). The FineWeb-Edu ablation model (trained on 350B tokens) is available at [https://huggingface.co/HuggingFaceFW/ablation-model-fineweb-edu](https://huggingface.co/HuggingFaceFW/ablation-model-fineweb-edu).
+
+## Considerations for Using the Data
+This section is copied from the parent dataset: [FineWeb](https://huggingface.co/datasets/HuggingFaceFW/fineweb).
+
+### Social Impact of Dataset
+
+With the release of this dataset we aim to make model training more accessible to the machine learning community at large. 
+
+While multiple open-weights models with strong performance have been publicly released in the past, more often than not these releases are not accompanied by the corresponding training dataset. This is unfortunate as the dataset specificities and characteristics have been demonstrated to have a very large impact and role in the performances of the models. As the creation of a high quality training dataset is a fundamental requirement to training an LLM capable of excelling at downstream tasks, with 🍷 FineWeb we (a) not only make the dataset creation process more transparent, by sharing our entire processing setup including the codebase used, we also (b) help alleviate the costs of dataset curation, both in time and in compute, for model creators by publicly releasing our dataset with the community.
+
+### Discussion of Biases
+
+Efforts were made to minimize the amount of NSFW and toxic content present in the dataset by employing filtering on the URL level. However, there are still a significant number of documents present in the final dataset that could be considered toxic or contain harmful content. As 🍷 FineWeb was sourced from the web as a whole, any harmful biases typically present in it may be reproduced on our dataset.
+
+We deliberately avoided using machine learning filtering methods that define text quality based on the similarity to a “gold” source such as wikipedia or toxicity classifiers as these methods have been known to [disproportionately remove content in specific dialects](https://aclanthology.org/D16-1120/) and [overclassify as toxic text related to specific social identities](https://arxiv.org/pdf/2109.07445.pdf), respectively.
+
+### Other Known Limitations
+
+As a consequence of some of the filtering steps applied, it is likely that code content is not prevalent in our dataset. If you are training a model that should also perform code tasks, we recommend you use 🍷 FineWeb with a code dataset, such as [The Stack v2](https://huggingface.co/datasets/bigcode/the-stack-v2). You should also probably consider complementing 🍷 FineWeb with specialized curated sources (such as Wikipedia, for example) as they will likely have better formatting than the wikipedia content included in 🍷 FineWeb (we did not tailor the processing to individual websites).
+
+## Additional Information
+
+### Licensing Information
+
+The dataset is released under the **Open Data Commons Attribution License (ODC-By) v1.0** [license](https://opendatacommons.org/licenses/by/1-0/). The use of this dataset is also subject to [CommonCrawl's Terms of Use](https://commoncrawl.org/terms-of-use).
+
+### Future work
+
+We plan to work on better educational classifier to improve the quality of FineWeb-Edu.
+
+### Citation Information
+
+You can cite our paper https://arxiv.org/abs/2406.17557 or this dataset:
+
+```
+@misc{lozhkov2024fineweb-edu,
+    author       = { Lozhkov, Anton and Ben Allal, Loubna and von Werra, Leandro and Wolf, Thomas },  
+    title        = { FineWeb-Edu: the Finest Collection of Educational Content }, 
+    year         = 2024,  
+    url          = { https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu },  
+    doi          = { 10.57967/hf/2497 },
+	publisher    = { Hugging Face }
+}
+```

hf_cache/hub/datasets--Rowan--hellaswag/blobs/29f11d90eb3a5b319cfe8ce2a4e78d9f2a1aea3f

@@ -0,0 +1,218 @@
+---
+language:
+- en
+paperswithcode_id: hellaswag
+pretty_name: HellaSwag
+dataset_info:
+  features:
+  - name: ind
+    dtype: int32
+  - name: activity_label
+    dtype: string
+  - name: ctx_a
+    dtype: string
+  - name: ctx_b
+    dtype: string
+  - name: ctx
+    dtype: string
+  - name: endings
+    sequence: string
+  - name: source_id
+    dtype: string
+  - name: split
+    dtype: string
+  - name: split_type
+    dtype: string
+  - name: label
+    dtype: string
+  splits:
+  - name: train
+    num_bytes: 43232624
+    num_examples: 39905
+  - name: test
+    num_bytes: 10791853
+    num_examples: 10003
+  - name: validation
+    num_bytes: 11175717
+    num_examples: 10042
+  download_size: 36793872
+  dataset_size: 65200194
+configs:
+- config_name: default
+  data_files:
+  - split: train
+    path: data/train-*
+  - split: test
+    path: data/test-*
+  - split: validation
+    path: data/validation-*
+---
+
+# Dataset Card for "hellaswag"
+
+## Table of Contents
+- [Dataset Description](#dataset-description)
+  - [Dataset Summary](#dataset-summary)
+  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
+  - [Languages](#languages)
+- [Dataset Structure](#dataset-structure)
+  - [Data Instances](#data-instances)
+  - [Data Fields](#data-fields)
+  - [Data Splits](#data-splits)
+- [Dataset Creation](#dataset-creation)
+  - [Curation Rationale](#curation-rationale)
+  - [Source Data](#source-data)
+  - [Annotations](#annotations)
+  - [Personal and Sensitive Information](#personal-and-sensitive-information)
+- [Considerations for Using the Data](#considerations-for-using-the-data)
+  - [Social Impact of Dataset](#social-impact-of-dataset)
+  - [Discussion of Biases](#discussion-of-biases)
+  - [Other Known Limitations](#other-known-limitations)
+- [Additional Information](#additional-information)
+  - [Dataset Curators](#dataset-curators)
+  - [Licensing Information](#licensing-information)
+  - [Citation Information](#citation-information)
+  - [Contributions](#contributions)
+
+## Dataset Description
+
+- **Homepage:** [https://rowanzellers.com/hellaswag/](https://rowanzellers.com/hellaswag/)
+- **Repository:** [https://github.com/rowanz/hellaswag/](https://github.com/rowanz/hellaswag/)
+- **Paper:** [HellaSwag: Can a Machine Really Finish Your Sentence?](https://arxiv.org/abs/1905.07830)
+- **Point of Contact:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+- **Size of downloaded dataset files:** 71.49 MB
+- **Size of the generated dataset:** 65.32 MB
+- **Total amount of disk used:** 136.81 MB
+
+### Dataset Summary
+
+HellaSwag: Can a Machine Really Finish Your Sentence? is a new dataset for commonsense NLI. A paper was published at ACL2019.
+
+### Supported Tasks and Leaderboards
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Languages
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Dataset Structure
+
+### Data Instances
+
+#### default
+
+- **Size of downloaded dataset files:** 71.49 MB
+- **Size of the generated dataset:** 65.32 MB
+- **Total amount of disk used:** 136.81 MB
+
+An example of 'train' looks as follows.
+```
+This example was too long and was cropped:
+
+{
+    "activity_label": "Removing ice from car",
+    "ctx": "Then, the man writes over the snow covering the window of a car, and a woman wearing winter clothes smiles. then",
+    "ctx_a": "Then, the man writes over the snow covering the window of a car, and a woman wearing winter clothes smiles.",
+    "ctx_b": "then",
+    "endings": "[\", the man adds wax to the windshield and cuts it.\", \", a person board a ski lift, while two men supporting the head of the per...",
+    "ind": 4,
+    "label": "3",
+    "source_id": "activitynet~v_-1IBHYS3L-Y",
+    "split": "train",
+    "split_type": "indomain"
+}
+```
+
+### Data Fields
+
+The data fields are the same among all splits.
+
+#### default
+- `ind`: a `int32` feature.
+- `activity_label`: a `string` feature.
+- `ctx_a`: a `string` feature.
+- `ctx_b`: a `string` feature.
+- `ctx`: a `string` feature.
+- `endings`: a `list` of `string` features.
+- `source_id`: a `string` feature.
+- `split`: a `string` feature.
+- `split_type`: a `string` feature.
+- `label`: a `string` feature.
+
+### Data Splits
+
+| name  |train|validation|test |
+|-------|----:|---------:|----:|
+|default|39905|     10042|10003|
+
+## Dataset Creation
+
+### Curation Rationale
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Source Data
+
+#### Initial Data Collection and Normalization
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+#### Who are the source language producers?
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Annotations
+
+#### Annotation process
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+#### Who are the annotators?
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Personal and Sensitive Information
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Considerations for Using the Data
+
+### Social Impact of Dataset
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Discussion of Biases
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Other Known Limitations
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Additional Information
+
+### Dataset Curators
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Licensing Information
+
+MIT https://github.com/rowanz/hellaswag/blob/master/LICENSE
+
+### Citation Information
+
+```
+@inproceedings{zellers2019hellaswag,
+    title={HellaSwag: Can a Machine Really Finish Your Sentence?},
+    author={Zellers, Rowan and Holtzman, Ari and Bisk, Yonatan and Farhadi, Ali and Choi, Yejin},
+    booktitle ={Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
+    year={2019}
+}
+
+```
+
+
+### Contributions
+
+Thanks to [@albertvillanova](https://github.com/albertvillanova), [@mariamabarham](https://github.com/mariamabarham), [@thomwolf](https://github.com/thomwolf), [@patrickvonplaten](https://github.com/patrickvonplaten), [@lewtun](https://github.com/lewtun) for adding this dataset.

hf_cache/hub/datasets--Rowan--hellaswag/refs/main

@@ -0,0 +1 @@
+218ec52e09a7e7462a5400043bb9a69a41d06b76

hf_cache/hub/datasets--Rowan--hellaswag/snapshots/218ec52e09a7e7462a5400043bb9a69a41d06b76/README.md

@@ -0,0 +1,218 @@
+---
+language:
+- en
+paperswithcode_id: hellaswag
+pretty_name: HellaSwag
+dataset_info:
+  features:
+  - name: ind
+    dtype: int32
+  - name: activity_label
+    dtype: string
+  - name: ctx_a
+    dtype: string
+  - name: ctx_b
+    dtype: string
+  - name: ctx
+    dtype: string
+  - name: endings
+    sequence: string
+  - name: source_id
+    dtype: string
+  - name: split
+    dtype: string
+  - name: split_type
+    dtype: string
+  - name: label
+    dtype: string
+  splits:
+  - name: train
+    num_bytes: 43232624
+    num_examples: 39905
+  - name: test
+    num_bytes: 10791853
+    num_examples: 10003
+  - name: validation
+    num_bytes: 11175717
+    num_examples: 10042
+  download_size: 36793872
+  dataset_size: 65200194
+configs:
+- config_name: default
+  data_files:
+  - split: train
+    path: data/train-*
+  - split: test
+    path: data/test-*
+  - split: validation
+    path: data/validation-*
+---
+
+# Dataset Card for "hellaswag"
+
+## Table of Contents
+- [Dataset Description](#dataset-description)
+  - [Dataset Summary](#dataset-summary)
+  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
+  - [Languages](#languages)
+- [Dataset Structure](#dataset-structure)
+  - [Data Instances](#data-instances)
+  - [Data Fields](#data-fields)
+  - [Data Splits](#data-splits)
+- [Dataset Creation](#dataset-creation)
+  - [Curation Rationale](#curation-rationale)
+  - [Source Data](#source-data)
+  - [Annotations](#annotations)
+  - [Personal and Sensitive Information](#personal-and-sensitive-information)
+- [Considerations for Using the Data](#considerations-for-using-the-data)
+  - [Social Impact of Dataset](#social-impact-of-dataset)
+  - [Discussion of Biases](#discussion-of-biases)
+  - [Other Known Limitations](#other-known-limitations)
+- [Additional Information](#additional-information)
+  - [Dataset Curators](#dataset-curators)
+  - [Licensing Information](#licensing-information)
+  - [Citation Information](#citation-information)
+  - [Contributions](#contributions)
+
+## Dataset Description
+
+- **Homepage:** [https://rowanzellers.com/hellaswag/](https://rowanzellers.com/hellaswag/)
+- **Repository:** [https://github.com/rowanz/hellaswag/](https://github.com/rowanz/hellaswag/)
+- **Paper:** [HellaSwag: Can a Machine Really Finish Your Sentence?](https://arxiv.org/abs/1905.07830)
+- **Point of Contact:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+- **Size of downloaded dataset files:** 71.49 MB
+- **Size of the generated dataset:** 65.32 MB
+- **Total amount of disk used:** 136.81 MB
+
+### Dataset Summary
+
+HellaSwag: Can a Machine Really Finish Your Sentence? is a new dataset for commonsense NLI. A paper was published at ACL2019.
+
+### Supported Tasks and Leaderboards
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Languages
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Dataset Structure
+
+### Data Instances
+
+#### default
+
+- **Size of downloaded dataset files:** 71.49 MB
+- **Size of the generated dataset:** 65.32 MB
+- **Total amount of disk used:** 136.81 MB
+
+An example of 'train' looks as follows.
+```
+This example was too long and was cropped:
+
+{
+    "activity_label": "Removing ice from car",
+    "ctx": "Then, the man writes over the snow covering the window of a car, and a woman wearing winter clothes smiles. then",
+    "ctx_a": "Then, the man writes over the snow covering the window of a car, and a woman wearing winter clothes smiles.",
+    "ctx_b": "then",
+    "endings": "[\", the man adds wax to the windshield and cuts it.\", \", a person board a ski lift, while two men supporting the head of the per...",
+    "ind": 4,
+    "label": "3",
+    "source_id": "activitynet~v_-1IBHYS3L-Y",
+    "split": "train",
+    "split_type": "indomain"
+}
+```
+
+### Data Fields
+
+The data fields are the same among all splits.
+
+#### default
+- `ind`: a `int32` feature.
+- `activity_label`: a `string` feature.
+- `ctx_a`: a `string` feature.
+- `ctx_b`: a `string` feature.
+- `ctx`: a `string` feature.
+- `endings`: a `list` of `string` features.
+- `source_id`: a `string` feature.
+- `split`: a `string` feature.
+- `split_type`: a `string` feature.
+- `label`: a `string` feature.
+
+### Data Splits
+
+| name  |train|validation|test |
+|-------|----:|---------:|----:|
+|default|39905|     10042|10003|
+
+## Dataset Creation
+
+### Curation Rationale
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Source Data
+
+#### Initial Data Collection and Normalization
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+#### Who are the source language producers?
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Annotations
+
+#### Annotation process
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+#### Who are the annotators?
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Personal and Sensitive Information
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Considerations for Using the Data
+
+### Social Impact of Dataset
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Discussion of Biases
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Other Known Limitations
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Additional Information
+
+### Dataset Curators
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Licensing Information
+
+MIT https://github.com/rowanz/hellaswag/blob/master/LICENSE
+
+### Citation Information
+
+```
+@inproceedings{zellers2019hellaswag,
+    title={HellaSwag: Can a Machine Really Finish Your Sentence?},
+    author={Zellers, Rowan and Holtzman, Ari and Bisk, Yonatan and Farhadi, Ali and Choi, Yejin},
+    booktitle ={Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
+    year={2019}
+}
+
+```
+
+
+### Contributions
+
+Thanks to [@albertvillanova](https://github.com/albertvillanova), [@mariamabarham](https://github.com/mariamabarham), [@thomwolf](https://github.com/thomwolf), [@patrickvonplaten](https://github.com/patrickvonplaten), [@lewtun](https://github.com/lewtun) for adding this dataset.

hf_cache/hub/datasets--Salesforce--wikitext/blobs/2a4fec2bc8df76c9d4da1c8e8865b625eb221c76

@@ -0,0 +1,344 @@
+---
+annotations_creators:
+- no-annotation
+language_creators:
+- crowdsourced
+language:
+- en
+license:
+- cc-by-sa-3.0
+- gfdl
+multilinguality:
+- monolingual
+size_categories:
+- 1M<n<10M
+source_datasets:
+- original
+task_categories:
+- text-generation
+- fill-mask
+task_ids:
+- language-modeling
+- masked-language-modeling
+paperswithcode_id: wikitext-2
+pretty_name: WikiText
+dataset_info:
+- config_name: wikitext-103-raw-v1
+  features:
+  - name: text
+    dtype: string
+  splits:
+  - name: test
+    num_bytes: 1305088
+    num_examples: 4358
+  - name: train
+    num_bytes: 546500949
+    num_examples: 1801350
+  - name: validation
+    num_bytes: 1159288
+    num_examples: 3760
+  download_size: 315466397
+  dataset_size: 548965325
+- config_name: wikitext-103-v1
+  features:
+  - name: text
+    dtype: string
+  splits:
+  - name: test
+    num_bytes: 1295575
+    num_examples: 4358
+  - name: train
+    num_bytes: 545141915
+    num_examples: 1801350
+  - name: validation
+    num_bytes: 1154751
+    num_examples: 3760
+  download_size: 313093838
+  dataset_size: 547592241
+- config_name: wikitext-2-raw-v1
+  features:
+  - name: text
+    dtype: string
+  splits:
+  - name: test
+    num_bytes: 1305088
+    num_examples: 4358
+  - name: train
+    num_bytes: 11061717
+    num_examples: 36718
+  - name: validation
+    num_bytes: 1159288
+    num_examples: 3760
+  download_size: 7747362
+  dataset_size: 13526093
+- config_name: wikitext-2-v1
+  features:
+  - name: text
+    dtype: string
+  splits:
+  - name: test
+    num_bytes: 1270947
+    num_examples: 4358
+  - name: train
+    num_bytes: 10918118
+    num_examples: 36718
+  - name: validation
+    num_bytes: 1134123
+    num_examples: 3760
+  download_size: 7371282
+  dataset_size: 13323188
+configs:
+- config_name: wikitext-103-raw-v1
+  data_files:
+  - split: test
+    path: wikitext-103-raw-v1/test-*
+  - split: train
+    path: wikitext-103-raw-v1/train-*
+  - split: validation
+    path: wikitext-103-raw-v1/validation-*
+- config_name: wikitext-103-v1
+  data_files:
+  - split: test
+    path: wikitext-103-v1/test-*
+  - split: train
+    path: wikitext-103-v1/train-*
+  - split: validation
+    path: wikitext-103-v1/validation-*
+- config_name: wikitext-2-raw-v1
+  data_files:
+  - split: test
+    path: wikitext-2-raw-v1/test-*
+  - split: train
+    path: wikitext-2-raw-v1/train-*
+  - split: validation
+    path: wikitext-2-raw-v1/validation-*
+- config_name: wikitext-2-v1
+  data_files:
+  - split: test
+    path: wikitext-2-v1/test-*
+  - split: train
+    path: wikitext-2-v1/train-*
+  - split: validation
+    path: wikitext-2-v1/validation-*
+---
+
+# Dataset Card for "wikitext"
+
+## Table of Contents
+- [Dataset Description](#dataset-description)
+  - [Dataset Summary](#dataset-summary)
+  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
+  - [Languages](#languages)
+- [Dataset Structure](#dataset-structure)
+  - [Data Instances](#data-instances)
+  - [Data Fields](#data-fields)
+  - [Data Splits](#data-splits)
+- [Dataset Creation](#dataset-creation)
+  - [Curation Rationale](#curation-rationale)
+  - [Source Data](#source-data)
+  - [Annotations](#annotations)
+  - [Personal and Sensitive Information](#personal-and-sensitive-information)
+- [Considerations for Using the Data](#considerations-for-using-the-data)
+  - [Social Impact of Dataset](#social-impact-of-dataset)
+  - [Discussion of Biases](#discussion-of-biases)
+  - [Other Known Limitations](#other-known-limitations)
+- [Additional Information](#additional-information)
+  - [Dataset Curators](#dataset-curators)
+  - [Licensing Information](#licensing-information)
+  - [Citation Information](#citation-information)
+  - [Contributions](#contributions)
+
+## Dataset Description
+
+- **Homepage:** [https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/](https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/)
+- **Repository:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+- **Paper:** [Pointer Sentinel Mixture Models](https://arxiv.org/abs/1609.07843)
+- **Point of Contact:** [Stephen Merity](mailto:smerity@salesforce.com)
+- **Size of downloaded dataset files:** 391.41 MB
+- **Size of the generated dataset:** 1.12 GB
+- **Total amount of disk used:** 1.52 GB
+
+### Dataset Summary
+
+ The WikiText language modeling dataset is a collection of over 100 million tokens extracted from the set of verified
+ Good and Featured articles on Wikipedia. The dataset is available under the Creative Commons Attribution-ShareAlike License.
+
+Compared to the preprocessed version of Penn Treebank (PTB), WikiText-2 is over 2 times larger and WikiText-103 is over
+110 times larger. The WikiText dataset also features a far larger vocabulary and retains the original case, punctuation
+and numbers - all of which are removed in PTB. As it is composed of full articles, the dataset is well suited for models
+that can take advantage of long term dependencies.
+
+Each subset comes in two different variants:
+- Raw (for character level work) contain the raw tokens, before the addition of the <unk> (unknown) tokens.
+- Non-raw (for word level work) contain only the tokens in their vocabulary (wiki.train.tokens, wiki.valid.tokens, and wiki.test.tokens).
+  The out-of-vocabulary tokens have been replaced with the the <unk> token.
+
+
+### Supported Tasks and Leaderboards
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Languages
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Dataset Structure
+
+### Data Instances
+
+#### wikitext-103-raw-v1
+
+- **Size of downloaded dataset files:** 191.98 MB
+- **Size of the generated dataset:** 549.42 MB
+- **Total amount of disk used:** 741.41 MB
+
+An example of 'validation' looks as follows.
+```
+This example was too long and was cropped:
+
+{
+    "text": "\" The gold dollar or gold one @-@ dollar piece was a coin struck as a regular issue by the United States Bureau of the Mint from..."
+}
+```
+
+#### wikitext-103-v1
+
+- **Size of downloaded dataset files:** 190.23 MB
+- **Size of the generated dataset:** 548.05 MB
+- **Total amount of disk used:** 738.27 MB
+
+An example of 'train' looks as follows.
+```
+This example was too long and was cropped:
+
+{
+    "text": "\" Senjō no Valkyria 3 : <unk> Chronicles ( Japanese : 戦場のヴァルキュリア3 , lit . Valkyria of the Battlefield 3 ) , commonly referred to..."
+}
+```
+
+#### wikitext-2-raw-v1
+
+- **Size of downloaded dataset files:** 4.72 MB
+- **Size of the generated dataset:** 13.54 MB
+- **Total amount of disk used:** 18.26 MB
+
+An example of 'train' looks as follows.
+```
+This example was too long and was cropped:
+
+{
+    "text": "\" The Sinclair Scientific Programmable was introduced in 1975 , with the same case as the Sinclair Oxford . It was larger than t..."
+}
+```
+
+#### wikitext-2-v1
+
+- **Size of downloaded dataset files:** 4.48 MB
+- **Size of the generated dataset:** 13.34 MB
+- **Total amount of disk used:** 17.82 MB
+
+An example of 'train' looks as follows.
+```
+This example was too long and was cropped:
+
+{
+    "text": "\" Senjō no Valkyria 3 : <unk> Chronicles ( Japanese : 戦場のヴァルキュリア3 , lit . Valkyria of the Battlefield 3 ) , commonly referred to..."
+}
+```
+
+### Data Fields
+
+The data fields are the same among all splits.
+
+#### wikitext-103-raw-v1
+- `text`: a `string` feature.
+
+#### wikitext-103-v1
+- `text`: a `string` feature.
+
+#### wikitext-2-raw-v1
+- `text`: a `string` feature.
+
+#### wikitext-2-v1
+- `text`: a `string` feature.
+
+### Data Splits
+
+|       name        | train |validation|test|
+|-------------------|------:|---------:|---:|
+|wikitext-103-raw-v1|1801350|      3760|4358|
+|wikitext-103-v1    |1801350|      3760|4358|
+|wikitext-2-raw-v1  |  36718|      3760|4358|
+|wikitext-2-v1      |  36718|      3760|4358|
+
+## Dataset Creation
+
+### Curation Rationale
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Source Data
+
+#### Initial Data Collection and Normalization
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+#### Who are the source language producers?
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Annotations
+
+#### Annotation process
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+#### Who are the annotators?
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Personal and Sensitive Information
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Considerations for Using the Data
+
+### Social Impact of Dataset
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Discussion of Biases
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Other Known Limitations
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Additional Information
+
+### Dataset Curators
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Licensing Information
+
+The dataset is available under the [Creative Commons Attribution-ShareAlike License (CC BY-SA 4.0)](https://creativecommons.org/licenses/by-sa/4.0/).
+
+### Citation Information
+
+```
+@misc{merity2016pointer,
+      title={Pointer Sentinel Mixture Models},
+      author={Stephen Merity and Caiming Xiong and James Bradbury and Richard Socher},
+      year={2016},
+      eprint={1609.07843},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+```
+
+
+### Contributions
+
+Thanks to [@thomwolf](https://github.com/thomwolf), [@lewtun](https://github.com/lewtun), [@patrickvonplaten](https://github.com/patrickvonplaten), [@mariamabarham](https://github.com/mariamabarham) for adding this dataset.

hf_cache/hub/datasets--Salesforce--wikitext/refs/main

@@ -0,0 +1 @@
+b08601e04326c79dfdd32d625aee71d232d685c3

hf_cache/hub/datasets--Salesforce--wikitext/snapshots/b08601e04326c79dfdd32d625aee71d232d685c3/README.md

@@ -0,0 +1,344 @@
+---
+annotations_creators:
+- no-annotation
+language_creators:
+- crowdsourced
+language:
+- en
+license:
+- cc-by-sa-3.0
+- gfdl
+multilinguality:
+- monolingual
+size_categories:
+- 1M<n<10M
+source_datasets:
+- original
+task_categories:
+- text-generation
+- fill-mask
+task_ids:
+- language-modeling
+- masked-language-modeling
+paperswithcode_id: wikitext-2
+pretty_name: WikiText
+dataset_info:
+- config_name: wikitext-103-raw-v1
+  features:
+  - name: text
+    dtype: string
+  splits:
+  - name: test
+    num_bytes: 1305088
+    num_examples: 4358
+  - name: train
+    num_bytes: 546500949
+    num_examples: 1801350
+  - name: validation
+    num_bytes: 1159288
+    num_examples: 3760
+  download_size: 315466397
+  dataset_size: 548965325
+- config_name: wikitext-103-v1
+  features:
+  - name: text
+    dtype: string
+  splits:
+  - name: test
+    num_bytes: 1295575
+    num_examples: 4358
+  - name: train
+    num_bytes: 545141915
+    num_examples: 1801350
+  - name: validation
+    num_bytes: 1154751
+    num_examples: 3760
+  download_size: 313093838
+  dataset_size: 547592241
+- config_name: wikitext-2-raw-v1
+  features:
+  - name: text
+    dtype: string
+  splits:
+  - name: test
+    num_bytes: 1305088
+    num_examples: 4358
+  - name: train
+    num_bytes: 11061717
+    num_examples: 36718
+  - name: validation
+    num_bytes: 1159288
+    num_examples: 3760
+  download_size: 7747362
+  dataset_size: 13526093
+- config_name: wikitext-2-v1
+  features:
+  - name: text
+    dtype: string
+  splits:
+  - name: test
+    num_bytes: 1270947
+    num_examples: 4358
+  - name: train
+    num_bytes: 10918118
+    num_examples: 36718
+  - name: validation
+    num_bytes: 1134123
+    num_examples: 3760
+  download_size: 7371282
+  dataset_size: 13323188
+configs:
+- config_name: wikitext-103-raw-v1
+  data_files:
+  - split: test
+    path: wikitext-103-raw-v1/test-*
+  - split: train
+    path: wikitext-103-raw-v1/train-*
+  - split: validation
+    path: wikitext-103-raw-v1/validation-*
+- config_name: wikitext-103-v1
+  data_files:
+  - split: test
+    path: wikitext-103-v1/test-*
+  - split: train
+    path: wikitext-103-v1/train-*
+  - split: validation
+    path: wikitext-103-v1/validation-*
+- config_name: wikitext-2-raw-v1
+  data_files:
+  - split: test
+    path: wikitext-2-raw-v1/test-*
+  - split: train
+    path: wikitext-2-raw-v1/train-*
+  - split: validation
+    path: wikitext-2-raw-v1/validation-*
+- config_name: wikitext-2-v1
+  data_files:
+  - split: test
+    path: wikitext-2-v1/test-*
+  - split: train
+    path: wikitext-2-v1/train-*
+  - split: validation
+    path: wikitext-2-v1/validation-*
+---
+
+# Dataset Card for "wikitext"
+
+## Table of Contents
+- [Dataset Description](#dataset-description)
+  - [Dataset Summary](#dataset-summary)
+  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
+  - [Languages](#languages)
+- [Dataset Structure](#dataset-structure)
+  - [Data Instances](#data-instances)
+  - [Data Fields](#data-fields)
+  - [Data Splits](#data-splits)
+- [Dataset Creation](#dataset-creation)
+  - [Curation Rationale](#curation-rationale)
+  - [Source Data](#source-data)
+  - [Annotations](#annotations)
+  - [Personal and Sensitive Information](#personal-and-sensitive-information)
+- [Considerations for Using the Data](#considerations-for-using-the-data)
+  - [Social Impact of Dataset](#social-impact-of-dataset)
+  - [Discussion of Biases](#discussion-of-biases)
+  - [Other Known Limitations](#other-known-limitations)
+- [Additional Information](#additional-information)
+  - [Dataset Curators](#dataset-curators)
+  - [Licensing Information](#licensing-information)
+  - [Citation Information](#citation-information)
+  - [Contributions](#contributions)
+
+## Dataset Description
+
+- **Homepage:** [https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/](https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/)
+- **Repository:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+- **Paper:** [Pointer Sentinel Mixture Models](https://arxiv.org/abs/1609.07843)
+- **Point of Contact:** [Stephen Merity](mailto:smerity@salesforce.com)
+- **Size of downloaded dataset files:** 391.41 MB
+- **Size of the generated dataset:** 1.12 GB
+- **Total amount of disk used:** 1.52 GB
+
+### Dataset Summary
+
+ The WikiText language modeling dataset is a collection of over 100 million tokens extracted from the set of verified
+ Good and Featured articles on Wikipedia. The dataset is available under the Creative Commons Attribution-ShareAlike License.
+
+Compared to the preprocessed version of Penn Treebank (PTB), WikiText-2 is over 2 times larger and WikiText-103 is over
+110 times larger. The WikiText dataset also features a far larger vocabulary and retains the original case, punctuation
+and numbers - all of which are removed in PTB. As it is composed of full articles, the dataset is well suited for models
+that can take advantage of long term dependencies.
+
+Each subset comes in two different variants:
+- Raw (for character level work) contain the raw tokens, before the addition of the <unk> (unknown) tokens.
+- Non-raw (for word level work) contain only the tokens in their vocabulary (wiki.train.tokens, wiki.valid.tokens, and wiki.test.tokens).
+  The out-of-vocabulary tokens have been replaced with the the <unk> token.
+
+
+### Supported Tasks and Leaderboards
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Languages
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Dataset Structure
+
+### Data Instances
+
+#### wikitext-103-raw-v1
+
+- **Size of downloaded dataset files:** 191.98 MB
+- **Size of the generated dataset:** 549.42 MB
+- **Total amount of disk used:** 741.41 MB
+
+An example of 'validation' looks as follows.
+```
+This example was too long and was cropped:
+
+{
+    "text": "\" The gold dollar or gold one @-@ dollar piece was a coin struck as a regular issue by the United States Bureau of the Mint from..."
+}
+```
+
+#### wikitext-103-v1
+
+- **Size of downloaded dataset files:** 190.23 MB
+- **Size of the generated dataset:** 548.05 MB
+- **Total amount of disk used:** 738.27 MB
+
+An example of 'train' looks as follows.
+```
+This example was too long and was cropped:
+
+{
+    "text": "\" Senjō no Valkyria 3 : <unk> Chronicles ( Japanese : 戦場のヴァルキュリア3 , lit . Valkyria of the Battlefield 3 ) , commonly referred to..."
+}
+```
+
+#### wikitext-2-raw-v1
+
+- **Size of downloaded dataset files:** 4.72 MB
+- **Size of the generated dataset:** 13.54 MB
+- **Total amount of disk used:** 18.26 MB
+
+An example of 'train' looks as follows.
+```
+This example was too long and was cropped:
+
+{
+    "text": "\" The Sinclair Scientific Programmable was introduced in 1975 , with the same case as the Sinclair Oxford . It was larger than t..."
+}
+```
+
+#### wikitext-2-v1
+
+- **Size of downloaded dataset files:** 4.48 MB
+- **Size of the generated dataset:** 13.34 MB
+- **Total amount of disk used:** 17.82 MB
+
+An example of 'train' looks as follows.
+```
+This example was too long and was cropped:
+
+{
+    "text": "\" Senjō no Valkyria 3 : <unk> Chronicles ( Japanese : 戦場のヴァルキュリア3 , lit . Valkyria of the Battlefield 3 ) , commonly referred to..."
+}
+```
+
+### Data Fields
+
+The data fields are the same among all splits.
+
+#### wikitext-103-raw-v1
+- `text`: a `string` feature.
+
+#### wikitext-103-v1
+- `text`: a `string` feature.
+
+#### wikitext-2-raw-v1
+- `text`: a `string` feature.
+
+#### wikitext-2-v1
+- `text`: a `string` feature.
+
+### Data Splits
+
+|       name        | train |validation|test|
+|-------------------|------:|---------:|---:|
+|wikitext-103-raw-v1|1801350|      3760|4358|
+|wikitext-103-v1    |1801350|      3760|4358|
+|wikitext-2-raw-v1  |  36718|      3760|4358|
+|wikitext-2-v1      |  36718|      3760|4358|
+
+## Dataset Creation
+
+### Curation Rationale
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Source Data
+
+#### Initial Data Collection and Normalization
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+#### Who are the source language producers?
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Annotations
+
+#### Annotation process
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+#### Who are the annotators?
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Personal and Sensitive Information
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Considerations for Using the Data
+
+### Social Impact of Dataset
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Discussion of Biases
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Other Known Limitations
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Additional Information
+
+### Dataset Curators
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Licensing Information
+
+The dataset is available under the [Creative Commons Attribution-ShareAlike License (CC BY-SA 4.0)](https://creativecommons.org/licenses/by-sa/4.0/).
+
+### Citation Information
+
+```
+@misc{merity2016pointer,
+      title={Pointer Sentinel Mixture Models},
+      author={Stephen Merity and Caiming Xiong and James Bradbury and Richard Socher},
+      year={2016},
+      eprint={1609.07843},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL}
+}
+```
+
+
+### Contributions
+
+Thanks to [@thomwolf](https://github.com/thomwolf), [@lewtun](https://github.com/lewtun), [@patrickvonplaten](https://github.com/patrickvonplaten), [@mariamabarham](https://github.com/mariamabarham) for adding this dataset.

hf_cache/hub/datasets--abisee--cnn_dailymail/blobs/feadf7d245b4f6818e20b4cf65841d03b5703d47

@@ -0,0 +1,305 @@
+---
+annotations_creators:
+- no-annotation
+language_creators:
+- found
+language:
+- en
+license:
+- apache-2.0
+multilinguality:
+- monolingual
+size_categories:
+- 100K<n<1M
+source_datasets:
+- original
+task_categories:
+- summarization
+task_ids:
+- news-articles-summarization
+paperswithcode_id: cnn-daily-mail-1
+pretty_name: CNN / Daily Mail
+dataset_info:
+- config_name: 1.0.0
+  features:
+  - name: article
+    dtype: string
+  - name: highlights
+    dtype: string
+  - name: id
+    dtype: string
+  splits:
+  - name: train
+    num_bytes: 1261703785
+    num_examples: 287113
+  - name: validation
+    num_bytes: 57732412
+    num_examples: 13368
+  - name: test
+    num_bytes: 49925732
+    num_examples: 11490
+  download_size: 836927248
+  dataset_size: 1369361929
+- config_name: 2.0.0
+  features:
+  - name: article
+    dtype: string
+  - name: highlights
+    dtype: string
+  - name: id
+    dtype: string
+  splits:
+  - name: train
+    num_bytes: 1261703785
+    num_examples: 287113
+  - name: validation
+    num_bytes: 57732412
+    num_examples: 13368
+  - name: test
+    num_bytes: 49925732
+    num_examples: 11490
+  download_size: 837094602
+  dataset_size: 1369361929
+- config_name: 3.0.0
+  features:
+  - name: article
+    dtype: string
+  - name: highlights
+    dtype: string
+  - name: id
+    dtype: string
+  splits:
+  - name: train
+    num_bytes: 1261703785
+    num_examples: 287113
+  - name: validation
+    num_bytes: 57732412
+    num_examples: 13368
+  - name: test
+    num_bytes: 49925732
+    num_examples: 11490
+  download_size: 837094602
+  dataset_size: 1369361929
+configs:
+- config_name: 1.0.0
+  data_files:
+  - split: train
+    path: 1.0.0/train-*
+  - split: validation
+    path: 1.0.0/validation-*
+  - split: test
+    path: 1.0.0/test-*
+- config_name: 2.0.0
+  data_files:
+  - split: train
+    path: 2.0.0/train-*
+  - split: validation
+    path: 2.0.0/validation-*
+  - split: test
+    path: 2.0.0/test-*
+- config_name: 3.0.0
+  data_files:
+  - split: train
+    path: 3.0.0/train-*
+  - split: validation
+    path: 3.0.0/validation-*
+  - split: test
+    path: 3.0.0/test-*
+train-eval-index:
+- config: 3.0.0
+  task: summarization
+  task_id: summarization
+  splits:
+    eval_split: test
+  col_mapping:
+    article: text
+    highlights: target
+---
+# Dataset Card for CNN Dailymail Dataset
+
+## Table of Contents
+- [Dataset Description](#dataset-description)
+  - [Dataset Summary](#dataset-summary)
+  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
+  - [Languages](#languages)
+- [Dataset Structure](#dataset-structure)
+  - [Data Instances](#data-instances)
+  - [Data Fields](#data-fields)
+  - [Data Splits](#data-splits)
+- [Dataset Creation](#dataset-creation)
+  - [Curation Rationale](#curation-rationale)
+  - [Source Data](#source-data)
+  - [Annotations](#annotations)
+  - [Personal and Sensitive Information](#personal-and-sensitive-information)
+- [Considerations for Using the Data](#considerations-for-using-the-data)
+  - [Social Impact of Dataset](#social-impact-of-dataset)
+  - [Discussion of Biases](#discussion-of-biases)
+  - [Other Known Limitations](#other-known-limitations)
+- [Additional Information](#additional-information)
+  - [Dataset Curators](#dataset-curators)
+  - [Licensing Information](#licensing-information)
+  - [Citation Information](#citation-information)
+  - [Contributions](#contributions)
+
+## Dataset Description
+
+- **Homepage:**
+- **Repository:** [CNN / DailyMail Dataset repository](https://github.com/abisee/cnn-dailymail)
+- **Paper:** [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://papers.nips.cc/paper/5945-teaching-machines-to-read-and-comprehend.pdf), [Get To The Point: Summarization with Pointer-Generator Networks](https://www.aclweb.org/anthology/K16-1028.pdf)
+- **Leaderboard:** [Papers with Code leaderboard for CNN / Dailymail Dataset](https://paperswithcode.com/sota/document-summarization-on-cnn-daily-mail)
+- **Point of Contact:** [Abigail See](mailto:abisee@stanford.edu)
+
+### Dataset Summary
+
+The CNN / DailyMail Dataset is an English-language dataset containing just over 300k unique news articles as written by journalists at CNN and the Daily Mail. The current version supports both extractive and abstractive summarization, though the original version was created for machine reading and comprehension and abstractive question answering. 
+
+### Supported Tasks and Leaderboards
+
+- 'summarization': [Versions 2.0.0 and 3.0.0 of the CNN / DailyMail Dataset](https://www.aclweb.org/anthology/K16-1028.pdf) can be used to train a model for abstractive and extractive summarization ([Version 1.0.0](https://papers.nips.cc/paper/5945-teaching-machines-to-read-and-comprehend.pdf) was developed for machine reading and comprehension and abstractive question answering). The model performance is measured by how high the output summary's [ROUGE](https://huggingface.co/metrics/rouge) score for a given article is when compared to the highlight as written by the original article author. [Zhong et al (2020)](https://www.aclweb.org/anthology/2020.acl-main.552.pdf) report a ROUGE-1 score of 44.41 when testing a model trained for extractive summarization. See the [Papers With Code leaderboard](https://paperswithcode.com/sota/document-summarization-on-cnn-daily-mail) for more models. 
+
+### Languages
+
+The BCP-47 code for English as generally spoken in the United States is en-US and the BCP-47 code for English as generally spoken in the United Kingdom is en-GB. It is unknown if other varieties of English are represented in the data.
+
+## Dataset Structure
+
+### Data Instances
+
+For each instance, there is a string for the article, a string for the highlights, and a string for the id. See the [CNN / Daily Mail dataset viewer](https://huggingface.co/datasets/viewer/?dataset=cnn_dailymail&config=3.0.0) to explore more examples.
+
+```
+{'id': '0054d6d30dbcad772e20b22771153a2a9cbeaf62',
+ 'article': '(CNN) -- An American woman died aboard a cruise ship that docked at Rio de Janeiro on Tuesday, the same ship on which 86 passengers previously fell ill, according to the state-run Brazilian news agency, Agencia Brasil. The American tourist died aboard the MS Veendam, owned by cruise operator Holland America. Federal Police told Agencia Brasil that forensic doctors were investigating her death. The ship's doctors told police that the woman was elderly and suffered from diabetes and hypertension, according the agency. The other passengers came down with diarrhea prior to her death during an earlier part of the trip, the ship's doctors said. The Veendam left New York 36 days ago for a South America tour.'
+ 'highlights': 'The elderly woman suffered from diabetes and hypertension, ship's doctors say .\nPreviously, 86 passengers had fallen ill on the ship, Agencia Brasil says .'}
+```
+
+The average token count for the articles and the highlights are provided below:
+
+| Feature    | Mean Token Count |
+| ---------- | ---------------- |
+| Article    | 781              |
+| Highlights | 56               |
+
+### Data Fields
+
+- `id`: a string containing the heximal formated SHA1 hash of the url where the story was retrieved from
+- `article`: a string containing the body of the news article 
+- `highlights`: a string containing the highlight of the article as written by the article author
+
+### Data Splits
+
+The CNN/DailyMail dataset has 3 splits: _train_, _validation_, and _test_. Below are the statistics for Version 3.0.0 of the dataset.
+
+| Dataset Split | Number of Instances in Split                |
+| ------------- | ------------------------------------------- |
+| Train         | 287,113                                     |
+| Validation    | 13,368                                      |
+| Test          | 11,490                                      |
+
+## Dataset Creation
+
+### Curation Rationale
+
+Version 1.0.0 aimed to support supervised neural methodologies for machine reading and question answering with a large amount of real natural language training data and released about 313k unique articles and nearly 1M Cloze style questions to go with the articles. Versions 2.0.0 and 3.0.0 changed the structure of the dataset to support summarization rather than question answering. Version 3.0.0 provided a non-anonymized version of the data, whereas both the previous versions were preprocessed to replace named entities with unique identifier labels. 
+
+### Source Data
+
+#### Initial Data Collection and Normalization
+
+The data consists of news articles and highlight sentences. In the question answering setting of the data, the articles are used as the context and entities are hidden one at a time in the highlight sentences, producing Cloze style questions where the goal of the model is to correctly guess which entity in the context has been hidden in the highlight. In the summarization setting, the highlight sentences are concatenated to form a summary of the article. The CNN articles were written between April 2007 and April 2015. The Daily Mail articles were written between June 2010 and April 2015. 
+
+The code for the original data collection is available at <https://github.com/deepmind/rc-data>. The articles were downloaded using archives of <www.cnn.com> and <www.dailymail.co.uk> on the Wayback Machine. Articles were not included in the Version 1.0.0 collection if they exceeded 2000 tokens. Due to accessibility issues with the Wayback Machine, Kyunghyun Cho has made the datasets available at <https://cs.nyu.edu/~kcho/DMQA/>. An updated version of the code that does not anonymize the data is available at <https://github.com/abisee/cnn-dailymail>. 
+
+Hermann et al provided their own tokenization script. The script provided by See uses the PTBTokenizer. It also lowercases the text and adds periods to lines missing them.
+
+#### Who are the source language producers?
+
+The text was written by journalists at CNN and the Daily Mail. 
+
+### Annotations
+
+The dataset does not contain any additional annotations.
+
+#### Annotation process
+
+[N/A]
+
+#### Who are the annotators?
+
+[N/A]
+
+### Personal and Sensitive Information
+
+Version 3.0 is not anonymized, so individuals' names can be found in the dataset. Information about the original author is not included in the dataset.
+
+## Considerations for Using the Data
+
+### Social Impact of Dataset
+
+The purpose of this dataset is to help develop models that can summarize long paragraphs of text in one or two sentences.
+
+This task is useful for efficiently presenting information given a large quantity of text. It should be made clear that any summarizations produced by models trained on this dataset are reflective of the language used in the articles, but are in fact automatically generated.
+
+### Discussion of Biases
+
+[Bordia and Bowman (2019)](https://www.aclweb.org/anthology/N19-3002.pdf) explore measuring gender bias and debiasing techniques in the CNN / Dailymail dataset, the Penn Treebank, and WikiText-2. They find the CNN / Dailymail dataset to have a slightly lower gender bias based on their metric compared to the other datasets, but still show evidence of gender bias when looking at words such as 'fragile'.
+
+Because the articles were written by and for people in the US and the UK, they will likely present specifically US and UK perspectives and feature events that are considered relevant to those populations during the time that the articles were published. 
+
+### Other Known Limitations
+
+News articles have been shown to conform to writing conventions in which important information is primarily presented in the first third of the article [(Kryściński et al, 2019)](https://www.aclweb.org/anthology/D19-1051.pdf). [Chen et al (2016)](https://www.aclweb.org/anthology/P16-1223.pdf) conducted a manual study of 100 random instances of the first version of the dataset and found 25% of the samples to be difficult even for humans to answer correctly due to ambiguity and coreference errors. 
+
+It should also be noted that machine-generated summarizations, even when extractive, may differ in truth values when compared to the original articles. 
+
+## Additional Information
+
+### Dataset Curators
+
+The data was originally collected by Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom of Google DeepMind. Tomáš Kočiský and Phil Blunsom are also affiliated with the University of Oxford. They released scripts to collect and process the data into the question answering format. 
+
+Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, and Bing Xiang of IMB Watson and Çağlar Gu̇lçehre of Université de Montréal modified Hermann et al's collection scripts to restore the data to a summary format. They also produced both anonymized and non-anonymized versions.
+
+The code for the non-anonymized version is made publicly available by Abigail See of Stanford University, Peter J. Liu of Google Brain and Christopher D. Manning of Stanford University at <https://github.com/abisee/cnn-dailymail>. The work at Stanford University was supported by the DARPA DEFT ProgramAFRL contract no. FA8750-13-2-0040.
+
+### Licensing Information
+
+The CNN / Daily Mail dataset version 1.0.0 is released under the [Apache-2.0 License](http://www.apache.org/licenses/LICENSE-2.0). 
+
+### Citation Information
+
+```
+@inproceedings{see-etal-2017-get,
+    title = "Get To The Point: Summarization with Pointer-Generator Networks",
+    author = "See, Abigail  and
+      Liu, Peter J.  and
+      Manning, Christopher D.",
+    booktitle = "Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
+    month = jul,
+    year = "2017",
+    address = "Vancouver, Canada",
+    publisher = "Association for Computational Linguistics",
+    url = "https://www.aclweb.org/anthology/P17-1099",
+    doi = "10.18653/v1/P17-1099",
+    pages = "1073--1083",
+    abstract = "Neural sequence-to-sequence models have provided a viable new approach for abstractive text summarization (meaning they are not restricted to simply selecting and rearranging passages from the original text). However, these models have two shortcomings: they are liable to reproduce factual details inaccurately, and they tend to repeat themselves. In this work we propose a novel architecture that augments the standard sequence-to-sequence attentional model in two orthogonal ways. First, we use a hybrid pointer-generator network that can copy words from the source text via pointing, which aids accurate reproduction of information, while retaining the ability to produce novel words through the generator. Second, we use coverage to keep track of what has been summarized, which discourages repetition. We apply our model to the CNN / Daily Mail summarization task, outperforming the current abstractive state-of-the-art by at least 2 ROUGE points.",
+}
+```
+
+```
+@inproceedings{DBLP:conf/nips/HermannKGEKSB15,
+  author={Karl Moritz Hermann and Tomás Kociský and Edward Grefenstette and Lasse Espeholt and Will Kay and Mustafa Suleyman and Phil Blunsom},
+  title={Teaching Machines to Read and Comprehend},
+  year={2015},
+  cdate={1420070400000},
+  pages={1693-1701},
+  url={http://papers.nips.cc/paper/5945-teaching-machines-to-read-and-comprehend},
+  booktitle={NIPS},
+  crossref={conf/nips/2015}
+}
+
+```
+
+### Contributions
+
+Thanks to [@thomwolf](https://github.com/thomwolf), [@lewtun](https://github.com/lewtun), [@jplu](https://github.com/jplu), [@jbragg](https://github.com/jbragg), [@patrickvonplaten](https://github.com/patrickvonplaten) and [@mcmillanmajora](https://github.com/mcmillanmajora) for adding this dataset.

hf_cache/hub/datasets--abisee--cnn_dailymail/refs/main

@@ -0,0 +1 @@
+96df5e686bee6baa90b8bee7c28b81fa3fa6223d

hf_cache/hub/datasets--abisee--cnn_dailymail/snapshots/96df5e686bee6baa90b8bee7c28b81fa3fa6223d/README.md

@@ -0,0 +1,305 @@
+---
+annotations_creators:
+- no-annotation
+language_creators:
+- found
+language:
+- en
+license:
+- apache-2.0
+multilinguality:
+- monolingual
+size_categories:
+- 100K<n<1M
+source_datasets:
+- original
+task_categories:
+- summarization
+task_ids:
+- news-articles-summarization
+paperswithcode_id: cnn-daily-mail-1
+pretty_name: CNN / Daily Mail
+dataset_info:
+- config_name: 1.0.0
+  features:
+  - name: article
+    dtype: string
+  - name: highlights
+    dtype: string
+  - name: id
+    dtype: string
+  splits:
+  - name: train
+    num_bytes: 1261703785
+    num_examples: 287113
+  - name: validation
+    num_bytes: 57732412
+    num_examples: 13368
+  - name: test
+    num_bytes: 49925732
+    num_examples: 11490
+  download_size: 836927248
+  dataset_size: 1369361929
+- config_name: 2.0.0
+  features:
+  - name: article
+    dtype: string
+  - name: highlights
+    dtype: string
+  - name: id
+    dtype: string
+  splits:
+  - name: train
+    num_bytes: 1261703785
+    num_examples: 287113
+  - name: validation
+    num_bytes: 57732412
+    num_examples: 13368
+  - name: test
+    num_bytes: 49925732
+    num_examples: 11490
+  download_size: 837094602
+  dataset_size: 1369361929
+- config_name: 3.0.0
+  features:
+  - name: article
+    dtype: string
+  - name: highlights
+    dtype: string
+  - name: id
+    dtype: string
+  splits:
+  - name: train
+    num_bytes: 1261703785
+    num_examples: 287113
+  - name: validation
+    num_bytes: 57732412
+    num_examples: 13368
+  - name: test
+    num_bytes: 49925732
+    num_examples: 11490
+  download_size: 837094602
+  dataset_size: 1369361929
+configs:
+- config_name: 1.0.0
+  data_files:
+  - split: train
+    path: 1.0.0/train-*
+  - split: validation
+    path: 1.0.0/validation-*
+  - split: test
+    path: 1.0.0/test-*
+- config_name: 2.0.0
+  data_files:
+  - split: train
+    path: 2.0.0/train-*
+  - split: validation
+    path: 2.0.0/validation-*
+  - split: test
+    path: 2.0.0/test-*
+- config_name: 3.0.0
+  data_files:
+  - split: train
+    path: 3.0.0/train-*
+  - split: validation
+    path: 3.0.0/validation-*
+  - split: test
+    path: 3.0.0/test-*
+train-eval-index:
+- config: 3.0.0
+  task: summarization
+  task_id: summarization
+  splits:
+    eval_split: test
+  col_mapping:
+    article: text
+    highlights: target
+---
+# Dataset Card for CNN Dailymail Dataset
+
+## Table of Contents
+- [Dataset Description](#dataset-description)
+  - [Dataset Summary](#dataset-summary)
+  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
+  - [Languages](#languages)
+- [Dataset Structure](#dataset-structure)
+  - [Data Instances](#data-instances)
+  - [Data Fields](#data-fields)
+  - [Data Splits](#data-splits)
+- [Dataset Creation](#dataset-creation)
+  - [Curation Rationale](#curation-rationale)
+  - [Source Data](#source-data)
+  - [Annotations](#annotations)
+  - [Personal and Sensitive Information](#personal-and-sensitive-information)
+- [Considerations for Using the Data](#considerations-for-using-the-data)
+  - [Social Impact of Dataset](#social-impact-of-dataset)
+  - [Discussion of Biases](#discussion-of-biases)
+  - [Other Known Limitations](#other-known-limitations)
+- [Additional Information](#additional-information)
+  - [Dataset Curators](#dataset-curators)
+  - [Licensing Information](#licensing-information)
+  - [Citation Information](#citation-information)
+  - [Contributions](#contributions)
+
+## Dataset Description
+
+- **Homepage:**
+- **Repository:** [CNN / DailyMail Dataset repository](https://github.com/abisee/cnn-dailymail)
+- **Paper:** [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://papers.nips.cc/paper/5945-teaching-machines-to-read-and-comprehend.pdf), [Get To The Point: Summarization with Pointer-Generator Networks](https://www.aclweb.org/anthology/K16-1028.pdf)
+- **Leaderboard:** [Papers with Code leaderboard for CNN / Dailymail Dataset](https://paperswithcode.com/sota/document-summarization-on-cnn-daily-mail)
+- **Point of Contact:** [Abigail See](mailto:abisee@stanford.edu)
+
+### Dataset Summary
+
+The CNN / DailyMail Dataset is an English-language dataset containing just over 300k unique news articles as written by journalists at CNN and the Daily Mail. The current version supports both extractive and abstractive summarization, though the original version was created for machine reading and comprehension and abstractive question answering. 
+
+### Supported Tasks and Leaderboards
+
+- 'summarization': [Versions 2.0.0 and 3.0.0 of the CNN / DailyMail Dataset](https://www.aclweb.org/anthology/K16-1028.pdf) can be used to train a model for abstractive and extractive summarization ([Version 1.0.0](https://papers.nips.cc/paper/5945-teaching-machines-to-read-and-comprehend.pdf) was developed for machine reading and comprehension and abstractive question answering). The model performance is measured by how high the output summary's [ROUGE](https://huggingface.co/metrics/rouge) score for a given article is when compared to the highlight as written by the original article author. [Zhong et al (2020)](https://www.aclweb.org/anthology/2020.acl-main.552.pdf) report a ROUGE-1 score of 44.41 when testing a model trained for extractive summarization. See the [Papers With Code leaderboard](https://paperswithcode.com/sota/document-summarization-on-cnn-daily-mail) for more models. 
+
+### Languages
+
+The BCP-47 code for English as generally spoken in the United States is en-US and the BCP-47 code for English as generally spoken in the United Kingdom is en-GB. It is unknown if other varieties of English are represented in the data.
+
+## Dataset Structure
+
+### Data Instances
+
+For each instance, there is a string for the article, a string for the highlights, and a string for the id. See the [CNN / Daily Mail dataset viewer](https://huggingface.co/datasets/viewer/?dataset=cnn_dailymail&config=3.0.0) to explore more examples.
+
+```
+{'id': '0054d6d30dbcad772e20b22771153a2a9cbeaf62',
+ 'article': '(CNN) -- An American woman died aboard a cruise ship that docked at Rio de Janeiro on Tuesday, the same ship on which 86 passengers previously fell ill, according to the state-run Brazilian news agency, Agencia Brasil. The American tourist died aboard the MS Veendam, owned by cruise operator Holland America. Federal Police told Agencia Brasil that forensic doctors were investigating her death. The ship's doctors told police that the woman was elderly and suffered from diabetes and hypertension, according the agency. The other passengers came down with diarrhea prior to her death during an earlier part of the trip, the ship's doctors said. The Veendam left New York 36 days ago for a South America tour.'
+ 'highlights': 'The elderly woman suffered from diabetes and hypertension, ship's doctors say .\nPreviously, 86 passengers had fallen ill on the ship, Agencia Brasil says .'}
+```
+
+The average token count for the articles and the highlights are provided below:
+
+| Feature    | Mean Token Count |
+| ---------- | ---------------- |
+| Article    | 781              |
+| Highlights | 56               |
+
+### Data Fields
+
+- `id`: a string containing the heximal formated SHA1 hash of the url where the story was retrieved from
+- `article`: a string containing the body of the news article 
+- `highlights`: a string containing the highlight of the article as written by the article author
+
+### Data Splits
+
+The CNN/DailyMail dataset has 3 splits: _train_, _validation_, and _test_. Below are the statistics for Version 3.0.0 of the dataset.
+
+| Dataset Split | Number of Instances in Split                |
+| ------------- | ------------------------------------------- |
+| Train         | 287,113                                     |
+| Validation    | 13,368                                      |
+| Test          | 11,490                                      |
+
+## Dataset Creation
+
+### Curation Rationale
+
+Version 1.0.0 aimed to support supervised neural methodologies for machine reading and question answering with a large amount of real natural language training data and released about 313k unique articles and nearly 1M Cloze style questions to go with the articles. Versions 2.0.0 and 3.0.0 changed the structure of the dataset to support summarization rather than question answering. Version 3.0.0 provided a non-anonymized version of the data, whereas both the previous versions were preprocessed to replace named entities with unique identifier labels. 
+
+### Source Data
+
+#### Initial Data Collection and Normalization
+
+The data consists of news articles and highlight sentences. In the question answering setting of the data, the articles are used as the context and entities are hidden one at a time in the highlight sentences, producing Cloze style questions where the goal of the model is to correctly guess which entity in the context has been hidden in the highlight. In the summarization setting, the highlight sentences are concatenated to form a summary of the article. The CNN articles were written between April 2007 and April 2015. The Daily Mail articles were written between June 2010 and April 2015. 
+
+The code for the original data collection is available at <https://github.com/deepmind/rc-data>. The articles were downloaded using archives of <www.cnn.com> and <www.dailymail.co.uk> on the Wayback Machine. Articles were not included in the Version 1.0.0 collection if they exceeded 2000 tokens. Due to accessibility issues with the Wayback Machine, Kyunghyun Cho has made the datasets available at <https://cs.nyu.edu/~kcho/DMQA/>. An updated version of the code that does not anonymize the data is available at <https://github.com/abisee/cnn-dailymail>. 
+
+Hermann et al provided their own tokenization script. The script provided by See uses the PTBTokenizer. It also lowercases the text and adds periods to lines missing them.
+
+#### Who are the source language producers?
+
+The text was written by journalists at CNN and the Daily Mail. 
+
+### Annotations
+
+The dataset does not contain any additional annotations.
+
+#### Annotation process
+
+[N/A]
+
+#### Who are the annotators?
+
+[N/A]
+
+### Personal and Sensitive Information
+
+Version 3.0 is not anonymized, so individuals' names can be found in the dataset. Information about the original author is not included in the dataset.
+
+## Considerations for Using the Data
+
+### Social Impact of Dataset
+
+The purpose of this dataset is to help develop models that can summarize long paragraphs of text in one or two sentences.
+
+This task is useful for efficiently presenting information given a large quantity of text. It should be made clear that any summarizations produced by models trained on this dataset are reflective of the language used in the articles, but are in fact automatically generated.
+
+### Discussion of Biases
+
+[Bordia and Bowman (2019)](https://www.aclweb.org/anthology/N19-3002.pdf) explore measuring gender bias and debiasing techniques in the CNN / Dailymail dataset, the Penn Treebank, and WikiText-2. They find the CNN / Dailymail dataset to have a slightly lower gender bias based on their metric compared to the other datasets, but still show evidence of gender bias when looking at words such as 'fragile'.
+
+Because the articles were written by and for people in the US and the UK, they will likely present specifically US and UK perspectives and feature events that are considered relevant to those populations during the time that the articles were published. 
+
+### Other Known Limitations
+
+News articles have been shown to conform to writing conventions in which important information is primarily presented in the first third of the article [(Kryściński et al, 2019)](https://www.aclweb.org/anthology/D19-1051.pdf). [Chen et al (2016)](https://www.aclweb.org/anthology/P16-1223.pdf) conducted a manual study of 100 random instances of the first version of the dataset and found 25% of the samples to be difficult even for humans to answer correctly due to ambiguity and coreference errors. 
+
+It should also be noted that machine-generated summarizations, even when extractive, may differ in truth values when compared to the original articles. 
+
+## Additional Information
+
+### Dataset Curators
+
+The data was originally collected by Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom of Google DeepMind. Tomáš Kočiský and Phil Blunsom are also affiliated with the University of Oxford. They released scripts to collect and process the data into the question answering format. 
+
+Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, and Bing Xiang of IMB Watson and Çağlar Gu̇lçehre of Université de Montréal modified Hermann et al's collection scripts to restore the data to a summary format. They also produced both anonymized and non-anonymized versions.
+
+The code for the non-anonymized version is made publicly available by Abigail See of Stanford University, Peter J. Liu of Google Brain and Christopher D. Manning of Stanford University at <https://github.com/abisee/cnn-dailymail>. The work at Stanford University was supported by the DARPA DEFT ProgramAFRL contract no. FA8750-13-2-0040.
+
+### Licensing Information
+
+The CNN / Daily Mail dataset version 1.0.0 is released under the [Apache-2.0 License](http://www.apache.org/licenses/LICENSE-2.0). 
+
+### Citation Information
+
+```
+@inproceedings{see-etal-2017-get,
+    title = "Get To The Point: Summarization with Pointer-Generator Networks",
+    author = "See, Abigail  and
+      Liu, Peter J.  and
+      Manning, Christopher D.",
+    booktitle = "Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
+    month = jul,
+    year = "2017",
+    address = "Vancouver, Canada",
+    publisher = "Association for Computational Linguistics",
+    url = "https://www.aclweb.org/anthology/P17-1099",
+    doi = "10.18653/v1/P17-1099",
+    pages = "1073--1083",
+    abstract = "Neural sequence-to-sequence models have provided a viable new approach for abstractive text summarization (meaning they are not restricted to simply selecting and rearranging passages from the original text). However, these models have two shortcomings: they are liable to reproduce factual details inaccurately, and they tend to repeat themselves. In this work we propose a novel architecture that augments the standard sequence-to-sequence attentional model in two orthogonal ways. First, we use a hybrid pointer-generator network that can copy words from the source text via pointing, which aids accurate reproduction of information, while retaining the ability to produce novel words through the generator. Second, we use coverage to keep track of what has been summarized, which discourages repetition. We apply our model to the CNN / Daily Mail summarization task, outperforming the current abstractive state-of-the-art by at least 2 ROUGE points.",
+}
+```
+
+```
+@inproceedings{DBLP:conf/nips/HermannKGEKSB15,
+  author={Karl Moritz Hermann and Tomás Kociský and Edward Grefenstette and Lasse Espeholt and Will Kay and Mustafa Suleyman and Phil Blunsom},
+  title={Teaching Machines to Read and Comprehend},
+  year={2015},
+  cdate={1420070400000},
+  pages={1693-1701},
+  url={http://papers.nips.cc/paper/5945-teaching-machines-to-read-and-comprehend},
+  booktitle={NIPS},
+  crossref={conf/nips/2015}
+}
+
+```
+
+### Contributions
+
+Thanks to [@thomwolf](https://github.com/thomwolf), [@lewtun](https://github.com/lewtun), [@jplu](https://github.com/jplu), [@jbragg](https://github.com/jbragg), [@patrickvonplaten](https://github.com/patrickvonplaten) and [@mcmillanmajora](https://github.com/mcmillanmajora) for adding this dataset.

hf_cache/hub/datasets--allenai--openbookqa/blobs/08128898cc0433b97f7a9c9ff09c5054c8587e3e

@@ -0,0 +1,301 @@
+---
+annotations_creators:
+- crowdsourced
+- expert-generated
+language_creators:
+- expert-generated
+language:
+- en
+license:
+- unknown
+multilinguality:
+- monolingual
+size_categories:
+- 1K<n<10K
+source_datasets:
+- original
+task_categories:
+- question-answering
+task_ids:
+- open-domain-qa
+paperswithcode_id: openbookqa
+pretty_name: OpenBookQA
+dataset_info:
+- config_name: additional
+  features:
+  - name: id
+    dtype: string
+  - name: question_stem
+    dtype: string
+  - name: choices
+    sequence:
+    - name: text
+      dtype: string
+    - name: label
+      dtype: string
+  - name: answerKey
+    dtype: string
+  - name: fact1
+    dtype: string
+  - name: humanScore
+    dtype: float32
+  - name: clarity
+    dtype: float32
+  - name: turkIdAnonymized
+    dtype: string
+  splits:
+  - name: train
+    num_bytes: 1288577
+    num_examples: 4957
+  - name: validation
+    num_bytes: 135916
+    num_examples: 500
+  - name: test
+    num_bytes: 130701
+    num_examples: 500
+  download_size: 783789
+  dataset_size: 1555194
+- config_name: main
+  features:
+  - name: id
+    dtype: string
+  - name: question_stem
+    dtype: string
+  - name: choices
+    sequence:
+    - name: text
+      dtype: string
+    - name: label
+      dtype: string
+  - name: answerKey
+    dtype: string
+  splits:
+  - name: train
+    num_bytes: 895386
+    num_examples: 4957
+  - name: validation
+    num_bytes: 95428
+    num_examples: 500
+  - name: test
+    num_bytes: 91759
+    num_examples: 500
+  download_size: 609613
+  dataset_size: 1082573
+configs:
+- config_name: additional
+  data_files:
+  - split: train
+    path: additional/train-*
+  - split: validation
+    path: additional/validation-*
+  - split: test
+    path: additional/test-*
+- config_name: main
+  data_files:
+  - split: train
+    path: main/train-*
+  - split: validation
+    path: main/validation-*
+  - split: test
+    path: main/test-*
+  default: true
+---
+
+# Dataset Card for OpenBookQA
+
+## Table of Contents
+- [Dataset Description](#dataset-description)
+  - [Dataset Summary](#dataset-summary)
+  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
+  - [Languages](#languages)
+- [Dataset Structure](#dataset-structure)
+  - [Data Instances](#data-instances)
+  - [Data Fields](#data-fields)
+  - [Data Splits](#data-splits)
+- [Dataset Creation](#dataset-creation)
+  - [Curation Rationale](#curation-rationale)
+  - [Source Data](#source-data)
+  - [Annotations](#annotations)
+  - [Personal and Sensitive Information](#personal-and-sensitive-information)
+- [Considerations for Using the Data](#considerations-for-using-the-data)
+  - [Social Impact of Dataset](#social-impact-of-dataset)
+  - [Discussion of Biases](#discussion-of-biases)
+  - [Other Known Limitations](#other-known-limitations)
+- [Additional Information](#additional-information)
+  - [Dataset Curators](#dataset-curators)
+  - [Licensing Information](#licensing-information)
+  - [Citation Information](#citation-information)
+  - [Contributions](#contributions)
+
+## Dataset Description
+
+- **Homepage:** [https://allenai.org/data/open-book-qa](https://allenai.org/data/open-book-qa)
+- **Repository:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+- **Paper:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+- **Point of Contact:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+- **Size of downloaded dataset files:** 2.89 MB
+- **Size of the generated dataset:** 2.88 MB
+- **Total amount of disk used:** 5.78 MB
+
+### Dataset Summary
+
+OpenBookQA aims to promote research in advanced question-answering, probing a deeper understanding of both the topic
+(with salient facts summarized as an open book, also provided with the dataset) and the language it is expressed in. In
+particular, it contains questions that require multi-step reasoning, use of additional common and commonsense knowledge,
+and rich text comprehension.
+OpenBookQA is a new kind of question-answering dataset modeled after open book exams for assessing human understanding of
+a subject.
+
+### Supported Tasks and Leaderboards
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Languages
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Dataset Structure
+
+### Data Instances
+
+#### main
+
+- **Size of downloaded dataset files:** 1.45 MB
+- **Size of the generated dataset:** 1.45 MB
+- **Total amount of disk used:** 2.88 MB
+
+An example of 'train' looks as follows:
+```
+{'id': '7-980',
+ 'question_stem': 'The sun is responsible for',
+ 'choices': {'text': ['puppies learning new tricks',
+   'children growing up and getting old',
+   'flowers wilting in a vase',
+   'plants sprouting, blooming and wilting'],
+  'label': ['A', 'B', 'C', 'D']},
+ 'answerKey': 'D'}
+```
+
+#### additional
+
+- **Size of downloaded dataset files:** 1.45 MB
+- **Size of the generated dataset:** 1.45 MB
+- **Total amount of disk used:** 2.88 MB
+
+An example of 'train' looks as follows:
+```
+{'id': '7-980',
+ 'question_stem': 'The sun is responsible for',
+ 'choices': {'text': ['puppies learning new tricks',
+   'children growing up and getting old',
+   'flowers wilting in a vase',
+   'plants sprouting, blooming and wilting'],
+  'label': ['A', 'B', 'C', 'D']},
+ 'answerKey': 'D',
+ 'fact1': 'the sun is the source of energy for physical cycles on Earth',
+ 'humanScore': 1.0,
+ 'clarity': 2.0,
+ 'turkIdAnonymized': 'b356d338b7'}
+```
+
+### Data Fields
+
+The data fields are the same among all splits.
+
+#### main
+- `id`: a `string` feature.
+- `question_stem`: a `string` feature.
+- `choices`: a dictionary feature containing:
+  - `text`: a `string` feature.
+  - `label`: a `string` feature.
+- `answerKey`: a `string` feature.
+
+#### additional
+- `id`: a `string` feature.
+- `question_stem`: a `string` feature.
+- `choices`: a dictionary feature containing:
+  - `text`: a `string` feature.
+  - `label`: a `string` feature.
+- `answerKey`: a `string` feature.
+- `fact1` (`str`): oOriginating common knowledge core fact associated to the question.
+- `humanScore` (`float`): Human accuracy score.
+- `clarity` (`float`): Clarity score.
+- `turkIdAnonymized` (`str`): Anonymized crowd-worker ID.
+
+### Data Splits
+
+| name       | train | validation | test |
+|------------|------:|-----------:|-----:|
+| main       |  4957 |        500 |  500 |
+| additional |  4957 |        500 |  500 |
+
+## Dataset Creation
+
+### Curation Rationale
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Source Data
+
+#### Initial Data Collection and Normalization
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+#### Who are the source language producers?
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Annotations
+
+#### Annotation process
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+#### Who are the annotators?
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Personal and Sensitive Information
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Considerations for Using the Data
+
+### Social Impact of Dataset
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Discussion of Biases
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Other Known Limitations
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Additional Information
+
+### Dataset Curators
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Licensing Information
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Citation Information
+
+```
+@inproceedings{OpenBookQA2018,
+ title={Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering},
+ author={Todor Mihaylov and Peter Clark and Tushar Khot and Ashish Sabharwal},
+ booktitle={EMNLP},
+ year={2018}
+}
+
+```
+
+
+### Contributions
+
+Thanks to [@thomwolf](https://github.com/thomwolf), [@patrickvonplaten](https://github.com/patrickvonplaten), [@lewtun](https://github.com/lewtun) for adding this dataset.

hf_cache/hub/datasets--allenai--openbookqa/refs/main

@@ -0,0 +1 @@
+388097ea7776314e93a529163e0fea805b8a6454

hf_cache/hub/datasets--allenai--openbookqa/snapshots/388097ea7776314e93a529163e0fea805b8a6454/README.md

@@ -0,0 +1,301 @@
+---
+annotations_creators:
+- crowdsourced
+- expert-generated
+language_creators:
+- expert-generated
+language:
+- en
+license:
+- unknown
+multilinguality:
+- monolingual
+size_categories:
+- 1K<n<10K
+source_datasets:
+- original
+task_categories:
+- question-answering
+task_ids:
+- open-domain-qa
+paperswithcode_id: openbookqa
+pretty_name: OpenBookQA
+dataset_info:
+- config_name: additional
+  features:
+  - name: id
+    dtype: string
+  - name: question_stem
+    dtype: string
+  - name: choices
+    sequence:
+    - name: text
+      dtype: string
+    - name: label
+      dtype: string
+  - name: answerKey
+    dtype: string
+  - name: fact1
+    dtype: string
+  - name: humanScore
+    dtype: float32
+  - name: clarity
+    dtype: float32
+  - name: turkIdAnonymized
+    dtype: string
+  splits:
+  - name: train
+    num_bytes: 1288577
+    num_examples: 4957
+  - name: validation
+    num_bytes: 135916
+    num_examples: 500
+  - name: test
+    num_bytes: 130701
+    num_examples: 500
+  download_size: 783789
+  dataset_size: 1555194
+- config_name: main
+  features:
+  - name: id
+    dtype: string
+  - name: question_stem
+    dtype: string
+  - name: choices
+    sequence:
+    - name: text
+      dtype: string
+    - name: label
+      dtype: string
+  - name: answerKey
+    dtype: string
+  splits:
+  - name: train
+    num_bytes: 895386
+    num_examples: 4957
+  - name: validation
+    num_bytes: 95428
+    num_examples: 500
+  - name: test
+    num_bytes: 91759
+    num_examples: 500
+  download_size: 609613
+  dataset_size: 1082573
+configs:
+- config_name: additional
+  data_files:
+  - split: train
+    path: additional/train-*
+  - split: validation
+    path: additional/validation-*
+  - split: test
+    path: additional/test-*
+- config_name: main
+  data_files:
+  - split: train
+    path: main/train-*
+  - split: validation
+    path: main/validation-*
+  - split: test
+    path: main/test-*
+  default: true
+---
+
+# Dataset Card for OpenBookQA
+
+## Table of Contents
+- [Dataset Description](#dataset-description)
+  - [Dataset Summary](#dataset-summary)
+  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
+  - [Languages](#languages)
+- [Dataset Structure](#dataset-structure)
+  - [Data Instances](#data-instances)
+  - [Data Fields](#data-fields)
+  - [Data Splits](#data-splits)
+- [Dataset Creation](#dataset-creation)
+  - [Curation Rationale](#curation-rationale)
+  - [Source Data](#source-data)
+  - [Annotations](#annotations)
+  - [Personal and Sensitive Information](#personal-and-sensitive-information)
+- [Considerations for Using the Data](#considerations-for-using-the-data)
+  - [Social Impact of Dataset](#social-impact-of-dataset)
+  - [Discussion of Biases](#discussion-of-biases)
+  - [Other Known Limitations](#other-known-limitations)
+- [Additional Information](#additional-information)
+  - [Dataset Curators](#dataset-curators)
+  - [Licensing Information](#licensing-information)
+  - [Citation Information](#citation-information)
+  - [Contributions](#contributions)
+
+## Dataset Description
+
+- **Homepage:** [https://allenai.org/data/open-book-qa](https://allenai.org/data/open-book-qa)
+- **Repository:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+- **Paper:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+- **Point of Contact:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+- **Size of downloaded dataset files:** 2.89 MB
+- **Size of the generated dataset:** 2.88 MB
+- **Total amount of disk used:** 5.78 MB
+
+### Dataset Summary
+
+OpenBookQA aims to promote research in advanced question-answering, probing a deeper understanding of both the topic
+(with salient facts summarized as an open book, also provided with the dataset) and the language it is expressed in. In
+particular, it contains questions that require multi-step reasoning, use of additional common and commonsense knowledge,
+and rich text comprehension.
+OpenBookQA is a new kind of question-answering dataset modeled after open book exams for assessing human understanding of
+a subject.
+
+### Supported Tasks and Leaderboards
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Languages
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Dataset Structure
+
+### Data Instances
+
+#### main
+
+- **Size of downloaded dataset files:** 1.45 MB
+- **Size of the generated dataset:** 1.45 MB
+- **Total amount of disk used:** 2.88 MB
+
+An example of 'train' looks as follows:
+```
+{'id': '7-980',
+ 'question_stem': 'The sun is responsible for',
+ 'choices': {'text': ['puppies learning new tricks',
+   'children growing up and getting old',
+   'flowers wilting in a vase',
+   'plants sprouting, blooming and wilting'],
+  'label': ['A', 'B', 'C', 'D']},
+ 'answerKey': 'D'}
+```
+
+#### additional
+
+- **Size of downloaded dataset files:** 1.45 MB
+- **Size of the generated dataset:** 1.45 MB
+- **Total amount of disk used:** 2.88 MB
+
+An example of 'train' looks as follows:
+```
+{'id': '7-980',
+ 'question_stem': 'The sun is responsible for',
+ 'choices': {'text': ['puppies learning new tricks',
+   'children growing up and getting old',
+   'flowers wilting in a vase',
+   'plants sprouting, blooming and wilting'],
+  'label': ['A', 'B', 'C', 'D']},
+ 'answerKey': 'D',
+ 'fact1': 'the sun is the source of energy for physical cycles on Earth',
+ 'humanScore': 1.0,
+ 'clarity': 2.0,
+ 'turkIdAnonymized': 'b356d338b7'}
+```
+
+### Data Fields
+
+The data fields are the same among all splits.
+
+#### main
+- `id`: a `string` feature.
+- `question_stem`: a `string` feature.
+- `choices`: a dictionary feature containing:
+  - `text`: a `string` feature.
+  - `label`: a `string` feature.
+- `answerKey`: a `string` feature.
+
+#### additional
+- `id`: a `string` feature.
+- `question_stem`: a `string` feature.
+- `choices`: a dictionary feature containing:
+  - `text`: a `string` feature.
+  - `label`: a `string` feature.
+- `answerKey`: a `string` feature.
+- `fact1` (`str`): oOriginating common knowledge core fact associated to the question.
+- `humanScore` (`float`): Human accuracy score.
+- `clarity` (`float`): Clarity score.
+- `turkIdAnonymized` (`str`): Anonymized crowd-worker ID.
+
+### Data Splits
+
+| name       | train | validation | test |
+|------------|------:|-----------:|-----:|
+| main       |  4957 |        500 |  500 |
+| additional |  4957 |        500 |  500 |
+
+## Dataset Creation
+
+### Curation Rationale
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Source Data
+
+#### Initial Data Collection and Normalization
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+#### Who are the source language producers?
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Annotations
+
+#### Annotation process
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+#### Who are the annotators?
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Personal and Sensitive Information
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Considerations for Using the Data
+
+### Social Impact of Dataset
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Discussion of Biases
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Other Known Limitations
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Additional Information
+
+### Dataset Curators
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Licensing Information
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Citation Information
+
+```
+@inproceedings{OpenBookQA2018,
+ title={Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering},
+ author={Todor Mihaylov and Peter Clark and Tushar Khot and Ashish Sabharwal},
+ booktitle={EMNLP},
+ year={2018}
+}
+
+```
+
+
+### Contributions
+
+Thanks to [@thomwolf](https://github.com/thomwolf), [@patrickvonplaten](https://github.com/patrickvonplaten), [@lewtun](https://github.com/lewtun) for adding this dataset.

hf_cache/hub/datasets--allenai--sciq/blobs/c644057869cabcde87a2b5ab9665ec0d0bd1405b

@@ -0,0 +1,216 @@
+---
+annotations_creators:
+- no-annotation
+language_creators:
+- crowdsourced
+language:
+- en
+license:
+- cc-by-nc-3.0
+multilinguality:
+- monolingual
+size_categories:
+- 10K<n<100K
+source_datasets:
+- original
+task_categories:
+- question-answering
+task_ids:
+- closed-domain-qa
+paperswithcode_id: sciq
+pretty_name: SciQ
+dataset_info:
+  features:
+  - name: question
+    dtype: string
+  - name: distractor3
+    dtype: string
+  - name: distractor1
+    dtype: string
+  - name: distractor2
+    dtype: string
+  - name: correct_answer
+    dtype: string
+  - name: support
+    dtype: string
+  splits:
+  - name: train
+    num_bytes: 6546183
+    num_examples: 11679
+  - name: validation
+    num_bytes: 554120
+    num_examples: 1000
+  - name: test
+    num_bytes: 563927
+    num_examples: 1000
+  download_size: 4674410
+  dataset_size: 7664230
+configs:
+- config_name: default
+  data_files:
+  - split: train
+    path: data/train-*
+  - split: validation
+    path: data/validation-*
+  - split: test
+    path: data/test-*
+---
+
+# Dataset Card for "sciq"
+
+## Table of Contents
+- [Dataset Description](#dataset-description)
+  - [Dataset Summary](#dataset-summary)
+  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
+  - [Languages](#languages)
+- [Dataset Structure](#dataset-structure)
+  - [Data Instances](#data-instances)
+  - [Data Fields](#data-fields)
+  - [Data Splits](#data-splits)
+- [Dataset Creation](#dataset-creation)
+  - [Curation Rationale](#curation-rationale)
+  - [Source Data](#source-data)
+  - [Annotations](#annotations)
+  - [Personal and Sensitive Information](#personal-and-sensitive-information)
+- [Considerations for Using the Data](#considerations-for-using-the-data)
+  - [Social Impact of Dataset](#social-impact-of-dataset)
+  - [Discussion of Biases](#discussion-of-biases)
+  - [Other Known Limitations](#other-known-limitations)
+- [Additional Information](#additional-information)
+  - [Dataset Curators](#dataset-curators)
+  - [Licensing Information](#licensing-information)
+  - [Citation Information](#citation-information)
+  - [Contributions](#contributions)
+
+## Dataset Description
+
+- **Homepage:** [https://allenai.org/data/sciq](https://allenai.org/data/sciq)
+- **Repository:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+- **Paper:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+- **Point of Contact:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+- **Size of downloaded dataset files:** 2.82 MB
+- **Size of the generated dataset:** 7.68 MB
+- **Total amount of disk used:** 10.50 MB
+
+### Dataset Summary
+
+The SciQ dataset contains 13,679 crowdsourced science exam questions about Physics, Chemistry and Biology, among others. The questions are in multiple-choice format with 4 answer options each. For the majority of the questions, an additional paragraph with supporting evidence for the correct answer is provided.
+
+### Supported Tasks and Leaderboards
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Languages
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Dataset Structure
+
+### Data Instances
+
+#### default
+
+- **Size of downloaded dataset files:** 2.82 MB
+- **Size of the generated dataset:** 7.68 MB
+- **Total amount of disk used:** 10.50 MB
+
+An example of 'train' looks as follows.
+```
+This example was too long and was cropped:
+
+{
+    "correct_answer": "coriolis effect",
+    "distractor1": "muon effect",
+    "distractor2": "centrifugal effect",
+    "distractor3": "tropical effect",
+    "question": "What phenomenon makes global winds blow northeast to southwest or the reverse in the northern hemisphere and northwest to southeast or the reverse in the southern hemisphere?",
+    "support": "\"Without Coriolis Effect the global winds would blow north to south or south to north. But Coriolis makes them blow northeast to..."
+}
+```
+
+### Data Fields
+
+The data fields are the same among all splits.
+
+#### default
+- `question`: a `string` feature.
+- `distractor3`: a `string` feature.
+- `distractor1`: a `string` feature.
+- `distractor2`: a `string` feature.
+- `correct_answer`: a `string` feature.
+- `support`: a `string` feature.
+
+### Data Splits
+
+| name  |train|validation|test|
+|-------|----:|---------:|---:|
+|default|11679|      1000|1000|
+
+## Dataset Creation
+
+### Curation Rationale
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Source Data
+
+#### Initial Data Collection and Normalization
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+#### Who are the source language producers?
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Annotations
+
+#### Annotation process
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+#### Who are the annotators?
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Personal and Sensitive Information
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Considerations for Using the Data
+
+### Social Impact of Dataset
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Discussion of Biases
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Other Known Limitations
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Additional Information
+
+### Dataset Curators
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Licensing Information
+
+The dataset is licensed under the [Creative Commons Attribution-NonCommercial 3.0 Unported License](http://creativecommons.org/licenses/by-nc/3.0/).
+
+### Citation Information
+
+```
+@inproceedings{SciQ,
+    title={Crowdsourcing Multiple Choice Science Questions},
+    author={Johannes Welbl, Nelson F. Liu, Matt Gardner},
+    year={2017},
+    journal={arXiv:1707.06209v1}
+}
+```
+
+### Contributions
+
+Thanks to [@patrickvonplaten](https://github.com/patrickvonplaten), [@lewtun](https://github.com/lewtun), [@thomwolf](https://github.com/thomwolf) for adding this dataset.

hf_cache/hub/datasets--allenai--sciq/refs/main

@@ -0,0 +1 @@
+2c94ad3e1aafab77146f384e23536f97a4849815

hf_cache/hub/datasets--allenai--sciq/snapshots/2c94ad3e1aafab77146f384e23536f97a4849815/README.md

@@ -0,0 +1,216 @@
+---
+annotations_creators:
+- no-annotation
+language_creators:
+- crowdsourced
+language:
+- en
+license:
+- cc-by-nc-3.0
+multilinguality:
+- monolingual
+size_categories:
+- 10K<n<100K
+source_datasets:
+- original
+task_categories:
+- question-answering
+task_ids:
+- closed-domain-qa
+paperswithcode_id: sciq
+pretty_name: SciQ
+dataset_info:
+  features:
+  - name: question
+    dtype: string
+  - name: distractor3
+    dtype: string
+  - name: distractor1
+    dtype: string
+  - name: distractor2
+    dtype: string
+  - name: correct_answer
+    dtype: string
+  - name: support
+    dtype: string
+  splits:
+  - name: train
+    num_bytes: 6546183
+    num_examples: 11679
+  - name: validation
+    num_bytes: 554120
+    num_examples: 1000
+  - name: test
+    num_bytes: 563927
+    num_examples: 1000
+  download_size: 4674410
+  dataset_size: 7664230
+configs:
+- config_name: default
+  data_files:
+  - split: train
+    path: data/train-*
+  - split: validation
+    path: data/validation-*
+  - split: test
+    path: data/test-*
+---
+
+# Dataset Card for "sciq"
+
+## Table of Contents
+- [Dataset Description](#dataset-description)
+  - [Dataset Summary](#dataset-summary)
+  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
+  - [Languages](#languages)
+- [Dataset Structure](#dataset-structure)
+  - [Data Instances](#data-instances)
+  - [Data Fields](#data-fields)
+  - [Data Splits](#data-splits)
+- [Dataset Creation](#dataset-creation)
+  - [Curation Rationale](#curation-rationale)
+  - [Source Data](#source-data)
+  - [Annotations](#annotations)
+  - [Personal and Sensitive Information](#personal-and-sensitive-information)
+- [Considerations for Using the Data](#considerations-for-using-the-data)
+  - [Social Impact of Dataset](#social-impact-of-dataset)
+  - [Discussion of Biases](#discussion-of-biases)
+  - [Other Known Limitations](#other-known-limitations)
+- [Additional Information](#additional-information)
+  - [Dataset Curators](#dataset-curators)
+  - [Licensing Information](#licensing-information)
+  - [Citation Information](#citation-information)
+  - [Contributions](#contributions)
+
+## Dataset Description
+
+- **Homepage:** [https://allenai.org/data/sciq](https://allenai.org/data/sciq)
+- **Repository:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+- **Paper:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+- **Point of Contact:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+- **Size of downloaded dataset files:** 2.82 MB
+- **Size of the generated dataset:** 7.68 MB
+- **Total amount of disk used:** 10.50 MB
+
+### Dataset Summary
+
+The SciQ dataset contains 13,679 crowdsourced science exam questions about Physics, Chemistry and Biology, among others. The questions are in multiple-choice format with 4 answer options each. For the majority of the questions, an additional paragraph with supporting evidence for the correct answer is provided.
+
+### Supported Tasks and Leaderboards
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Languages
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Dataset Structure
+
+### Data Instances
+
+#### default
+
+- **Size of downloaded dataset files:** 2.82 MB
+- **Size of the generated dataset:** 7.68 MB
+- **Total amount of disk used:** 10.50 MB
+
+An example of 'train' looks as follows.
+```
+This example was too long and was cropped:
+
+{
+    "correct_answer": "coriolis effect",
+    "distractor1": "muon effect",
+    "distractor2": "centrifugal effect",
+    "distractor3": "tropical effect",
+    "question": "What phenomenon makes global winds blow northeast to southwest or the reverse in the northern hemisphere and northwest to southeast or the reverse in the southern hemisphere?",
+    "support": "\"Without Coriolis Effect the global winds would blow north to south or south to north. But Coriolis makes them blow northeast to..."
+}
+```
+
+### Data Fields
+
+The data fields are the same among all splits.
+
+#### default
+- `question`: a `string` feature.
+- `distractor3`: a `string` feature.
+- `distractor1`: a `string` feature.
+- `distractor2`: a `string` feature.
+- `correct_answer`: a `string` feature.
+- `support`: a `string` feature.
+
+### Data Splits
+
+| name  |train|validation|test|
+|-------|----:|---------:|---:|
+|default|11679|      1000|1000|
+
+## Dataset Creation
+
+### Curation Rationale
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Source Data
+
+#### Initial Data Collection and Normalization
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+#### Who are the source language producers?
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Annotations
+
+#### Annotation process
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+#### Who are the annotators?
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Personal and Sensitive Information
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Considerations for Using the Data
+
+### Social Impact of Dataset
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Discussion of Biases
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Other Known Limitations
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+## Additional Information
+
+### Dataset Curators
+
+[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
+
+### Licensing Information
+
+The dataset is licensed under the [Creative Commons Attribution-NonCommercial 3.0 Unported License](http://creativecommons.org/licenses/by-nc/3.0/).
+
+### Citation Information
+
+```
+@inproceedings{SciQ,
+    title={Crowdsourcing Multiple Choice Science Questions},
+    author={Johannes Welbl, Nelson F. Liu, Matt Gardner},
+    year={2017},
+    journal={arXiv:1707.06209v1}
+}
+```
+
+### Contributions
+
+Thanks to [@patrickvonplaten](https://github.com/patrickvonplaten), [@lewtun](https://github.com/lewtun), [@thomwolf](https://github.com/thomwolf) for adding this dataset.

hf_cache/hub/datasets--cais--mmlu/blobs/08de94c560ad7420252bff6e4729f1d1683def4f

@@ -0,0 +1,2299 @@
+---
+annotations_creators:
+- no-annotation
+language_creators:
+- expert-generated
+language:
+- en
+license:
+- mit
+multilinguality:
+- monolingual
+size_categories:
+- 10K<n<100K
+source_datasets:
+- original
+task_categories:
+- question-answering
+task_ids:
+- multiple-choice-qa
+paperswithcode_id: mmlu
+pretty_name: Measuring Massive Multitask Language Understanding
+language_bcp47:
+- en-US
+dataset_info:
+- config_name: abstract_algebra
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 49618.6654322746
+    num_examples: 100
+  - name: validation
+    num_bytes: 5485.515349444808
+    num_examples: 11
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 17143
+  dataset_size: 57303.3562203159
+- config_name: all
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 6967453
+    num_examples: 14042
+  - name: validation
+    num_bytes: 763484
+    num_examples: 1531
+  - name: dev
+    num_bytes: 125353
+    num_examples: 285
+  - name: auxiliary_train
+    num_bytes: 161000625
+    num_examples: 99842
+  download_size: 51503402
+  dataset_size: 168856915
+- config_name: anatomy
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 66985.19833357072
+    num_examples: 135
+  - name: validation
+    num_bytes: 6981.5649902024825
+    num_examples: 14
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 28864
+  dataset_size: 76165.9387623697
+- config_name: astronomy
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 75420.3714570574
+    num_examples: 152
+  - name: validation
+    num_bytes: 7978.931417374265
+    num_examples: 16
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 39316
+  dataset_size: 85598.47831302814
+- config_name: auxiliary_train
+  features:
+  - name: train
+    struct:
+    - name: answer
+      dtype: int64
+    - name: choices
+      sequence: string
+    - name: question
+      dtype: string
+    - name: subject
+      dtype: string
+  splits:
+  - name: train
+    num_bytes: 161000625
+    num_examples: 99842
+  download_size: 47518592
+  dataset_size: 161000625
+- config_name: business_ethics
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 49618.6654322746
+    num_examples: 100
+  - name: validation
+    num_bytes: 5485.515349444808
+    num_examples: 11
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 31619
+  dataset_size: 57303.3562203159
+- config_name: clinical_knowledge
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 131489.4633955277
+    num_examples: 265
+  - name: validation
+    num_bytes: 14461.813193990856
+    num_examples: 29
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 51655
+  dataset_size: 148150.45202811505
+- config_name: college_biology
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 71450.87822247542
+    num_examples: 144
+  - name: validation
+    num_bytes: 7978.931417374265
+    num_examples: 16
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 43017
+  dataset_size: 81628.98507844617
+- config_name: college_chemistry
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 49618.6654322746
+    num_examples: 100
+  - name: validation
+    num_bytes: 3989.4657086871325
+    num_examples: 8
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 26781
+  dataset_size: 55807.30657955822
+- config_name: college_computer_science
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 49618.6654322746
+    num_examples: 100
+  - name: validation
+    num_bytes: 5485.515349444808
+    num_examples: 11
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 41132
+  dataset_size: 57303.3562203159
+- config_name: college_mathematics
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 49618.6654322746
+    num_examples: 100
+  - name: validation
+    num_bytes: 5485.515349444808
+    num_examples: 11
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 26779
+  dataset_size: 57303.3562203159
+- config_name: college_medicine
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 85840.29119783506
+    num_examples: 173
+  - name: validation
+    num_bytes: 10971.030698889615
+    num_examples: 22
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 56303
+  dataset_size: 99010.49733532117
+- config_name: college_physics
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 50611.0387409201
+    num_examples: 102
+  - name: validation
+    num_bytes: 5485.515349444808
+    num_examples: 11
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 29539
+  dataset_size: 58295.7295289614
+- config_name: computer_security
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 49618.6654322746
+    num_examples: 100
+  - name: validation
+    num_bytes: 5485.515349444808
+    num_examples: 11
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 30150
+  dataset_size: 57303.3562203159
+- config_name: conceptual_physics
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 116603.86376584532
+    num_examples: 235
+  - name: validation
+    num_bytes: 12965.76355323318
+    num_examples: 26
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 34968
+  dataset_size: 131768.802757675
+- config_name: econometrics
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 56565.27859279305
+    num_examples: 114
+  - name: validation
+    num_bytes: 5984.198563030699
+    num_examples: 12
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 36040
+  dataset_size: 64748.652594420244
+- config_name: electrical_engineering
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 71947.06487679818
+    num_examples: 145
+  - name: validation
+    num_bytes: 7978.931417374265
+    num_examples: 16
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 26746
+  dataset_size: 82125.17173276893
+- config_name: elementary_mathematics
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 187558.555333998
+    num_examples: 378
+  - name: validation
+    num_bytes: 20446.011757021555
+    num_examples: 41
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 54987
+  dataset_size: 210203.74252961605
+- config_name: formal_logic
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 62519.518444666
+    num_examples: 126
+  - name: validation
+    num_bytes: 6981.5649902024825
+    num_examples: 14
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 32884
+  dataset_size: 71700.25887346498
+- config_name: global_facts
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 49618.6654322746
+    num_examples: 100
+  - name: validation
+    num_bytes: 4986.8321358589155
+    num_examples: 10
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 19258
+  dataset_size: 56804.67300673001
+- config_name: high_school_biology
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 153817.86284005127
+    num_examples: 310
+  - name: validation
+    num_bytes: 15957.86283474853
+    num_examples: 32
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 78216
+  dataset_size: 171974.90111339628
+- config_name: high_school_chemistry
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 100725.89082751745
+    num_examples: 203
+  - name: validation
+    num_bytes: 10971.030698889615
+    num_examples: 22
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 45799
+  dataset_size: 113896.09696500355
+- config_name: high_school_computer_science
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 49618.6654322746
+    num_examples: 100
+  - name: validation
+    num_bytes: 4488.148922273024
+    num_examples: 9
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 39072
+  dataset_size: 56305.989793144116
+- config_name: high_school_european_history
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 81870.79796325309
+    num_examples: 165
+  - name: validation
+    num_bytes: 8976.297844546049
+    num_examples: 18
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 196270
+  dataset_size: 93046.27124639563
+- config_name: high_school_geography
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 98244.95755590372
+    num_examples: 198
+  - name: validation
+    num_bytes: 10971.030698889615
+    num_examples: 22
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 38255
+  dataset_size: 111415.16369338983
+- config_name: high_school_government_and_politics
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 95764.02428428999
+    num_examples: 193
+  - name: validation
+    num_bytes: 10472.347485303722
+    num_examples: 21
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 52963
+  dataset_size: 108435.5472081902
+- config_name: high_school_macroeconomics
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 193512.79518587096
+    num_examples: 390
+  - name: validation
+    num_bytes: 21443.378184193338
+    num_examples: 43
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 68758
+  dataset_size: 217155.34880866078
+- config_name: high_school_mathematics
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 133970.39666714144
+    num_examples: 270
+  - name: validation
+    num_bytes: 14461.813193990856
+    num_examples: 29
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 45210
+  dataset_size: 150631.38529972878
+- config_name: high_school_microeconomics
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 118092.42372881356
+    num_examples: 238
+  - name: validation
+    num_bytes: 12965.76355323318
+    num_examples: 26
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 49885
+  dataset_size: 133257.36272064323
+- config_name: high_school_physics
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 74924.18480273466
+    num_examples: 151
+  - name: validation
+    num_bytes: 8477.614630960157
+    num_examples: 17
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 45483
+  dataset_size: 85600.9748722913
+- config_name: high_school_psychology
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 270421.7266058966
+    num_examples: 545
+  - name: validation
+    num_bytes: 29920.992815153495
+    num_examples: 60
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 113158
+  dataset_size: 302541.8948596466
+- config_name: high_school_statistics
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 107176.31733371314
+    num_examples: 216
+  - name: validation
+    num_bytes: 11469.713912475507
+    num_examples: 23
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 74924
+  dataset_size: 120845.20668478514
+- config_name: high_school_us_history
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 101222.0774818402
+    num_examples: 204
+  - name: validation
+    num_bytes: 10971.030698889615
+    num_examples: 22
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 200043
+  dataset_size: 114392.2836193263
+- config_name: high_school_world_history
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 117596.23707449081
+    num_examples: 237
+  - name: validation
+    num_bytes: 12965.76355323318
+    num_examples: 26
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 250302
+  dataset_size: 132761.17606632048
+- config_name: human_aging
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 110649.62391397236
+    num_examples: 223
+  - name: validation
+    num_bytes: 11469.713912475507
+    num_examples: 23
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 41196
+  dataset_size: 124318.51326504436
+- config_name: human_sexuality
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 65000.451716279735
+    num_examples: 131
+  - name: validation
+    num_bytes: 5984.198563030699
+    num_examples: 12
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 32533
+  dataset_size: 73183.82571790692
+- config_name: international_law
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 60038.58517305227
+    num_examples: 121
+  - name: validation
+    num_bytes: 6482.88177661659
+    num_examples: 13
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 41592
+  dataset_size: 68720.64238826535
+- config_name: jurisprudence
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 53588.15866685657
+    num_examples: 108
+  - name: validation
+    num_bytes: 5485.515349444808
+    num_examples: 11
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 33578
+  dataset_size: 61272.84945489787
+- config_name: logical_fallacies
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 80878.4246546076
+    num_examples: 163
+  - name: validation
+    num_bytes: 8976.297844546049
+    num_examples: 18
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 33669
+  dataset_size: 92053.89793775014
+- config_name: machine_learning
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 55572.90528414756
+    num_examples: 112
+  - name: validation
+    num_bytes: 5485.515349444808
+    num_examples: 11
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 31121
+  dataset_size: 63257.596072188855
+- config_name: management
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 51107.225395242844
+    num_examples: 103
+  - name: validation
+    num_bytes: 5485.515349444808
+    num_examples: 11
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 22828
+  dataset_size: 58791.91618328414
+- config_name: marketing
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 116107.67711152257
+    num_examples: 234
+  - name: validation
+    num_bytes: 12467.08033964729
+    num_examples: 25
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 49747
+  dataset_size: 130773.93288976635
+- config_name: medical_genetics
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 49618.6654322746
+    num_examples: 100
+  - name: validation
+    num_bytes: 5485.515349444808
+    num_examples: 11
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 25775
+  dataset_size: 57303.3562203159
+- config_name: miscellaneous
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 388514.15033471014
+    num_examples: 783
+  - name: validation
+    num_bytes: 42886.756368386676
+    num_examples: 86
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 115097
+  dataset_size: 433600.08214169333
+- config_name: moral_disputes
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 171680.58239567012
+    num_examples: 346
+  - name: validation
+    num_bytes: 18949.96211626388
+    num_examples: 38
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 76043
+  dataset_size: 192829.71995053047
+- config_name: moral_scenarios
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 444087.05561885773
+    num_examples: 895
+  - name: validation
+    num_bytes: 49868.32135858916
+    num_examples: 100
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 109869
+  dataset_size: 496154.5524160434
+- config_name: nutrition
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 151833.1162227603
+    num_examples: 306
+  - name: validation
+    num_bytes: 16456.54604833442
+    num_examples: 33
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 69050
+  dataset_size: 170488.8377096912
+- config_name: philosophy
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 154314.04949437402
+    num_examples: 311
+  - name: validation
+    num_bytes: 16955.229261920314
+    num_examples: 34
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 61912
+  dataset_size: 173468.45419489083
+- config_name: prehistory
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 160764.47600056973
+    num_examples: 324
+  - name: validation
+    num_bytes: 17453.912475506204
+    num_examples: 35
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 68826
+  dataset_size: 180417.5639146724
+- config_name: professional_accounting
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 139924.6365190144
+    num_examples: 282
+  - name: validation
+    num_bytes: 15459.179621162639
+    num_examples: 31
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 87297
+  dataset_size: 157582.99157877354
+- config_name: professional_law
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 761150.3277310925
+    num_examples: 1534
+  - name: validation
+    num_bytes: 84776.14630960157
+    num_examples: 170
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 1167828
+  dataset_size: 848125.6494792906
+- config_name: professional_medicine
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 134962.7699757869
+    num_examples: 272
+  - name: validation
+    num_bytes: 15459.179621162639
+    num_examples: 31
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 153242
+  dataset_size: 152621.12503554605
+- config_name: professional_psychology
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 303666.2324455206
+    num_examples: 612
+  - name: validation
+    num_bytes: 34409.14173742652
+    num_examples: 69
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 159357
+  dataset_size: 340274.5496215436
+- config_name: public_relations
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 54580.53197550207
+    num_examples: 110
+  - name: validation
+    num_bytes: 5984.198563030699
+    num_examples: 12
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 31500
+  dataset_size: 62763.90597712925
+- config_name: security_studies
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 121565.73030907278
+    num_examples: 245
+  - name: validation
+    num_bytes: 13464.446766819072
+    num_examples: 27
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 140258
+  dataset_size: 137229.35251448833
+- config_name: sociology
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 99733.51751887196
+    num_examples: 201
+  - name: validation
+    num_bytes: 10971.030698889615
+    num_examples: 22
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 56480
+  dataset_size: 112903.72365635807
+- config_name: us_foreign_policy
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 49618.6654322746
+    num_examples: 100
+  - name: validation
+    num_bytes: 5485.515349444808
+    num_examples: 11
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 29027
+  dataset_size: 57303.3562203159
+- config_name: virology
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 82366.98461757584
+    num_examples: 166
+  - name: validation
+    num_bytes: 8976.297844546049
+    num_examples: 18
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 38229
+  dataset_size: 93542.45790071838
+- config_name: world_religions
+  features:
+  - name: question
+    dtype: string
+  - name: subject
+    dtype: string
+  - name: choices
+    sequence: string
+  - name: answer
+    dtype:
+      class_label:
+        names:
+          '0': A
+          '1': B
+          '2': C
+          '3': D
+  splits:
+  - name: test
+    num_bytes: 84847.91788918957
+    num_examples: 171
+  - name: validation
+    num_bytes: 9474.98105813194
+    num_examples: 19
+  - name: dev
+    num_bytes: 2199.1754385964914
+    num_examples: 5
+  download_size: 27165
+  dataset_size: 96522.07438591801
+configs:
+- config_name: abstract_algebra
+  data_files:
+  - split: test
+    path: abstract_algebra/test-*
+  - split: validation
+    path: abstract_algebra/validation-*
+  - split: dev
+    path: abstract_algebra/dev-*
+- config_name: all
+  data_files:
+  - split: test
+    path: all/test-*
+  - split: validation
+    path: all/validation-*
+  - split: dev
+    path: all/dev-*
+  - split: auxiliary_train
+    path: all/auxiliary_train-*
+- config_name: anatomy
+  data_files:
+  - split: test
+    path: anatomy/test-*
+  - split: validation
+    path: anatomy/validation-*
+  - split: dev
+    path: anatomy/dev-*
+- config_name: astronomy
+  data_files:
+  - split: test
+    path: astronomy/test-*
+  - split: validation
+    path: astronomy/validation-*
+  - split: dev
+    path: astronomy/dev-*
+- config_name: auxiliary_train
+  data_files:
+  - split: train
+    path: auxiliary_train/train-*
+- config_name: business_ethics
+  data_files:
+  - split: test
+    path: business_ethics/test-*
+  - split: validation
+    path: business_ethics/validation-*
+  - split: dev
+    path: business_ethics/dev-*
+- config_name: clinical_knowledge
+  data_files:
+  - split: test
+    path: clinical_knowledge/test-*
+  - split: validation
+    path: clinical_knowledge/validation-*
+  - split: dev
+    path: clinical_knowledge/dev-*
+- config_name: college_biology
+  data_files:
+  - split: test
+    path: college_biology/test-*
+  - split: validation
+    path: college_biology/validation-*
+  - split: dev
+    path: college_biology/dev-*
+- config_name: college_chemistry
+  data_files:
+  - split: test
+    path: college_chemistry/test-*
+  - split: validation
+    path: college_chemistry/validation-*
+  - split: dev
+    path: college_chemistry/dev-*
+- config_name: college_computer_science
+  data_files:
+  - split: test
+    path: college_computer_science/test-*
+  - split: validation
+    path: college_computer_science/validation-*
+  - split: dev
+    path: college_computer_science/dev-*
+- config_name: college_mathematics
+  data_files:
+  - split: test
+    path: college_mathematics/test-*
+  - split: validation
+    path: college_mathematics/validation-*
+  - split: dev
+    path: college_mathematics/dev-*
+- config_name: college_medicine
+  data_files:
+  - split: test
+    path: college_medicine/test-*
+  - split: validation
+    path: college_medicine/validation-*
+  - split: dev
+    path: college_medicine/dev-*
+- config_name: college_physics
+  data_files:
+  - split: test
+    path: college_physics/test-*
+  - split: validation
+    path: college_physics/validation-*
+  - split: dev
+    path: college_physics/dev-*
+- config_name: computer_security
+  data_files:
+  - split: test
+    path: computer_security/test-*
+  - split: validation
+    path: computer_security/validation-*
+  - split: dev
+    path: computer_security/dev-*
+- config_name: conceptual_physics
+  data_files:
+  - split: test
+    path: conceptual_physics/test-*
+  - split: validation
+    path: conceptual_physics/validation-*
+  - split: dev
+    path: conceptual_physics/dev-*
+- config_name: econometrics
+  data_files:
+  - split: test
+    path: econometrics/test-*
+  - split: validation
+    path: econometrics/validation-*
+  - split: dev
+    path: econometrics/dev-*
+- config_name: electrical_engineering
+  data_files:
+  - split: test
+    path: electrical_engineering/test-*
+  - split: validation
+    path: electrical_engineering/validation-*
+  - split: dev
+    path: electrical_engineering/dev-*
+- config_name: elementary_mathematics
+  data_files:
+  - split: test
+    path: elementary_mathematics/test-*
+  - split: validation
+    path: elementary_mathematics/validation-*
+  - split: dev
+    path: elementary_mathematics/dev-*
+- config_name: formal_logic
+  data_files:
+  - split: test
+    path: formal_logic/test-*
+  - split: validation
+    path: formal_logic/validation-*
+  - split: dev
+    path: formal_logic/dev-*
+- config_name: global_facts
+  data_files:
+  - split: test
+    path: global_facts/test-*
+  - split: validation
+    path: global_facts/validation-*
+  - split: dev
+    path: global_facts/dev-*
+- config_name: high_school_biology
+  data_files:
+  - split: test
+    path: high_school_biology/test-*
+  - split: validation
+    path: high_school_biology/validation-*
+  - split: dev
+    path: high_school_biology/dev-*
+- config_name: high_school_chemistry
+  data_files:
+  - split: test
+    path: high_school_chemistry/test-*
+  - split: validation
+    path: high_school_chemistry/validation-*
+  - split: dev
+    path: high_school_chemistry/dev-*
+- config_name: high_school_computer_science
+  data_files:
+  - split: test
+    path: high_school_computer_science/test-*
+  - split: validation
+    path: high_school_computer_science/validation-*
+  - split: dev
+    path: high_school_computer_science/dev-*
+- config_name: high_school_european_history
+  data_files:
+  - split: test
+    path: high_school_european_history/test-*
+  - split: validation
+    path: high_school_european_history/validation-*
+  - split: dev
+    path: high_school_european_history/dev-*
+- config_name: high_school_geography
+  data_files:
+  - split: test
+    path: high_school_geography/test-*
+  - split: validation
+    path: high_school_geography/validation-*
+  - split: dev
+    path: high_school_geography/dev-*
+- config_name: high_school_government_and_politics
+  data_files:
+  - split: test
+    path: high_school_government_and_politics/test-*
+  - split: validation
+    path: high_school_government_and_politics/validation-*
+  - split: dev
+    path: high_school_government_and_politics/dev-*
+- config_name: high_school_macroeconomics
+  data_files:
+  - split: test
+    path: high_school_macroeconomics/test-*
+  - split: validation
+    path: high_school_macroeconomics/validation-*
+  - split: dev
+    path: high_school_macroeconomics/dev-*
+- config_name: high_school_mathematics
+  data_files:
+  - split: test
+    path: high_school_mathematics/test-*
+  - split: validation
+    path: high_school_mathematics/validation-*
+  - split: dev
+    path: high_school_mathematics/dev-*
+- config_name: high_school_microeconomics
+  data_files:
+  - split: test
+    path: high_school_microeconomics/test-*
+  - split: validation
+    path: high_school_microeconomics/validation-*
+  - split: dev
+    path: high_school_microeconomics/dev-*
+- config_name: high_school_physics
+  data_files:
+  - split: test
+    path: high_school_physics/test-*
+  - split: validation
+    path: high_school_physics/validation-*
+  - split: dev
+    path: high_school_physics/dev-*
+- config_name: high_school_psychology
+  data_files:
+  - split: test
+    path: high_school_psychology/test-*
+  - split: validation
+    path: high_school_psychology/validation-*
+  - split: dev
+    path: high_school_psychology/dev-*
+- config_name: high_school_statistics
+  data_files:
+  - split: test
+    path: high_school_statistics/test-*
+  - split: validation
+    path: high_school_statistics/validation-*
+  - split: dev
+    path: high_school_statistics/dev-*
+- config_name: high_school_us_history
+  data_files:
+  - split: test
+    path: high_school_us_history/test-*
+  - split: validation
+    path: high_school_us_history/validation-*
+  - split: dev
+    path: high_school_us_history/dev-*
+- config_name: high_school_world_history
+  data_files:
+  - split: test
+    path: high_school_world_history/test-*
+  - split: validation
+    path: high_school_world_history/validation-*
+  - split: dev
+    path: high_school_world_history/dev-*
+- config_name: human_aging
+  data_files:
+  - split: test
+    path: human_aging/test-*
+  - split: validation
+    path: human_aging/validation-*
+  - split: dev
+    path: human_aging/dev-*
+- config_name: human_sexuality
+  data_files:
+  - split: test
+    path: human_sexuality/test-*
+  - split: validation
+    path: human_sexuality/validation-*
+  - split: dev
+    path: human_sexuality/dev-*
+- config_name: international_law
+  data_files:
+  - split: test
+    path: international_law/test-*
+  - split: validation
+    path: international_law/validation-*
+  - split: dev
+    path: international_law/dev-*
+- config_name: jurisprudence
+  data_files:
+  - split: test
+    path: jurisprudence/test-*
+  - split: validation
+    path: jurisprudence/validation-*
+  - split: dev
+    path: jurisprudence/dev-*
+- config_name: logical_fallacies
+  data_files:
+  - split: test
+    path: logical_fallacies/test-*
+  - split: validation
+    path: logical_fallacies/validation-*
+  - split: dev
+    path: logical_fallacies/dev-*
+- config_name: machine_learning
+  data_files:
+  - split: test
+    path: machine_learning/test-*
+  - split: validation
+    path: machine_learning/validation-*
+  - split: dev
+    path: machine_learning/dev-*
+- config_name: management
+  data_files:
+  - split: test
+    path: management/test-*
+  - split: validation
+    path: management/validation-*
+  - split: dev
+    path: management/dev-*
+- config_name: marketing
+  data_files:
+  - split: test
+    path: marketing/test-*
+  - split: validation
+    path: marketing/validation-*
+  - split: dev
+    path: marketing/dev-*
+- config_name: medical_genetics
+  data_files:
+  - split: test
+    path: medical_genetics/test-*
+  - split: validation
+    path: medical_genetics/validation-*
+  - split: dev
+    path: medical_genetics/dev-*
+- config_name: miscellaneous
+  data_files:
+  - split: test
+    path: miscellaneous/test-*
+  - split: validation
+    path: miscellaneous/validation-*
+  - split: dev
+    path: miscellaneous/dev-*
+- config_name: moral_disputes
+  data_files:
+  - split: test
+    path: moral_disputes/test-*
+  - split: validation
+    path: moral_disputes/validation-*
+  - split: dev
+    path: moral_disputes/dev-*
+- config_name: moral_scenarios
+  data_files:
+  - split: test
+    path: moral_scenarios/test-*
+  - split: validation
+    path: moral_scenarios/validation-*
+  - split: dev
+    path: moral_scenarios/dev-*
+- config_name: nutrition
+  data_files:
+  - split: test
+    path: nutrition/test-*
+  - split: validation
+    path: nutrition/validation-*
+  - split: dev
+    path: nutrition/dev-*
+- config_name: philosophy
+  data_files:
+  - split: test
+    path: philosophy/test-*
+  - split: validation
+    path: philosophy/validation-*
+  - split: dev
+    path: philosophy/dev-*
+- config_name: prehistory
+  data_files:
+  - split: test
+    path: prehistory/test-*
+  - split: validation
+    path: prehistory/validation-*
+  - split: dev
+    path: prehistory/dev-*
+- config_name: professional_accounting
+  data_files:
+  - split: test
+    path: professional_accounting/test-*
+  - split: validation
+    path: professional_accounting/validation-*
+  - split: dev
+    path: professional_accounting/dev-*
+- config_name: professional_law
+  data_files:
+  - split: test
+    path: professional_law/test-*
+  - split: validation
+    path: professional_law/validation-*
+  - split: dev
+    path: professional_law/dev-*
+- config_name: professional_medicine
+  data_files:
+  - split: test
+    path: professional_medicine/test-*
+  - split: validation
+    path: professional_medicine/validation-*
+  - split: dev
+    path: professional_medicine/dev-*
+- config_name: professional_psychology
+  data_files:
+  - split: test
+    path: professional_psychology/test-*
+  - split: validation
+    path: professional_psychology/validation-*
+  - split: dev
+    path: professional_psychology/dev-*
+- config_name: public_relations
+  data_files:
+  - split: test
+    path: public_relations/test-*
+  - split: validation
+    path: public_relations/validation-*
+  - split: dev
+    path: public_relations/dev-*
+- config_name: security_studies
+  data_files:
+  - split: test
+    path: security_studies/test-*
+  - split: validation
+    path: security_studies/validation-*
+  - split: dev
+    path: security_studies/dev-*
+- config_name: sociology
+  data_files:
+  - split: test
+    path: sociology/test-*
+  - split: validation
+    path: sociology/validation-*
+  - split: dev
+    path: sociology/dev-*
+- config_name: us_foreign_policy
+  data_files:
+  - split: test
+    path: us_foreign_policy/test-*
+  - split: validation
+    path: us_foreign_policy/validation-*
+  - split: dev
+    path: us_foreign_policy/dev-*
+- config_name: virology
+  data_files:
+  - split: test
+    path: virology/test-*
+  - split: validation
+    path: virology/validation-*
+  - split: dev
+    path: virology/dev-*
+- config_name: world_religions
+  data_files:
+  - split: test
+    path: world_religions/test-*
+  - split: validation
+    path: world_religions/validation-*
+  - split: dev
+    path: world_religions/dev-*
+---
+
+# Dataset Card for MMLU
+
+## Table of Contents
+- [Table of Contents](#table-of-contents)
+- [Dataset Description](#dataset-description)
+  - [Dataset Summary](#dataset-summary)
+  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
+  - [Languages](#languages)
+- [Dataset Structure](#dataset-structure)
+  - [Data Instances](#data-instances)
+  - [Data Fields](#data-fields)
+  - [Data Splits](#data-splits)
+- [Dataset Creation](#dataset-creation)
+  - [Curation Rationale](#curation-rationale)
+  - [Source Data](#source-data)
+  - [Annotations](#annotations)
+  - [Personal and Sensitive Information](#personal-and-sensitive-information)
+- [Considerations for Using the Data](#considerations-for-using-the-data)
+  - [Social Impact of Dataset](#social-impact-of-dataset)
+  - [Discussion of Biases](#discussion-of-biases)
+  - [Other Known Limitations](#other-known-limitations)
+- [Additional Information](#additional-information)
+  - [Dataset Curators](#dataset-curators)
+  - [Licensing Information](#licensing-information)
+  - [Citation Information](#citation-information)
+  - [Contributions](#contributions)
+
+## Dataset Description
+
+- **Repository**: https://github.com/hendrycks/test
+- **Paper**: https://arxiv.org/abs/2009.03300
+
+### Dataset Summary
+
+[Measuring Massive Multitask Language Understanding](https://arxiv.org/pdf/2009.03300) by [Dan Hendrycks](https://people.eecs.berkeley.edu/~hendrycks/), [Collin Burns](http://collinpburns.com), [Steven Basart](https://stevenbas.art), Andy Zou, Mantas Mazeika, [Dawn Song](https://people.eecs.berkeley.edu/~dawnsong/), and [Jacob Steinhardt](https://www.stat.berkeley.edu/~jsteinhardt/) (ICLR 2021).
+
+This is a massive multitask test consisting of multiple-choice questions from various branches of knowledge. The test spans subjects in the humanities, social sciences, hard sciences, and other areas that are important for some people to learn. This covers 57 tasks including elementary mathematics, US history, computer science, law, and more. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability.
+
+A complete list of tasks: ['abstract_algebra', 'anatomy', 'astronomy', 'business_ethics', 'clinical_knowledge', 'college_biology', 'college_chemistry', 'college_computer_science', 'college_mathematics', 'college_medicine', 'college_physics', 'computer_security', 'conceptual_physics', 'econometrics', 'electrical_engineering', 'elementary_mathematics', 'formal_logic', 'global_facts', 'high_school_biology', 'high_school_chemistry', 'high_school_computer_science', 'high_school_european_history', 'high_school_geography', 'high_school_government_and_politics', 'high_school_macroeconomics', 'high_school_mathematics', 'high_school_microeconomics', 'high_school_physics', 'high_school_psychology', 'high_school_statistics', 'high_school_us_history', 'high_school_world_history', 'human_aging', 'human_sexuality', 'international_law', 'jurisprudence', 'logical_fallacies', 'machine_learning', 'management', 'marketing', 'medical_genetics', 'miscellaneous', 'moral_disputes', 'moral_scenarios', 'nutrition', 'philosophy', 'prehistory', 'professional_accounting', 'professional_law', 'professional_medicine', 'professional_psychology', 'public_relations', 'security_studies', 'sociology', 'us_foreign_policy', 'virology', 'world_religions']
+
+### Supported Tasks and Leaderboards
+
+|                Model               | Authors |  Humanities |  Social Science  | STEM | Other | Average |
+|------------------------------------|----------|:-------:|:-------:|:-------:|:-------:|:-------:|
+| [UnifiedQA](https://arxiv.org/abs/2005.00700) | Khashabi et al., 2020 | 45.6 | 56.6 | 40.2 | 54.6 | 48.9
+| [GPT-3](https://arxiv.org/abs/2005.14165) (few-shot) | Brown et al., 2020 | 40.8 | 50.4 | 36.7 | 48.8 | 43.9
+| [GPT-2](https://arxiv.org/abs/2005.14165) | Radford et al., 2019 | 32.8 | 33.3 | 30.2 | 33.1 | 32.4
+| Random Baseline           | N/A | 25.0 | 25.0 | 25.0 | 25.0 | 25.0 | 25.0
+
+### Languages
+
+English
+
+## Dataset Structure
+
+### Data Instances
+
+An example from anatomy subtask looks as follows:
+```
+{
+  "question": "What is the embryological origin of the hyoid bone?",
+  "choices": ["The first pharyngeal arch", "The first and second pharyngeal arches", "The second pharyngeal arch", "The second and third pharyngeal arches"],
+  "answer": "D"
+}
+```
+
+### Data Fields
+
+- `question`: a string feature
+- `choices`: a list of 4 string features
+- `answer`: a ClassLabel feature
+
+### Data Splits
+
+- `auxiliary_train`: auxiliary multiple-choice training questions from ARC, MC_TEST, OBQA, RACE, etc.
+- `dev`: 5 examples per subtask, meant for few-shot setting
+- `test`: there are at least 100 examples per subtask
+
+|       | auxiliary_train   | dev | val | test |
+| ----- | :------: | :-----: | :-----: | :-----: |
+| TOTAL | 99842 | 285 | 1531 | 14042
+
+## Dataset Creation
+
+### Curation Rationale
+
+Transformer models have driven this recent progress by pretraining on massive text corpora, including all of Wikipedia, thousands of books, and numerous websites. These models consequently see extensive information about specialized topics, most of which is not assessed by existing NLP benchmarks. To bridge the gap between the wide-ranging knowledge that models see during pretraining and the existing measures of success, we introduce a new benchmark for assessing models across a diverse set of subjects that humans learn.
+
+### Source Data
+
+#### Initial Data Collection and Normalization
+
+[More Information Needed]
+
+#### Who are the source language producers?
+
+[More Information Needed]
+
+### Annotations
+
+#### Annotation process
+
+[More Information Needed]
+
+#### Who are the annotators?
+
+[More Information Needed]
+
+### Personal and Sensitive Information
+
+[More Information Needed]
+
+## Considerations for Using the Data
+
+### Social Impact of Dataset
+
+[More Information Needed]
+
+### Discussion of Biases
+
+[More Information Needed]
+
+### Other Known Limitations
+
+[More Information Needed]
+
+## Additional Information
+
+### Dataset Curators
+
+[More Information Needed]
+
+### Licensing Information
+
+[MIT License](https://github.com/hendrycks/test/blob/master/LICENSE)
+
+### Citation Information
+
+If you find this useful in your research, please consider citing the test and also the [ETHICS](https://arxiv.org/abs/2008.02275) dataset it draws from:
+```
+    @article{hendryckstest2021,
+      title={Measuring Massive Multitask Language Understanding},
+      author={Dan Hendrycks and Collin Burns and Steven Basart and Andy Zou and Mantas Mazeika and Dawn Song and Jacob Steinhardt},
+      journal={Proceedings of the International Conference on Learning Representations (ICLR)},
+      year={2021}
+    }
+
+    @article{hendrycks2021ethics,
+      title={Aligning AI With Shared Human Values},
+      author={Dan Hendrycks and Collin Burns and Steven Basart and Andrew Critch and Jerry Li and Dawn Song and Jacob Steinhardt},
+      journal={Proceedings of the International Conference on Learning Representations (ICLR)},
+      year={2021}
+    }
+```
+### Contributions
+
+Thanks to [@andyzoujm](https://github.com/andyzoujm) for adding this dataset.