Create 07_how_to_structure_curation.md

sections/07_how_to_structure_curation.md

@@ -0,0 +1,200 @@
+### **Structure of data in a HuggingFace datasets**
+
+#### Datasets, sub-datasets, splits
+
+* A HuggingFace dataset contains multiple sub-datasets e.g. at different filter/stringency levels.  
+* Each sub-dataset has one or more splits, typically ('train', 'validate', 'test'). If the data does not have splits it will be 'train'.  
+* The data in different splits of a single sub-dataset should non-overlapping  
+* Example:  
+  * The [MegaScale](https://huggingface.co/datasets/RosettaCommons/MegaScale) contains 6 datasets  
+    * dataset1                   \# all stability measurements  
+    * dataset2                   \# high-quality folding stabilities  
+    * dataset3                   \# ΔG measurements  
+    * dataset3\_single       \# ΔG measurements of single-point mutants with ThermoMPNN (Dieckhaus, et al., 2024\) splits  
+    * dataset3\_single\_cv \# 5-fold cross validation of ΔG measurements of single-point mutants with ThermoMPNN (Dieckhaus, et al., 2024\) splits  
+  * To load a specific subdataset:  
+    * datasets.load\_dataset(path \= "RosettaCommons/MegaScale", name \= "dataset1", data\_dir \= "dataset1")
+
+#### Example: One .csv file dataset
+
+One table named `outcomes.csv` to be pushed to HuggingFace dataset repository `maomlab/example_dataset`  
+First load the dataset locally then push it to the hub:
+
+import datasets  
+dataset \= datasets.load\_dataset(  
+    "csv",  
+    data\_files \="outcomes.csv",  
+    keep\_in\_memory \= True)
+
+dataset.push\_to\_hub(repo\_id \= "`maomlab/example_dataset`")
+
+This will create the following files in the repo
+
+data/  
+	train-00000-of-00001.parquet
+
+and add the following to the header of README.md
+
+dataset\_info:  
+  features:  
+    \- name: id  
+      dtype: int64  
+    \- name: value  
+      dtype: int64  
+  splits:  
+    \- name: train  
+      num\_bytes: 64  
+      num\_examples: 4  
+  download\_size: 1332  
+  dataset\_size: 64  
+configs:  
+  \- config\_name: default  
+    data\_files:  
+      \- split: train  
+        path: data/train-\*
+
+to load these data from HuggingFace
+
+`dataset = datasets.load_dataset("maomlab/example_dataset")`
+
+#### Example: train/valid/test split .csv files
+
+Three tables train.csv, valid.csv, test.csv to be pushed to HuggingFace dataset repository `maomlab/example_dataset`  
+load the three splits into one dataset and push it to the hub:
+
+import datasets  
+dataset \= datasets.load\_dataset(  
+'csv',  
+data\_dir \= "/tmp",  
+data\_files \= {  
+'train': 'train.csv',  
+'valid': 'valid.csv',  
+'test': 'test.csv'},  
+keep\_in\_memory \= True)
+
+	dataset.push\_to\_hub(repo\_id \= "maomlab/example\_dataset")
+
+This will create the following files in the repo
+
+data/  
+	train-00000-of-00001.parquet  
+	valid-00000-of-00001.parquet  
+	test-00000-of-00001.parquet
+
+and add the following to the header of the README.md
+
+dataset\_info:  
+  features:  
+    \- name: id  
+      dtype: int64  
+    \- name: value  
+      dtype: int64  
+  splits:  
+    \- name: train  
+      num\_bytes: 64  
+      num\_examples: 4  
+    \- name: valid  
+      num\_bytes: 64  
+      num\_examples: 4  
+    \- name: test  
+      num\_bytes: 64  
+      num\_examples: 4  
+  download\_size: 3996  
+  dataset\_size: 192  
+configs:  
+  \- config\_name: default  
+    data\_files:  
+      \- split: train  
+        path: data/train-\*  
+      \- split: valid  
+        path: data/valid-\*  
+      \- split: test  
+        path: data/test-\*
+
+to load these data from HuggingFace
+
+`dataset = datasets.load_dataset("maomlab/example_dataset")`
+
+#### Example: sub-datasets
+
+If you have different related datasets (`dataset1.csv`, `dataset2.csv`, `dataset3.csv`) that should go into a single repository but contain different types of data so they aren't just splits of the same dataset, then load each dataset separately and push it to the hub with a given config name.
+
+	import datasets  
+dataset1 \= datasets.load\_dataset('csv', data\_files \= '/tmp/dataset1.csv', keep\_in\_memory \= True)  
+dataset2 \= datasets.load\_dataset('csv', data\_files \= '/tmp/dataset2.csv', keep\_in\_memory \= True)  
+dataset3 \= datasets.load\_dataset('csv', data\_files \= '/tmp/dataset3.csv', keep\_in\_memory \= True)
+
+dataset1.push\_to\_hub(repo\_id \= "`maomlab/example_dataset`", config\_name \= 'dataset1', data\_dir \= 'dataset1/data')  
+dataset2.push\_to\_hub(repo\_id \= "`maomlab/example_dataset`", config\_name \= 'dataset2', data\_dir \= 'dataset2/data')  
+dataset3.push\_to\_hub(repo\_id \= "`maomlab/example_dataset`", config\_name \= 'dataset3', data\_dir \= 'dataset3/data')
+
+This will create the following files in the repo
+
+dataset1/  
+	data/  
+	train-00000-of-00001.parquet  
+dataset2/  
+	data/  
+	train-00000-of-00001.parquet  
+dataset3/  
+	data/  
+	train-00000-of-00001.parquet
+
+and add the following to the header of the README.md
+
+dataset\_info:  
+  \- config\_name: dataset1  
+    features:  
+      \- name: id  
+        dtype: int64  
+      \- name: value1  
+        dtype: int64  
+    splits:  
+      \- name: train  
+        num\_bytes: 64  
+        num\_examples: 4  
+    download\_size: 1344  
+    dataset\_size: 64  
+  \- config\_name: dataset2  
+    features:  
+      \- name: id  
+        dtype: int64  
+      \- name: value2  
+        dtype: int64  
+    splits:  
+      \- name: train  
+        num\_bytes: 64  
+        num\_examples: 4  
+    download\_size: 1344  
+    dataset\_size: 64  
+  \- config\_name: dataset3  
+    features:  
+      \- name: id  
+        dtype: int64  
+      \- name: value3  
+        dtype: int64  
+    splits:  
+      \- name: train  
+        num\_bytes: 64  
+        num\_examples: 4  
+    download\_size: 1344  
+    dataset\_size: 64  
+configs:  
+  \- config\_name: dataset1  
+    data\_files:  
+      \- split: train  
+        path: dataset1/data/train-\*  
+  \- config\_name: dataset2  
+    data\_files:  
+      \- split: train  
+        path: dataset2/data/train-\*  
+  \- config\_name: dataset3  
+    data\_files:  
+      \- split: train  
+        path: dataset3/data/train-\*
+
+to load these datasets from HuggingFace
+
+`dataset1 = datasets.load_dataset("maomlab/example_dataset", name = 'dataset1', data_dir = 'dataset1')`  
+	`dataset2 = datasets.load_dataset("maomlab/example_dataset", name = 'dataset2', data_dir = 'dataset2')`  
+	`dataset3 = datasets.load_dataset("maomlab/example_dataset", name = 'dataset3', data_dir = 'dataset3')`