## Practical Recommendations

### **Structure of data in a HuggingFace datasets**

#### Datasets, sub-datasets, splits

* A HuggingFace dataset contains multiple sub-datasets e.g. at different filter/stringency levels.  
* Each sub-dataset has one or more splits, typically ('train', 'validate', 'test'). If the data does not have splits it will be 'train'.  
* The data in different splits of a single sub-dataset should non-overlapping  
* Example:  
  * The [MegaScale](https://huggingface.co/datasets/RosettaCommons/MegaScale) contains 6 datasets  
    * dataset1                   \# all stability measurements  
    * dataset2                   \# high-quality folding stabilities  
    * dataset3                   \# ΔG measurements  
    * dataset3\_single       \# ΔG measurements of single-point mutants with ThermoMPNN (Dieckhaus, et al., 2024\) splits  
    * dataset3\_single\_cv \# 5-fold cross validation of ΔG measurements of single-point mutants with ThermoMPNN (Dieckhaus, et al., 2024\) splits  
  * To load a specific subdataset:
    ```
    datasets.load_dataset(path = "RosettaCommons/MegaScale", name = "dataset1", data_dir = "dataset1")
    ```

#### Example: One .csv file dataset

One table named `outcomes.csv` to be pushed to HuggingFace dataset repository `maomlab/example_dataset`  
First load the dataset locally then push it to the hub:

    import datasets  
    dataset = datasets.load_dataset(  
        "csv",  
        data_files ="outcomes.csv",  
        keep_in_memory = True)

    dataset.push_to_hub(repo_id = "`maomlab/example_dataset`")

This will create the following files in the repo

    data/  
	    train-00000-of-00001.parquet

and add the following to the header of README.md

    dataset_info:  
      features:  
        - name: id  
          dtype: int64  
        - name: value  
          dtype: int64  
      splits:  
        - name: train  
          num_bytes: 64  
          num_examples: 4  
      download_size: 1332  
      dataset_size: 64  
    configs:  
      - config_name: default  
        data_files:  
          - split: train  
            path: data/train-*

to load these data from HuggingFace

    dataset = datasets.load_dataset("maomlab/example_dataset")

#### Example: train/valid/test split .csv files

Three tables train.csv, valid.csv, test.csv to be pushed to HuggingFace dataset repository `maomlab/example_dataset`  
load the three splits into one dataset and push it to the hub:

    import datasets  
    dataset = datasets.load_dataset(  
        'csv',  
        data_dir = "/tmp",  
        data_files = {  
          'train': 'train.csv',  
          'valid': 'valid.csv',  
          'test': 'test.csv'},  
        keep_in_memory = True)

	dataset.push_to_hub(repo_id = "maomlab/example_dataset")

This will create the following files in the repo

    data/  
	    train-00000-of-00001.parquet  
	    valid-00000-of-00001.parquet  
	    test-00000-of-00001.parquet

and add the following to the header of the README.md

    dataset_info:  
      features:  
        - name: id  
          dtype: int64  
        - name: value  
          dtype: int64  
       splits:  
        - name: train  
          num_bytes: 64  
          num_examples: 4  
        - name: valid  
          num_bytes: 64  
          num_examples: 4  
        - name: test  
          num_bytes: 64  
          num_examples: 4  
      download_size: 3996  
      dataset_size: 192  
    configs:  
      - config_name: default  
        data_files:  
          - split: train  
            path: data/train-*  
          - split: valid  
            path: data/valid-*  
          - split: test  
            path: data/test-*

to load these data from HuggingFace

    dataset = datasets.load_dataset("maomlab/example_dataset")

#### Example: sub-datasets

If you have different related datasets (`dataset1.csv`, `dataset2.csv`, `dataset3.csv`) that should go into a single repository but contain different types of data so they aren't just splits of the same dataset, then load each dataset separately and push it to the hub with a given config name.

	import datasets  
    dataset1 = datasets.load_dataset('csv', data_files = '/tmp/dataset1.csv', keep_in\_memory = True)  
    dataset2 = datasets.load_dataset('csv', data_files = '/tmp/dataset2.csv', keep_in\_memory = True)  
    dataset3 = datasets.load_dataset('csv', data_files = '/tmp/dataset3.csv', keep_in\_memory = True)

    dataset1.push_to_hub(repo_id = "`maomlab/example_dataset`", config_name = 'dataset1', data_dir = 'dataset1/data')  
    dataset2.push_to_hub(repo_id = "`maomlab/example_dataset`", config_name = 'dataset2', data_dir = 'dataset2/data')  
    dataset3.push_to_hub(repo_id = "`maomlab/example_dataset`", config_name = 'dataset3', data_dir = 'dataset3/data')

This will create the following files in the repo

    dataset1/  
	    data/  
	    train-00000-of-00001.parquet  
    dataset2/  
	    data/  
	    train-00000-of-00001.parquet  
    dataset3/  
	    data/  
	    train-00000-of-00001.parquet

and add the following to the header of the README.md

    dataset_info:  
      - config_name: dataset1  
        features:  
          - name: id  
            dtype: int64  
          - name: value1  
            dtype: int64  
        splits:  
          - name: train  
            num_bytes: 64  
            num_examples: 4  
        download_size: 1344  
        dataset_size: 64  
      - config_name: dataset2  
        features:  
          - name: id  
            dtype: int64  
          - name: value2  
            dtype: int64  
        splits:  
          - name: train  
            num_bytes: 64  
            num_examples: 4  
        download_size: 1344  
        dataset_size: 64  
      - config_name: dataset3  
        features:  
          - name: id  
            dtype: int64  
          - name: value3  
            dtype: int64  
        splits:  
          - name: train  
            num_bytes: 64  
            num_examples: 4  
        download_size: 1344  
        dataset_size: 64  
    configs:  
      - config_name: dataset1  
        data_files:  
          - split: train  
            path: dataset1/data/train-*  
      - config_name: dataset2  
        data_files:  
          - split: train  
            path: dataset2/data/train-*  
      - config_name: dataset3  
        data_files:  
          - split: train  
            path: dataset3/data/train-*

to load these datasets from HuggingFace

    dataset1 = datasets.load_dataset("maomlab/example_dataset", name = 'dataset1', data_dir = 'dataset1')  
	dataset2 = datasets.load_dataset("maomlab/example_dataset", name = 'dataset2', data_dir = 'dataset2')  
	dataset3 = datasets.load_dataset("maomlab/example_dataset", name = 'dataset3', data_dir = 'dataset3')


### **Format of a dataset**

A dataset should consist of a single table where each row is a single observation  
The columns should follow typical database design guidelines 

* Identifier columns  
  * sequential key  
    * For example: `[1, 2, 3, ...]`  
  * primary key  
    * single column that uniquely identify each row  
      * distinct for every row  
      * no non-missing values  
    * For example, for a dataset of protein structures from the Protein Data Bank, the PDB ID is the primary key  
  * composite key  
    * A set of columns that uniquely identify each row  
      * Either hierarchical or complementary ids that characterize the observation  
      * For example, for an observation of mutations, the (`structure_id`, `residue_id`, `mutation_aa`) is a unique identifier  
  * additional/foreign key identifiers  
    * identifiers to link the observation with other data  
    * For example  
      * for compounds identified by PubChem SubstanceID, the ZINC ID for the compound could be a foreign key  
      * FDA drug name or the IUPAC substance name  
* Tidy key/value columns  
  * [Tidy vs array data](https://vita.had.co.nz/papers/tidy-data.pdf)  
    * tidy data sometimes called (long) has one measurement per row  
      * Multiple columns can be used to give details for each measurement including type, units, metadata  
      * Often good for certain data science computational analysis workflows (e.g. tidyverse/dplyr)  
      * Can handle variable number of measurements per object  
      * Duplicates object identifier columns for each measurement  
    * array data sometimes called (wide) has one object per row and multiple measurements as different columns  
      * Typically each measurement is typically a single column  
      * More compact, i.e. no duplication of identifier columns  
      * Good for certain ML/matrix based computational workflows

#### Molecular formats

* Store molecular structure in standard text formats   
  * protein structure: PDB, mmCIF, modelCIF  
  * small molecule: SMILES, InChi  
  * use uncompressed, plaintext format  
    * Easier to computationally analyze  
    * the whole dataset will be compressed for data serialization  
* Filtering / Standardization / sanitization  
  * Be clear about process methods used to process the molecular data  
  * Be especially careful for inferred / aspects of the data  
    * protonation states,  
    * salt form, stereochemistry for small molecules  
    * data missingness including unstructured loops for proteins  
  * Tools  
    * MolVS is useful for small molecule sanitization

#### Computational data formats

* On disk formats  
  * parquet format disk format  
    * column oriented (so can load only data that is needed, easier to compress)  
    * robust reader/write codes from apache arrow for Python, R etc.  
  * ArrowTable  
    * In memory format closely aligned with the on disk parquet format   
    * Native format for datasets stored in datasets python package  
  * tab/comma separated table  
    * Prefer tab separated, more consistent parsing without needing escaping values  
    * Widely used row-oriented text format for storing tabular data to disk  
    * Does not store data format and often needs custom format conversion code/QC for loading into python/R  
    * Can be compressed on disk but row-oriented, so less compressible than .parquet  
  * .pickle / .Rdata  
    * language specific serialization of complex data structures  
    * Often very fast to read/write, but may not be robust for across language/OS versions  
    * Not easily interoperable across programming languages  
* In memory formats  
  * R `data.frame`/`dplyr::tibble`  
    * Widely used format for R data science  
    * Out of the box faster for tidyverse data manipulation, split-apply-combine workflows  
  * Python pandas DataFrame  
    * Widely used for python data science  
    * Out of the box not super fast for data science  
  * Python numpy array / R Matrix  
    * Uses single data type for all data  
    * Useful for efficient/matrix manipulation  
  * Python Pytorch dataset  
    * Format specifically geared for loading data for Pytorch deep-learning

Recommendations

* On disk  
  * For small, config level tables use .tsv  
  * For large data format use .parquet  
    * Smaller than .csv/.tsv  
    * Robust open source libraries in major language can read and write .parquet files faster than .csv/.tsv  
* In memory  
  * Use `dplyr::tibble` / pandas DataFrame for data science tables  
  * Use numpy array / pytorch dataset for machine learning