| --- |
| license: cc-by-4.0 |
| --- |
| |
| # UniPath (complete) |
|
|
| Full contents of the `UniPath` working directory: ~983 GB across checkpoints, datasets and code. |
|
|
| ## Layout |
|
|
| Directories with a small number of large files are stored as plain files, so any |
| single checkpoint can be downloaded on its own: |
|
|
| | Path | Files | Size | |
| |---|---|---| |
| | `ckpt/` | 2826 | 620 GB | |
| | `data/` | 337 | 202 GB | |
| | `ckpt_mol/` | 439 | 14.7 GB | |
| | `code_ignore/`, `code_ALIGNN_ignore/`, `code_mol_ignore/` | 473 | 2.2 GB | |
| | `ckpt_etflow/` | 1 | 0.1 GB | |
| | `data_mol/stable/qm9.tar.gz` | 1 | 0.8 GB | |
|
|
| Directories holding hundreds of thousands of tiny files are stored under |
| `archives/` as split tar volumes, because uploading them file-by-file is not |
| practical (`data_MD_baseline` alone contains 2.42 M files averaging 3.9 KB). |
|
|
| | Archive set | Volumes | Size | |
| |---|---|---| |
| | `data_MD_baseline` | 1 | 6.0 GB | |
| | `data_mol__stable__rdkit_folder` | 10 | 50.1 GB | |
| | `data_mol__stable__rdkit_folder_hf` | 10 | 50.1 GB | |
| | `data_mol__unstable__Trans_1x` | 3 | 11.9 GB | |
| | `data_mol__unstable__ANI_1x` | 2 | 6.2 GB | |
| | `data_mol__unstable__Trans_1x_sub_qm9` | 2 | 5.5 GB | |
| | `data_mol__unstable__ANI_1x_sub_qm9` | 1 | 2.6 GB | |
| | `data_mol__unstable__DPA2_Drug` | 1 | 2.6 GB | |
| | `data_mol__stable__QM9` | 1 | 2.0 GB | |
| | `data_mol__unstable__ANI_1x_Sub1M` | 1 | 1.3 GB | |
| | `data_mol__unstable__Trans_1x_sub1M` | 1 | 1.2 GB | |
| | `data_mol__stable__processed` | 1 | 0.7 GB | |
| | `code_ignore`, `code_ALIGNN_ignore`, `code_mol_ignore`, `ckpt_etflow` | 1 each | 2.5 GB | |
|
|
| `__` in an archive name is a path separator: `data_mol__stable__rdkit_folder` |
| unpacks to `data_mol/stable/rdkit_folder`. |
|
|
| The `code_*` archives duplicate the plain `code_*/` directories, with one |
| difference: the archives include each project's `.git` directory, which the |
| plain upload omits. |
|
|
| ## Restoring an archive set |
|
|
| Volumes are pieces of one tar stream and must be concatenated in order: |
|
|
| ```bash |
| hf download Wthinker/UniPath_Complete \ |
| --include "archives/data_mol__stable__rdkit_folder.tar.*" \ |
| --local-dir . |
| |
| cat archives/data_mol__stable__rdkit_folder.tar.* | tar -xf - |
| ``` |
|
|
| Single-volume sets work the same way: |
|
|
| ```bash |
| cat archives/data_MD_baseline.tar.000 | tar -xf - |
| ``` |
|
|
