UniPath (complete)
Full contents of the UniPath working directory: ~983 GB across checkpoints, datasets and code.
Layout
Directories with a small number of large files are stored as plain files, so any single checkpoint can be downloaded on its own:
| Path | Files | Size |
|---|---|---|
ckpt/ |
2826 | 620 GB |
data/ |
337 | 202 GB |
ckpt_mol/ |
439 | 14.7 GB |
code_ignore/, code_ALIGNN_ignore/, code_mol_ignore/ |
473 | 2.2 GB |
ckpt_etflow/ |
1 | 0.1 GB |
data_mol/stable/qm9.tar.gz |
1 | 0.8 GB |
Directories holding hundreds of thousands of tiny files are stored under
archives/ as split tar volumes, because uploading them file-by-file is not
practical (data_MD_baseline alone contains 2.42 M files averaging 3.9 KB).
| Archive set | Volumes | Size |
|---|---|---|
data_MD_baseline |
1 | 6.0 GB |
data_mol__stable__rdkit_folder |
10 | 50.1 GB |
data_mol__stable__rdkit_folder_hf |
10 | 50.1 GB |
data_mol__unstable__Trans_1x |
3 | 11.9 GB |
data_mol__unstable__ANI_1x |
2 | 6.2 GB |
data_mol__unstable__Trans_1x_sub_qm9 |
2 | 5.5 GB |
data_mol__unstable__ANI_1x_sub_qm9 |
1 | 2.6 GB |
data_mol__unstable__DPA2_Drug |
1 | 2.6 GB |
data_mol__stable__QM9 |
1 | 2.0 GB |
data_mol__unstable__ANI_1x_Sub1M |
1 | 1.3 GB |
data_mol__unstable__Trans_1x_sub1M |
1 | 1.2 GB |
data_mol__stable__processed |
1 | 0.7 GB |
code_ignore, code_ALIGNN_ignore, code_mol_ignore, ckpt_etflow |
1 each | 2.5 GB |
__ in an archive name is a path separator: data_mol__stable__rdkit_folder
unpacks to data_mol/stable/rdkit_folder.
The code_* archives duplicate the plain code_*/ directories, with one
difference: the archives include each project's .git directory, which the
plain upload omits.
Restoring an archive set
Volumes are pieces of one tar stream and must be concatenated in order:
hf download Wthinker/UniPath_Complete \
--include "archives/data_mol__stable__rdkit_folder.tar.*" \
--local-dir .
cat archives/data_mol__stable__rdkit_folder.tar.* | tar -xf -
Single-volume sets work the same way:
cat archives/data_MD_baseline.tar.000 | tar -xf -