Buckets:
| pretty_name: BFD Source Archive Index | |
| license: other | |
| tags: | |
| - biology | |
| - protein | |
| - sequence-database | |
| - bfd | |
| - fasta | |
| - archive | |
| - parquet | |
| configs: | |
| - config_name: default | |
| data_files: | |
| - split: train | |
| path: data/train-*.parquet | |
| - split: test | |
| path: data/test-*.parquet | |
| # BFD Source Archive Index | |
| BFD is a very large clustered protein sequence database built from UniProt and metagenomic sequence resources, commonly used for homology search and multiple sequence alignment generation. | |
| ## Splits | |
| The split is deterministic by chunk identifier: `sha256(index_id) % 10`. Bucket `0` is `test`; buckets `1` through `9` are `train`. | |
| | Split | Rows | | |
| |---|---:| | |
| | train | 255 | | |
| | test | 17 | | |
| | total | 272 | | |
| ## Source File | |
| | File | Size | | |
| |---|---:| | |
| | `bfd_metaclust_clu_complete_id30_c90_final_seq.sorted_opt.tar.gz` | 291,649,557,441 bytes | | |
| ## Usage | |
| ```bash | |
| pip install datasets | |
| ``` | |
| Load the archive index: | |
| ```python | |
| from datasets import load_dataset | |
| ds = load_dataset("LiteFold/BFD") | |
| print(ds) | |
| print(ds["train"][0]) | |
| ``` | |
| Load one split: | |
| ```python | |
| from datasets import load_dataset | |
| train = load_dataset("LiteFold/BFD", split="train") | |
| test = load_dataset("LiteFold/BFD", split="test") | |
| ``` | |
| Download the original archive: | |
| ```python | |
| from huggingface_hub import hf_hub_download | |
| archive_path = hf_hub_download( | |
| repo_id="LiteFold/BFD", | |
| repo_type="dataset", | |
| filename="bfd_metaclust_clu_complete_id30_c90_final_seq.sorted_opt.tar.gz", | |
| ) | |
| ``` | |
| Load the source-file manifest: | |
| ```python | |
| import pandas as pd | |
| from huggingface_hub import hf_hub_download | |
| path = hf_hub_download( | |
| repo_id="LiteFold/BFD", | |
| repo_type="dataset", | |
| filename="metadata/source_files.parquet", | |
| ) | |
| source_files = pd.read_parquet(path) | |
| print(source_files) | |
| ``` | |
| ## Columns | |
| | Column | Description | | |
| |---|---| | |
| | `index_id` | Stable row ID for one compressed chunk. | | |
| | `repo_id` | Hugging Face dataset repository. | | |
| | `source_file` | Source archive filename. | | |
| | `source_sha` | Source repository commit used to build the index. | | |
| | `source_format` | Source archive format. | | |
| | `chunk_index` | Zero-based compressed chunk index. | | |
| | `byte_start` | Inclusive compressed byte offset. | | |
| | `byte_end_exclusive` | Exclusive compressed byte offset. | | |
| | `chunk_size_bytes` | Compressed chunk size in bytes. | | |
| | `total_size_bytes` | Full compressed archive size. | | |
| | `chunk_size_gib` | Target chunk size in GiB. | | |
| | `is_first_chunk` | Whether the row indexes the first chunk. | | |
| | `is_last_chunk` | Whether the row indexes the final chunk. | | |
| | `access_note` | Note describing the index scope. | | |
| | `split_bucket` | Deterministic split bucket from `sha256(index_id) % 10`. | | |
| # Citation | |
| ``` | |
| @article{steinegger2019bfd, | |
| title = {Protein-level assembly increases protein sequence recovery from metagenomic samples manyfold}, | |
| author = {Steinegger, Martin and Mirdita, Milot and S{\"o}ding, Johannes}, | |
| journal = {Nature Methods}, | |
| volume = {16}, | |
| pages = {603--606}, | |
| year = {2019}, | |
| doi = {10.1038/s41592-019-0437-4} | |
| } | |
| ``` |
Xet Storage Details
- Size:
- 3.03 kB
- Xet hash:
- 22dc3d66f3328eb5d5a95bd88ba61c4f9edab9bd79c4ea04ec2faa646fab4554
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.