data/document_structures/contacts_v1/README.md

# contacts-v1 protein-document dataset

Training documents in the **contacts-v1** format (single-chain protein →
sequence section + side-chain *contact* statements), generated from
[`timodonnell/afdb-24M`](https://huggingface.co/datasets/timodonnell/afdb-24M)
by calling MarinFold's `contacts_v1` generator (no re-implementation). See the
format spec: [`marinfold/.../document_structures/contacts_v1`](https://github.com/Open-Athena/MarinFold/tree/main/marinfold/marinfold/document_structures/contacts_v1).

Produced by experiment [exp53](https://github.com/Open-Athena/MarinFold/issues/53)
(`marinfold` @ `dd025aab`, which includes the `min_seq_separation=6` contact filter).

## Structure (rounds)

Up to **5 examples per structural cluster**, organised into pLDDT **rounds**:
`round = 0` is the highest-pLDDT representative of each `struct_cluster_id`,
`round = 1` the next, etc. Structural clusters with **fewer than 3 usable
members are dropped** (so `round-0 == round-1 == round-2` in size). Shards are
laid out **round-descending** (round-4 first … round-0 last) so a sequential
reader trains on the highest-quality data **last**. The `round` column is
authoritative regardless of file order.

`split` (`train`/`val`/`test`) is inherited from afdb-24M and is
cluster-consistent (a whole structural cluster stays in one split → no
train/test leakage).

## Layout

```
<split>/contacts_v1-{shard:05d}-of-{total:05d}.parquet   # 2000 rows/shard, round-descending
tokenizer/                                               # contacts-v1 tokenizer (2845 tokens)
```

## Counts

**4,213,203 documents** over **960,054 structural clusters** (0 generation drops).

| split | round 0 | round 1 | round 2 | round 3 | round 4 | total |
|---|--:|--:|--:|--:|--:|--:|
| train | 941,028 | 941,028 | 941,028 | 719,519 | 587,079 | 4,129,682 |
| val | 9,558 | 9,558 | 9,558 | 7,316 | 5,964 | 41,954 |
| test | 9,468 | 9,468 | 9,468 | 7,248 | 5,915 | 41,567 |

**~4.67 B tokens in `train`** (4.77 B total; mean ~1,131 `num_tokens`/doc, max 8,192; 245 docs truncated at the budget). Mean ~200 contacts/doc.

## Columns

`document` (token string) · `structure` (`"contacts-v1"`) · `entry_id` · `round` ·
`struct_cluster_id` · `seq_cluster_id` · `split` · `global_plddt` · `seq_len` ·
`start_index` / `n_term_index` / `c_term_index` · `contacts_pre_filter` /
`contacts_passing_min_degree` / `contacts_emitted` / `contacts_excluded` ·
`truncated` · `highest_contact_degree` / `lowest_nonzero_contact_degree` /
`lowest_included_contact_degree` · `num_tokens` · `sha1` ·
`uniprot_accession` / `tax_id` / `organism_name`.

`sha1` = sha1 of `document`, so byte-equality with the MarinFold generator is a single column compare.