data/document_structures/contacts-v1.sequence_only/README.md

contacts-v1.sequence_only protein-document dataset

Sequence-only training documents in the contacts-v1 token space, generated from LiteFold/UniRef50 by calling MarinFold's generate_sequence_only_document (no re-implementation). Each document is the contacts-v1 sequence section only — same random wrap-around <pX> <AA> indexing, <n-term>/<c-term> markers, and shuffled statement order — under a new doc type <contacts-v1.sequence_only> and with no structure section (no contacts):

<contacts-v1.sequence_only> <begin_sequence> <p976> <GLY> <p572> <ASN> … <n-term> <p336> … <c-term> <p813> … <end>

That shared representation is the point: this corpus can be mixed with the contacts_v1 corpus under one tokenizer to (hypothesis, tested later) improve the contacts-v1 eval. See the format spec: marinfold/.../document_structures/contacts_v1 (the "Sequence-only variant" section of SPEC.md).

Produced by experiment exp64 (marinfold @ 8e6249a). The sequence section is byte-identical to what <contacts-v1> emits for the same entry_id — only the leading doc-type token and the absent structure section differ.

Splits

Arbitrary train/val/test, independent of the contacts-v1 splits (issue #64 allows this): bucket = sha1(entry_id) % 1000; [0,5) -> test, [5,10) -> val, the rest -> train (≈ 99 / 0.5 / 0.5%). Hashing on entry_id keeps each split length-balanced.

Layout

<split>/uniref50-<shard>-<chunk>.parquet   # ≤200k rows/file; <shard> = source UniRef50 shard (0-60)
tokenizer/                                  # unified tokenizer (2846 tokens; see below)

Ordering caveat. UniRef50's 61 source shards are globally sorted by length, longest first, and documents are written in that order, so the published files are length-banded (low <shard> numbers = longer sequences). Shuffle at training time (shuffle file order + a shuffle buffer) rather than reading the shards in order.

Tokenizer

The unified contacts-v1 tokenizer: contacts-v1's 2845 tokens plus the single <contacts-v1.sequence_only> doc-type token appended last (id 2845), so every pre-existing contacts-v1 token id is unchanged. A model can train on this corpus and the contacts-v1 corpus together with no tokenizer change.

Counts

60,004,535 documents — ~32.98 B tokens from 60,315,044 UniRef50 sequences (0 generation failures; 310,509 sequences = 0.51% dropped for falling outside the [2, 2000]-residue serializable range — almost all the

2000-residue giants in source shards 0-1). Mean ~550 num_tokens/doc (≈271 residues).

split	documents	tokens	files
train	59,403,434	32,653,114,680	301
val	300,982	165,485,468	61
test	300,119	164,819,873	61
total	60,004,535	32,983,420,021	423

Columns

document (token string) · structure ("contacts-v1.sequence_only") · entry_id (UniRef50 accession, e.g. UniRef50_P00350) · seq_len · start_index / n_term_index / c_term_index · num_tokens · sha1 · split.

The contact-statistics columns of the contacts-v1 corpus (contacts_emitted, highest_contact_degree, …) are omitted — there is no structure section. num_tokens == 2 * seq_len + 7 for every row.

sha1 = sha1 of document, so byte-equality with the MarinFold generator is a single-column compare.