---
license: cc-by-4.0
language:
  - en
tags:
  - tokenizer
  - protein-structure
  - protein-language-model
  - contacts-v1
---

# contacts-v1 tokenizer

Tokenizer for the [`contacts-v1`](https://github.com/Open-Athena/MarinFold/tree/main/marinfold/marinfold/document_structures/contacts_v1) protein-document format produced by [MarinFold](https://github.com/Open-Athena/MarinFold), used by the [`contacts-v1-5x` subset of `timodonnell/protein-docs`](https://huggingface.co/datasets/timodonnell/protein-docs/tree/main/contacts-v1-5x).

> **This is a mirror** for ease of `AutoTokenizer.from_pretrained("timodonnell/contacts-v1-tokenizer")`. **Canonical location** is co-located with the data at [`open-athena/MarinFold:data/document_structures/contacts_v1/tokenizer/`](https://huggingface.co/buckets/open-athena/MarinFold/tree/main/data/document_structures/contacts_v1/tokenizer) (new MarinFold convention: tokenizer lives in the bucket next to its data).

`load_dataset("timodonnell/protein-docs", "contacts-v1-5x")` documents tokenize directly with this tokenizer — no extra mapping needed.

## Vocab

**2,845 tokens total** (2,843 domain tokens + `<pad>` + `<eos>`). The vocab is **the union of contacts-v1's own 5 native tokens and the entire `contacts-and-distances-v1` vocab** — so a `contacts-v1` model can be fine-tuned on `contacts-and-distances-v1` documents (and vice versa) without swapping tokenizers.

- **5 contacts-v1 native tokens** — `<contacts-v1>`, `<n-term>`, `<c-term>`, `<contact>`, `<think>`.
- **`contacts-and-distances-v1` vocab (reused by emission)** — position tokens `<p0>`…`<p2700>`, section markers `<begin_sequence>` / `<begin_statements>`, amino-acid tokens `<ALA>`…`<VAL>`, `<UNK>`, `<end>`, plus the rest of `contacts-and-distances-v1`'s tokens (modes, distance bins, atoms, pLDDT bins) which contacts-v1 documents do not emit but the tokenizer can encode.

## Usage

```python
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("timodonnell/contacts-v1-tokenizer")
ids = tok.encode("<contacts-v1> <begin_sequence> <p20> <MET>", add_special_tokens=False)
print(ids)
print(tok.decode(ids))
```

## Provenance

Built by `marinfold` v0.1.0 at git rev `dd025aab` (`uv run contacts-v1 tokenizer --push …`), with the full vocab in [`marinfold/.../document_structures/contacts_v1/vocab.py`](https://github.com/Open-Athena/MarinFold/blob/main/marinfold/marinfold/document_structures/contacts_v1/vocab.py). Created as part of [MarinFold issue #53](https://github.com/Open-Athena/MarinFold/issues/53).