data/document_structures/contacts_v1/tokenizer/README.md

contacts-v1 tokenizer

Co-located with the corresponding contacts-v1 documents in this bucket (data/document_structures/contacts_v1/).

This bucket directory is the canonical location for this tokenizer.

• Vocab: 2,845 tokens — 5 contacts-v1 native (<contacts-v1>, <n-term>, <c-term>, <contact>, <think>) plus the full contacts-and-distances-v1 vocab (reused, so a contacts-v1 model can be fine-tuned on c-and-d-v1 documents without swapping tokenizers).
• Built by: marinfold @ git rev dd025aab (uv run contacts-v1 tokenizer --save-local ./tok/), full vocab in marinfold/.../contacts_v1/vocab.py.
• Produced by: MarinFold issue #53.

Loading

Buckets aren't directly addressable by AutoTokenizer.from_pretrained, so download then load:

hf buckets cp -r \
  hf://buckets/open-athena/MarinFold/data/document_structures/contacts_v1/tokenizer \
  ./contacts-v1-tokenizer

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("./contacts-v1-tokenizer")