data/document_structures/contacts_v1/tokenizer/README.md

# contacts-v1 tokenizer

Co-located with the corresponding `contacts-v1` documents in this bucket
(`data/document_structures/contacts_v1/`).

**This bucket directory is the canonical location for this tokenizer.**

- **Vocab**: 2,845 tokens — 5 `contacts-v1` native (`<contacts-v1>`, `<n-term>`,
  `<c-term>`, `<contact>`, `<think>`) plus the full
  `contacts-and-distances-v1` vocab (reused, so a contacts-v1 model can be
  fine-tuned on c-and-d-v1 documents without swapping tokenizers).
- **Built by**: `marinfold` @ git rev `dd025aab`
  (`uv run contacts-v1 tokenizer --save-local ./tok/`), full vocab in
  [`marinfold/.../contacts_v1/vocab.py`](https://github.com/Open-Athena/MarinFold/blob/main/marinfold/marinfold/document_structures/contacts_v1/vocab.py).
- **Produced by**: [MarinFold issue #53](https://github.com/Open-Athena/MarinFold/issues/53).

## Loading

Buckets aren't directly addressable by `AutoTokenizer.from_pretrained`, so
download then load:

```bash
hf buckets cp -r \
  hf://buckets/open-athena/MarinFold/data/document_structures/contacts_v1/tokenizer \
  ./contacts-v1-tokenizer
```

```python
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("./contacts-v1-tokenizer")
```