File size: 2,111 Bytes
db0d949 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 |
---
language: en
library_name: transformers
tags:
- tokenizer
- smiles
- protein
- molecule-to-protein
license: apache-2.0
---
# Mol2Pro-tokenizer
#### Paper: [`Generalise or Memorise? Benchmarking Ligand-Conditioned Protein Generation from Sequence-Only Data`](https://doi.org/10.64898/2026.02.06.704305)
## Tokenizer description
This repository provides the **paired tokenizers** used by Mol2Pro models:
- **`smiles/`**: tokenizer for molecule inputs (SMILES) used on the **encoder** side.
- **`aa/`**: tokenizer for protein sequence outputs used on the **decoder** side.
The two tokenizers are designed to be used together with the Mol2Pro sequence-to-sequence checkpoints (see the model card: [`AI4PD/Mol2Pro-base`](https://huggingface.co/AI4PD/Mol2Pro-base)).
## Offset vocabulary
Mol2Pro uses an offset token-id scheme so that SMILES tokens and amino-acid tokens do not collide in id space. Avoids sharing embeddings for identical token strings.
- The **AA** tokenizer uses its natural token id space.
- The **SMILES** tokenizer vocabulary ids are offset above the AA vocabulary ids.
## How to use
```python
from transformers import AutoTokenizer
tokenizer_id = "AI4PD/Mol2Pro-tokenizer"
# Load tokenizers
tokenizer_mol = AutoTokenizer.from_pretrained(tokenizer_id, subfolder="smiles")
tokenizer_aa = AutoTokenizer.from_pretrained(tokenizer_id, subfolder="aa")
# Example:
smiles = "CCO"
enc = tokenizer_mol(smiles, return_tensors="pt")
print("Encoder token ids:", enc.input_ids[0].tolist())
print("Encoder tokens:", tokenizer_mol.convert_ids_to_tokens(enc.input_ids[0]))
aa_text = tokenizer_aa.decode([0, 1, 2], skip_special_tokens=True)
print("Decoded protein sequence:", decoded)
```
## Citation
If you find this work useful, please cite:
```bibtex
@article{VicenteSola2026Generalise,
title = {Generalise or Memorise? Benchmarking Ligand-Conditioned Protein Generation from Sequence-Only Data},
author = {Vicente-Sola, Alex and Dornfeld, Lars and Coines, Joan and Ferruz, Noelia},
journal = {bioRxiv},
year = {2026},
doi = {10.64898/2026.02.06.704305},
}
```
|