File size: 879 Bytes
6425080 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 | # SMILES Tokenizer
This is a custom tokenizer for SMILES (Simplified Molecular Input Line Entry System) strings.
## Usage
```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('suku9/smiles-tokenizer')
# Tokenize a SMILES string
tokens = tokenizer.tokenize('CC(=O)OC1=CC=CC=C1C(=O)O')
print(tokens)
# Encode a SMILES string
encoded = tokenizer('CC(=O)OC1=CC=CC=C1C(=O)O', return_tensors='pt')
print(encoded)
# Use with a GPT-2 model
from transformers import GPT2LMHeadModel
model = GPT2LMHeadModel.from_pretrained('gpt2')
# Resize model embeddings to match tokenizer vocabulary size
model.resize_token_embeddings(len(tokenizer))
```
## Features
- Specialized for SMILES strings
- Compatible with Hugging Face's transformers library
- Designed to work with GPT-2 models
- Preserves all functionality of the original SMILES tokenizer
|