How to use marin-dna/tokenizer-char-bos-eos with Transformers:
# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("marin-dna/tokenizer-char-bos-eos", device_map="auto")
