How to use marin-dna/tokenizer-char-bos with Transformers:
# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("marin-dna/tokenizer-char-bos", device_map="auto")
