How to use Heimrih/code-tokenizer-0526 with Transformers:
# Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("Heimrih/code-tokenizer-0526", dtype="auto")
