Transformers How to use nolros/tokenizer_from_scratch with Transformers:
# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("nolros/tokenizer_from_scratch", device_map="auto") 
