How to use hmteams/teams-tokenizer with Transformers:
# Load model directly from transformers import AutoTokenizer, AutoModelForPreTraining tokenizer = AutoTokenizer.from_pretrained("hmteams/teams-tokenizer") model = AutoModelForPreTraining.from_pretrained("hmteams/teams-tokenizer")
# Load model directly from transformers import AutoTokenizer, AutoModelForPreTraining tokenizer = AutoTokenizer.from_pretrained("hmteams/teams-tokenizer") model = AutoModelForPreTraining.from_pretrained("hmteams/teams-tokenizer")