moonshotai
/

Moonlight-16B-A3B-Instruct

Text Generation

text-generation-inference

Model card Files Files and versions

Transformers v5 support

#16

by hmellor HF Staff - opened Jan 29

base: refs/heads/main

←

from: refs/pr/16

Discussion Files changed

Files changed (1) hide show

tokenization_moonshot.py +1 -1

tokenization_moonshot.py CHANGED Viewed

@@ -18,7 +18,7 @@ from tiktoken.load import load_tiktoken_bpe
 from tokenizers import AddedToken
 from transformers import  PreTrainedTokenizerFast
 from transformers.tokenization_utils import PreTrainedTokenizer
-from transformers.models.gpt2.tokenization_gpt2 import bytes_to_unicode

 from tokenizers import AddedToken
 from transformers import  PreTrainedTokenizerFast
 from transformers.tokenization_utils import PreTrainedTokenizer
+from transformers.convert_slow_tokenizer import bytes_to_unicode