internlm
/

Intern-S1-mini

Image-Text-to-Text

text-generation

Model card Files Files and versions

yczhang commited on 18 days ago

Commit

021778d

·

1 Parent(s): 633ca6d

Update: fix bug

Files changed (1) hide show

tokenization_interns1.py +1 -0

tokenization_interns1.py CHANGED Viewed

@@ -450,6 +450,7 @@ class InternS1Tokenizer(PreTrainedTokenizer):
         pad_token="<|endoftext|>",
         clean_up_tokenization_spaces=False,
         split_special_tokens=False,
         **kwargs,
     ):
         bos_token = (

         pad_token="<|endoftext|>",
         clean_up_tokenization_spaces=False,
         split_special_tokens=False,
+        special_tokens_pattern="none",
         **kwargs,
     ):
         bos_token = (