Salesforce
/

xgen-7b-8k-base

Text Generation

text-generation-inference

Model card Files Files and versions

psinger commited on Jun 30, 2023

Commit

db5edb9

·

1 Parent(s): 3568622

Update tokenization_xgen.py

Files changed (1) hide show

tokenization_xgen.py +1 -1

tokenization_xgen.py CHANGED Viewed

@@ -115,7 +115,7 @@ class XgenTokenizer(PreTrainedTokenizer):
     def __init__(
             self,
             pad_token=None,
-            eos_token=<|endoftext|>",
             add_eos_token=False,
             add_special_tokens=True,
             **kwargs,

     def __init__(
             self,
             pad_token=None,
+            eos_token="<|endoftext|>",
             add_eos_token=False,
             add_special_tokens=True,
             **kwargs,