Spaces:

FireRedTeam
/

FireRedASR2S

Running on Zero

FireRed Team commited on 25 days ago

Commit

034000c

verified ·

1 Parent(s): 970908d

Update fireredasr2s/fireredasr2/tokenizer/llm_tokenizer.py

Files changed (1) hide show

fireredasr2s/fireredasr2/tokenizer/llm_tokenizer.py CHANGED Viewed

@@ -66,7 +66,6 @@ class LlmTokenizerWrapper:
                     tokenize=True,
                     chat_template=TEMPLATE,
                     add_generation_prompt=False,
-                    padding="longest",
                     max_length=max_len,
                     truncation=True,
                 )
@@ -76,7 +75,7 @@ class LlmTokenizerWrapper:
         max_len_texts = max([len(text) for text in texts])
         if tokenizer.padding_side == "right":
             texts = [
-                text + [tokenizer.pad_token_id] * (max_len_texts - len(text))
                 for text in texts
             ]
         else:

                     tokenize=True,
                     chat_template=TEMPLATE,
                     add_generation_prompt=False,
                     max_length=max_len,
                     truncation=True,
                 )
         max_len_texts = max([len(text) for text in texts])
         if tokenizer.padding_side == "right":
             texts = [
+                list(text) + [tokenizer.pad_token_id] * (max_len_texts - len(text))
                 for text in texts
             ]
         else: