Processor Decode does not remove <end_of_turn> token
I'm using the example provided on the model's page. When I run:
decoded = processor.decode(generation, skip_special_tokens=True) print(decoded)
the output doesn't remove the "end_of_turn", which still appears in the decoded text.
This issue occurs specifically when I use the model.generate method:
input_len = inputs["input_ids"].shape[-1] with torch.inference_mode(): generation = model.generate(**inputs, max_new_tokens=500, do_sample=False) generation = generation[0][input_len:]
However, when I use the pipeline interface, the special tokens (including "end_of_turn") are properly removed and everything works as expected. Also, using a tokenizer instead of processor to decode the text works just fine.
I'm hitting this same behaviour
Hi @gka-arch ,
I have reproduced the issue in my local, whether to include the special tokens in the output or not depends on the skip_special_tokens parameter that passed to decode method. If we don't specify any value (True/False) explicitly to this skip_special_tokens parameter (by default set to False) or if set to False explicitly then the output of the model contains the special tokens like end_of_turn. If skip_special_tokens=True then these special tokens are excluded from the output.
Case 1: skip_special_tokens set to False - output contains the special tokens.
Case 2: Not specified any value to skip_special_tokens - output contains the special tokens.
Case 3: skip_special_tokens set to True - special tokens are excluded from the output.
Thanks.
This is resolved with the latest transformers version, just upgrade
Thanks for the confirmation on the resolution.