clean_up_tokenization_spaces in tokenizer_config.json?

by fahadh4ilyas - opened Jul 30, 2023

Why the value that should be for bos_token is moved to clean_up_tokenization_spaces while the value for bos_token becomes empty string?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment