Extra tokens for UL2
I have a question about the tokenizer, as I noticed that there are no additional_special_tokens with extra_id_{number} in the model's tokenizer config that was trained on the UL2 task. Is this intentional, or are you using different tokens for extra_id ? They were present in the previous UL2-trained model: https://huggingface.co/google/ul2/blob/main/tokenizer_config.json
Yes, It's intentional. The model uses token (ex- <unused0> to <unused98>) to serve the identical purpose of sentinel tokens for the UL2 denoising objective. These tokens are already integrated directly into the model's main vocabulary added_tokens_decoder and do not need to be separately listed under additional_special_tokens. Please have a look at this t5gemma-2b-2b-ul2 - tokenizer_config.json file.