pad_token_id value
Hello, thank you for sharing your fantastic model!
I'm curious about the pad_token_id config of your model.
The value of pad_token_id in this model is 3. I think the value of pad_token_id in the original roberta-base was 1. Is there any reason why the value of pad_token_id is 3?
Hi, there isn't a specific reason.
Hi tianyuz, thank you for your reply.
Actually, I found a github issue related to my question just now and had the same question with this (https://github.com/rinnakk/japanese-pretrained-models/issues/3).
I knew the huggingface's roberta model construct position_ids with position_ids as start, so I couldn't figure out why the pad_token_id is 3 (because the max_length will be 510).
Sorry for not having confirmed well this matter and the vague question, but I'm now understand.
Thank you!
Welcome, it is great that you have figured it out!