Token indices sequence length is longer than the specified maximum sequence length
#11
by bl0ckade - opened
Hi everyone.
I am trying to convert my text which is pretty large, but I run into this error:
Token indices sequence length is longer than the specified maximum sequence length for this model (510649 > 4096). Running this sequence through the model will result in indexing errors.
Is there any way to increase the maximum amount of tokens?
Thanks!