Token size
#60
by DeveloperZoyel - opened
How to increase the token size and what is the name of the parameter for the token size?
@developerZoyal did you solved the error?, I am getting same one, and I am unable to figure it out so.
@Harsh6519 I have used the concept of loop. broke into chunk of 1024 tokens and then passed into the model
what tokenizer can be used for this model.