Limit of tokens
#4
by francostan - opened
Hi everyone, im facing this problems that all the ai response generated through generate() are limited on 256 tokens:
Prompt: 707 tokens, 180.446 tokens-per-sec
Generation: 256 tokens, 20.440 tokens-per-sec
Peak memory: 4.440 GB
Respuesta generada:
...
Anyone know how to change this limit, should be on load() ?