Is this model not train for Yarn?

#11
by tarjintor - opened

I use 256K context works fine,but when I use yarn 1M context,the model sometime repeat output

Hi, thanks for the report!

This model has not been fine-tuned for longer contexts β€” the base model is trained on 262,144 positions, and YaRN is applied purely at inference time. That is why 256K works as you observed.

Beyond the trained length, some quality degradation is expected even with YaRN, and it grows with the extension factor. At 1M you are at 4x the trained context, and occasional repetition is a typical symptom of that extrapolation rather than a defect of this quant.

Our recommendation is to stay at or below 512K (YaRN factor 2) β€” at 512K the model behaves normally in our tests. Anything beyond that should be treated as experimental.

Sign up or log in to comment