Training Question: Context Length
#1
by AMDBartek - opened
Hi, thanks for releasing the real base model!
In config.json, I noticed that max_position_embeddings is 8192. The mid-train base, merged final base, and instruct tune has 262144.
Is this a configuration discrepancy or was the base-30T model actually trained with 8192 token context length?
Thanks!
base-30T model is trained with 8k context length and we do context extension training in midtrain phase