Should the "Max Positional Embeddings" be 4096?
#10
by barleyspectacular - opened
Noticed this in the config.json
"max_position_embeddings": 2048,
Should this be 4096? Or perhaps this is because the dataset is 2048 context length?
barleyspectacular changed discussion title from Should the "Max Positional Encoding" be 4096? to Should the "Max Positional Embeddings" be 4096?
