How Does NeMo Handle Sequences That Exceed the Max Sequence Length?
#6
by Zieksy - opened
I'm currently working with NeMo-Megatron and have a question about how it handles input sequences that exceed the maximum sequence length allowed by the model.
If a particular data entry exceeds the model's processing capacity (e.g., token count goes beyond max_sequence_length), does NeMo:
Discard the entire sequence?
Truncate it to fit within the limit?
Or handle it in some other way?
I want to ensure data integrity and understand whether I need to implement manual chunking/splitting before feeding data into the pipeline.
Thanks in advance!