tokenizer.json: 0%| | 0.00/11.4M [00:004096) (num_proc=4): 0%| | 0/25620 [00:004096) (num_proc=4): 4%|█▉ | 1000/25620 [00:00<00:05, 4469.12 examples/s] Dropping Long Sequences (>4096) (num_proc=4): 23%|███████████▍ | 6000/25620 [00:00<00:00, 21720.64 examples/s] Dropping Long Sequences (>4096) (num_proc=4): 43%|████████████████████▌ | 11000/25620 [00:00<00:00, 31422.54 examples/s] Dropping Long Sequences (>4096) (num_proc=4): 66%|███████████████████████████████▊ | 17000/25620 [00:00<00:00, 34083.34 examples/s] Dropping Long Sequences (>4096) (num_proc=4): 86%|█████████████████████████████████████████▏ | 22000/25620 [00:00<00:00, 38301.21 examples/s] Dropping Long Sequences (>4096) (num_proc=4): 100%|████████████████████████████████████████████████| 25620/25620 [00:00<00:00, 29329.19 examples/s] Saving the dataset (0/4 shards): 0%| | 0/25103 [00:00