Existing tokens: 199,971,619
1/3 - WikiText-103 (fixed API)...

Generating test split:   0%|          | 0/4358 [00:00<?, ? examples/s]
Generating test split: 100%|██████████| 4358/4358 [00:00<00:00, 346110.29 examples/s]

Generating train split:   0%|          | 0/1801350 [00:00<?, ? examples/s]
Generating train split:   3%|▎         | 50000/1801350 [00:00<00:03, 490798.37 examples/s]
Generating train split:   7%|▋         | 133000/1801350 [00:00<00:02, 684505.13 examples/s]
Generating train split:  13%|█▎        | 239000/1801350 [00:00<00:02, 692225.46 examples/s]
Generating train split:  18%|█▊        | 326000/1801350 [00:00<00:01, 749504.02 examples/s]
Generating train split:  24%|██▍       | 433000/1801350 [00:00<00:01, 730015.82 examples/s]
Generating train split:  29%|██▉       | 519000/1801350 [00:00<00:01, 762857.48 examples/s]
Generating train split:  34%|███▍      | 615000/1801350 [00:00<00:01, 711042.16 examples/s]
Generating train split:  40%|████      | 723000/1801350 [00:01<00:01, 710382.22 examples/s]
Generating train split:  46%|████▋     | 834000/1801350 [00:01<00:01, 717654.58 examples/s]
Generating train split:  52%|█████▏    | 942675/1801350 [00:01<00:01, 696870.70 examples/s]
Generating train split:  57%|█████▋    | 1028675/1801350 [00:01<00:01, 732840.22 examples/s]
Generating train split:  63%|██████▎   | 1136675/1801350 [00:01<00:00, 725021.27 examples/s]
Generating train split:  68%|██████▊   | 1219675/1801350 [00:01<00:00, 747761.05 examples/s]
Generating train split:  73%|███████▎  | 1322675/1801350 [00:01<00:00, 719268.86 examples/s]
Generating train split:  80%|███████▉  | 1435675/1801350 [00:01<00:00, 727812.57 examples/s]
Generating train split:  85%|████████▌ | 1538675/1801350 [00:02<00:00, 710518.96 examples/s]
Generating train split:  90%|█████████ | 1622675/1801350 [00:02<00:00, 737050.01 examples/s]
Generating train split:  96%|█████████▌| 1729675/1801350 [00:02<00:00, 724068.37 examples/s]
Generating train split: 100%|██████████| 1801350/1801350 [00:02<00:00, 721288.20 examples/s]

Generating validation split:   0%|          | 0/3760 [00:00<?, ? examples/s]
Generating validation split: 100%|██████████| 3760/3760 [00:00<00:00, 480444.27 examples/s]
  WikiText-103: 50,000 texts -> 22,635,189 tokens
  WikiText-103: 100,000 texts -> 45,823,471 tokens
  WikiText-103: 150,000 texts -> 69,308,297 tokens
  WikiText-103: 200,000 texts -> 92,105,968 tokens
  WikiText-103: 250,000 texts -> 115,047,336 tokens
  WikiText-103: 300,000 texts -> 138,601,321 tokens
  WikiText-103: 350,000 texts -> 161,788,432 tokens
  WikiText-103: 400,000 texts -> 184,963,474 tokens
  WikiText-103: 450,000 texts -> 207,852,194 tokens
  WikiText-103: 500,000 texts -> 231,214,640 tokens
  WikiText-103: 550,000 texts -> 254,905,316 tokens
  WikiText-103: 600,000 texts -> 278,402,058 tokens
  WikiText-103: 650,000 texts -> 301,290,874 tokens
  WikiText-103: 700,000 texts -> 324,414,818 tokens
  WikiText-103: 750,000 texts -> 347,964,383 tokens
  WikiText-103: 800,000 texts -> 370,967,929 tokens
  WikiText-103: 850,000 texts -> 394,148,337 tokens
  WikiText-103: 900,000 texts -> 417,556,771 tokens
  WikiText-103: 950,000 texts -> 441,075,122 tokens
  WikiText-103: 1,000,000 texts -> 464,091,223 tokens
  WikiText-103: 1,050,000 texts -> 487,133,309 tokens
  WikiText-103: 1,100,000 texts -> 510,482,618 tokens
  WikiText-103: 1,150,000 texts -> 533,526,405 tokens
  OK WikiText-103: 540,519,546 tokens
2/3 - C4 English...
  C4-EN: 50,000 texts -> 106,885,554 tokens
  OK C4-EN: 215,577,179 tokens
3/3 - FineWeb-Edu...
  FineWeb-Edu: 50,000 texts -> 242,392,039 tokens
⚠ Received signal 15, will save checkpoint after current step...
  OK FineWeb-Edu: 480,132,888 tokens
TOTAL TOKENS: 1,436,201,232
Train: 1,364,391,170 tokens (1364.4M)
Val:   71,810,062 tokens (71.8M)
MORE DATA COMPLETE!