| [2025-12-26 08:21:18,309] [INFO] [axolotl.utils.data.sft._load_raw_datasets:320] [PID:5436] Loading raw datasets... | |
| [2025-12-26 08:21:21,409] [INFO] [axolotl.utils.data.wrappers.get_dataset_wrapper:87] [PID:5436] Loading dataset: darwinkernelpanic/luau_corpus_axolotl with base_type: completion and prompt_style: None | |
| Tokenizing Prompts (num_proc=64): 0%| | 0/22633 [00:00<?, ? examples/s] Tokenizing Prompts (num_proc=64): 2%|█▏ | 354/22633 [00:01<01:34, 234.52 examples/s] Tokenizing Prompts (num_proc=64): 5%|███▎ | 1062/22633 [00:01<00:26, 803.01 examples/s] Tokenizing Prompts (num_proc=64): 8%|█████▍ | 1770/22633 [00:01<00:14, 1431.89 examples/s] Tokenizing Prompts (num_proc=64): 11%|███████▋ | 2478/22633 [00:01<00:09, 2057.21 examples/s] Tokenizing Prompts (num_proc=64): 14%|█████████▊ | 3186/22633 [00:02<00:07, 2676.87 examples/s] Tokenizing Prompts (num_proc=64): 17%|████████████ | 3894/22633 [00:02<00:05, 3209.08 examples/s] Tokenizing Prompts (num_proc=64): 20%|██████████████▏ | 4602/22633 [00:02<00:04, 3700.39 examples/s] Tokenizing Prompts (num_proc=64): 23%|████████████████▍ | 5310/22633 [00:02<00:04, 4119.88 examples/s] Tokenizing Prompts (num_proc=64): 27%|██████████████████▌ | 6018/22633 [00:02<00:03, 4370.31 examples/s] Tokenizing Prompts (num_proc=64): 30%|████████████████████▊ | 6726/22633 [00:02<00:03, 4510.41 examples/s] Tokenizing Prompts (num_proc=64): 33%|██████████████████████▉ | 7434/22633 [00:02<00:03, 4650.63 examples/s] Tokenizing Prompts (num_proc=64): 36%|█████████████████████████▏ | 8142/22633 [00:03<00:03, 4703.03 examples/s] Tokenizing Prompts (num_proc=64): 39%|███████████████████████████▎ | 8850/22633 [00:03<00:02, 4743.14 examples/s] Tokenizing Prompts (num_proc=64): 42%|█████████████████████████████▌ | 9558/22633 [00:03<00:02, 4776.23 examples/s] Tokenizing Prompts (num_proc=64): 45%|███████████████████████████████▎ | 10266/22633 [00:03<00:02, 4400.59 examples/s] Tokenizing Prompts (num_proc=64): 48%|█████████████████████████████████▍ | 10974/22633 [00:03<00:02, 4289.45 examples/s] Tokenizing Prompts (num_proc=64): 53%|████████████████████████████████████▋ | 12036/22633 [00:03<00:02, 5202.46 examples/s] Tokenizing Prompts (num_proc=64): 56%|██████████████████████████████████████▊ | 12744/22633 [00:04<00:02, 4492.49 examples/s] Tokenizing Prompts (num_proc=64): 59%|█████████████████████████████████████████ | 13452/22633 [00:04<00:02, 4575.30 examples/s] Tokenizing Prompts (num_proc=64): 63%|███████████████████████████████████████████▏ | 14160/22633 [00:04<00:01, 4637.54 examples/s] Tokenizing Prompts (num_proc=64): 66%|█████████████████████████████████████████████▎ | 14867/22633 [00:04<00:01, 4708.44 examples/s] Tokenizing Prompts (num_proc=64): 69%|███████████████████████████████████████████████▍ | 15573/22633 [00:04<00:01, 4717.78 examples/s] Tokenizing Prompts (num_proc=64): 72%|█████████████████████████████████████████████████▋ | 16279/22633 [00:04<00:01, 4733.48 examples/s] Tokenizing Prompts (num_proc=64): 75%|███████████████████████████████████████████████████▊ | 16985/22633 [00:04<00:01, 4739.48 examples/s] Tokenizing Prompts (num_proc=64): 78%|█████████████████████████████████████████████████████▉ | 17691/22633 [00:05<00:01, 4808.79 examples/s] Tokenizing Prompts (num_proc=64): 81%|████████████████████████████████████████████████████████ | 18397/22633 [00:05<00:00, 4777.92 examples/s] Tokenizing Prompts (num_proc=64): 84%|██████████████████████████████████████████████████████████▏ | 19103/22633 [00:05<00:00, 4793.38 examples/s] Tokenizing Prompts (num_proc=64): 88%|████████████████████████████████████████████████████████████▍ | 19809/22633 [00:05<00:00, 4805.58 examples/s] Tokenizing Prompts (num_proc=64): 91%|██████████████████████████████████████████████████████████████▌ | 20515/22633 [00:05<00:00, 4803.13 examples/s] Tokenizing Prompts (num_proc=64): 94%|████████████████████████████████████████████████████████████████▋ | 21221/22633 [00:05<00:00, 5103.36 examples/s] Tokenizing Prompts (num_proc=64): 97%|██████████████████████████████████████████████████████████████████▊ | 21927/22633 [00:05<00:00, 5144.60 examples/s] Tokenizing Prompts (num_proc=64): 100%|█████████████████████████████████████████████████████████████████████| 22633/22633 [00:06<00:00, 5121.25 examples/s] Tokenizing Prompts (num_proc=64): 100%|█████████████████████████████████████████████████████████████████████| 22633/22633 [00:06<00:00, 3469.95 examples/s] | |
| Dropping Long Sequences (>2048) (num_proc=64): 0%| | 0/22640 [00:00<?, ? examples/s] Dropping Long Sequences (>2048) (num_proc=64): 2%|▉ | 354/22640 [00:01<01:14, 299.24 examples/s] Dropping Long Sequences (>2048) (num_proc=64): 6%|███▌ | 1416/22640 [00:01<00:15, 1392.33 examples/s] Dropping Long Sequences (>2048) (num_proc=64): 100%|███████████████████████████████████████████████████████| 22640/22640 [00:01<00:00, 14654.26 examples/s] | |
| Drop Samples with Zero Trainable Tokens (num_proc=64): 0%| | 0/22640 [00:00<?, ? examples/s] Drop Samples with Zero Trainable Tokens (num_proc=64): 2%|▊ | 354/22640 [00:01<01:15, 297.14 examples/s] Drop Samples with Zero Trainable Tokens (num_proc=64): 8%|███▊ | 1770/22640 [00:01<00:11, 1778.86 examples/s] Drop Samples with Zero Trainable Tokens (num_proc=64): 17%|████████▍ | 3894/22640 [00:01<00:04, 4157.04 examples/s] Drop Samples with Zero Trainable Tokens (num_proc=64): 39%|██████████████████▊ | 8850/22640 [00:01<00:01, 11087.04 examples/s] Drop Samples with Zero Trainable Tokens (num_proc=64): 100%|███████████████████████████████████████████████| 22640/22640 [00:01<00:00, 12773.18 examples/s] | |
| Add position_id column (Sample Packing) (num_proc=64): 0%| | 0/22640 [00:00<?, ? examples/s] Add position_id column (Sample Packing) (num_proc=64): 2%|▊ | 354/22640 [00:01<01:14, 299.14 examples/s] Add position_id column (Sample Packing) (num_proc=64): 6%|███ | 1416/22640 [00:01<00:15, 1379.16 examples/s] Add position_id column (Sample Packing) (num_proc=64): 80%|█████████████████████████████████████▍ | 18051/22640 [00:01<00:00, 22934.39 examples/s] Add position_id column (Sample Packing) (num_proc=64): 100%|███████████████████████████████████████████████| 22640/22640 [00:01<00:00, 13969.82 examples/s] | |
| Saving the dataset (0/64 shards): 0%| | 0/22640 [00:00<?, ? examples/s] Saving the dataset (0/64 shards): 2%|█▏ | 354/22640 [00:01<01:19, 280.97 examples/s] Saving the dataset (1/64 shards): 2%|█▏ | 354/22640 [00:01<01:19, 280.97 examples/s] Saving the dataset (2/64 shards): 3%|██▎ | 708/22640 [00:01<01:18, 280.97 examples/s] Saving the dataset (3/64 shards): 5%|███▎ | 1062/22640 [00:01<01:16, 280.97 examples/s] Saving the dataset (4/64 shards): 6%|████▍ | 1416/22640 [00:01<01:15, 280.97 examples/s] Saving the dataset (5/64 shards): 8%|█████▌ | 1770/22640 [00:01<01:14, 280.97 examples/s] Saving the dataset (6/64 shards): 9%|██████▋ | 2124/22640 [00:01<01:13, 280.97 examples/s] Saving the dataset (7/64 shards): 11%|███████▊ | 2478/22640 [00:01<01:11, 280.97 examples/s] Saving the dataset (8/64 shards): 13%|████████▉ | 2832/22640 [00:01<01:10, 280.97 examples/s] Saving the dataset (9/64 shards): 14%|█████████▉ | 3186/22640 [00:01<01:09, 280.97 examples/s] Saving the dataset (10/64 shards): 16%|██████████▉ | 3540/22640 [00:01<01:07, 280.97 examples/s] Saving the dataset (11/64 shards): 17%|████████████ | 3894/22640 [00:01<01:06, 280.97 examples/s] Saving the dataset (12/64 shards): 19%|█████████████▏ | 4248/22640 [00:01<01:05, 280.97 examples/s] Saving the dataset (13/64 shards): 20%|██████████████▏ | 4602/22640 [00:01<01:04, 280.97 examples/s] Saving the dataset (14/64 shards): 22%|███████████████▎ | 4956/22640 [00:01<01:02, 280.97 examples/s] Saving the dataset (15/64 shards): 23%|████████████████▍ | 5310/22640 [00:01<01:01, 280.97 examples/s] Saving the dataset (16/64 shards): 25%|█████████████████▌ | 5664/22640 [00:01<01:00, 280.97 examples/s] Saving the dataset (17/64 shards): 27%|██████████████████▌ | 6018/22640 [00:01<00:59, 280.97 examples/s] Saving the dataset (18/64 shards): 28%|███████████████████▋ | 6372/22640 [00:01<00:57, 280.97 examples/s] Saving the dataset (19/64 shards): 30%|████████████████████▊ | 6726/22640 [00:01<00:56, 280.97 examples/s] Saving the dataset (20/64 shards): 31%|█████████████████████▉ | 7080/22640 [00:01<00:55, 280.97 examples/s] Saving the dataset (21/64 shards): 33%|██████████████████████▉ | 7434/22640 [00:01<00:54, 280.97 examples/s] Saving the dataset (22/64 shards): 34%|████████████████████████ | 7788/22640 [00:01<00:52, 280.97 examples/s] Saving the dataset (23/64 shards): 36%|█████████████████████████▏ | 8142/22640 [00:01<00:51, 280.97 examples/s] Saving the dataset (24/64 shards): 38%|██████████████████████████▎ | 8496/22640 [00:01<00:50, 280.97 examples/s] Saving the dataset (25/64 shards): 39%|███████████████████████████▎ | 8850/22640 [00:01<00:49, 280.97 examples/s] Saving the dataset (26/64 shards): 41%|████████████████████████████▍ | 9204/22640 [00:01<00:47, 280.97 examples/s] Saving the dataset (27/64 shards): 42%|█████████████████████████████▌ | 9558/22640 [00:01<00:46, 280.97 examples/s] Saving the dataset (28/64 shards): 44%|██████████████████████████████▋ | 9912/22640 [00:01<00:45, 280.97 examples/s] Saving the dataset (29/64 shards): 45%|███████████████████████████████▎ | 10266/22640 [00:01<00:44, 280.97 examples/s] Saving the dataset (30/64 shards): 47%|████████████████████████████████▎ | 10620/22640 [00:01<00:42, 280.97 examples/s] Saving the dataset (31/64 shards): 48%|█████████████████████████████████▍ | 10974/22640 [00:01<00:41, 280.97 examples/s] Saving the dataset (32/64 shards): 50%|██████████████████████████████████▌ | 11328/22640 [00:01<00:40, 280.97 examples/s] Saving the dataset (33/64 shards): 53%|████████████████████████████████████▋ | 12036/22640 [00:01<00:37, 280.97 examples/s] Saving the dataset (34/64 shards): 53%|████████████████████████████████████▋ | 12036/22640 [00:01<00:37, 280.97 examples/s] Saving the dataset (35/64 shards): 55%|█████████████████████████████████████▊ | 12390/22640 [00:01<00:36, 280.97 examples/s] Saving the dataset (36/64 shards): 56%|██████████████████████████████████████▊ | 12744/22640 [00:01<00:35, 280.97 examples/s] Saving the dataset (37/64 shards): 58%|███████████████████████████████████████▉ | 13098/22640 [00:01<00:33, 280.97 examples/s] Saving the dataset (38/64 shards): 59%|████████████████████████████████████████▉ | 13452/22640 [00:01<00:32, 280.97 examples/s] Saving the dataset (39/64 shards): 61%|██████████████████████████████████████████ | 13806/22640 [00:01<00:31, 280.97 examples/s] Saving the dataset (40/64 shards): 64%|████████████████████████████████████████████▏ | 14514/22640 [00:01<00:28, 280.97 examples/s] Saving the dataset (41/64 shards): 64%|████████████████████████████████████████████▏ | 14514/22640 [00:01<00:28, 280.97 examples/s] Saving the dataset (42/64 shards): 66%|█████████████████████████████████████████████▎ | 14868/22640 [00:01<00:27, 280.97 examples/s] Saving the dataset (43/64 shards): 67%|██████████████████████████████████████████████▍ | 15222/22640 [00:01<00:26, 280.97 examples/s] Saving the dataset (44/64 shards): 70%|████████████████████████████████████████████████▌ | 15930/22640 [00:01<00:23, 280.97 examples/s] Saving the dataset (45/64 shards): 70%|████████████████████████████████████████████████▌ | 15930/22640 [00:01<00:23, 280.97 examples/s] Saving the dataset (46/64 shards): 72%|█████████████████████████████████████████████████▋ | 16284/22640 [00:01<00:22, 280.97 examples/s] Saving the dataset (47/64 shards): 73%|██████████████████████████████████████████████████▋ | 16638/22640 [00:01<00:21, 280.97 examples/s] Saving the dataset (48/64 shards): 75%|███████████████████████████████████████████████████▊ | 16992/22640 [00:01<00:20, 280.97 examples/s] Saving the dataset (49/64 shards): 78%|█████████████████████████████████████████████████████▉ | 17698/22640 [00:01<00:17, 280.97 examples/s] Saving the dataset (50/64 shards): 78%|█████████████████████████████████████████████████████▉ | 17698/22640 [00:01<00:17, 280.97 examples/s] Saving the dataset (51/64 shards): 80%|███████████████████████████████████████████████████████ | 18051/22640 [00:01<00:16, 280.97 examples/s] Saving the dataset (52/64 shards): 81%|████████████████████████████████████████████████████████ | 18404/22640 [00:01<00:15, 280.97 examples/s] Saving the dataset (53/64 shards): 83%|█████████████████████████████████████████████████████████▏ | 18757/22640 [00:01<00:13, 280.97 examples/s] Saving the dataset (54/64 shards): 84%|██████████████████████████████████████████████████████████▏ | 19110/22640 [00:01<00:12, 280.97 examples/s] Saving the dataset (55/64 shards): 86%|███████████████████████████████████████████████████████████▎ | 19463/22640 [00:01<00:11, 280.97 examples/s] Saving the dataset (56/64 shards): 88%|████████████████████████████████████████████████████████████▍ | 19816/22640 [00:01<00:10, 280.97 examples/s] Saving the dataset (57/64 shards): 89%|█████████████████████████████████████████████████████████████▍ | 20169/22640 [00:01<00:08, 280.97 examples/s] Saving the dataset (58/64 shards): 91%|██████████████████████████████████████████████████████████████▌ | 20522/22640 [00:01<00:07, 280.97 examples/s] Saving the dataset (59/64 shards): 92%|███████████████████████████████████████████████████████████████▌ | 20875/22640 [00:01<00:06, 280.97 examples/s] Saving the dataset (60/64 shards): 94%|████████████████████████████████████████████████████████████████▋ | 21228/22640 [00:01<00:05, 280.97 examples/s] Saving the dataset (61/64 shards): 95%|█████████████████████████████████████████████████████████████████▊ | 21581/22640 [00:01<00:03, 280.97 examples/s] Saving the dataset (62/64 shards): 97%|██████████████████████████████████████████████████████████████████▊ | 21934/22640 [00:01<00:02, 280.97 examples/s] Saving the dataset (63/64 shards): 98%|███████████████████████████████████████████████████████████████████▉ | 22287/22640 [00:01<00:01, 280.97 examples/s] Saving the dataset (64/64 shards): 100%|█████████████████████████████████████████████████████████████████████| 22640/22640 [00:01<00:00, 280.97 examples/s] Saving the dataset (64/64 shards): 100%|███████████████████████████████████████████████████████████████████| 22640/22640 [00:01<00:00, 16486.64 examples/s] | |
| Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s] Loading checkpoint shards: 25%|███████████████████████▎ | 1/4 [00:03<00:09, 3.18s/it] Loading checkpoint shards: 50%|██████████████████████████████████████████████▌ | 2/4 [00:07<00:07, 3.62s/it] Loading checkpoint shards: 75%|█████████████████████████████████████████████████████████████████████▊ | 3/4 [00:10<00:03, 3.53s/it] Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:10<00:00, 2.20s/it] Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:10<00:00, 2.67s/it] | |
| [2025-12-26 08:23:32,340] [WARNING] [py.warnings._showwarnmsg:110] [PID:5436] <string>:204: RuntimeWarning: Mean of empty slice | |
| [2025-12-26 08:26:22,565] [WARNING] [py.warnings._showwarnmsg:110] [PID:5436] /root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:678: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html . | |
| warnings.warn( | |
| [2025-12-26 08:26:40,186] [WARNING] [py.warnings._showwarnmsg:110] [PID:5436] /root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:860: UserWarning: `_get_pg_default_device` will be deprecated, it only stays for backward-compatiblity reason. If you need to find a device for object collectives, please use `_get_object_coll_device`. If you need to query the device types supported by group, please use `_device_capability(group)`. | |
| warnings.warn( | |
| [2025-12-26 08:26:40,186] [WARNING] [py.warnings._showwarnmsg:110] [PID:5436] /root/miniconda3/envs/py3.11/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:904: UserWarning: Multiple backends are registered with this ProcessGroup. We cannot determine which one is the default. Returning cpu. Please consider using other APIs. | |
| warnings.warn( | |