nohup: ignoring input
W0405 10:13:37.253000 1134 site-packages/torch/distributed/run.py:803] 
W0405 10:13:37.253000 1134 site-packages/torch/distributed/run.py:803] *****************************************
W0405 10:13:37.253000 1134 site-packages/torch/distributed/run.py:803] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
W0405 10:13:37.253000 1134 site-packages/torch/distributed/run.py:803] *****************************************
Set TORCH_CUDA_ARCH_LIST to 9.0Set TORCH_CUDA_ARCH_LIST to 9.0Set TORCH_CUDA_ARCH_LIST to 9.0


Set TORCH_CUDA_ARCH_LIST to 9.0
Set TORCH_CUDA_ARCH_LIST to 9.0
Set TORCH_CUDA_ARCH_LIST to 9.0Set TORCH_CUDA_ARCH_LIST to 9.0

Set TORCH_CUDA_ARCH_LIST to 9.0
/workspace/hanrui/SpecForge/specforge/modeling/draft/llama3_eagle.py:29: UserWarning: flash_attn is not found, falling back to flex_attention. Please install flash_attn if you want to use the flash attention backend.
  warnings.warn(
/workspace/hanrui/SpecForge/specforge/modeling/draft/llama3_eagle.py:29: UserWarning: flash_attn is not found, falling back to flex_attention. Please install flash_attn if you want to use the flash attention backend.
  warnings.warn(
/workspace/hanrui/SpecForge/specforge/modeling/draft/llama3_eagle.py:29: UserWarning: flash_attn is not found, falling back to flex_attention. Please install flash_attn if you want to use the flash attention backend.
  warnings.warn(
/workspace/hanrui/SpecForge/specforge/modeling/draft/llama3_eagle.py:29: UserWarning: flash_attn is not found, falling back to flex_attention. Please install flash_attn if you want to use the flash attention backend.
  warnings.warn(
/workspace/hanrui/SpecForge/specforge/modeling/draft/llama3_eagle.py:29: UserWarning: flash_attn is not found, falling back to flex_attention. Please install flash_attn if you want to use the flash attention backend.
  warnings.warn(
/workspace/hanrui/SpecForge/specforge/modeling/draft/llama3_eagle.py:29: UserWarning: flash_attn is not found, falling back to flex_attention. Please install flash_attn if you want to use the flash attention backend.
  warnings.warn(
/workspace/hanrui/SpecForge/specforge/modeling/draft/llama3_eagle.py:29: UserWarning: flash_attn is not found, falling back to flex_attention. Please install flash_attn if you want to use the flash attention backend.
  warnings.warn(
/workspace/hanrui/SpecForge/specforge/modeling/draft/llama3_eagle.py:29: UserWarning: flash_attn is not found, falling back to flex_attention. Please install flash_attn if you want to use the flash attention backend.
  warnings.warn(
<frozen importlib._bootstrap_external>:1241: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
<frozen importlib._bootstrap_external>:1241: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
<frozen importlib._bootstrap_external>:1241: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
<frozen importlib._bootstrap_external>:1241: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
<frozen importlib._bootstrap_external>:1241: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
<frozen importlib._bootstrap_external>:1241: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
<frozen importlib._bootstrap_external>:1241: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
<frozen importlib._bootstrap_external>:1241: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
<frozen importlib._bootstrap_external>:1241: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
<frozen importlib._bootstrap_external>:1241: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
<frozen importlib._bootstrap_external>:1241: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
<frozen importlib._bootstrap_external>:1241: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
<frozen importlib._bootstrap_external>:1241: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
<frozen importlib._bootstrap_external>:1241: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
<frozen importlib._bootstrap_external>:1241: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
<frozen importlib._bootstrap_external>:1241: FutureWarning: The cuda.nvrtc module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.nvrtc module instead.
INFO:specforge.utils:rank 0: bind to device 0
INFO:specforge.utils:rank 0: device mesh: DeviceMesh((dp=8, tp=1), device: 'cuda', stride: (1, 1))
INFO:specforge.utils:rank 0: Initialized distributed
INFO:specforge.utils:Loading target model from /workspace/models/Qwen3-8B using hf backend
`torch_dtype` is deprecated! Use `dtype` instead!
INFO:specforge.utils:rank 3: bind to device 3
The following generation flags are not valid and may be ignored: ['output_hidden_states']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
INFO:specforge.utils:rank 4: bind to device 4
INFO:specforge.utils:rank 2: bind to device 2
INFO:specforge.utils:rank 3: device mesh: DeviceMesh((dp=8, tp=1), device: 'cuda', stride: (1, 1))
INFO:specforge.utils:rank 3: Initialized distributed
`torch_dtype` is deprecated! Use `dtype` instead!
INFO:specforge.utils:rank 4: device mesh: DeviceMesh((dp=8, tp=1), device: 'cuda', stride: (1, 1))
INFO:specforge.utils:rank 4: Initialized distributed
`torch_dtype` is deprecated! Use `dtype` instead!
INFO:specforge.utils:rank 2: device mesh: DeviceMesh((dp=8, tp=1), device: 'cuda', stride: (1, 1))
INFO:specforge.utils:rank 2: Initialized distributed
The following generation flags are not valid and may be ignored: ['output_hidden_states']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
`torch_dtype` is deprecated! Use `dtype` instead!
The following generation flags are not valid and may be ignored: ['output_hidden_states']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
The following generation flags are not valid and may be ignored: ['output_hidden_states']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
INFO:specforge.utils:rank 5: bind to device 5
INFO:specforge.utils:rank 5: device mesh: DeviceMesh((dp=8, tp=1), device: 'cuda', stride: (1, 1))
INFO:specforge.utils:rank 5: Initialized distributed
`torch_dtype` is deprecated! Use `dtype` instead!
The following generation flags are not valid and may be ignored: ['output_hidden_states']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
INFO:specforge.utils:rank 7: bind to device 7
INFO:specforge.utils:rank 1: bind to device 1
INFO:specforge.utils:rank 6: bind to device 6
INFO:specforge.utils:rank 7: device mesh: DeviceMesh((dp=8, tp=1), device: 'cuda', stride: (1, 1))
INFO:specforge.utils:rank 1: device mesh: DeviceMesh((dp=8, tp=1), device: 'cuda', stride: (1, 1))
INFO:specforge.utils:rank 7: Initialized distributed
INFO:specforge.utils:rank 1: Initialized distributed
`torch_dtype` is deprecated! Use `dtype` instead!
`torch_dtype` is deprecated! Use `dtype` instead!
INFO:specforge.utils:rank 6: device mesh: DeviceMesh((dp=8, tp=1), device: 'cuda', stride: (1, 1))
INFO:specforge.utils:rank 6: Initialized distributed
`torch_dtype` is deprecated! Use `dtype` instead!
The following generation flags are not valid and may be ignored: ['output_hidden_states']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
The following generation flags are not valid and may be ignored: ['output_hidden_states']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
The following generation flags are not valid and may be ignored: ['output_hidden_states']. Set `TRANSFORMERS_VERBOSITY=info` for more details.

Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards:  20%|██        | 1/5 [00:00<00:01,  2.37it/s]
Loading checkpoint shards:  20%|██        | 1/5 [00:00<00:01,  2.41it/s]
Loading checkpoint shards:  20%|██        | 1/5 [00:00<00:01,  2.44it/s]
Loading checkpoint shards:  20%|██        | 1/5 [00:00<00:01,  2.31it/s]
Loading checkpoint shards:  20%|██        | 1/5 [00:00<00:01,  2.27it/s]
Loading checkpoint shards:  20%|██        | 1/5 [00:00<00:01,  2.41it/s]
Loading checkpoint shards:  20%|██        | 1/5 [00:00<00:01,  2.29it/s]
Loading checkpoint shards:  20%|██        | 1/5 [00:00<00:01,  2.33it/s]
Loading checkpoint shards:  40%|████      | 2/5 [00:01<00:01,  1.82it/s]
Loading checkpoint shards:  40%|████      | 2/5 [00:01<00:01,  1.81it/s]
Loading checkpoint shards:  40%|████      | 2/5 [00:01<00:01,  1.85it/s]
Loading checkpoint shards:  40%|████      | 2/5 [00:01<00:01,  1.84it/s]
Loading checkpoint shards:  40%|████      | 2/5 [00:01<00:01,  1.86it/s]
Loading checkpoint shards:  40%|████      | 2/5 [00:01<00:01,  1.83it/s]
Loading checkpoint shards:  40%|████      | 2/5 [00:01<00:01,  1.85it/s]
Loading checkpoint shards:  40%|████      | 2/5 [00:01<00:01,  1.82it/s]
Loading checkpoint shards:  60%|██████    | 3/5 [00:01<00:01,  1.79it/s]
Loading checkpoint shards:  60%|██████    | 3/5 [00:01<00:01,  1.78it/s]
Loading checkpoint shards:  60%|██████    | 3/5 [00:01<00:01,  1.80it/s]
Loading checkpoint shards:  60%|██████    | 3/5 [00:01<00:01,  1.80it/s]
Loading checkpoint shards:  60%|██████    | 3/5 [00:01<00:01,  1.80it/s]
Loading checkpoint shards:  60%|██████    | 3/5 [00:01<00:01,  1.81it/s]
Loading checkpoint shards:  60%|██████    | 3/5 [00:01<00:01,  1.79it/s]
Loading checkpoint shards:  60%|██████    | 3/5 [00:01<00:01,  1.79it/s]
Loading checkpoint shards:  80%|████████  | 4/5 [00:02<00:00,  1.86it/s]
Loading checkpoint shards:  80%|████████  | 4/5 [00:02<00:00,  1.87it/s]
Loading checkpoint shards:  80%|████████  | 4/5 [00:02<00:00,  1.87it/s]
Loading checkpoint shards:  80%|████████  | 4/5 [00:02<00:00,  1.87it/s]
Loading checkpoint shards:  80%|████████  | 4/5 [00:02<00:00,  1.87it/s]
Loading checkpoint shards:  80%|████████  | 4/5 [00:02<00:00,  1.86it/s]
Loading checkpoint shards:  80%|████████  | 4/5 [00:02<00:00,  1.88it/s]
Loading checkpoint shards:  80%|████████  | 4/5 [00:02<00:00,  1.87it/s]
Loading checkpoint shards: 100%|██████████| 5/5 [00:02<00:00,  2.33it/s]

Loading checkpoint shards: 100%|██████████| 5/5 [00:02<00:00,  2.34it/s]
Loading checkpoint shards: 100%|██████████| 5/5 [00:02<00:00,  2.36it/s]


Loading checkpoint shards: 100%|██████████| 5/5 [00:02<00:00,  2.35it/s]
Loading checkpoint shards: 100%|██████████| 5/5 [00:02<00:00,  2.36it/s]

Loading checkpoint shards: 100%|██████████| 5/5 [00:02<00:00,  2.34it/s]

Loading checkpoint shards: 100%|██████████| 5/5 [00:02<00:00,  2.34it/s]
Loading checkpoint shards: 100%|██████████| 5/5 [00:02<00:00,  2.37it/s]


INFO:specforge.utils:Loaded draft config from /workspace/hanrui/SpecForge/configs/qwen3-8b-dflash.json
INFO:specforge.utils:Using attention backend: flex_attention
INFO:specforge.utils:Draft config: block_size=16, num_hidden_layers=5, num_target_layers=36
INFO:specforge.utils:Draft model parameters: 1,048,626,432
INFO:specforge.utils:Using mask_token_id: 151669
INFO:specforge.utils:dflash_config: {'mask_token_id': 151669, 'target_layer_ids': [1, 9, 17, 25, 33]}

Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1415 examples [00:00, 8811.58 examples/s]
Generating train split: 3122 examples [00:00, 11082.04 examples/s]
Generating train split: 4875 examples [00:00, 12069.85 examples/s]
Generating train split: 6641 examples [00:00, 13076.22 examples/s]
Generating train split: 8414 examples [00:00, 13857.65 examples/s]
Generating train split: 10156 examples [00:00, 14143.44 examples/s]
Generating train split: 11890 examples [00:00, 14012.42 examples/s]
Generating train split: 13682 examples [00:01, 9161.16 examples/s] 
Generating train split: 15433 examples [00:01, 10065.15 examples/s]
Generating train split: 16762 examples [00:01, 9910.32 examples/s] 
Generating train split: 17694 examples [00:01, 9631.51 examples/s]
[rank0]: Traceback (most recent call last):
[rank0]:   File "/workspace/miniconda3/envs/specforge/lib/python3.11/site-packages/datasets/builder.py", line 1890, in _prepare_split_single
[rank0]:     writer.write_table(table)
[rank0]:   File "/workspace/miniconda3/envs/specforge/lib/python3.11/site-packages/datasets/arrow_writer.py", line 680, in write_table
[rank0]:     self.pa_writer.write_table(pa_table, writer_batch_size)
[rank0]:   File "pyarrow/ipc.pxi", line 616, in pyarrow.lib._CRecordBatchWriter.write_table
[rank0]:   File "pyarrow/error.pxi", line 89, in pyarrow.lib.check_status
[rank0]:   File "/workspace/miniconda3/envs/specforge/lib/python3.11/site-packages/fsspec/implementations/local.py", line 473, in write
[rank0]:     return self.f.write(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: OSError: [Errno 122] Disk quota exceeded

[rank0]: During handling of the above exception, another exception occurred:

[rank0]: Traceback (most recent call last):
[rank0]:   File "/workspace/miniconda3/envs/specforge/lib/python3.11/site-packages/datasets/builder.py", line 1911, in _prepare_split_single
[rank0]:     num_examples, num_bytes = writer.finalize()
[rank0]:                               ^^^^^^^^^^^^^^^^^
[rank0]:   File "/workspace/miniconda3/envs/specforge/lib/python3.11/site-packages/datasets/arrow_writer.py", line 693, in finalize
[rank0]:     self.stream.close()
[rank0]:   File "/workspace/miniconda3/envs/specforge/lib/python3.11/site-packages/fsspec/implementations/local.py", line 491, in close
[rank0]:     return self.f.close()
[rank0]:            ^^^^^^^^^^^^^^
[rank0]: OSError: [Errno 122] Disk quota exceeded

[rank0]: The above exception was the direct cause of the following exception:

[rank0]: Traceback (most recent call last):
[rank0]:   File "/workspace/hanrui/SpecForge/scripts/train_dflash.py", line 562, in <module>
[rank0]:     main()
[rank0]:   File "/workspace/hanrui/SpecForge/scripts/train_dflash.py", line 415, in main
[rank0]:     train_dataloader, eval_dataloader = build_dataloader(args, tokenizer)
[rank0]:                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/workspace/hanrui/SpecForge/scripts/train_dflash.py", line 222, in build_dataloader
[rank0]:     train_dataset = load_dataset("json", data_files=args.train_data_path)["train"]
[rank0]:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/workspace/miniconda3/envs/specforge/lib/python3.11/site-packages/datasets/load.py", line 1505, in load_dataset
[rank0]:     builder_instance.download_and_prepare(
[rank0]:   File "/workspace/miniconda3/envs/specforge/lib/python3.11/site-packages/datasets/builder.py", line 884, in download_and_prepare
[rank0]:     self._download_and_prepare(
[rank0]:   File "/workspace/miniconda3/envs/specforge/lib/python3.11/site-packages/datasets/builder.py", line 947, in _download_and_prepare
[rank0]:     self._prepare_split(split_generator, **prepare_split_kwargs)
[rank0]:   File "/workspace/miniconda3/envs/specforge/lib/python3.11/site-packages/datasets/builder.py", line 1739, in _prepare_split
[rank0]:     for job_id, done, content in self._prepare_split_single(
[rank0]:   File "/workspace/miniconda3/envs/specforge/lib/python3.11/site-packages/datasets/builder.py", line 1922, in _prepare_split_single
[rank0]:     raise DatasetGenerationError("An error occurred while generating the dataset") from e
[rank0]: datasets.exceptions.DatasetGenerationError: An error occurred while generating the dataset

Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1837 examples [00:00, 11719.94 examples/s]
Generating train split: 3552 examples [00:00, 11699.05 examples/s]
Generating train split: 5305 examples [00:00, 12818.69 examples/s]
Generating train split: 7092 examples [00:00, 13468.71 examples/s][rank0]:[W405 10:15:29.947786516 ProcessGroupNCCL.cpp:1524] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())

Generating train split: 8810 examples [00:00, 13834.01 examples/s]
Generating train split: 10577 examples [00:00, 14054.90 examples/s]
Generating train split: 12339 examples [00:00, 12805.47 examples/s]
Generating train split: 13682 examples [00:01, 9900.64 examples/s] 
Generating train split: 14998 examples [00:01, 10026.49 examples/s]
Generating train split: 16762 examples [00:01, 8287.30 examples/s] 
Generating train split: 17694 examples [00:01, 7877.81 examples/s]
Generating train split: 17694 examples [00:01, 10199.12 examples/s]
[rank6]: Traceback (most recent call last):
[rank6]:   File "/workspace/miniconda3/envs/specforge/lib/python3.11/site-packages/datasets/builder.py", line 1890, in _prepare_split_single
[rank6]:     writer.write_table(table)
[rank6]:   File "/workspace/miniconda3/envs/specforge/lib/python3.11/site-packages/datasets/arrow_writer.py", line 680, in write_table
[rank6]:     self.pa_writer.write_table(pa_table, writer_batch_size)
[rank6]:   File "pyarrow/ipc.pxi", line 616, in pyarrow.lib._CRecordBatchWriter.write_table
[rank6]:   File "pyarrow/error.pxi", line 89, in pyarrow.lib.check_status
[rank6]:   File "/workspace/miniconda3/envs/specforge/lib/python3.11/site-packages/fsspec/implementations/local.py", line 473, in write
[rank6]:     return self.f.write(*args, **kwargs)
[rank6]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]: OSError: [Errno 122] Disk quota exceeded

[rank6]: During handling of the above exception, another exception occurred:

[rank6]: Traceback (most recent call last):
[rank6]:   File "/workspace/miniconda3/envs/specforge/lib/python3.11/site-packages/datasets/builder.py", line 1911, in _prepare_split_single
[rank6]:     num_examples, num_bytes = writer.finalize()
[rank6]:                               ^^^^^^^^^^^^^^^^^
[rank6]:   File "/workspace/miniconda3/envs/specforge/lib/python3.11/site-packages/datasets/arrow_writer.py", line 693, in finalize
[rank6]:     self.stream.close()
[rank6]:   File "/workspace/miniconda3/envs/specforge/lib/python3.11/site-packages/fsspec/implementations/local.py", line 491, in close
[rank6]:     return self.f.close()
[rank6]:            ^^^^^^^^^^^^^^
[rank6]: OSError: [Errno 122] Disk quota exceeded

[rank6]: The above exception was the direct cause of the following exception:

[rank6]: Traceback (most recent call last):
[rank6]:   File "/workspace/hanrui/SpecForge/scripts/train_dflash.py", line 562, in <module>
[rank6]:     main()
[rank6]:   File "/workspace/hanrui/SpecForge/scripts/train_dflash.py", line 415, in main
[rank6]:     train_dataloader, eval_dataloader = build_dataloader(args, tokenizer)
[rank6]:                                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]:   File "/workspace/hanrui/SpecForge/scripts/train_dflash.py", line 222, in build_dataloader
[rank6]:     train_dataset = load_dataset("json", data_files=args.train_data_path)["train"]
[rank6]:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank6]:   File "/workspace/miniconda3/envs/specforge/lib/python3.11/site-packages/datasets/load.py", line 1505, in load_dataset
[rank6]:     builder_instance.download_and_prepare(
[rank6]:   File "/workspace/miniconda3/envs/specforge/lib/python3.11/site-packages/datasets/builder.py", line 884, in download_and_prepare
[rank6]:     self._download_and_prepare(
[rank6]:   File "/workspace/miniconda3/envs/specforge/lib/python3.11/site-packages/datasets/builder.py", line 947, in _download_and_prepare
[rank6]:     self._prepare_split(split_generator, **prepare_split_kwargs)
[rank6]:   File "/workspace/miniconda3/envs/specforge/lib/python3.11/site-packages/datasets/builder.py", line 1739, in _prepare_split
[rank6]:     for job_id, done, content in self._prepare_split_single(
[rank6]:   File "/workspace/miniconda3/envs/specforge/lib/python3.11/site-packages/datasets/builder.py", line 1922, in _prepare_split_single
[rank6]:     raise DatasetGenerationError("An error occurred while generating the dataset") from e
[rank6]: datasets.exceptions.DatasetGenerationError: An error occurred while generating the dataset

Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 1415 examples [00:00, 9349.59 examples/s]
Generating train split: 3122 examples [00:00, 10389.63 examples/s]W0405 10:15:30.839000 1134 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 1221 closing signal SIGTERM
W0405 10:15:30.840000 1134 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 1222 closing signal SIGTERM
W0405 10:15:30.841000 1134 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 1223 closing signal SIGTERM
W0405 10:15:30.841000 1134 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 1224 closing signal SIGTERM
W0405 10:15:30.842000 1134 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 1225 closing signal SIGTERM
W0405 10:15:30.842000 1134 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 1226 closing signal SIGTERM
W0405 10:15:30.843000 1134 site-packages/torch/distributed/elastic/multiprocessing/api.py:908] Sending process 1227 closing signal SIGTERM
E0405 10:15:32.009000 1134 site-packages/torch/distributed/elastic/multiprocessing/api.py:882] failed (exitcode: 1) local_rank: 0 (pid: 1220) of binary: /workspace/miniconda3/envs/specforge/bin/python3.11
Traceback (most recent call last):
  File "/workspace/miniconda3/envs/specforge/bin/torchrun", line 6, in <module>
    sys.exit(main())
             ^^^^^^
  File "/workspace/miniconda3/envs/specforge/lib/python3.11/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 357, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/workspace/miniconda3/envs/specforge/lib/python3.11/site-packages/torch/distributed/run.py", line 936, in main
    run(args)
  File "/workspace/miniconda3/envs/specforge/lib/python3.11/site-packages/torch/distributed/run.py", line 927, in run
    elastic_launch(
  File "/workspace/miniconda3/envs/specforge/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 156, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/miniconda3/envs/specforge/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 293, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
/workspace/hanrui/SpecForge/scripts/train_dflash.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2026-04-05_10:15:30
  host      : job-006ce80a7c47-20260302193512-7694985998-5ng4c
  rank      : 0 (local_rank: 0)
  exitcode  : 1 (pid: 1220)
  error_file: <N/A>
  traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================