Hugging Face's logo

AdwolfCzar
/
anima-nextscene-archive

TensorBoard
Safetensors
Model card Files
xet
 Metrics Community
anima-nextscene-archive
File size: 25,159 Bytes
fbf35c9
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
/venv/main/lib/python3.12/site-packages/torch/jit/_script.py:1491: FutureWarning: `torch.jit.script` is deprecated. Please switch to `torch.compile` or `torch.export`.
  warnings.warn(
/venv/main/lib/python3.12/site-packages/torch/jit/_script.py:1491: FutureWarning: `torch.jit.script` is deprecated. Please switch to `torch.compile` or `torch.export`.
  warnings.warn(
[2026-09-30 06:12:54,353] [WARNING] [runner.py:232:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2026-09-30 06:12:54,353] [INFO] [runner.py:630:main] cmd = /venv/main/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMF19 --master_addr=127.0.0.1 --master_port=29500 --enable_each_rank_log=None --log_level=info train.py --deepspeed --config examples/anima_nextscene/gpu_20260930/smoke_b1.toml
/venv/main/lib/python3.12/site-packages/torch/jit/_script.py:1491: FutureWarning: `torch.jit.script` is deprecated. Please switch to `torch.compile` or `torch.export`.
  warnings.warn(
/venv/main/lib/python3.12/site-packages/torch/jit/_script.py:1491: FutureWarning: `torch.jit.script` is deprecated. Please switch to `torch.compile` or `torch.export`.
  warnings.warn(
[2026-09-30 06:12:57,437] [INFO] [launch.py:155:main] 0 NCCL_P2P_DISABLE=1
[2026-09-30 06:12:57,437] [INFO] [launch.py:155:main] 0 NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.28.3-1+cuda13.0
[2026-09-30 06:12:57,437] [INFO] [launch.py:155:main] 0 NV_LIBNCCL_DEV_PACKAGE_VERSION=2.28.3-1
[2026-09-30 06:12:57,437] [INFO] [launch.py:155:main] 0 NCCL_VERSION=2.28.3-1
[2026-09-30 06:12:57,437] [INFO] [launch.py:155:main] 0 NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
[2026-09-30 06:12:57,437] [INFO] [launch.py:155:main] 0 NV_LIBNCCL_PACKAGE=libnccl2=2.28.3-1+cuda13.0
[2026-09-30 06:12:57,437] [INFO] [launch.py:155:main] 0 NV_LIBNCCL_PACKAGE_NAME=libnccl2
[2026-09-30 06:12:57,437] [INFO] [launch.py:155:main] 0 NV_LIBNCCL_PACKAGE_VERSION=2.28.3-1
[2026-09-30 06:12:57,437] [INFO] [launch.py:162:main] WORLD INFO DICT: {'localhost': [0]}
[2026-09-30 06:12:57,437] [INFO] [launch.py:168:main] nnodes=1, num_local_procs=1, node_rank=0
[2026-09-30 06:12:57,437] [INFO] [launch.py:179:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0]})
[2026-09-30 06:12:57,437] [INFO] [launch.py:180:main] dist_world_size=1
[2026-09-30 06:12:57,437] [INFO] [launch.py:184:main] Setting CUDA_VISIBLE_DEVICES=0
[2026-09-30 06:12:57,437] [INFO] [launch.py:272:main] process 19657 spawned with command: ['/venv/main/bin/python', '-u', 'train.py', '--local_rank=0', '--deepspeed', '--config', 'examples/anima_nextscene/gpu_20260930/smoke_b1.toml']
/venv/main/lib/python3.12/site-packages/torch/jit/_script.py:1491: FutureWarning: `torch.jit.script` is deprecated. Please switch to `torch.compile` or `torch.export`.
  warnings.warn(
/venv/main/lib/python3.12/site-packages/torch/jit/_script.py:1491: FutureWarning: `torch.jit.script` is deprecated. Please switch to `torch.compile` or `torch.export`.
  warnings.warn(
[2026-09-30 06:13:01,417] [INFO] [comm.py:821:init_distributed] cdb=None
[2026-09-30 06:13:01,417] [INFO] [comm.py:852:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
using video_clip_mode=single_beginning
using framerate=16
Grouped metadata is not cached. Computing ungrouped metadata and then grouping.
Intermediate metadata is not cached. Enumerating all files.

  0%|          | 0/32 [00:00<?, ?it/s]
100%|██████████| 32/32 [00:00<00:00, 118987.35it/s]
Saving intermediate metadata dataset.

Saving the dataset (0/1 shards):   0%|          | 0/16 [00:00<?, ? examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 16/16 [00:00<00:00, 9918.54 examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 16/16 [00:00<00:00, 9447.96 examples/s]
Loading intermediate metadata dataset.
Caching ungrouped metadata.

Map (num_proc=8):   0%|          | 0/16 [00:00<?, ? examples/s]
Map (num_proc=8): 100%|██████████| 16/16 [00:00<00:00, 88.55 examples/s]

Grouping examples:   0%|          | 0/16 [00:00<?, ?it/s]
Grouping examples: 100%|██████████| 16/16 [00:00<00:00, 43919.41it/s]

Saving the dataset (0/1 shards):   0%|          | 0/16 [00:00<?, ? examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 16/16 [00:00<00:00, 10903.15 examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 16/16 [00:00<00:00, 10407.70 examples/s]
Loading grouped metadata with grouping key (1.0, 1)
caching latents: /workspace/ns_smoke/target
caching latents: (1.0, 1)

Adding size bucket:   0%|          | 0/16 [00:00<?, ? examples/s]
Adding size bucket: 100%|██████████| 16/16 [00:00<00:00, 7710.12 examples/s]
caching latents: (1.0, 512, 512, 1)
[CACHE] Initializing
[CACHE] Storing new fingerprint: 729cf0b7f391da14
[CACHE] Existing cache length: 0

  0%|          | 0/4 [00:00<?, ?it/s][CACHE] Creating new shard: shard_0

 25%|██▌       | 1/4 [00:00<00:02,  1.36it/s]
 50%|█████     | 2/4 [00:00<00:00,  2.60it/s]
100%|██████████| 4/4 [00:01<00:00,  5.32it/s]
100%|██████████| 4/4 [00:01<00:00,  3.94it/s]
Building iteration order

Saving the dataset (0/1 shards):   0%|          | 0/16 [00:00<?, ? examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 16/16 [00:00<00:00, 12593.14 examples/s]
Saving the dataset (1/1 shards): 100%|██████████| 16/16 [00:00<00:00, 11831.61 examples/s]
caching text embeddings: /workspace/ns_smoke/target
caching text embeddings: (1.0, 1)

Map:   0%|          | 0/16 [00:00<?, ? examples/s]
Map: 100%|██████████| 16/16 [00:00<00:00, 10446.59 examples/s]
[CACHE] Initializing
[CACHE] Storing new fingerprint: 85f0af65ef677478
[CACHE] Existing cache length: 0

  0%|          | 0/4 [00:00<?, ?it/s][CACHE] Creating new shard: shard_0

 25%|██▌       | 1/4 [00:00<00:02,  1.08it/s]
100%|██████████| 4/4 [00:01<00:00,  3.98it/s]
[CACHE] Initializing
[CACHE] Storing new fingerprint: b072e77f045c8ad1
[CACHE] Existing cache length: 0

  0%|          | 0/1 [00:00<?, ?it/s][CACHE] Creating new shard: shard_0

100%|██████████| 1/1 [00:00<00:00, 57.87it/s]
Found grouped metadata cache. Directly loading it.
Loading grouped metadata with grouping key [1.0, 1]
caching latents: /workspace/ns_smoke/target
caching latents: [1.0, 1]
caching latents: (1.0, 512, 512, 1)
[CACHE] Initializing
[CACHE] Existing cache has fingerprint 729cf0b7f391da14
[CACHE] Existing cache length: 16
caching text embeddings: /workspace/ns_smoke/target
caching text embeddings: [1.0, 1]

Map:   0%|          | 0/16 [00:00<?, ? examples/s]
Map: 100%|██████████| 16/16 [00:00<00:00, 7067.81 examples/s]
[CACHE] Initializing
[CACHE] Existing cache has fingerprint 85f0af65ef677478
[CACHE] Existing cache length: 16
[CACHE] Initializing
[CACHE] Existing cache has fingerprint b072e77f045c8ad1
[CACHE] Existing cache length: 1
[Anima NextScene] RoPE layout=disjoint_w ref_temporal_index=1
[Anima NextScene] LoRA targets: 280 linears (112 cross_attn; adaln/llm_adapter excluded)
trainable params: 96,468,992 || all params: 2,187,537,920 || trainable%: 4.4099
SEED_LAYERS=False BASE_SEED=1234 SEED_FN=None
Using topology: {ProcessCoord(pipe=0, data=0): 0}
[2026-09-30 06:13:05,584] [INFO] [module.py:398:_partition_layers] Partitioning pipeline stages with method parameters
stage=0 layers=31
     0: InitialLayer
     1: LLMAdapterLayer
     2: TransformerLayer
     3: TransformerLayer
     4: TransformerLayer
     5: TransformerLayer
     6: TransformerLayer
     7: TransformerLayer
     8: TransformerLayer
     9: TransformerLayer
    10: TransformerLayer
    11: TransformerLayer
    12: TransformerLayer
    13: TransformerLayer
    14: TransformerLayer
    15: TransformerLayer
    16: TransformerLayer
    17: TransformerLayer
    18: TransformerLayer
    19: TransformerLayer
    20: TransformerLayer
    21: TransformerLayer
    22: TransformerLayer
    23: TransformerLayer
    24: TransformerLayer
    25: TransformerLayer
    26: TransformerLayer
    27: TransformerLayer
    28: TransformerLayer
    29: TransformerLayer
    30: FinalLayer
  loss: loss_fn
[2026-09-30 06:13:05,947] [INFO] [logging.py:123:log_dist] [Rank 0] DeepSpeed info: version=0.18.4, git-hash=unknown, git-branch=unknown
[2026-09-30 06:13:05,947] [INFO] [config.py:687:__init__] Config mesh_device None world_size = 1
[2026-09-30 06:13:05,965] [INFO] [engine.py:1415:_configure_distributed_model] ********** distributed groups summary **********
	 self.dp_world_size=1
	 self.mp_world_size=1
	 self.seq_dp_world_size=1
	 self.sequence_parallel_size=1
***********************************************
[2026-09-30 06:13:05,974] [INFO] [logging.py:123:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
[2026-09-30 06:13:05,978] [INFO] [logging.py:123:log_dist] [Rank 0] [TorchCheckpointEngine] Initialized with serialization = True
[2026-09-30 06:13:05,978] [INFO] [config.py:958:print] DeepSpeedEngine configuration:
[2026-09-30 06:13:05,978] [INFO] [config.py:962:print]   activation_checkpointing_config  {
    "partition_activations": false, 
    "contiguous_memory_optimization": false, 
    "cpu_checkpointing": false, 
    "number_checkpoints": null, 
    "synchronize_checkpoint_boundary": false, 
    "profile": false
}
[2026-09-30 06:13:05,978] [INFO] [config.py:962:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'intra_op_parallelism': 1, 'single_submit': False, 'overlap_events': True, 'use_gds': False}
[2026-09-30 06:13:05,978] [INFO] [config.py:962:print]   amp_enabled .................. False
[2026-09-30 06:13:05,978] [INFO] [config.py:962:print]   amp_params ................... False
[2026-09-30 06:13:05,978] [INFO] [config.py:962:print]   autotuning_config ............ {
    "enabled": false, 
    "start_step": null, 
    "end_step": null, 
    "metric_path": null, 
    "arg_mappings": null, 
    "metric": "throughput", 
    "model_info": null, 
    "results_dir": "autotuning_results", 
    "exps_dir": "autotuning_exps", 
    "overwrite": true, 
    "fast": true, 
    "start_profile_step": 3, 
    "end_profile_step": 5, 
    "tuner_type": "gridsearch", 
    "tuner_early_stopping": 5, 
    "tuner_num_trials": 50, 
    "model_info_path": null, 
    "mp_size": 1, 
    "max_train_batch_size": null, 
    "min_train_batch_size": 1, 
    "max_train_micro_batch_size_per_gpu": 1.024000e+03, 
    "min_train_micro_batch_size_per_gpu": 1, 
    "num_tuning_micro_batch_sizes": 3
}
[2026-09-30 06:13:05,978] [INFO] [config.py:962:print]   bfloat16_config .............. enabled=False immediate_grad_update=False check_grad_overflow=False bf16_master_weights_and_grads=False bf16_optimizer_states=False
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   checkpoint_config ............ {'tag_validation': 'WARN', 'checkpoint_serialization': True, 'writer': None}
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   checkpoint_parallel_write_pipeline  False
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   checkpoint_tag_validation_enabled  True
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   checkpoint_tag_validation_fail  False
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7e09e56e1af0>
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   communication_data_type ...... None
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   compile_config ............... deepcompile=False free_activation=False free_activation_threshold=10485760 offload_activation=False offload_opt_states=False double_buffer=True symmetric_memory=False debug_log=False offload_parameters=False sync_before_reduce=False sync_after_reduce=False sync_before_allgather=False sync_after_allgather=False keep_int_input_tensors=True keep_all_input_tensors=False
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   curriculum_enabled_legacy .... False
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   curriculum_params_legacy ..... False
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'pin_memory': False, 'curriculum_learning': {'enabled': False}, 'dynamic_batching': {'enabled': False, 'lr_scaling_method': 'linear', 'min_batch_size': 1, 'max_batch_size': None, 'sequence_picking_order': 'dataloader', 'verbose': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   data_efficiency_enabled ...... False
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   dataloader_drop_last ......... False
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   datastates_config ............ {
    "enabled": false, 
    "config": null
}
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   disable_allgather ............ False
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   dump_state ................... False
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   eigenvalue_enabled ........... False
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   eigenvalue_gas_boundary_resolution  1
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   eigenvalue_layer_name ........ bert.encoder.layer
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   eigenvalue_layer_num ......... 0
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   eigenvalue_max_iter .......... 100
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   eigenvalue_stability ......... 1e-06
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   eigenvalue_tol ............... 0.01
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   eigenvalue_verbose ........... False
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   elasticity_enabled ........... False
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   float16_config ............... enabled=False auto_cast=False loss_scale=0.0 initial_scale_power=16 loss_scale_window=1000 hysteresis=2 consecutive_hysteresis=False min_loss_scale=1 fp16_master_weights_and_grads=False
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   flops_profiler_config ........ {
    "enabled": false, 
    "recompute_fwd_factor": 0.0, 
    "profile_step": 1, 
    "module_depth": -1, 
    "top_modules": 1, 
    "detailed": true, 
    "output_file": null
}
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   global_rank .................. 0
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   grad_accum_dtype ............. None
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   gradient_accumulation_steps .. 1
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   gradient_clipping ............ 1.0
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   gradient_predivide_factor .... 1.0
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   graph_harvesting ............. False
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   load_universal_checkpoint .... False
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   memory_breakdown ............. False
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   mics_hierarchial_params_gather  False
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   mics_shard_size .............. -1
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') comet=CometConfig(enabled=False, samples_log_interval=100, project=None, workspace=None, api_key=None, experiment_name=None, experiment_key=None, online=None, mode=None) wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName')
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   nebula_config ................ {
    "enabled": false, 
    "persistent_storage_path": null, 
    "persistent_time_interval": 100, 
    "num_of_version_in_retention": 2, 
    "enable_nebula_load": true, 
    "load_path": null
}
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   optimizer_legacy_fusion ...... False
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   optimizer_name ............... None
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   optimizer_params ............. None
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': True, 'grad_partitioned': True}
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   pld_enabled .................. False
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   pld_params ................... False
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   prescale_gradients ........... False
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   scheduler_name ............... None
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   scheduler_params ............. None
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   seq_parallel_communication_data_type  torch.float32
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   sparse_attention ............. None
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   sparse_gradients_enabled ..... False
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   steps_per_print .............. 1
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   tensor_parallel_config ....... dtype=torch.float16 autotp_size=0 tp_overlap_comm=False tensor_parallel=TPConfig(tp_size=1, tp_grain_size=1, mpu=None, tp_group=None) injection_policy_tuple=None keep_module_on_host=False replace_with_kernel_inject=False
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   timers_config ................ enabled=True synchronized=True
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   torch_autocast_dtype ......... None
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   torch_autocast_enabled ....... False
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   torch_autocast_lower_precision_safe_modules  None
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   train_batch_size ............. 1
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   train_micro_batch_size_per_gpu  1
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   use_data_before_expert_parallel_  False
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   use_node_local_storage ....... False
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   wall_clock_breakdown ......... False
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   weight_quantization_config ... None
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   world_size ................... 1
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   zero_allow_untested_optimizer  False
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   zero_config .................. stage=0 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=500000000 use_multi_rank_bucket_allreduce=True allgather_partitions=True allgather_bucket_size=500000000 overlap_comm=False load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=None zenflow=None sub_group_size=1000000000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50000000 param_persistence_threshold=100000 model_persistence_threshold=9223372036854775807 max_live_parameters=1000000000 max_reuse_distance=1000000000 gather_16bit_weights_on_model_save=False module_granularity_threshold=0 use_all_reduce_for_fetch_params=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False zeropp_loco_param=None mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True log_trace_cache_warnings=False enable_sanity_checks=False leaf_module=DeepSpeedZeroLeafModuleConfig(classes=['transformers.models.mixtral.modeling_mixtral.MixtralSparseMoeBlock', 'transformers.models.qwen2_moe.modeling_qwen2_moe.Qwen2MoeSparseMoeBlock', 'transformers.models.qwen3_moe.modeling_qwen3_moe.Qwen3MoeSparseMoeBlock'], names=[], name_suffixes=[])
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   zero_enabled ................. False
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   zero_force_ds_cpu_optimizer .. True
[2026-09-30 06:13:05,979] [INFO] [config.py:962:print]   zero_optimization_stage ...... 0
[2026-09-30 06:13:05,979] [INFO] [config.py:948:print_user_config]   json = {
    "train_micro_batch_size_per_gpu": 1, 
    "gradient_accumulation_steps": 1, 
    "gradient_clipping": 1.0, 
    "steps_per_print": 1
}
[2026-09-30 06:13:05,979] [INFO] [engine.py:104:__init__] CONFIG: micro_batches=1 micro_batch_size=1
[2026-09-30 06:13:05,979] [INFO] [engine.py:145:__init__] is_pipe_partitioned= False is_grad_partitioned= False
[2026-09-30 06:13:05,982] [INFO] [engine.py:164:__init__] RANK=0 STAGE=0 LAYERS=31 [0, 31) STAGE_PARAMS=96468992 (96.469M) TOTAL_PARAMS=96468992 (96.469M) UNIQUE_PARAMS=96468992 (96.469M)Global batch size = 1Using base_lr=5e-05, self_attn_lr=5e-05, cross_attn_lr=5e-05, mlp_lr=5e-05, mod_lr=5e-05, llm_adapter_lr=5e-05Num base params: 0Num self_attn params: 224Num cross_attn params: 224Num mlp params: 112Num mod params: 0Num llm_adapter params: 72[2026-09-30 06:13:06,008] [INFO] [logging.py:123:log_dist] [Rank 0] Using client callable to create basic optimizer
[2026-09-30 06:13:06,008] [INFO] [logging.py:123:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer
[2026-09-30 06:13:06,089] [INFO] [logging.py:123:log_dist] [Rank 0] DeepSpeed Basic Optimizer = AdamW
[2026-09-30 06:13:06,089] [INFO] [logging.py:123:log_dist] [Rank 0] DeepSpeed Final Optimizer = AdamW
[rank0]: Traceback (most recent call last):
[rank0]:   File "/workspace/diffusion-pipe-easycontrol/train.py", line 1010, in <module>
[rank0]:     iterator = get_data_iterator_for_step(train_dataloader, model_engine)
[rank0]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/workspace/diffusion-pipe-easycontrol/train.py", line 172, in get_data_iterator_for_step
[rank0]:     items = [next(dataloader_iter) for _ in range(num_micro_batches)]
[rank0]:              ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/workspace/diffusion-pipe-easycontrol/utils/dataset.py", line 1494, in __next__
[rank0]:     self.next_micro_batch = next(self.data)
[rank0]:                             ^^^^^^^^^^^^^^^
[rank0]:   File "/workspace/diffusion-pipe-easycontrol/utils/dataset.py", line 1531, in _pull_batches_from_dataloader
[rank0]:     target, mask = label
[rank0]:     ^^^^^^^^^^^^
[rank0]: ValueError: too many values to unpack (expected 2)
[2026-09-30 06:13:08,438] [INFO] [launch.py:335:sigkill_handler] Killing subprocess 19657
[2026-09-30 06:13:08,439] [ERROR] [launch.py:341:sigkill_handler] ['/venv/main/bin/python', '-u', 'train.py', '--local_rank=0', '--deepspeed', '--config', 'examples/anima_nextscene/gpu_20260930/smoke_b1.toml'] exits with return code = 1