torchrun --nproc_per_node 1 --nnodes 1 --node_rank 0 --master_addr localhost --master_port 35632 /mnt/ssd/lvzhihao/PostTrain/YuLan-Pretrain/scripts/distributed_checkpoints_convertor/impl/convert.py --tokenizer-type HuggingFaceTokenizer --tokenizer-model /mnt/ssd/cache_tmp/tmp/tmp.1mZMUIgeRZ --hf-dir /mnt/ssd/cache_tmp/tmp/tmp.1mZMUIgeRZ --mcore2hf --use-gpu --bf16 --normalization RMSNorm --swiglu --disable-bias-linear --seq-length 1 --max-position-embeddings 490000 --attention-backend auto --position-embedding-type rope --kv-channels 64 --group-query-attention --add-qkv-bias --num-layers 56 --hidden-size 1920 --ffn-hidden-size 4800 --num-attention-heads 30 --untie-embeddings-and-output-weights --rotary-base 490000 --rotary-percent 1.00 --num-query-groups 6 --normalization RMSNorm --norm-epsilon 1e-6 --linear-attention-type gated_delta_net --linear-attention-freq [1,1,1,1,1,1,1,1,1,1,1,1,0,1,1,1,1,1,1,1,0,0,1,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,1,0,0,1,1,1,1,1,1] --linear-conv-kernel-dim 4 --linear-key-head-dim 64 --linear-value-head-dim 64 --linear-num-key-heads 8 --linear-num-value-heads 32 --micro-batch-size 1 --global-batch-size 1024 --train-iters 500000 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.95 --init-method-std 0.006 --clip-grad 1.0 --lr 2.0e-5 --lr-decay-style cosine --min-lr 6.0e-6 --lr-warmup-fraction .001 --lr-decay-iters 430000 --bf16 --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --expert-tensor-parallel-size 1 --expert-model-parallel-size 1 --log-interval 100 --save-interval 10000 --eval-interval 1000 --eval-iters 10 --model-type GPT --load-dir /mnt/hdd/lvzhihao/mcore_models/Dist-mathcode10b-s1randg-sch1-CPT-200b-stage3-r640k-GDN2.9b-A7-12_20_21_23_46_48_49-sl32768bs128lr2e5-2e5 --save-dir /mnt/hdd/lvzhihao/mcore_models/Dist-mathcode10b-s1randg-sch1-CPT-200b-stage3-r640k-GDN2.9b-A7-12_20_21_23_46_48_49-sl32768bs128lr2e5-2e5/iter_71525-hf --dist-ckpt-optim-fully-reshardable --skip-train --use-cpu-initialization --padded-vocab-size 99000 --no-load-optim --no-load-rng --logging-level 1 --attention-backend auto --synchronizer mcore_gdn_moe --pretrain-script mcore_gdn_moe.model_provider --debug --max-shard-size 20GB --ckpt-step 71525 W0316 15:31:46.709000 11298 .venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. W0316 15:31:46.709000 11298 .venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures. fused_indices_to_multihot has reached end of life. Please migrate to a non-experimental function. Current Python version 3.10 is below the recommended 3.11 version. It is recommended to upgrade to Python 3.11 or higher for the best experience. Warning: Pai-Megatron-Patch arguments not available, some arguments may not be recognized using world size: 1, data-parallel size: 1, context-parallel size: 1, hierarchical context-parallel sizes: None, tensor-model-parallel size: 1, pipeline-model-parallel size: 1 Number of virtual stages per pipeline stage: None accumulate and all-reduce gradients in fp32 for bfloat16 data type. using torch.bfloat16 for parameters ... ------------------------ arguments ------------------------ account_for_embedding_in_pipeline_split ......... False account_for_loss_in_pipeline_split .............. False accumulate_allreduce_grads_in_fp32 .............. True activation_func_clamp_value ..................... None adam_beta1 ...................................... 0.9 adam_beta2 ...................................... 0.95 adam_eps ........................................ 1e-08 adamw_lr_mup_scaler ............................. False add_bias_linear ................................. False add_position_embedding .......................... True add_qkv_bias .................................... True adlr_autoresume ................................. False adlr_autoresume_interval ........................ 1000 align_grad_reduce ............................... True align_param_gather .............................. False allow_ambiguous_pad_tokens ...................... False app_tag_run_name ................................ None app_tag_run_version ............................. 0.0.0 apply_layernorm_1p .............................. False apply_query_key_layer_scaling ................... False apply_residual_connection_post_layernorm ........ False apply_rope_fusion ............................... True async_save ...................................... None async_tensor_model_parallel_allreduce ........... True attention_backend ............................... AttnBackend.auto attention_dropout ............................... 0.1 attention_output_gate ........................... False attention_softmax_in_fp32 ....................... False attn_k_token_shift .............................. None attn_output_gate ................................ None attn_output_gate_rand_init ...................... False attn_q_token_shift .............................. None attn_token_shift ................................ None attn_v_token_shift .............................. None auto_detect_ckpt_format ......................... False auto_generate_cu_seqlens ........................ False auto_model ...................................... AutoModelForCausalLM barrier_with_L1_time ............................ True benchmark_eval .................................. False benchmark_global_batch .......................... None benchmark_interval .............................. None benchmark_micro_batch ........................... None benchmark_sequence_length ....................... None benchmark_tasks ................................. None bert_binary_head ................................ True bert_embedder_type .............................. megatron bert_load ....................................... None bf16 ............................................ True bias_dropout_fusion ............................. True bias_gelu_fusion ................................ False bias_swiglu_fusion .............................. True biencoder_projection_dim ........................ 0 biencoder_shared_query_context_model ............ False block_data_path ................................. None cache_mla_latents ............................... False calc_ft_timeouts ................................ False calculate_per_token_loss ........................ False check_for_large_grads ........................... False check_for_nan_in_loss_and_grad .................. True check_for_spiky_loss ............................ False check_weight_hash_across_dp_replicas_interval ... None ckpt_assume_constant_structure .................. False ckpt_convert_format ............................. None ckpt_convert_save ............................... None ckpt_convert_update_legacy_dist_opt_format ...... False ckpt_format ..................................... torch_dist ckpt_fully_parallel_load ........................ False ckpt_fully_parallel_save ........................ True ckpt_fully_parallel_save_deprecated ............. False ckpt_step ....................................... 71525 classes_fraction ................................ 1.0 clip_grad ....................................... 1.0 clone_scatter_output_in_embedding ............... True config_logger_dir ............................... consumed_train_samples .......................... 0 consumed_valid_samples .......................... 0 context_parallel_size ........................... 1 cp_comm_type .................................... ['p2p'] create_attention_mask_in_dataloader ............. True cross_entropy_fusion_impl ....................... native cross_entropy_loss_fusion ....................... False cuda_graph_impl ................................. none cuda_graph_scope ................................ [] cuda_graph_warmup_steps ......................... 3 data_args_path .................................. None data_cache_path ................................. None data_parallel_random_init ....................... False data_parallel_sharding_strategy ................. no_shard data_parallel_size .............................. 1 data_path ....................................... None data_per_class_fraction ......................... 1.0 data_sharding ................................... True dataloader_type ................................. single ddp_average_in_collective ....................... False ddp_bucket_size ................................. None ddp_num_buckets ................................. None ddp_pad_buckets_for_high_nccl_busbw ............. False debug ........................................... True decode_only_cuda_graphs ......................... False decoder_first_pipeline_num_layers ............... None decoder_last_pipeline_num_layers ................ None decoder_num_layers .............................. None decoder_seq_length .............................. None decoupled_lr .................................... None decoupled_min_lr ................................ None decrease_batch_size_if_needed ................... False defer_embedding_wgrad_compute ................... False delay_wgrad_compute ............................. False deprecated_use_mcore_models ..................... False deterministic_mode .............................. False dino_bottleneck_size ............................ 256 dino_freeze_last_layer .......................... 1 dino_head_hidden_size ........................... 2048 dino_local_crops_number ......................... 10 dino_local_img_size ............................. 96 dino_norm_last_layer ............................ False dino_teacher_temp ............................... 0.07 dino_warmup_teacher_temp ........................ 0.04 dino_warmup_teacher_temp_epochs ................. 30 disable_attn_output_gate ........................ False disable_bf16_reduced_precision_matmul ........... False disable_chunked_prefill ......................... False disable_explicit_attention_mask ................. False disable_mamba_mem_eff_path ...................... False disable_straggler_on_startup .................... False disable_symmetric_registration .................. False dist_ckpt_format_deprecated ..................... None dist_ckpt_optim_fully_reshardable ............... True dist_ckpt_save_pre_mcore_014 .................... False dist_ckpt_strictness ............................ assume_ok_unexpected distrib_optim_fully_reshardable_mem_efficient ... False distribute_saved_activations .................... False distributed_backend ............................. nccl distributed_timeout_minutes ..................... 10 distributed_timeout_seconds_after_init .......... None document_packing_algorithm ...................... random dryrun .......................................... False dump_param_to_param_group_map ................... None emb_deviation_loss_coeff ........................ 0 emb_deviation_type .............................. None embedding_init_method_std ....................... None embedding_path .................................. None empty_unused_memory_level ....................... 0 enable_cuda_graph ............................... False enable_debug_logging ............................ False enable_experimental ............................. False enable_ft_package ............................... False enable_full_sharding_in_hsdp .................... False enable_gloo_process_groups ...................... True enable_msc ...................................... True enable_one_logger ............................... True encoder_num_layers .............................. 56 encoder_seq_length .............................. 1 end_weight_decay ................................ 0.1 eod_mask_loss ................................... False error_injection_rate ............................ 0 error_injection_type ............................ transient_error eval_interval ................................... 1000 eval_iters ...................................... 10 evidence_data_path .............................. None exit_duration_in_mins ........................... None exit_interval ................................... None exit_on_missing_checkpoint ...................... False exit_signal_handler ............................. False exp_avg_dtype ................................... torch.float32 exp_avg_sq_dtype ................................ torch.float32 expert_model_parallel_size ...................... 1 expert_tensor_parallel_size ..................... 1 external_cuda_graph ............................. False ffn_hidden_size ................................. 4800 ffn_intermediate_token_shift .................... None ffn_token_shift ................................. None fine_grained_activation_offloading .............. False finetune ........................................ False first_last_layers_bf16 .......................... False flash_decode .................................... False fp16 ............................................ False fp16_lm_cross_entropy ........................... False fp32_residual_connection ........................ False fp4 ............................................. None fp4_param ....................................... False fp4_recipe ...................................... nvfp4 fp8 ............................................. None fp8_amax_compute_algo ........................... most_recent fp8_amax_history_len ............................ 1 fp8_interval .................................... 1 fp8_margin ...................................... 0 fp8_param_gather ................................ False fp8_recipe ...................................... delayed fp8_wgrad ....................................... True freeze_layernorm_weight ......................... False freeze_non_mamba ................................ False fsdp_double_buffer .............................. False full_validation ................................. False geglu ........................................... False global_batch_size ............................... 1024 glu_linear_offset ............................... 0.0 grad_reduce_in_bf16 ............................. False gradient_accumulation_fusion .................... True gradient_reduce_div_fusion ...................... True group_query_attention ........................... True grpo_clamp_eps_lower ............................ 0.01 grpo_clamp_eps_upper ............................ 0.01 grpo_default_temperature ........................ 1.0 grpo_default_top_p .............................. 0 grpo_entropy_term_weight ........................ 0.0 grpo_filter_groups_with_same_reward ............. False grpo_group_size ................................. 2 grpo_iterations ................................. 2 grpo_kl_beta .................................... 0.001 grpo_prompts_per_step ........................... 32 head_lr_mult .................................... 1.0 heterogeneous_layers_config_encoded_json ........ None heterogeneous_layers_config_path ................ None hf_dir .......................................... /mnt/ssd/cache_tmp/tmp/tmp.1mZMUIgeRZ hidden_dropout .................................. 0.1 hidden_size ..................................... 1920 hierarchical_context_parallel_sizes ............. None high_priority_stream_groups ..................... [] hybrid_attention_ratio .......................... 0.0 hybrid_mlp_ratio ................................ 0.0 hybrid_override_pattern ......................... None hysteresis ...................................... 2 ict_head_size ................................... None ict_load ........................................ None img_h ........................................... 224 img_w ........................................... 224 increase_log_level_interval ..................... 1000 increase_log_level_iters ........................ 5 indexer_batch_size .............................. 128 indexer_log_interval ............................ 1000 inference_batch_times_seqlen_threshold .......... -1 inference_dynamic_batching ...................... False inference_dynamic_batching_block_size ........... 256 inference_dynamic_batching_buffer_guaranteed_fraction 0.2 inference_dynamic_batching_buffer_overflow_factor None inference_dynamic_batching_buffer_size_gb ....... 40.0 inference_dynamic_batching_max_requests_override None inference_dynamic_batching_max_tokens_override .. None inference_dynamic_batching_num_cuda_graphs ...... 16 inference_dynamic_batching_track_paused_request_events False inference_dynamic_batching_unified_memory_level . 0 inference_max_batch_size ........................ 8 inference_max_seq_length ........................ 2560 inference_rng_tracker ........................... False init_method_std ................................. 0.006 init_method_xavier_uniform ...................... False init_model_with_meta_device ..................... False initial_loss_scale .............................. 4294967296 inprocess_active_world_size ..................... 1 inprocess_barrier_timeout ....................... 120 inprocess_completion_timeout .................... 120 inprocess_empty_cuda_cache ...................... False inprocess_granularity ........................... node inprocess_hard_timeout .......................... 90 inprocess_heartbeat_interval .................... 30 inprocess_heartbeat_timeout ..................... 60 inprocess_last_call_wait ........................ 1 inprocess_max_iterations ........................ None inprocess_monitor_process_interval .............. 1.0 inprocess_monitor_thread_interval ............... 1.0 inprocess_progress_watchdog_interval ............ 1.0 inprocess_restart ............................... False inprocess_soft_timeout .......................... 60 inprocess_termination_grace_time ................ 1 is_hybrid_model ................................. False iter_per_epoch .................................. 1250 iterations_to_skip .............................. [] keep_fp8_transpose_cache ........................ False kitchen_config_file ............................. None kitchen_recipe_number ........................... None kv_channels ..................................... 64 kv_lora_rank .................................... 32 langrl_env_config ............................... None langrl_external_server .......................... False langrl_inference_server_conversation_template ... None langrl_inference_server_type .................... inplace_megatron lazy_mpu_init ................................... None legacy_tokenizer ................................ False linear_attention_freq ........................... [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1] linear_attention_type ........................... gated_delta_net linear_conv_kernel_dim .......................... 4 linear_key_head_dim ............................. 64 linear_num_key_heads ............................ 8 linear_num_value_heads .......................... 32 linear_value_head_dim ........................... 64 load ............................................ None load_complemental_dataset ....................... None load_dir ........................................ /mnt/hdd/lvzhihao/mcore_models/Dist-mathcode10b-s1randg-sch1-CPT-200b-stage3-r640k-GDN2.9b-A7-12_20_21_23_46_48_49-sl32768bs128lr2e5-2e5 load_main_params_from_ckpt ...................... None local_rank ...................................... 0 log_energy ...................................... False log_hidden_states ............................... [] log_interval .................................... 100 log_loss_scale_to_tensorboard ................... True log_memory_to_tensorboard ....................... False log_num_zeros_in_grad ........................... False log_params ...................................... [] log_params_norm ................................. False log_per_module_grad_rms ......................... False log_per_module_update_rms ....................... False log_progress .................................... False log_straggler ................................... False log_throughput .................................. False log_timers_to_tensorboard ....................... False log_validation_ppl_to_tensorboard ............... False log_world_size_to_tensorboard ................... False logging_level ................................... 1 loss_scale ...................................... None loss_scale_window ............................... 1000 lr .............................................. 2e-05 lr_decay_iters .................................. 430000 lr_decay_samples ................................ None lr_decay_style .................................. cosine lr_warmup_fraction .............................. 0.001 lr_warmup_init .................................. 0.0 lr_warmup_iters ................................. 0 lr_warmup_samples ............................... 0 lr_wsd_decay_iters .............................. None lr_wsd_decay_samples ............................ None lr_wsd_decay_style .............................. exponential main_grads_dtype ................................ torch.float32 main_params_dtype ............................... torch.float32 make_vocab_size_divisible_by .................... 128 mamba_disable_cp ................................ False mamba_expand .................................... 2 mamba_head_dim .................................. 64 mamba_num_groups ................................ 8 mamba_num_heads ................................. None mamba_state_dim ................................. 128 manual_gc ....................................... False manual_gc_eval .................................. True manual_gc_interval .............................. 0 mask_factor ..................................... 1.0 mask_prob ....................................... 0.15 mask_type ....................................... random masked_softmax_fusion ........................... True max_position_embeddings ......................... 490000 max_shard_size .................................. 20GB max_tokens_to_oom ............................... 12000 mcore2hf ........................................ True memory_snapshot_path ............................ None merge_file ...................................... None micro_batch_size ................................ 1 microbatch_group_size_per_vp_stage .............. None mid_level_dataset_surplus ....................... 0.005 min_loss_scale .................................. 1.0 min_lr .......................................... 6e-06 min_offloaded_tensor_size ....................... 1048576 mlp_chunks_for_prefill .......................... 1 mmap_bin_files .................................. True mock_data ....................................... False model_type ...................................... GPT moe_apply_probs_on_input ........................ False moe_aux_loss_coeff .............................. 0.0 moe_deepep_num_sms .............................. 20 moe_enable_deepep ............................... False moe_expert_capacity_factor ...................... None moe_extended_tp ................................. False moe_ffn_hidden_size ............................. None moe_flex_dispatcher_backend ..................... deepep moe_grouped_gemm ................................ False moe_hybridep_num_sms ............................ 16 moe_input_jitter_eps ............................ None moe_layer_freq .................................. 1 moe_layer_recompute ............................. False moe_pad_expert_input_to_capacity ................ False moe_pad_experts_for_cuda_graph_inference ........ False moe_per_layer_logging ........................... False moe_permute_fusion .............................. False moe_router_bias_update_method ................... sign moe_router_bias_update_rate ..................... 0.001 moe_router_dtype ................................ None moe_router_enable_expert_bias ................... False moe_router_force_load_balancing ................. False moe_router_fusion ............................... False moe_router_group_topk ........................... None moe_router_load_balancing_type .................. aux_loss moe_router_num_groups ........................... None moe_router_padding_for_fp8 ...................... False moe_router_padding_for_quantization ............. False moe_router_pre_softmax .......................... False moe_router_score_function ....................... softmax moe_router_topk ................................. 2 moe_router_topk_scaling_factor .................. None moe_shared_expert_gate .......................... False moe_shared_expert_intermediate_size ............. None moe_shared_expert_overlap ....................... False moe_token_dispatcher_type ....................... allgather moe_token_drop_policy ........................... probs moe_upcycling_granularity ....................... 1 moe_use_legacy_grouped_gemm ..................... False moe_use_upcycling ............................... False moe_z_loss_coeff ................................ None mrope_section ................................... None mscale .......................................... 1.0 mscale_all_dim .................................. 0.0 mtp_linear_attention_type ....................... None mtp_loss_scaling_factor ......................... 0.1 mtp_num_layers .................................. None multi_latent_attention .......................... False multiple_validation_sets ........................ False muon_ball_momentum .............................. 0.9 muon_ball_msign_steps ........................... 5 muon_ball_power_iteration_steps ................. 10 muon_ball_qkv_split_mode ........................ component muon_ball_radius_mode ........................... spectral_mup muon_ball_retract_alpha ......................... 0.05 muon_ball_retract_mode .......................... hard muon_ball_scale_mode ............................ spectral_mup muon_ball_split_fc1 ............................. True muon_ball_split_moe_experts ..................... True muon_ball_split_qkv ............................. True muon_ball_use_nesterov .......................... True muon_extra_scale_factor ......................... 1.0 muon_fp32_matmul_prec ........................... medium muon_momentum ................................... 0.9 muon_num_ns_steps ............................... 5 muon_qkv_split_mode ............................. component muon_scale_mode ................................. spectral_mup muon_scale_vectorized_mode ...................... full muon_split_fc1 .................................. True muon_split_moe_experts .......................... True muon_split_qkv .................................. True muon_tp_mode .................................... blockwise muon_use_nesterov ............................... False muon_vectorize .................................. [] muon_vectorize_attn_dim ......................... hidden_size nccl_all_reduce_for_prefill ..................... False nccl_communicator_config_path ................... None nccl_ub ......................................... False no_load_optim ................................... True no_load_rng ..................................... True no_load_scheduler ............................... None no_persist_layer_norm ........................... False no_rope_freq .................................... None no_save_optim ................................... None no_save_rng ..................................... None no_save_step_one ................................ None no_weight_decay_cond_type ....................... None non_persistent_ckpt_type ........................ None non_persistent_global_ckpt_dir .................. None non_persistent_local_ckpt_algo .................. fully_parallel non_persistent_local_ckpt_dir ................... None non_persistent_save_interval .................... None norm_epsilon .................................... 1e-06 normalization ................................... RMSNorm num_attention_heads ............................. 30 num_channels .................................... 3 num_classes ..................................... 1000 num_dataset_builder_threads ..................... 1 num_distributed_optimizer_instances ............. 1 num_experts ..................................... None num_hf_saver .................................... None num_layers ...................................... 56 num_layers_at_end_in_bf16 ....................... 1 num_layers_at_start_in_bf16 ..................... 1 num_layers_per_virtual_pipeline_stage ........... None num_query_groups ................................ 6 num_virtual_stages_per_pipeline_rank ............ None num_workers ..................................... 2 object_storage_cache_path ....................... None offload_modules ................................. [] one_logger_async ................................ False one_logger_project .............................. megatron-lm one_logger_run_name ............................. None onnx_safe ....................................... None openai_gelu ..................................... False optimizer ....................................... adam optimizer_cpu_offload ........................... False optimizer_offload_fraction ...................... 1.0 output_bert_embeddings .......................... False overlap_cpu_optimizer_d2h_h2d ................... False overlap_grad_reduce ............................. False overlap_moe_expert_parallel_comm ................ False overlap_p2p_comm ................................ False overlap_p2p_comm_warmup_flush ................... False overlap_param_gather ............................ False overlap_param_gather_with_optimizer_step ........ False override_hf_eod_token_id ........................ None override_opt_param_scheduler .................... False padded_vocab_size ............................... 99000 params_dtype .................................... torch.bfloat16 patch_dim ....................................... 16 per_split_data_args_path ........................ None perform_initialization .......................... True perform_rl_step ................................. False pin_cpu_grads ................................... True pin_cpu_params .................................. True pipeline_model_parallel_comm_backend ............ None pipeline_model_parallel_layout .................. None pipeline_model_parallel_size .................... 1 position_embedding_type ......................... rope pretrain_script ................................. mcore_gdn_moe.model_provider pretrained_checkpoint ........................... None profile ......................................... False profile_ranks ................................... [0] profile_step_end ................................ 12 profile_step_start .............................. 10 q_lora_rank ..................................... None qk_head_dim ..................................... 128 qk_l2_norm ...................................... False qk_layernorm .................................... False qk_pos_emb_head_dim ............................. 64 query_in_block_prob ............................. 0.1 quick_geglu ..................................... False rampup_batch_size ............................... None rank ............................................ 0 recompute_granularity ........................... None recompute_method ................................ None recompute_modules ............................... None recompute_num_layers ............................ None record_memory_history ........................... False relative_attention_max_distance ................. 128 relative_attention_num_buckets .................. 32 reparam_checkpoint .............................. None reparam_fallback_value .......................... None reparam_keys .................................... None replication ..................................... False replication_factor .............................. 2 replication_jump ................................ None rerun_mode ...................................... validate_results reset_attention_mask ............................ False reset_iteration_one_to_zero ..................... False reset_position_ids .............................. False result_rejected_tracker_filename ................ None retriever_report_topk_accuracies ................ [] retriever_score_scaling ......................... False retriever_seq_length ............................ 256 retro_add_retriever ............................. False retro_attention_gate ............................ 1 retro_cyclic_train_iters ........................ None retro_encoder_attention_dropout ................. 0.1 retro_encoder_hidden_dropout .................... 0.1 retro_encoder_layers ............................ 2 retro_num_neighbors ............................. 2 retro_num_retrieved_chunks ...................... 2 retro_project_dir ............................... None retro_verify_neighbor_count ..................... True reuse_grad_buf_for_mxfp8_param_ag ............... False rl_calculate_intra_group_similarity ............. False rl_importance_sampling_truncation_coef .......... None rl_inference_logprobs_is_correction ............. False rl_offload_kv_cache_during_training ............. False rl_offload_optimizer_during_inference ........... False rl_partial_rollouts ............................. False rl_prompts_per_eval ............................. 32 rl_remove_kv_cache_during_training .............. False rl_reset_cuda_graphs ............................ False rl_sequence_packing_algo ........................ fifo rl_sequence_packing_bin_size .................... 8192 rl_use_sequence_packing ......................... False rope_scaling_factor ............................. 8.0 rope_type ....................................... None rotary_base ..................................... 490000 rotary_interleaved .............................. False rotary_percent .................................. 1.0 rotary_scaling_factor ........................... 1.0 rotary_seq_len_interpolation_factor ............. None run_workload_inspector_server ................... False sample_rate ..................................... 1.0 save ............................................ None save_dir ........................................ /mnt/hdd/lvzhihao/mcore_models/Dist-mathcode10b-s1randg-sch1-CPT-200b-stage3-r640k-GDN2.9b-A7-12_20_21_23_46_48_49-sl32768bs128lr2e5-2e5/iter_71525-hf save_interval ................................... 10000 save_retain_interval ............................ None scatter_gather_tensors_in_pipeline .............. True seed ............................................ 1234 seq_length ...................................... 1 sequence_parallel ............................... False sft ............................................. False sft_tokenizer_prompt_format ..................... nemotron-h-aligned sgd_momentum .................................... 0.9 sharp_enabled_group ............................. None short_seq_prob .................................. 0.1 skip_train ...................................... True skipped_train_samples ........................... 0 softmax_type .................................... vanilla spec ............................................ None spectral_ball_momentum .......................... 0.9 spectral_ball_msign_steps ....................... 8 spectral_ball_power_iteration_steps ............. 20 spectral_ball_qkv_split_mode .................... component spectral_ball_radius_mode ....................... spectral_mup spectral_ball_retract_alpha ..................... 0.05 spectral_ball_retract_mode ...................... hard spectral_ball_scale_mode ........................ spectral_mup spectral_ball_solver ............................ bisection spectral_ball_solver_max_iterations ............. 20 spectral_ball_solver_tolerance_f ................ 1e-08 spectral_ball_split_fc1 ......................... True spectral_ball_split_moe_experts ................. True spectral_ball_split_qkv ......................... True spectral_ball_use_nesterov ...................... True spectral_mup_init ............................... False split ........................................... None split_expert_init ............................... True split_fc1_init .................................. True split_qkv_init .................................. True split_qkv_init_mode ............................. group sqreglu ......................................... False squared_relu .................................... False start_samples ................................... None start_weight_decay .............................. 0.1 straggler_ctrlr_port ............................ 65535 straggler_minmax_count .......................... 1 strict_fsdp_dtensor_load ........................ True suggested_communication_unit_size ............... None swanlab_exp_name ................................ swanlab_project ................................. swanlab_save_dir ................................ swanlab_workspace ............................... swiglu .......................................... True swin_backbone_type .............................. tiny symmetric_ar_type ............................... None synchronizer .................................... mcore_gdn_moe target_ckpt_format .............................. torch_dist te_rng_tracker .................................. False tensor_model_parallel_size ...................... 1 tensorboard_dir ................................. None tensorboard_log_interval ........................ 1 tensorboard_queue_size .......................... 1000 test_data_path .................................. None test_mode ....................................... False tiktoken_num_special_tokens ..................... 1000 tiktoken_pattern ................................ None tiktoken_special_tokens ......................... None timing_log_level ................................ 0 timing_log_option ............................... minmax titles_data_path ................................ None token_shift_conv_init ........................... default token_shift_conv_size ........................... 4 tokenizer_metadata .............................. None tokenizer_model ................................. /mnt/ssd/cache_tmp/tmp/tmp.1mZMUIgeRZ tokenizer_type .................................. HuggingFaceTokenizer torch_fsdp2_reshard_after_forward ............... True tp_comm_bootstrap_backend ....................... nccl tp_comm_bulk_dgrad .............................. True tp_comm_bulk_wgrad .............................. True tp_comm_overlap ................................. False tp_comm_overlap_ag .............................. True tp_comm_overlap_cfg ............................. None tp_comm_overlap_rs .............................. True tp_comm_overlap_rs_dgrad ........................ False tp_comm_split_ag ................................ True tp_comm_split_rs ................................ True train_data_path ................................. None train_iters ..................................... 500000 train_samples ................................... None train_sync_interval ............................. None transformer_impl ................................ transformer_engine transformer_pipeline_model_parallel_size ........ 1 trust_remote_code ............................... False untie_embeddings_and_output_weights ............. True use_checkpoint_args ............................. False use_checkpoint_opt_param_scheduler .............. False use_cpu_initialization .......................... True use_dist_ckpt ................................... True use_dist_ckpt_deprecated ........................ False use_distributed_optimizer ....................... False use_flash_attn .................................. False use_fused_weighted_squared_relu ................. False use_gpu ......................................... True use_legacy_models ............................... False use_megatron_fsdp ............................... False use_mp_args_from_checkpoint_args ................ False use_one_sent_docs ............................... False use_persistent_ckpt_worker ...................... False use_precision_aware_optimizer ................... False use_pytorch_profiler ............................ False use_ring_exchange_p2p ........................... False use_rope_scaling ................................ False use_rotary_position_embeddings .................. False use_sharp ....................................... False use_te_activation_func .......................... False use_tokenizer_model_from_checkpoint_args ........ True use_torch_fsdp2 ................................. False use_torch_optimizer_for_cpu_offload ............. False use_tp_pp_dp_mapping ............................ False v_head_dim ...................................... 128 valid_data_path ................................. None variable_seq_lengths ............................ False virtual_pipeline_model_parallel_size ............ None vision_backbone_type ............................ vit vision_pretraining .............................. False vision_pretraining_type ......................... classify vocab_extra_ids ................................. 0 vocab_file ...................................... None vocab_size ...................................... None wandb_entity .................................... wandb_exp_name .................................. wandb_project ................................... wandb_save_dir .................................. weight_decay .................................... 0.1 weight_decay_incr_style ......................... constant wgrad_deferral_limit ............................ 0 window_attn_skip_freq ........................... None window_size ..................................... None word_embedding_dropout_prob ..................... 0.0 world_size ...................................... 1 yaml_cfg ........................................ None -------------------- end of arguments --------------------- INFO:megatron.core.num_microbatches_calculator:setting number of microbatches to constant 1024 > building HuggingFaceTokenizer tokenizer ... You are using the default legacy behaviour of the . This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 - if you loaded a llama tokenizer from a GGUF file you can ignore this message. WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it INFO:megatron.training.initialize:Setting logging level to 1 WARNING:megatron.core.rerun_state_machine:RerunStateMachine initialized in mode validate_results > initializing torch distributed ... [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 > initialized tensor model parallel with size 1 > initialized pipeline model parallel with size 1 > setting random seeds to 1234 ... > compiling dataset index builder ... make: Entering directory '/mnt/ssd/lvzhihao/PostTrain/YuLan-Pretrain/megatron/core/datasets' make: Nothing to be done for 'default'. make: Leaving directory '/mnt/ssd/lvzhihao/PostTrain/YuLan-Pretrain/megatron/core/datasets' >>> done with dataset index builder. Compilation time: 0.114 seconds WARNING: constraints for invoking optimized fused softmax kernel are not met. We default back to unfused kernel invocations. > compiling and loading fused kernels ... /mnt/ssd/lvzhihao/PostTrain/YuLan-Pretrain/.venv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4807: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user. warnings.warn( # warn only once [rank0]:[W316 15:31:58.513326746 ProcessGroupNCCL.cpp:5023] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device. >>> done with compiling and loading fused kernels. Compilation time: 0.440 seconds WORLD_SIZE: 1, RANK: 0, LOCAL_RANK: 0 building GPT model ... `torch_dtype` is deprecated! Use `dtype` instead! `torch_dtype` is deprecated! Use `dtype` instead! INFO:transformers_modules.tmp_dot_1mZMUIgeRZ.modeling_qwen3_next:[Qwen3Next custom] attn_position_embedding_type=rope, rnn_position_embedding_type=nope, attn_logits_scaling=None Qwen3NextForCausalLM( (model): Qwen3NextModel( (embed_tokens): Embedding(99000, 1920) (layers): ModuleList( (0-11): 12 x Qwen3NextDecoderLayer( (linear_attn): Qwen3NextGatedDeltaNet( (act): SiLUActivation() (conv1d): Conv1d(3072, 3072, kernel_size=(4,), stride=(1,), padding=(3,), groups=3072, bias=False) (in_proj_qkvz): Linear(in_features=1920, out_features=5120, bias=False) (in_proj_ba): Linear(in_features=1920, out_features=64, bias=False) (norm): FusedRMSNormGated(64, eps=1e-06, activation=silu) (out_proj): Linear(in_features=2048, out_features=1920, bias=False) ) (mlp): Qwen3NextMLP( (gate_proj): Linear(in_features=1920, out_features=4800, bias=False) (up_proj): Linear(in_features=1920, out_features=4800, bias=False) (down_proj): Linear(in_features=4800, out_features=1920, bias=False) (act_fn): SiLUActivation() ) (input_layernorm): LlamaRMSNorm() (post_attention_layernorm): LlamaRMSNorm() ) (12): Qwen3NextDecoderLayer( (self_attn): Qwen3NextAttention( (q_proj): Linear(in_features=1920, out_features=1920, bias=True) (k_proj): Linear(in_features=1920, out_features=384, bias=True) (v_proj): Linear(in_features=1920, out_features=384, bias=True) (o_proj): Linear(in_features=1920, out_features=1920, bias=False) ) (mlp): Qwen3NextMLP( (gate_proj): Linear(in_features=1920, out_features=4800, bias=False) (up_proj): Linear(in_features=1920, out_features=4800, bias=False) (down_proj): Linear(in_features=4800, out_features=1920, bias=False) (act_fn): SiLUActivation() ) (input_layernorm): LlamaRMSNorm() (post_attention_layernorm): LlamaRMSNorm() ) (13-19): 7 x Qwen3NextDecoderLayer( (linear_attn): Qwen3NextGatedDeltaNet( (act): SiLUActivation() (conv1d): Conv1d(3072, 3072, kernel_size=(4,), stride=(1,), padding=(3,), groups=3072, bias=False) (in_proj_qkvz): Linear(in_features=1920, out_features=5120, bias=False) (in_proj_ba): Linear(in_features=1920, out_features=64, bias=False) (norm): FusedRMSNormGated(64, eps=1e-06, activation=silu) (out_proj): Linear(in_features=2048, out_features=1920, bias=False) ) (mlp): Qwen3NextMLP( (gate_proj): Linear(in_features=1920, out_features=4800, bias=False) (up_proj): Linear(in_features=1920, out_features=4800, bias=False) (down_proj): Linear(in_features=4800, out_features=1920, bias=False) (act_fn): SiLUActivation() ) (input_layernorm): LlamaRMSNorm() (post_attention_layernorm): LlamaRMSNorm() ) (20-21): 2 x Qwen3NextDecoderLayer( (self_attn): Qwen3NextAttention( (q_proj): Linear(in_features=1920, out_features=1920, bias=True) (k_proj): Linear(in_features=1920, out_features=384, bias=True) (v_proj): Linear(in_features=1920, out_features=384, bias=True) (o_proj): Linear(in_features=1920, out_features=1920, bias=False) ) (mlp): Qwen3NextMLP( (gate_proj): Linear(in_features=1920, out_features=4800, bias=False) (up_proj): Linear(in_features=1920, out_features=4800, bias=False) (down_proj): Linear(in_features=4800, out_features=1920, bias=False) (act_fn): SiLUActivation() ) (input_layernorm): LlamaRMSNorm() (post_attention_layernorm): LlamaRMSNorm() ) (22): Qwen3NextDecoderLayer( (linear_attn): Qwen3NextGatedDeltaNet( (act): SiLUActivation() (conv1d): Conv1d(3072, 3072, kernel_size=(4,), stride=(1,), padding=(3,), groups=3072, bias=False) (in_proj_qkvz): Linear(in_features=1920, out_features=5120, bias=False) (in_proj_ba): Linear(in_features=1920, out_features=64, bias=False) (norm): FusedRMSNormGated(64, eps=1e-06, activation=silu) (out_proj): Linear(in_features=2048, out_features=1920, bias=False) ) (mlp): Qwen3NextMLP( (gate_proj): Linear(in_features=1920, out_features=4800, bias=False) (up_proj): Linear(in_features=1920, out_features=4800, bias=False) (down_proj): Linear(in_features=4800, out_features=1920, bias=False) (act_fn): SiLUActivation() ) (input_layernorm): LlamaRMSNorm() (post_attention_layernorm): LlamaRMSNorm() ) (23): Qwen3NextDecoderLayer( (self_attn): Qwen3NextAttention( (q_proj): Linear(in_features=1920, out_features=1920, bias=True) (k_proj): Linear(in_features=1920, out_features=384, bias=True) (v_proj): Linear(in_features=1920, out_features=384, bias=True) (o_proj): Linear(in_features=1920, out_features=1920, bias=False) ) (mlp): Qwen3NextMLP( (gate_proj): Linear(in_features=1920, out_features=4800, bias=False) (up_proj): Linear(in_features=1920, out_features=4800, bias=False) (down_proj): Linear(in_features=4800, out_features=1920, bias=False) (act_fn): SiLUActivation() ) (input_layernorm): LlamaRMSNorm() (post_attention_layernorm): LlamaRMSNorm() ) (24-45): 22 x Qwen3NextDecoderLayer( (linear_attn): Qwen3NextGatedDeltaNet( (act): SiLUActivation() (conv1d): Conv1d(3072, 3072, kernel_size=(4,), stride=(1,), padding=(3,), groups=3072, bias=False) (in_proj_qkvz): Linear(in_features=1920, out_features=5120, bias=False) (in_proj_ba): Linear(in_features=1920, out_features=64, bias=False) (norm): FusedRMSNormGated(64, eps=1e-06, activation=silu) (out_proj): Linear(in_features=2048, out_features=1920, bias=False) ) (mlp): Qwen3NextMLP( (gate_proj): Linear(in_features=1920, out_features=4800, bias=False) (up_proj): Linear(in_features=1920, out_features=4800, bias=False) (down_proj): Linear(in_features=4800, out_features=1920, bias=False) (act_fn): SiLUActivation() ) (input_layernorm): LlamaRMSNorm() (post_attention_layernorm): LlamaRMSNorm() ) (46): Qwen3NextDecoderLayer( (self_attn): Qwen3NextAttention( (q_proj): Linear(in_features=1920, out_features=1920, bias=True) (k_proj): Linear(in_features=1920, out_features=384, bias=True) (v_proj): Linear(in_features=1920, out_features=384, bias=True) (o_proj): Linear(in_features=1920, out_features=1920, bias=False) ) (mlp): Qwen3NextMLP( (gate_proj): Linear(in_features=1920, out_features=4800, bias=False) (up_proj): Linear(in_features=1920, out_features=4800, bias=False) (down_proj): Linear(in_features=4800, out_features=1920, bias=False) (act_fn): SiLUActivation() ) (input_layernorm): LlamaRMSNorm() (post_attention_layernorm): LlamaRMSNorm() ) (47): Qwen3NextDecoderLayer( (linear_attn): Qwen3NextGatedDeltaNet( (act): SiLUActivation() (conv1d): Conv1d(3072, 3072, kernel_size=(4,), stride=(1,), padding=(3,), groups=3072, bias=False) (in_proj_qkvz): Linear(in_features=1920, out_features=5120, bias=False) (in_proj_ba): Linear(in_features=1920, out_features=64, bias=False) (norm): FusedRMSNormGated(64, eps=1e-06, activation=silu) (out_proj): Linear(in_features=2048, out_features=1920, bias=False) ) (mlp): Qwen3NextMLP( (gate_proj): Linear(in_features=1920, out_features=4800, bias=False) (up_proj): Linear(in_features=1920, out_features=4800, bias=False) (down_proj): Linear(in_features=4800, out_features=1920, bias=False) (act_fn): SiLUActivation() ) (input_layernorm): LlamaRMSNorm() (post_attention_layernorm): LlamaRMSNorm() ) (48-49): 2 x Qwen3NextDecoderLayer( (self_attn): Qwen3NextAttention( (q_proj): Linear(in_features=1920, out_features=1920, bias=True) (k_proj): Linear(in_features=1920, out_features=384, bias=True) (v_proj): Linear(in_features=1920, out_features=384, bias=True) (o_proj): Linear(in_features=1920, out_features=1920, bias=False) ) (mlp): Qwen3NextMLP( (gate_proj): Linear(in_features=1920, out_features=4800, bias=False) (up_proj): Linear(in_features=1920, out_features=4800, bias=False) (down_proj): Linear(in_features=4800, out_features=1920, bias=False) (act_fn): SiLUActivation() ) (input_layernorm): LlamaRMSNorm() (post_attention_layernorm): LlamaRMSNorm() ) (50-55): 6 x Qwen3NextDecoderLayer( (linear_attn): Qwen3NextGatedDeltaNet( (act): SiLUActivation() (conv1d): Conv1d(3072, 3072, kernel_size=(4,), stride=(1,), padding=(3,), groups=3072, bias=False) (in_proj_qkvz): Linear(in_features=1920, out_features=5120, bias=False) (in_proj_ba): Linear(in_features=1920, out_features=64, bias=False) (norm): FusedRMSNormGated(64, eps=1e-06, activation=silu) (out_proj): Linear(in_features=2048, out_features=1920, bias=False) ) (mlp): Qwen3NextMLP( (gate_proj): Linear(in_features=1920, out_features=4800, bias=False) (up_proj): Linear(in_features=1920, out_features=4800, bias=False) (down_proj): Linear(in_features=4800, out_features=1920, bias=False) (act_fn): SiLUActivation() ) (input_layernorm): LlamaRMSNorm() (post_attention_layernorm): LlamaRMSNorm() ) ) (norm): LlamaRMSNorm() (rotary_emb): Qwen3NextRotaryEmbedding() ) (lm_head): Linear(in_features=1920, out_features=99000, bias=False) ) GPTModel( (embedding): LanguageModelEmbedding( (word_embeddings): VocabParallelEmbedding() (embedding_dropout): Dropout(p=0.1, inplace=False) ) (rotary_pos_emb): RotaryEmbedding() (decoder): TransformerBlock( (layers): ModuleList( (0-11): 12 x TransformerLayer( (input_layernorm): IdentityOp() (self_attention): GatedDeltaNet( (in_proj): TELayerNormColumnParallelLinear(in_features=1920, out_features=5184, bias=False, TP=1) (conv1d): Conv1d(3072, 3072, kernel_size=(4,), stride=(1,), padding=(3,), groups=3072, bias=False) (out_norm): RMSNorm() (out_proj): TERowParallelLinear(in_features=2048, out_features=1920, bias=False, TP=1) ) (pre_cross_attn_layernorm): IdentityOp() (cross_attention): IdentityOp() (cross_attn_bda): IdentityFuncOp() (pre_mlp_layernorm): IdentityOp() (mlp): MLP( (linear_fc1): TELayerNormColumnParallelLinear(in_features=1920, out_features=9600, bias=False, TP=1) (linear_fc2): TERowParallelLinear(in_features=4800, out_features=1920, bias=False, TP=1) ) ) (12): TransformerLayer( (input_layernorm): IdentityOp() (self_attention): SelfAttention( (core_attention): TEDotProductAttention( (flash_attention): FlashAttention() (fused_attention): FusedAttention() (unfused_attention): UnfusedDotProductAttention( (scale_mask_softmax): FusedScaleMaskSoftmax() (attention_dropout): Dropout(p=0.1, inplace=False) ) ) (linear_proj): TERowParallelLinear(in_features=1920, out_features=1920, bias=False, TP=1) (linear_qkv): TELayerNormColumnParallelLinear(in_features=1920, out_features=2688, bias=True, TP=1) (q_layernorm): IdentityOp() (k_layernorm): IdentityOp() ) (pre_cross_attn_layernorm): IdentityOp() (cross_attention): IdentityOp() (cross_attn_bda): IdentityFuncOp() (pre_mlp_layernorm): IdentityOp() (mlp): MLP( (linear_fc1): TELayerNormColumnParallelLinear(in_features=1920, out_features=9600, bias=False, TP=1) (linear_fc2): TERowParallelLinear(in_features=4800, out_features=1920, bias=False, TP=1) ) ) (13-19): 7 x TransformerLayer( (input_layernorm): IdentityOp() (self_attention): GatedDeltaNet( (in_proj): TELayerNormColumnParallelLinear(in_features=1920, out_features=5184, bias=False, TP=1) (conv1d): Conv1d(3072, 3072, kernel_size=(4,), stride=(1,), padding=(3,), groups=3072, bias=False) (out_norm): RMSNorm() (out_proj): TERowParallelLinear(in_features=2048, out_features=1920, bias=False, TP=1) ) (pre_cross_attn_layernorm): IdentityOp() (cross_attention): IdentityOp() (cross_attn_bda): IdentityFuncOp() (pre_mlp_layernorm): IdentityOp() (mlp): MLP( (linear_fc1): TELayerNormColumnParallelLinear(in_features=1920, out_features=9600, bias=False, TP=1) (linear_fc2): TERowParallelLinear(in_features=4800, out_features=1920, bias=False, TP=1) ) ) (20-21): 2 x TransformerLayer( (input_layernorm): IdentityOp() (self_attention): SelfAttention( (core_attention): TEDotProductAttention( (flash_attention): FlashAttention() (fused_attention): FusedAttention() (unfused_attention): UnfusedDotProductAttention( (scale_mask_softmax): FusedScaleMaskSoftmax() (attention_dropout): Dropout(p=0.1, inplace=False) ) ) (linear_proj): TERowParallelLinear(in_features=1920, out_features=1920, bias=False, TP=1) (linear_qkv): TELayerNormColumnParallelLinear(in_features=1920, out_features=2688, bias=True, TP=1) (q_layernorm): IdentityOp() (k_layernorm): IdentityOp() ) (pre_cross_attn_layernorm): IdentityOp() (cross_attention): IdentityOp() (cross_attn_bda): IdentityFuncOp() (pre_mlp_layernorm): IdentityOp() (mlp): MLP( (linear_fc1): TELayerNormColumnParallelLinear(in_features=1920, out_features=9600, bias=False, TP=1) (linear_fc2): TERowParallelLinear(in_features=4800, out_features=1920, bias=False, TP=1) ) ) (22): TransformerLayer( (input_layernorm): IdentityOp() (self_attention): GatedDeltaNet( (in_proj): TELayerNormColumnParallelLinear(in_features=1920, out_features=5184, bias=False, TP=1) (conv1d): Conv1d(3072, 3072, kernel_size=(4,), stride=(1,), padding=(3,), groups=3072, bias=False) (out_norm): RMSNorm() (out_proj): TERowParallelLinear(in_features=2048, out_features=1920, bias=False, TP=1) ) (pre_cross_attn_layernorm): IdentityOp() (cross_attention): IdentityOp() (cross_attn_bda): IdentityFuncOp() (pre_mlp_layernorm): IdentityOp() (mlp): MLP( (linear_fc1): TELayerNormColumnParallelLinear(in_features=1920, out_features=9600, bias=False, TP=1) (linear_fc2): TERowParallelLinear(in_features=4800, out_features=1920, bias=False, TP=1) ) ) (23): TransformerLayer( (input_layernorm): IdentityOp() (self_attention): SelfAttention( (core_attention): TEDotProductAttention( (flash_attention): FlashAttention() (fused_attention): FusedAttention() (unfused_attention): UnfusedDotProductAttention( (scale_mask_softmax): FusedScaleMaskSoftmax() (attention_dropout): Dropout(p=0.1, inplace=False) ) ) (linear_proj): TERowParallelLinear(in_features=1920, out_features=1920, bias=False, TP=1) (linear_qkv): TELayerNormColumnParallelLinear(in_features=1920, out_features=2688, bias=True, TP=1) (q_layernorm): IdentityOp() (k_layernorm): IdentityOp() ) (pre_cross_attn_layernorm): IdentityOp() (cross_attention): IdentityOp() (cross_attn_bda): IdentityFuncOp() (pre_mlp_layernorm): IdentityOp() (mlp): MLP( (linear_fc1): TELayerNormColumnParallelLinear(in_features=1920, out_features=9600, bias=False, TP=1) (linear_fc2): TERowParallelLinear(in_features=4800, out_features=1920, bias=False, TP=1) ) ) (24-45): 22 x TransformerLayer( (input_layernorm): IdentityOp() (self_attention): GatedDeltaNet( (in_proj): TELayerNormColumnParallelLinear(in_features=1920, out_features=5184, bias=False, TP=1) (conv1d): Conv1d(3072, 3072, kernel_size=(4,), stride=(1,), padding=(3,), groups=3072, bias=False) (out_norm): RMSNorm() (out_proj): TERowParallelLinear(in_features=2048, out_features=1920, bias=False, TP=1) ) (pre_cross_attn_layernorm): IdentityOp() (cross_attention): IdentityOp() (cross_attn_bda): IdentityFuncOp() (pre_mlp_layernorm): IdentityOp() (mlp): MLP( (linear_fc1): TELayerNormColumnParallelLinear(in_features=1920, out_features=9600, bias=False, TP=1) (linear_fc2): TERowParallelLinear(in_features=4800, out_features=1920, bias=False, TP=1) ) ) (46): TransformerLayer( (input_layernorm): IdentityOp() (self_attention): SelfAttention( (core_attention): TEDotProductAttention( (flash_attention): FlashAttention() (fused_attention): FusedAttention() (unfused_attention): UnfusedDotProductAttention( (scale_mask_softmax): FusedScaleMaskSoftmax() (attention_dropout): Dropout(p=0.1, inplace=False) ) ) (linear_proj): TERowParallelLinear(in_features=1920, out_features=1920, bias=False, TP=1) (linear_qkv): TELayerNormColumnParallelLinear(in_features=1920, out_features=2688, bias=True, TP=1) (q_layernorm): IdentityOp() (k_layernorm): IdentityOp() ) (pre_cross_attn_layernorm): IdentityOp() (cross_attention): IdentityOp() (cross_attn_bda): IdentityFuncOp() (pre_mlp_layernorm): IdentityOp() (mlp): MLP( (linear_fc1): TELayerNormColumnParallelLinear(in_features=1920, out_features=9600, bias=False, TP=1) (linear_fc2): TERowParallelLinear(in_features=4800, out_features=1920, bias=False, TP=1) ) ) (47): TransformerLayer( (input_layernorm): IdentityOp() (self_attention): GatedDeltaNet( (in_proj): TELayerNormColumnParallelLinear(in_features=1920, out_features=5184, bias=False, TP=1) (conv1d): Conv1d(3072, 3072, kernel_size=(4,), stride=(1,), padding=(3,), groups=3072, bias=False) (out_norm): RMSNorm() (out_proj): TERowParallelLinear(in_features=2048, out_features=1920, bias=False, TP=1) ) (pre_cross_attn_layernorm): IdentityOp() (cross_attention): IdentityOp() (cross_attn_bda): IdentityFuncOp() (pre_mlp_layernorm): IdentityOp() (mlp): MLP( (linear_fc1): TELayerNormColumnParallelLinear(in_features=1920, out_features=9600, bias=False, TP=1) (linear_fc2): TERowParallelLinear(in_features=4800, out_features=1920, bias=False, TP=1) ) ) (48-49): 2 x TransformerLayer( (input_layernorm): IdentityOp() (self_attention): SelfAttention( (core_attention): TEDotProductAttention( (flash_attention): FlashAttention() (fused_attention): FusedAttention() (unfused_attention): UnfusedDotProductAttention( (scale_mask_softmax): FusedScaleMaskSoftmax() (attention_dropout): Dropout(p=0.1, inplace=False) ) ) (linear_proj): TERowParallelLinear(in_features=1920, out_features=1920, bias=False, TP=1) (linear_qkv): TELayerNormColumnParallelLinear(in_features=1920, out_features=2688, bias=True, TP=1) (q_layernorm): IdentityOp() (k_layernorm): IdentityOp() ) (pre_cross_attn_layernorm): IdentityOp() (cross_attention): IdentityOp() (cross_attn_bda): IdentityFuncOp() (pre_mlp_layernorm): IdentityOp() (mlp): MLP( (linear_fc1): TELayerNormColumnParallelLinear(in_features=1920, out_features=9600, bias=False, TP=1) (linear_fc2): TERowParallelLinear(in_features=4800, out_features=1920, bias=False, TP=1) ) ) (50-55): 6 x TransformerLayer( (input_layernorm): IdentityOp() (self_attention): GatedDeltaNet( (in_proj): TELayerNormColumnParallelLinear(in_features=1920, out_features=5184, bias=False, TP=1) (conv1d): Conv1d(3072, 3072, kernel_size=(4,), stride=(1,), padding=(3,), groups=3072, bias=False) (out_norm): RMSNorm() (out_proj): TERowParallelLinear(in_features=2048, out_features=1920, bias=False, TP=1) ) (pre_cross_attn_layernorm): IdentityOp() (cross_attention): IdentityOp() (cross_attn_bda): IdentityFuncOp() (pre_mlp_layernorm): IdentityOp() (mlp): MLP( (linear_fc1): TELayerNormColumnParallelLinear(in_features=1920, out_features=9600, bias=False, TP=1) (linear_fc2): TERowParallelLinear(in_features=4800, out_features=1920, bias=False, TP=1) ) ) ) (final_layernorm): RMSNorm() ) (output_layer): ColumnParallelLinear(in_features=1920, out_features=99000, bias=False, TP=1) ) /mnt/ssd/lvzhihao/PostTrain/YuLan-Pretrain/megatron/core/dist_checkpointing/strategies/common.py:89: UserWarning: Environment variable TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD detected, since the`weights_only` argument was not explicitly passed to `torch.load`, forcing weights_only=False. return torch.load(load_path, map_location='cpu') (TP, PP) mismatch after resume ((1, 1) vs (2, 1) from checkpoint): RNG state will be ignored sharded_state_dict metadata loaded from the checkpoint: {'singleton_local_shards': True, 'distrib_optim_sharding_type': 'fully_reshardable', 'distrib_optim_fully_reshardable_mem_efficient': False, 'chained_optim_avoid_prefix': True} Job sharding has changed: Rerun state will be ignored loading distributed checkpoint from /mnt/hdd/lvzhihao/mcore_models/Dist-mathcode10b-s1randg-sch1-CPT-200b-stage3-r640k-GDN2.9b-A7-12_20_21_23_46_48_49-sl32768bs128lr2e5-2e5 at iteration 71525 /mnt/ssd/lvzhihao/PostTrain/YuLan-Pretrain/megatron/core/dist_checkpointing/strategies/torch.py:956: FutureWarning: `load_state_dict` is deprecated and will be removed in future versions. Please use `load` instead. checkpoint.load_state_dict( checkpoint version 3.0 successfully loaded checkpoint from /mnt/hdd/lvzhihao/mcore_models/Dist-mathcode10b-s1randg-sch1-CPT-200b-stage3-r640k-GDN2.9b-A7-12_20_21_23_46_48_49-sl32768bs128lr2e5-2e5 [ t 1/1, p 1/1 ] at iteration 71525 INFO:root:Converting layer 0 is_gdn=True is_not_moe=True INFO:root:Converting layer 1 is_gdn=True is_not_moe=True INFO:root:Converting layer 2 is_gdn=True is_not_moe=True INFO:root:Converting layer 3 is_gdn=True is_not_moe=True INFO:root:Converting layer 4 is_gdn=True is_not_moe=True INFO:root:Converting layer 5 is_gdn=True is_not_moe=True INFO:root:Converting layer 6 is_gdn=True is_not_moe=True INFO:root:Converting layer 7 is_gdn=True is_not_moe=True INFO:root:Converting layer 8 is_gdn=True is_not_moe=True INFO:root:Converting layer 9 is_gdn=True is_not_moe=True INFO:root:Converting layer 10 is_gdn=True is_not_moe=True INFO:root:Converting layer 11 is_gdn=True is_not_moe=True INFO:root:Converting layer 12 is_gdn=False is_not_moe=True INFO:root:[DEBUG] Layer 12: args.attention_output_gate=False INFO:root:[DEBUG] set_gated_selfattn_state: args.attention_output_gate=False INFO:root:[DEBUG] set_gated_selfattn_state: attention_output_gate=False, linear_layer=TELayerNormColumnParallelLinear INFO:root:Converting layer 13 is_gdn=True is_not_moe=True INFO:root:Converting layer 14 is_gdn=True is_not_moe=True INFO:root:Converting layer 15 is_gdn=True is_not_moe=True INFO:root:Converting layer 16 is_gdn=True is_not_moe=True INFO:root:Converting layer 17 is_gdn=True is_not_moe=True INFO:root:Converting layer 18 is_gdn=True is_not_moe=True INFO:root:Converting layer 19 is_gdn=True is_not_moe=True INFO:root:Converting layer 20 is_gdn=False is_not_moe=True INFO:root:[DEBUG] Layer 20: args.attention_output_gate=False INFO:root:[DEBUG] set_gated_selfattn_state: args.attention_output_gate=False INFO:root:[DEBUG] set_gated_selfattn_state: attention_output_gate=False, linear_layer=TELayerNormColumnParallelLinear INFO:root:Converting layer 21 is_gdn=False is_not_moe=True INFO:root:[DEBUG] Layer 21: args.attention_output_gate=False INFO:root:[DEBUG] set_gated_selfattn_state: args.attention_output_gate=False INFO:root:[DEBUG] set_gated_selfattn_state: attention_output_gate=False, linear_layer=TELayerNormColumnParallelLinear INFO:root:Converting layer 22 is_gdn=True is_not_moe=True INFO:root:Converting layer 23 is_gdn=False is_not_moe=True INFO:root:[DEBUG] Layer 23: args.attention_output_gate=False INFO:root:[DEBUG] set_gated_selfattn_state: args.attention_output_gate=False INFO:root:[DEBUG] set_gated_selfattn_state: attention_output_gate=False, linear_layer=TELayerNormColumnParallelLinear INFO:root:Converting layer 24 is_gdn=True is_not_moe=True INFO:root:Converting layer 25 is_gdn=True is_not_moe=True INFO:root:Converting layer 26 is_gdn=True is_not_moe=True INFO:root:Converting layer 27 is_gdn=True is_not_moe=True INFO:root:Converting layer 28 is_gdn=True is_not_moe=True INFO:root:Converting layer 29 is_gdn=True is_not_moe=True INFO:root:Converting layer 30 is_gdn=True is_not_moe=True INFO:root:Converting layer 31 is_gdn=True is_not_moe=True INFO:root:Converting layer 32 is_gdn=True is_not_moe=True INFO:root:Converting layer 33 is_gdn=True is_not_moe=True INFO:root:Converting layer 34 is_gdn=True is_not_moe=True INFO:root:Converting layer 35 is_gdn=True is_not_moe=True INFO:root:Converting layer 36 is_gdn=True is_not_moe=True INFO:root:Converting layer 37 is_gdn=True is_not_moe=True INFO:root:Converting layer 38 is_gdn=True is_not_moe=True INFO:root:Converting layer 39 is_gdn=True is_not_moe=True INFO:root:Converting layer 40 is_gdn=True is_not_moe=True INFO:root:Converting layer 41 is_gdn=True is_not_moe=True INFO:root:Converting layer 42 is_gdn=True is_not_moe=True INFO:root:Converting layer 43 is_gdn=True is_not_moe=True INFO:root:Converting layer 44 is_gdn=True is_not_moe=True INFO:root:Converting layer 45 is_gdn=True is_not_moe=True INFO:root:Converting layer 46 is_gdn=False is_not_moe=True INFO:root:[DEBUG] Layer 46: args.attention_output_gate=False INFO:root:[DEBUG] set_gated_selfattn_state: args.attention_output_gate=False INFO:root:[DEBUG] set_gated_selfattn_state: attention_output_gate=False, linear_layer=TELayerNormColumnParallelLinear INFO:root:Converting layer 47 is_gdn=True is_not_moe=True INFO:root:Converting layer 48 is_gdn=False is_not_moe=True INFO:root:[DEBUG] Layer 48: args.attention_output_gate=False INFO:root:[DEBUG] set_gated_selfattn_state: args.attention_output_gate=False INFO:root:[DEBUG] set_gated_selfattn_state: attention_output_gate=False, linear_layer=TELayerNormColumnParallelLinear INFO:root:Converting layer 49 is_gdn=False is_not_moe=True INFO:root:[DEBUG] Layer 49: args.attention_output_gate=False INFO:root:[DEBUG] set_gated_selfattn_state: args.attention_output_gate=False INFO:root:[DEBUG] set_gated_selfattn_state: attention_output_gate=False, linear_layer=TELayerNormColumnParallelLinear INFO:root:Converting layer 50 is_gdn=True is_not_moe=True INFO:root:Converting layer 51 is_gdn=True is_not_moe=True INFO:root:Converting layer 52 is_gdn=True is_not_moe=True INFO:root:Converting layer 53 is_gdn=True is_not_moe=True INFO:root:Converting layer 54 is_gdn=True is_not_moe=True INFO:root:Converting layer 55 is_gdn=True is_not_moe=True DEBUG:root:[RANK 0] 0 send op & 0 recv op. INFO:root:[Iters 0 RANK 0] starts synchronizing parameters with other ranks... INFO:root:[Iters 0 RANK 0] finishes synchronizing [Iters 0 RANK 0] model.safetensors is saved. DEBUG:root:[Iters 0 RANK 0] joined Conversion finished in 70.92923641204834 seconds. [rank0]:[W316 15:33:10.354704836 ProcessGroupNCCL.cpp:1538] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())