| torchrun --nproc_per_node 1 --nnodes 1 --node_rank 0 --master_addr localhost --master_port 35632 /mnt/ssd/lvzhihao/PostTrain/YuLan-Pretrain/scripts/distributed_checkpoints_convertor/impl/convert.py --tokenizer-type HuggingFaceTokenizer --tokenizer-model /mnt/ssd/cache_tmp/tmp/tmp.1mZMUIgeRZ --hf-dir /mnt/ssd/cache_tmp/tmp/tmp.1mZMUIgeRZ --mcore2hf --use-gpu --bf16 --normalization RMSNorm --swiglu --disable-bias-linear --seq-length 1 --max-position-embeddings 490000 --attention-backend auto --position-embedding-type rope --kv-channels 64 --group-query-attention --add-qkv-bias --num-layers 56 --hidden-size 1920 --ffn-hidden-size 4800 --num-attention-heads 30 --untie-embeddings-and-output-weights --rotary-base 490000 --rotary-percent 1.00 --num-query-groups 6 --normalization RMSNorm --norm-epsilon 1e-6 --linear-attention-type gated_delta_net --linear-attention-freq [1,1,1,1,1,1,1,1,1,1,1,1,0,1,1,1,1,1,1,1,0,0,1,0,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,1,0,0,1,1,1,1,1,1] --linear-conv-kernel-dim 4 --linear-key-head-dim 64 --linear-value-head-dim 64 --linear-num-key-heads 8 --linear-num-value-heads 32 --micro-batch-size 1 --global-batch-size 1024 --train-iters 500000 --weight-decay 0.1 --adam-beta1 0.9 --adam-beta2 0.95 --init-method-std 0.006 --clip-grad 1.0 --lr 2.0e-5 --lr-decay-style cosine --min-lr 6.0e-6 --lr-warmup-fraction .001 --lr-decay-iters 430000 --bf16 --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --expert-tensor-parallel-size 1 --expert-model-parallel-size 1 --log-interval 100 --save-interval 10000 --eval-interval 1000 --eval-iters 10 --model-type GPT --load-dir /mnt/hdd/lvzhihao/mcore_models/Dist-mathcode10b-s1randg-sch1-CPT-200b-stage3-r640k-GDN2.9b-A7-12_20_21_23_46_48_49-sl32768bs128lr2e5-2e5 --save-dir /mnt/hdd/lvzhihao/mcore_models/Dist-mathcode10b-s1randg-sch1-CPT-200b-stage3-r640k-GDN2.9b-A7-12_20_21_23_46_48_49-sl32768bs128lr2e5-2e5/iter_71525-hf --dist-ckpt-optim-fully-reshardable --skip-train --use-cpu-initialization --padded-vocab-size 99000 --no-load-optim --no-load-rng --logging-level 1 --attention-backend auto --synchronizer mcore_gdn_moe --pretrain-script mcore_gdn_moe.model_provider --debug --max-shard-size 20GB --ckpt-step 71525 |
| W0316 15:31:46.709000 11298 .venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. |
| W0316 15:31:46.709000 11298 .venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures. |
| fused_indices_to_multihot has reached end of life. Please migrate to a non-experimental function. |
| Current Python version 3.10 is below the recommended 3.11 version. It is recommended to upgrade to Python 3.11 or higher for the best experience. |
| Warning: Pai-Megatron-Patch arguments not available, some arguments may not be recognized |
| using world size: 1, data-parallel size: 1, context-parallel size: 1, hierarchical context-parallel sizes: None, tensor-model-parallel size: 1, pipeline-model-parallel size: 1 |
| Number of virtual stages per pipeline stage: None |
| accumulate and all-reduce gradients in fp32 for bfloat16 data type. |
| using torch.bfloat16 for parameters ... |
| ------------------------ arguments ------------------------ |
| account_for_embedding_in_pipeline_split ......... False |
| account_for_loss_in_pipeline_split .............. False |
| accumulate_allreduce_grads_in_fp32 .............. True |
| activation_func_clamp_value ..................... None |
| adam_beta1 ...................................... 0.9 |
| adam_beta2 ...................................... 0.95 |
| adam_eps ........................................ 1e-08 |
| adamw_lr_mup_scaler ............................. False |
| add_bias_linear ................................. False |
| add_position_embedding .......................... True |
| add_qkv_bias .................................... True |
| adlr_autoresume ................................. False |
| adlr_autoresume_interval ........................ 1000 |
| align_grad_reduce ............................... True |
| align_param_gather .............................. False |
| allow_ambiguous_pad_tokens ...................... False |
| app_tag_run_name ................................ None |
| app_tag_run_version ............................. 0.0.0 |
| apply_layernorm_1p .............................. False |
| apply_query_key_layer_scaling ................... False |
| apply_residual_connection_post_layernorm ........ False |
| apply_rope_fusion ............................... True |
| async_save ...................................... None |
| async_tensor_model_parallel_allreduce ........... True |
| attention_backend ............................... AttnBackend.auto |
| attention_dropout ............................... 0.1 |
| attention_output_gate ........................... False |
| attention_softmax_in_fp32 ....................... False |
| attn_k_token_shift .............................. None |
| attn_output_gate ................................ None |
| attn_output_gate_rand_init ...................... False |
| attn_q_token_shift .............................. None |
| attn_token_shift ................................ None |
| attn_v_token_shift .............................. None |
| auto_detect_ckpt_format ......................... False |
| auto_generate_cu_seqlens ........................ False |
| auto_model ...................................... AutoModelForCausalLM |
| barrier_with_L1_time ............................ True |
| benchmark_eval .................................. False |
| benchmark_global_batch .......................... None |
| benchmark_interval .............................. None |
| benchmark_micro_batch ........................... None |
| benchmark_sequence_length ....................... None |
| benchmark_tasks ................................. None |
| bert_binary_head ................................ True |
| bert_embedder_type .............................. megatron |
| bert_load ....................................... None |
| bf16 ............................................ True |
| bias_dropout_fusion ............................. True |
| bias_gelu_fusion ................................ False |
| bias_swiglu_fusion .............................. True |
| biencoder_projection_dim ........................ 0 |
| biencoder_shared_query_context_model ............ False |
| block_data_path ................................. None |
| cache_mla_latents ............................... False |
| calc_ft_timeouts ................................ False |
| calculate_per_token_loss ........................ False |
| check_for_large_grads ........................... False |
| check_for_nan_in_loss_and_grad .................. True |
| check_for_spiky_loss ............................ False |
| check_weight_hash_across_dp_replicas_interval ... None |
| ckpt_assume_constant_structure .................. False |
| ckpt_convert_format ............................. None |
| ckpt_convert_save ............................... None |
| ckpt_convert_update_legacy_dist_opt_format ...... False |
| ckpt_format ..................................... torch_dist |
| ckpt_fully_parallel_load ........................ False |
| ckpt_fully_parallel_save ........................ True |
| ckpt_fully_parallel_save_deprecated ............. False |
| ckpt_step ....................................... 71525 |
| classes_fraction ................................ 1.0 |
| clip_grad ....................................... 1.0 |
| clone_scatter_output_in_embedding ............... True |
| config_logger_dir ............................... |
| consumed_train_samples .......................... 0 |
| consumed_valid_samples .......................... 0 |
| context_parallel_size ........................... 1 |
| cp_comm_type .................................... ['p2p'] |
| create_attention_mask_in_dataloader ............. True |
| cross_entropy_fusion_impl ....................... native |
| cross_entropy_loss_fusion ....................... False |
| cuda_graph_impl ................................. none |
| cuda_graph_scope ................................ [] |
| cuda_graph_warmup_steps ......................... 3 |
| data_args_path .................................. None |
| data_cache_path ................................. None |
| data_parallel_random_init ....................... False |
| data_parallel_sharding_strategy ................. no_shard |
| data_parallel_size .............................. 1 |
| data_path ....................................... None |
| data_per_class_fraction ......................... 1.0 |
| data_sharding ................................... True |
| dataloader_type ................................. single |
| ddp_average_in_collective ....................... False |
| ddp_bucket_size ................................. None |
| ddp_num_buckets ................................. None |
| ddp_pad_buckets_for_high_nccl_busbw ............. False |
| debug ........................................... True |
| decode_only_cuda_graphs ......................... False |
| decoder_first_pipeline_num_layers ............... None |
| decoder_last_pipeline_num_layers ................ None |
| decoder_num_layers .............................. None |
| decoder_seq_length .............................. None |
| decoupled_lr .................................... None |
| decoupled_min_lr ................................ None |
| decrease_batch_size_if_needed ................... False |
| defer_embedding_wgrad_compute ................... False |
| delay_wgrad_compute ............................. False |
| deprecated_use_mcore_models ..................... False |
| deterministic_mode .............................. False |
| dino_bottleneck_size ............................ 256 |
| dino_freeze_last_layer .......................... 1 |
| dino_head_hidden_size ........................... 2048 |
| dino_local_crops_number ......................... 10 |
| dino_local_img_size ............................. 96 |
| dino_norm_last_layer ............................ False |
| dino_teacher_temp ............................... 0.07 |
| dino_warmup_teacher_temp ........................ 0.04 |
| dino_warmup_teacher_temp_epochs ................. 30 |
| disable_attn_output_gate ........................ False |
| disable_bf16_reduced_precision_matmul ........... False |
| disable_chunked_prefill ......................... False |
| disable_explicit_attention_mask ................. False |
| disable_mamba_mem_eff_path ...................... False |
| disable_straggler_on_startup .................... False |
| disable_symmetric_registration .................. False |
| dist_ckpt_format_deprecated ..................... None |
| dist_ckpt_optim_fully_reshardable ............... True |
| dist_ckpt_save_pre_mcore_014 .................... False |
| dist_ckpt_strictness ............................ assume_ok_unexpected |
| distrib_optim_fully_reshardable_mem_efficient ... False |
| distribute_saved_activations .................... False |
| distributed_backend ............................. nccl |
| distributed_timeout_minutes ..................... 10 |
| distributed_timeout_seconds_after_init .......... None |
| document_packing_algorithm ...................... random |
| dryrun .......................................... False |
| dump_param_to_param_group_map ................... None |
| emb_deviation_loss_coeff ........................ 0 |
| emb_deviation_type .............................. None |
| embedding_init_method_std ....................... None |
| embedding_path .................................. None |
| empty_unused_memory_level ....................... 0 |
| enable_cuda_graph ............................... False |
| enable_debug_logging ............................ False |
| enable_experimental ............................. False |
| enable_ft_package ............................... False |
| enable_full_sharding_in_hsdp .................... False |
| enable_gloo_process_groups ...................... True |
| enable_msc ...................................... True |
| enable_one_logger ............................... True |
| encoder_num_layers .............................. 56 |
| encoder_seq_length .............................. 1 |
| end_weight_decay ................................ 0.1 |
| eod_mask_loss ................................... False |
| error_injection_rate ............................ 0 |
| error_injection_type ............................ transient_error |
| eval_interval ................................... 1000 |
| eval_iters ...................................... 10 |
| evidence_data_path .............................. None |
| exit_duration_in_mins ........................... None |
| exit_interval ................................... None |
| exit_on_missing_checkpoint ...................... False |
| exit_signal_handler ............................. False |
| exp_avg_dtype ................................... torch.float32 |
| exp_avg_sq_dtype ................................ torch.float32 |
| expert_model_parallel_size ...................... 1 |
| expert_tensor_parallel_size ..................... 1 |
| external_cuda_graph ............................. False |
| ffn_hidden_size ................................. 4800 |
| ffn_intermediate_token_shift .................... None |
| ffn_token_shift ................................. None |
| fine_grained_activation_offloading .............. False |
| finetune ........................................ False |
| first_last_layers_bf16 .......................... False |
| flash_decode .................................... False |
| fp16 ............................................ False |
| fp16_lm_cross_entropy ........................... False |
| fp32_residual_connection ........................ False |
| fp4 ............................................. None |
| fp4_param ....................................... False |
| fp4_recipe ...................................... nvfp4 |
| fp8 ............................................. None |
| fp8_amax_compute_algo ........................... most_recent |
| fp8_amax_history_len ............................ 1 |
| fp8_interval .................................... 1 |
| fp8_margin ...................................... 0 |
| fp8_param_gather ................................ False |
| fp8_recipe ...................................... delayed |
| fp8_wgrad ....................................... True |
| freeze_layernorm_weight ......................... False |
| freeze_non_mamba ................................ False |
| fsdp_double_buffer .............................. False |
| full_validation ................................. False |
| geglu ........................................... False |
| global_batch_size ............................... 1024 |
| glu_linear_offset ............................... 0.0 |
| grad_reduce_in_bf16 ............................. False |
| gradient_accumulation_fusion .................... True |
| gradient_reduce_div_fusion ...................... True |
| group_query_attention ........................... True |
| grpo_clamp_eps_lower ............................ 0.01 |
| grpo_clamp_eps_upper ............................ 0.01 |
| grpo_default_temperature ........................ 1.0 |
| grpo_default_top_p .............................. 0 |
| grpo_entropy_term_weight ........................ 0.0 |
| grpo_filter_groups_with_same_reward ............. False |
| grpo_group_size ................................. 2 |
| grpo_iterations ................................. 2 |
| grpo_kl_beta .................................... 0.001 |
| grpo_prompts_per_step ........................... 32 |
| head_lr_mult .................................... 1.0 |
| heterogeneous_layers_config_encoded_json ........ None |
| heterogeneous_layers_config_path ................ None |
| hf_dir .......................................... /mnt/ssd/cache_tmp/tmp/tmp.1mZMUIgeRZ |
| hidden_dropout .................................. 0.1 |
| hidden_size ..................................... 1920 |
| hierarchical_context_parallel_sizes ............. None |
| high_priority_stream_groups ..................... [] |
| hybrid_attention_ratio .......................... 0.0 |
| hybrid_mlp_ratio ................................ 0.0 |
| hybrid_override_pattern ......................... None |
| hysteresis ...................................... 2 |
| ict_head_size ................................... None |
| ict_load ........................................ None |
| img_h ........................................... 224 |
| img_w ........................................... 224 |
| increase_log_level_interval ..................... 1000 |
| increase_log_level_iters ........................ 5 |
| indexer_batch_size .............................. 128 |
| indexer_log_interval ............................ 1000 |
| inference_batch_times_seqlen_threshold .......... -1 |
| inference_dynamic_batching ...................... False |
| inference_dynamic_batching_block_size ........... 256 |
| inference_dynamic_batching_buffer_guaranteed_fraction 0.2 |
| inference_dynamic_batching_buffer_overflow_factor None |
| inference_dynamic_batching_buffer_size_gb ....... 40.0 |
| inference_dynamic_batching_max_requests_override None |
| inference_dynamic_batching_max_tokens_override .. None |
| inference_dynamic_batching_num_cuda_graphs ...... 16 |
| inference_dynamic_batching_track_paused_request_events False |
| inference_dynamic_batching_unified_memory_level . 0 |
| inference_max_batch_size ........................ 8 |
| inference_max_seq_length ........................ 2560 |
| inference_rng_tracker ........................... False |
| init_method_std ................................. 0.006 |
| init_method_xavier_uniform ...................... False |
| init_model_with_meta_device ..................... False |
| initial_loss_scale .............................. 4294967296 |
| inprocess_active_world_size ..................... 1 |
| inprocess_barrier_timeout ....................... 120 |
| inprocess_completion_timeout .................... 120 |
| inprocess_empty_cuda_cache ...................... False |
| inprocess_granularity ........................... node |
| inprocess_hard_timeout .......................... 90 |
| inprocess_heartbeat_interval .................... 30 |
| inprocess_heartbeat_timeout ..................... 60 |
| inprocess_last_call_wait ........................ 1 |
| inprocess_max_iterations ........................ None |
| inprocess_monitor_process_interval .............. 1.0 |
| inprocess_monitor_thread_interval ............... 1.0 |
| inprocess_progress_watchdog_interval ............ 1.0 |
| inprocess_restart ............................... False |
| inprocess_soft_timeout .......................... 60 |
| inprocess_termination_grace_time ................ 1 |
| is_hybrid_model ................................. False |
| iter_per_epoch .................................. 1250 |
| iterations_to_skip .............................. [] |
| keep_fp8_transpose_cache ........................ False |
| kitchen_config_file ............................. None |
| kitchen_recipe_number ........................... None |
| kv_channels ..................................... 64 |
| kv_lora_rank .................................... 32 |
| langrl_env_config ............................... None |
| langrl_external_server .......................... False |
| langrl_inference_server_conversation_template ... None |
| langrl_inference_server_type .................... inplace_megatron |
| lazy_mpu_init ................................... None |
| legacy_tokenizer ................................ False |
| linear_attention_freq ........................... [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1] |
| linear_attention_type ........................... gated_delta_net |
| linear_conv_kernel_dim .......................... 4 |
| linear_key_head_dim ............................. 64 |
| linear_num_key_heads ............................ 8 |
| linear_num_value_heads .......................... 32 |
| linear_value_head_dim ........................... 64 |
| load ............................................ None |
| load_complemental_dataset ....................... None |
| load_dir ........................................ /mnt/hdd/lvzhihao/mcore_models/Dist-mathcode10b-s1randg-sch1-CPT-200b-stage3-r640k-GDN2.9b-A7-12_20_21_23_46_48_49-sl32768bs128lr2e5-2e5 |
| load_main_params_from_ckpt ...................... None |
| local_rank ...................................... 0 |
| log_energy ...................................... False |
| log_hidden_states ............................... [] |
| log_interval .................................... 100 |
| log_loss_scale_to_tensorboard ................... True |
| log_memory_to_tensorboard ....................... False |
| log_num_zeros_in_grad ........................... False |
| log_params ...................................... [] |
| log_params_norm ................................. False |
| log_per_module_grad_rms ......................... False |
| log_per_module_update_rms ....................... False |
| log_progress .................................... False |
| log_straggler ................................... False |
| log_throughput .................................. False |
| log_timers_to_tensorboard ....................... False |
| log_validation_ppl_to_tensorboard ............... False |
| log_world_size_to_tensorboard ................... False |
| logging_level ................................... 1 |
| loss_scale ...................................... None |
| loss_scale_window ............................... 1000 |
| lr .............................................. 2e-05 |
| lr_decay_iters .................................. 430000 |
| lr_decay_samples ................................ None |
| lr_decay_style .................................. cosine |
| lr_warmup_fraction .............................. 0.001 |
| lr_warmup_init .................................. 0.0 |
| lr_warmup_iters ................................. 0 |
| lr_warmup_samples ............................... 0 |
| lr_wsd_decay_iters .............................. None |
| lr_wsd_decay_samples ............................ None |
| lr_wsd_decay_style .............................. exponential |
| main_grads_dtype ................................ torch.float32 |
| main_params_dtype ............................... torch.float32 |
| make_vocab_size_divisible_by .................... 128 |
| mamba_disable_cp ................................ False |
| mamba_expand .................................... 2 |
| mamba_head_dim .................................. 64 |
| mamba_num_groups ................................ 8 |
| mamba_num_heads ................................. None |
| mamba_state_dim ................................. 128 |
| manual_gc ....................................... False |
| manual_gc_eval .................................. True |
| manual_gc_interval .............................. 0 |
| mask_factor ..................................... 1.0 |
| mask_prob ....................................... 0.15 |
| mask_type ....................................... random |
| masked_softmax_fusion ........................... True |
| max_position_embeddings ......................... 490000 |
| max_shard_size .................................. 20GB |
| max_tokens_to_oom ............................... 12000 |
| mcore2hf ........................................ True |
| memory_snapshot_path ............................ None |
| merge_file ...................................... None |
| micro_batch_size ................................ 1 |
| microbatch_group_size_per_vp_stage .............. None |
| mid_level_dataset_surplus ....................... 0.005 |
| min_loss_scale .................................. 1.0 |
| min_lr .......................................... 6e-06 |
| min_offloaded_tensor_size ....................... 1048576 |
| mlp_chunks_for_prefill .......................... 1 |
| mmap_bin_files .................................. True |
| mock_data ....................................... False |
| model_type ...................................... GPT |
| moe_apply_probs_on_input ........................ False |
| moe_aux_loss_coeff .............................. 0.0 |
| moe_deepep_num_sms .............................. 20 |
| moe_enable_deepep ............................... False |
| moe_expert_capacity_factor ...................... None |
| moe_extended_tp ................................. False |
| moe_ffn_hidden_size ............................. None |
| moe_flex_dispatcher_backend ..................... deepep |
| moe_grouped_gemm ................................ False |
| moe_hybridep_num_sms ............................ 16 |
| moe_input_jitter_eps ............................ None |
| moe_layer_freq .................................. 1 |
| moe_layer_recompute ............................. False |
| moe_pad_expert_input_to_capacity ................ False |
| moe_pad_experts_for_cuda_graph_inference ........ False |
| moe_per_layer_logging ........................... False |
| moe_permute_fusion .............................. False |
| moe_router_bias_update_method ................... sign |
| moe_router_bias_update_rate ..................... 0.001 |
| moe_router_dtype ................................ None |
| moe_router_enable_expert_bias ................... False |
| moe_router_force_load_balancing ................. False |
| moe_router_fusion ............................... False |
| moe_router_group_topk ........................... None |
| moe_router_load_balancing_type .................. aux_loss |
| moe_router_num_groups ........................... None |
| moe_router_padding_for_fp8 ...................... False |
| moe_router_padding_for_quantization ............. False |
| moe_router_pre_softmax .......................... False |
| moe_router_score_function ....................... softmax |
| moe_router_topk ................................. 2 |
| moe_router_topk_scaling_factor .................. None |
| moe_shared_expert_gate .......................... False |
| moe_shared_expert_intermediate_size ............. None |
| moe_shared_expert_overlap ....................... False |
| moe_token_dispatcher_type ....................... allgather |
| moe_token_drop_policy ........................... probs |
| moe_upcycling_granularity ....................... 1 |
| moe_use_legacy_grouped_gemm ..................... False |
| moe_use_upcycling ............................... False |
| moe_z_loss_coeff ................................ None |
| mrope_section ................................... None |
| mscale .......................................... 1.0 |
| mscale_all_dim .................................. 0.0 |
| mtp_linear_attention_type ....................... None |
| mtp_loss_scaling_factor ......................... 0.1 |
| mtp_num_layers .................................. None |
| multi_latent_attention .......................... False |
| multiple_validation_sets ........................ False |
| muon_ball_momentum .............................. 0.9 |
| muon_ball_msign_steps ........................... 5 |
| muon_ball_power_iteration_steps ................. 10 |
| muon_ball_qkv_split_mode ........................ component |
| muon_ball_radius_mode ........................... spectral_mup |
| muon_ball_retract_alpha ......................... 0.05 |
| muon_ball_retract_mode .......................... hard |
| muon_ball_scale_mode ............................ spectral_mup |
| muon_ball_split_fc1 ............................. True |
| muon_ball_split_moe_experts ..................... True |
| muon_ball_split_qkv ............................. True |
| muon_ball_use_nesterov .......................... True |
| muon_extra_scale_factor ......................... 1.0 |
| muon_fp32_matmul_prec ........................... medium |
| muon_momentum ................................... 0.9 |
| muon_num_ns_steps ............................... 5 |
| muon_qkv_split_mode ............................. component |
| muon_scale_mode ................................. spectral_mup |
| muon_scale_vectorized_mode ...................... full |
| muon_split_fc1 .................................. True |
| muon_split_moe_experts .......................... True |
| muon_split_qkv .................................. True |
| muon_tp_mode .................................... blockwise |
| muon_use_nesterov ............................... False |
| muon_vectorize .................................. [] |
| muon_vectorize_attn_dim ......................... hidden_size |
| nccl_all_reduce_for_prefill ..................... False |
| nccl_communicator_config_path ................... None |
| nccl_ub ......................................... False |
| no_load_optim ................................... True |
| no_load_rng ..................................... True |
| no_load_scheduler ............................... None |
| no_persist_layer_norm ........................... False |
| no_rope_freq .................................... None |
| no_save_optim ................................... None |
| no_save_rng ..................................... None |
| no_save_step_one ................................ None |
| no_weight_decay_cond_type ....................... None |
| non_persistent_ckpt_type ........................ None |
| non_persistent_global_ckpt_dir .................. None |
| non_persistent_local_ckpt_algo .................. fully_parallel |
| non_persistent_local_ckpt_dir ................... None |
| non_persistent_save_interval .................... None |
| norm_epsilon .................................... 1e-06 |
| normalization ................................... RMSNorm |
| num_attention_heads ............................. 30 |
| num_channels .................................... 3 |
| num_classes ..................................... 1000 |
| num_dataset_builder_threads ..................... 1 |
| num_distributed_optimizer_instances ............. 1 |
| num_experts ..................................... None |
| num_hf_saver .................................... None |
| num_layers ...................................... 56 |
| num_layers_at_end_in_bf16 ....................... 1 |
| num_layers_at_start_in_bf16 ..................... 1 |
| num_layers_per_virtual_pipeline_stage ........... None |
| num_query_groups ................................ 6 |
| num_virtual_stages_per_pipeline_rank ............ None |
| num_workers ..................................... 2 |
| object_storage_cache_path ....................... None |
| offload_modules ................................. [] |
| one_logger_async ................................ False |
| one_logger_project .............................. megatron-lm |
| one_logger_run_name ............................. None |
| onnx_safe ....................................... None |
| openai_gelu ..................................... False |
| optimizer ....................................... adam |
| optimizer_cpu_offload ........................... False |
| optimizer_offload_fraction ...................... 1.0 |
| output_bert_embeddings .......................... False |
| overlap_cpu_optimizer_d2h_h2d ................... False |
| overlap_grad_reduce ............................. False |
| overlap_moe_expert_parallel_comm ................ False |
| overlap_p2p_comm ................................ False |
| overlap_p2p_comm_warmup_flush ................... False |
| overlap_param_gather ............................ False |
| overlap_param_gather_with_optimizer_step ........ False |
| override_hf_eod_token_id ........................ None |
| override_opt_param_scheduler .................... False |
| padded_vocab_size ............................... 99000 |
| params_dtype .................................... torch.bfloat16 |
| patch_dim ....................................... 16 |
| per_split_data_args_path ........................ None |
| perform_initialization .......................... True |
| perform_rl_step ................................. False |
| pin_cpu_grads ................................... True |
| pin_cpu_params .................................. True |
| pipeline_model_parallel_comm_backend ............ None |
| pipeline_model_parallel_layout .................. None |
| pipeline_model_parallel_size .................... 1 |
| position_embedding_type ......................... rope |
| pretrain_script ................................. mcore_gdn_moe.model_provider |
| pretrained_checkpoint ........................... None |
| profile ......................................... False |
| profile_ranks ................................... [0] |
| profile_step_end ................................ 12 |
| profile_step_start .............................. 10 |
| q_lora_rank ..................................... None |
| qk_head_dim ..................................... 128 |
| qk_l2_norm ...................................... False |
| qk_layernorm .................................... False |
| qk_pos_emb_head_dim ............................. 64 |
| query_in_block_prob ............................. 0.1 |
| quick_geglu ..................................... False |
| rampup_batch_size ............................... None |
| rank ............................................ 0 |
| recompute_granularity ........................... None |
| recompute_method ................................ None |
| recompute_modules ............................... None |
| recompute_num_layers ............................ None |
| record_memory_history ........................... False |
| relative_attention_max_distance ................. 128 |
| relative_attention_num_buckets .................. 32 |
| reparam_checkpoint .............................. None |
| reparam_fallback_value .......................... None |
| reparam_keys .................................... None |
| replication ..................................... False |
| replication_factor .............................. 2 |
| replication_jump ................................ None |
| rerun_mode ...................................... validate_results |
| reset_attention_mask ............................ False |
| reset_iteration_one_to_zero ..................... False |
| reset_position_ids .............................. False |
| result_rejected_tracker_filename ................ None |
| retriever_report_topk_accuracies ................ [] |
| retriever_score_scaling ......................... False |
| retriever_seq_length ............................ 256 |
| retro_add_retriever ............................. False |
| retro_attention_gate ............................ 1 |
| retro_cyclic_train_iters ........................ None |
| retro_encoder_attention_dropout ................. 0.1 |
| retro_encoder_hidden_dropout .................... 0.1 |
| retro_encoder_layers ............................ 2 |
| retro_num_neighbors ............................. 2 |
| retro_num_retrieved_chunks ...................... 2 |
| retro_project_dir ............................... None |
| retro_verify_neighbor_count ..................... True |
| reuse_grad_buf_for_mxfp8_param_ag ............... False |
| rl_calculate_intra_group_similarity ............. False |
| rl_importance_sampling_truncation_coef .......... None |
| rl_inference_logprobs_is_correction ............. False |
| rl_offload_kv_cache_during_training ............. False |
| rl_offload_optimizer_during_inference ........... False |
| rl_partial_rollouts ............................. False |
| rl_prompts_per_eval ............................. 32 |
| rl_remove_kv_cache_during_training .............. False |
| rl_reset_cuda_graphs ............................ False |
| rl_sequence_packing_algo ........................ fifo |
| rl_sequence_packing_bin_size .................... 8192 |
| rl_use_sequence_packing ......................... False |
| rope_scaling_factor ............................. 8.0 |
| rope_type ....................................... None |
| rotary_base ..................................... 490000 |
| rotary_interleaved .............................. False |
| rotary_percent .................................. 1.0 |
| rotary_scaling_factor ........................... 1.0 |
| rotary_seq_len_interpolation_factor ............. None |
| run_workload_inspector_server ................... False |
| sample_rate ..................................... 1.0 |
| save ............................................ None |
| save_dir ........................................ /mnt/hdd/lvzhihao/mcore_models/Dist-mathcode10b-s1randg-sch1-CPT-200b-stage3-r640k-GDN2.9b-A7-12_20_21_23_46_48_49-sl32768bs128lr2e5-2e5/iter_71525-hf |
| save_interval ................................... 10000 |
| save_retain_interval ............................ None |
| scatter_gather_tensors_in_pipeline .............. True |
| seed ............................................ 1234 |
| seq_length ...................................... 1 |
| sequence_parallel ............................... False |
| sft ............................................. False |
| sft_tokenizer_prompt_format ..................... nemotron-h-aligned |
| sgd_momentum .................................... 0.9 |
| sharp_enabled_group ............................. None |
| short_seq_prob .................................. 0.1 |
| skip_train ...................................... True |
| skipped_train_samples ........................... 0 |
| softmax_type .................................... vanilla |
| spec ............................................ None |
| spectral_ball_momentum .......................... 0.9 |
| spectral_ball_msign_steps ....................... 8 |
| spectral_ball_power_iteration_steps ............. 20 |
| spectral_ball_qkv_split_mode .................... component |
| spectral_ball_radius_mode ....................... spectral_mup |
| spectral_ball_retract_alpha ..................... 0.05 |
| spectral_ball_retract_mode ...................... hard |
| spectral_ball_scale_mode ........................ spectral_mup |
| spectral_ball_solver ............................ bisection |
| spectral_ball_solver_max_iterations ............. 20 |
| spectral_ball_solver_tolerance_f ................ 1e-08 |
| spectral_ball_split_fc1 ......................... True |
| spectral_ball_split_moe_experts ................. True |
| spectral_ball_split_qkv ......................... True |
| spectral_ball_use_nesterov ...................... True |
| spectral_mup_init ............................... False |
| split ........................................... None |
| split_expert_init ............................... True |
| split_fc1_init .................................. True |
| split_qkv_init .................................. True |
| split_qkv_init_mode ............................. group |
| sqreglu ......................................... False |
| squared_relu .................................... False |
| start_samples ................................... None |
| start_weight_decay .............................. 0.1 |
| straggler_ctrlr_port ............................ 65535 |
| straggler_minmax_count .......................... 1 |
| strict_fsdp_dtensor_load ........................ True |
| suggested_communication_unit_size ............... None |
| swanlab_exp_name ................................ |
| swanlab_project ................................. |
| swanlab_save_dir ................................ |
| swanlab_workspace ............................... |
| swiglu .......................................... True |
| swin_backbone_type .............................. tiny |
| symmetric_ar_type ............................... None |
| synchronizer .................................... mcore_gdn_moe |
| target_ckpt_format .............................. torch_dist |
| te_rng_tracker .................................. False |
| tensor_model_parallel_size ...................... 1 |
| tensorboard_dir ................................. None |
| tensorboard_log_interval ........................ 1 |
| tensorboard_queue_size .......................... 1000 |
| test_data_path .................................. None |
| test_mode ....................................... False |
| tiktoken_num_special_tokens ..................... 1000 |
| tiktoken_pattern ................................ None |
| tiktoken_special_tokens ......................... None |
| timing_log_level ................................ 0 |
| timing_log_option ............................... minmax |
| titles_data_path ................................ None |
| token_shift_conv_init ........................... default |
| token_shift_conv_size ........................... 4 |
| tokenizer_metadata .............................. None |
| tokenizer_model ................................. /mnt/ssd/cache_tmp/tmp/tmp.1mZMUIgeRZ |
| tokenizer_type .................................. HuggingFaceTokenizer |
| torch_fsdp2_reshard_after_forward ............... True |
| tp_comm_bootstrap_backend ....................... nccl |
| tp_comm_bulk_dgrad .............................. True |
| tp_comm_bulk_wgrad .............................. True |
| tp_comm_overlap ................................. False |
| tp_comm_overlap_ag .............................. True |
| tp_comm_overlap_cfg ............................. None |
| tp_comm_overlap_rs .............................. True |
| tp_comm_overlap_rs_dgrad ........................ False |
| tp_comm_split_ag ................................ True |
| tp_comm_split_rs ................................ True |
| train_data_path ................................. None |
| train_iters ..................................... 500000 |
| train_samples ................................... None |
| train_sync_interval ............................. None |
| transformer_impl ................................ transformer_engine |
| transformer_pipeline_model_parallel_size ........ 1 |
| trust_remote_code ............................... False |
| untie_embeddings_and_output_weights ............. True |
| use_checkpoint_args ............................. False |
| use_checkpoint_opt_param_scheduler .............. False |
| use_cpu_initialization .......................... True |
| use_dist_ckpt ................................... True |
| use_dist_ckpt_deprecated ........................ False |
| use_distributed_optimizer ....................... False |
| use_flash_attn .................................. False |
| use_fused_weighted_squared_relu ................. False |
| use_gpu ......................................... True |
| use_legacy_models ............................... False |
| use_megatron_fsdp ............................... False |
| use_mp_args_from_checkpoint_args ................ False |
| use_one_sent_docs ............................... False |
| use_persistent_ckpt_worker ...................... False |
| use_precision_aware_optimizer ................... False |
| use_pytorch_profiler ............................ False |
| use_ring_exchange_p2p ........................... False |
| use_rope_scaling ................................ False |
| use_rotary_position_embeddings .................. False |
| use_sharp ....................................... False |
| use_te_activation_func .......................... False |
| use_tokenizer_model_from_checkpoint_args ........ True |
| use_torch_fsdp2 ................................. False |
| use_torch_optimizer_for_cpu_offload ............. False |
| use_tp_pp_dp_mapping ............................ False |
| v_head_dim ...................................... 128 |
| valid_data_path ................................. None |
| variable_seq_lengths ............................ False |
| virtual_pipeline_model_parallel_size ............ None |
| vision_backbone_type ............................ vit |
| vision_pretraining .............................. False |
| vision_pretraining_type ......................... classify |
| vocab_extra_ids ................................. 0 |
| vocab_file ...................................... None |
| vocab_size ...................................... None |
| wandb_entity .................................... |
| wandb_exp_name .................................. |
| wandb_project ................................... |
| wandb_save_dir .................................. |
| weight_decay .................................... 0.1 |
| weight_decay_incr_style ......................... constant |
| wgrad_deferral_limit ............................ 0 |
| window_attn_skip_freq ........................... None |
| window_size ..................................... None |
| word_embedding_dropout_prob ..................... 0.0 |
| world_size ...................................... 1 |
| yaml_cfg ........................................ None |
| -------------------- end of arguments --------------------- |
| INFO:megatron.core.num_microbatches_calculator:setting number of microbatches to constant 1024 |
| > building HuggingFaceTokenizer tokenizer ... |
| You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama_fast.LlamaTokenizerFast'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565 - if you loaded a llama tokenizer from a GGUF file you can ignore this message. |
| WARNING: one_logger package is required to enable e2e metrics tracking. please go to https://confluence.nvidia.com/display/MLWFO/Package+Repositories for details to install it |
| INFO:megatron.training.initialize:Setting logging level to 1 |
| WARNING:megatron.core.rerun_state_machine:RerunStateMachine initialized in mode validate_results |
| > initializing torch distributed ... |
| [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 |
| [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 |
| [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0 |
| > initialized tensor model parallel with size 1 |
| > initialized pipeline model parallel with size 1 |
| > setting random seeds to 1234 ... |
| > compiling dataset index builder ... |
| make: Entering directory '/mnt/ssd/lvzhihao/PostTrain/YuLan-Pretrain/megatron/core/datasets' |
| make: Nothing to be done for 'default'. |
| make: Leaving directory '/mnt/ssd/lvzhihao/PostTrain/YuLan-Pretrain/megatron/core/datasets' |
| >>> done with dataset index builder. Compilation time: 0.114 seconds |
| WARNING: constraints for invoking optimized fused softmax kernel are not met. We default back to unfused kernel invocations. |
| > compiling and loading fused kernels ... |
| /mnt/ssd/lvzhihao/PostTrain/YuLan-Pretrain/.venv/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:4807: UserWarning: No device id is provided via `init_process_group` or `barrier `. Using the current device set by the user. |
| warnings.warn( # warn only once |
| [rank0]:[W316 15:31:58.513326746 ProcessGroupNCCL.cpp:5023] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 as device used by this process is currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect. You can specify device_id in init_process_group() to force use of a particular device. |
| >>> done with compiling and loading fused kernels. Compilation time: 0.440 seconds |
| WORLD_SIZE: 1, RANK: 0, LOCAL_RANK: 0 |
| building GPT model ... |
| `torch_dtype` is deprecated! Use `dtype` instead! |
| `torch_dtype` is deprecated! Use `dtype` instead! |
| INFO:transformers_modules.tmp_dot_1mZMUIgeRZ.modeling_qwen3_next:[92m[Qwen3Next custom][0m [96mattn_position_embedding_type[0m=[93mrope[0m, [96mrnn_position_embedding_type[0m=[93mnope[0m, [96mattn_logits_scaling[0m=[93mNone[0m |
| Qwen3NextForCausalLM( |
| (model): Qwen3NextModel( |
| (embed_tokens): Embedding(99000, 1920) |
| (layers): ModuleList( |
| (0-11): 12 x Qwen3NextDecoderLayer( |
| (linear_attn): Qwen3NextGatedDeltaNet( |
| (act): SiLUActivation() |
| (conv1d): Conv1d(3072, 3072, kernel_size=(4,), stride=(1,), padding=(3,), groups=3072, bias=False) |
| (in_proj_qkvz): Linear(in_features=1920, out_features=5120, bias=False) |
| (in_proj_ba): Linear(in_features=1920, out_features=64, bias=False) |
| (norm): FusedRMSNormGated(64, eps=1e-06, activation=silu) |
| (out_proj): Linear(in_features=2048, out_features=1920, bias=False) |
| ) |
| (mlp): Qwen3NextMLP( |
| (gate_proj): Linear(in_features=1920, out_features=4800, bias=False) |
| (up_proj): Linear(in_features=1920, out_features=4800, bias=False) |
| (down_proj): Linear(in_features=4800, out_features=1920, bias=False) |
| (act_fn): SiLUActivation() |
| ) |
| (input_layernorm): LlamaRMSNorm() |
| (post_attention_layernorm): LlamaRMSNorm() |
| ) |
| (12): Qwen3NextDecoderLayer( |
| (self_attn): Qwen3NextAttention( |
| (q_proj): Linear(in_features=1920, out_features=1920, bias=True) |
| (k_proj): Linear(in_features=1920, out_features=384, bias=True) |
| (v_proj): Linear(in_features=1920, out_features=384, bias=True) |
| (o_proj): Linear(in_features=1920, out_features=1920, bias=False) |
| ) |
| (mlp): Qwen3NextMLP( |
| (gate_proj): Linear(in_features=1920, out_features=4800, bias=False) |
| (up_proj): Linear(in_features=1920, out_features=4800, bias=False) |
| (down_proj): Linear(in_features=4800, out_features=1920, bias=False) |
| (act_fn): SiLUActivation() |
| ) |
| (input_layernorm): LlamaRMSNorm() |
| (post_attention_layernorm): LlamaRMSNorm() |
| ) |
| (13-19): 7 x Qwen3NextDecoderLayer( |
| (linear_attn): Qwen3NextGatedDeltaNet( |
| (act): SiLUActivation() |
| (conv1d): Conv1d(3072, 3072, kernel_size=(4,), stride=(1,), padding=(3,), groups=3072, bias=False) |
| (in_proj_qkvz): Linear(in_features=1920, out_features=5120, bias=False) |
| (in_proj_ba): Linear(in_features=1920, out_features=64, bias=False) |
| (norm): FusedRMSNormGated(64, eps=1e-06, activation=silu) |
| (out_proj): Linear(in_features=2048, out_features=1920, bias=False) |
| ) |
| (mlp): Qwen3NextMLP( |
| (gate_proj): Linear(in_features=1920, out_features=4800, bias=False) |
| (up_proj): Linear(in_features=1920, out_features=4800, bias=False) |
| (down_proj): Linear(in_features=4800, out_features=1920, bias=False) |
| (act_fn): SiLUActivation() |
| ) |
| (input_layernorm): LlamaRMSNorm() |
| (post_attention_layernorm): LlamaRMSNorm() |
| ) |
| (20-21): 2 x Qwen3NextDecoderLayer( |
| (self_attn): Qwen3NextAttention( |
| (q_proj): Linear(in_features=1920, out_features=1920, bias=True) |
| (k_proj): Linear(in_features=1920, out_features=384, bias=True) |
| (v_proj): Linear(in_features=1920, out_features=384, bias=True) |
| (o_proj): Linear(in_features=1920, out_features=1920, bias=False) |
| ) |
| (mlp): Qwen3NextMLP( |
| (gate_proj): Linear(in_features=1920, out_features=4800, bias=False) |
| (up_proj): Linear(in_features=1920, out_features=4800, bias=False) |
| (down_proj): Linear(in_features=4800, out_features=1920, bias=False) |
| (act_fn): SiLUActivation() |
| ) |
| (input_layernorm): LlamaRMSNorm() |
| (post_attention_layernorm): LlamaRMSNorm() |
| ) |
| (22): Qwen3NextDecoderLayer( |
| (linear_attn): Qwen3NextGatedDeltaNet( |
| (act): SiLUActivation() |
| (conv1d): Conv1d(3072, 3072, kernel_size=(4,), stride=(1,), padding=(3,), groups=3072, bias=False) |
| (in_proj_qkvz): Linear(in_features=1920, out_features=5120, bias=False) |
| (in_proj_ba): Linear(in_features=1920, out_features=64, bias=False) |
| (norm): FusedRMSNormGated(64, eps=1e-06, activation=silu) |
| (out_proj): Linear(in_features=2048, out_features=1920, bias=False) |
| ) |
| (mlp): Qwen3NextMLP( |
| (gate_proj): Linear(in_features=1920, out_features=4800, bias=False) |
| (up_proj): Linear(in_features=1920, out_features=4800, bias=False) |
| (down_proj): Linear(in_features=4800, out_features=1920, bias=False) |
| (act_fn): SiLUActivation() |
| ) |
| (input_layernorm): LlamaRMSNorm() |
| (post_attention_layernorm): LlamaRMSNorm() |
| ) |
| (23): Qwen3NextDecoderLayer( |
| (self_attn): Qwen3NextAttention( |
| (q_proj): Linear(in_features=1920, out_features=1920, bias=True) |
| (k_proj): Linear(in_features=1920, out_features=384, bias=True) |
| (v_proj): Linear(in_features=1920, out_features=384, bias=True) |
| (o_proj): Linear(in_features=1920, out_features=1920, bias=False) |
| ) |
| (mlp): Qwen3NextMLP( |
| (gate_proj): Linear(in_features=1920, out_features=4800, bias=False) |
| (up_proj): Linear(in_features=1920, out_features=4800, bias=False) |
| (down_proj): Linear(in_features=4800, out_features=1920, bias=False) |
| (act_fn): SiLUActivation() |
| ) |
| (input_layernorm): LlamaRMSNorm() |
| (post_attention_layernorm): LlamaRMSNorm() |
| ) |
| (24-45): 22 x Qwen3NextDecoderLayer( |
| (linear_attn): Qwen3NextGatedDeltaNet( |
| (act): SiLUActivation() |
| (conv1d): Conv1d(3072, 3072, kernel_size=(4,), stride=(1,), padding=(3,), groups=3072, bias=False) |
| (in_proj_qkvz): Linear(in_features=1920, out_features=5120, bias=False) |
| (in_proj_ba): Linear(in_features=1920, out_features=64, bias=False) |
| (norm): FusedRMSNormGated(64, eps=1e-06, activation=silu) |
| (out_proj): Linear(in_features=2048, out_features=1920, bias=False) |
| ) |
| (mlp): Qwen3NextMLP( |
| (gate_proj): Linear(in_features=1920, out_features=4800, bias=False) |
| (up_proj): Linear(in_features=1920, out_features=4800, bias=False) |
| (down_proj): Linear(in_features=4800, out_features=1920, bias=False) |
| (act_fn): SiLUActivation() |
| ) |
| (input_layernorm): LlamaRMSNorm() |
| (post_attention_layernorm): LlamaRMSNorm() |
| ) |
| (46): Qwen3NextDecoderLayer( |
| (self_attn): Qwen3NextAttention( |
| (q_proj): Linear(in_features=1920, out_features=1920, bias=True) |
| (k_proj): Linear(in_features=1920, out_features=384, bias=True) |
| (v_proj): Linear(in_features=1920, out_features=384, bias=True) |
| (o_proj): Linear(in_features=1920, out_features=1920, bias=False) |
| ) |
| (mlp): Qwen3NextMLP( |
| (gate_proj): Linear(in_features=1920, out_features=4800, bias=False) |
| (up_proj): Linear(in_features=1920, out_features=4800, bias=False) |
| (down_proj): Linear(in_features=4800, out_features=1920, bias=False) |
| (act_fn): SiLUActivation() |
| ) |
| (input_layernorm): LlamaRMSNorm() |
| (post_attention_layernorm): LlamaRMSNorm() |
| ) |
| (47): Qwen3NextDecoderLayer( |
| (linear_attn): Qwen3NextGatedDeltaNet( |
| (act): SiLUActivation() |
| (conv1d): Conv1d(3072, 3072, kernel_size=(4,), stride=(1,), padding=(3,), groups=3072, bias=False) |
| (in_proj_qkvz): Linear(in_features=1920, out_features=5120, bias=False) |
| (in_proj_ba): Linear(in_features=1920, out_features=64, bias=False) |
| (norm): FusedRMSNormGated(64, eps=1e-06, activation=silu) |
| (out_proj): Linear(in_features=2048, out_features=1920, bias=False) |
| ) |
| (mlp): Qwen3NextMLP( |
| (gate_proj): Linear(in_features=1920, out_features=4800, bias=False) |
| (up_proj): Linear(in_features=1920, out_features=4800, bias=False) |
| (down_proj): Linear(in_features=4800, out_features=1920, bias=False) |
| (act_fn): SiLUActivation() |
| ) |
| (input_layernorm): LlamaRMSNorm() |
| (post_attention_layernorm): LlamaRMSNorm() |
| ) |
| (48-49): 2 x Qwen3NextDecoderLayer( |
| (self_attn): Qwen3NextAttention( |
| (q_proj): Linear(in_features=1920, out_features=1920, bias=True) |
| (k_proj): Linear(in_features=1920, out_features=384, bias=True) |
| (v_proj): Linear(in_features=1920, out_features=384, bias=True) |
| (o_proj): Linear(in_features=1920, out_features=1920, bias=False) |
| ) |
| (mlp): Qwen3NextMLP( |
| (gate_proj): Linear(in_features=1920, out_features=4800, bias=False) |
| (up_proj): Linear(in_features=1920, out_features=4800, bias=False) |
| (down_proj): Linear(in_features=4800, out_features=1920, bias=False) |
| (act_fn): SiLUActivation() |
| ) |
| (input_layernorm): LlamaRMSNorm() |
| (post_attention_layernorm): LlamaRMSNorm() |
| ) |
| (50-55): 6 x Qwen3NextDecoderLayer( |
| (linear_attn): Qwen3NextGatedDeltaNet( |
| (act): SiLUActivation() |
| (conv1d): Conv1d(3072, 3072, kernel_size=(4,), stride=(1,), padding=(3,), groups=3072, bias=False) |
| (in_proj_qkvz): Linear(in_features=1920, out_features=5120, bias=False) |
| (in_proj_ba): Linear(in_features=1920, out_features=64, bias=False) |
| (norm): FusedRMSNormGated(64, eps=1e-06, activation=silu) |
| (out_proj): Linear(in_features=2048, out_features=1920, bias=False) |
| ) |
| (mlp): Qwen3NextMLP( |
| (gate_proj): Linear(in_features=1920, out_features=4800, bias=False) |
| (up_proj): Linear(in_features=1920, out_features=4800, bias=False) |
| (down_proj): Linear(in_features=4800, out_features=1920, bias=False) |
| (act_fn): SiLUActivation() |
| ) |
| (input_layernorm): LlamaRMSNorm() |
| (post_attention_layernorm): LlamaRMSNorm() |
| ) |
| ) |
| (norm): LlamaRMSNorm() |
| (rotary_emb): Qwen3NextRotaryEmbedding() |
| ) |
| (lm_head): Linear(in_features=1920, out_features=99000, bias=False) |
| ) |
| GPTModel( |
| (embedding): LanguageModelEmbedding( |
| (word_embeddings): VocabParallelEmbedding() |
| (embedding_dropout): Dropout(p=0.1, inplace=False) |
| ) |
| (rotary_pos_emb): RotaryEmbedding() |
| (decoder): TransformerBlock( |
| (layers): ModuleList( |
| (0-11): 12 x TransformerLayer( |
| (input_layernorm): IdentityOp() |
| (self_attention): GatedDeltaNet( |
| (in_proj): TELayerNormColumnParallelLinear(in_features=1920, out_features=5184, bias=False, TP=1) |
| (conv1d): Conv1d(3072, 3072, kernel_size=(4,), stride=(1,), padding=(3,), groups=3072, bias=False) |
| (out_norm): RMSNorm() |
| (out_proj): TERowParallelLinear(in_features=2048, out_features=1920, bias=False, TP=1) |
| ) |
| (pre_cross_attn_layernorm): IdentityOp() |
| (cross_attention): IdentityOp() |
| (cross_attn_bda): IdentityFuncOp() |
| (pre_mlp_layernorm): IdentityOp() |
| (mlp): MLP( |
| (linear_fc1): TELayerNormColumnParallelLinear(in_features=1920, out_features=9600, bias=False, TP=1) |
| (linear_fc2): TERowParallelLinear(in_features=4800, out_features=1920, bias=False, TP=1) |
| ) |
| ) |
| (12): TransformerLayer( |
| (input_layernorm): IdentityOp() |
| (self_attention): SelfAttention( |
| (core_attention): TEDotProductAttention( |
| (flash_attention): FlashAttention() |
| (fused_attention): FusedAttention() |
| (unfused_attention): UnfusedDotProductAttention( |
| (scale_mask_softmax): FusedScaleMaskSoftmax() |
| (attention_dropout): Dropout(p=0.1, inplace=False) |
| ) |
| ) |
| (linear_proj): TERowParallelLinear(in_features=1920, out_features=1920, bias=False, TP=1) |
| (linear_qkv): TELayerNormColumnParallelLinear(in_features=1920, out_features=2688, bias=True, TP=1) |
| (q_layernorm): IdentityOp() |
| (k_layernorm): IdentityOp() |
| ) |
| (pre_cross_attn_layernorm): IdentityOp() |
| (cross_attention): IdentityOp() |
| (cross_attn_bda): IdentityFuncOp() |
| (pre_mlp_layernorm): IdentityOp() |
| (mlp): MLP( |
| (linear_fc1): TELayerNormColumnParallelLinear(in_features=1920, out_features=9600, bias=False, TP=1) |
| (linear_fc2): TERowParallelLinear(in_features=4800, out_features=1920, bias=False, TP=1) |
| ) |
| ) |
| (13-19): 7 x TransformerLayer( |
| (input_layernorm): IdentityOp() |
| (self_attention): GatedDeltaNet( |
| (in_proj): TELayerNormColumnParallelLinear(in_features=1920, out_features=5184, bias=False, TP=1) |
| (conv1d): Conv1d(3072, 3072, kernel_size=(4,), stride=(1,), padding=(3,), groups=3072, bias=False) |
| (out_norm): RMSNorm() |
| (out_proj): TERowParallelLinear(in_features=2048, out_features=1920, bias=False, TP=1) |
| ) |
| (pre_cross_attn_layernorm): IdentityOp() |
| (cross_attention): IdentityOp() |
| (cross_attn_bda): IdentityFuncOp() |
| (pre_mlp_layernorm): IdentityOp() |
| (mlp): MLP( |
| (linear_fc1): TELayerNormColumnParallelLinear(in_features=1920, out_features=9600, bias=False, TP=1) |
| (linear_fc2): TERowParallelLinear(in_features=4800, out_features=1920, bias=False, TP=1) |
| ) |
| ) |
| (20-21): 2 x TransformerLayer( |
| (input_layernorm): IdentityOp() |
| (self_attention): SelfAttention( |
| (core_attention): TEDotProductAttention( |
| (flash_attention): FlashAttention() |
| (fused_attention): FusedAttention() |
| (unfused_attention): UnfusedDotProductAttention( |
| (scale_mask_softmax): FusedScaleMaskSoftmax() |
| (attention_dropout): Dropout(p=0.1, inplace=False) |
| ) |
| ) |
| (linear_proj): TERowParallelLinear(in_features=1920, out_features=1920, bias=False, TP=1) |
| (linear_qkv): TELayerNormColumnParallelLinear(in_features=1920, out_features=2688, bias=True, TP=1) |
| (q_layernorm): IdentityOp() |
| (k_layernorm): IdentityOp() |
| ) |
| (pre_cross_attn_layernorm): IdentityOp() |
| (cross_attention): IdentityOp() |
| (cross_attn_bda): IdentityFuncOp() |
| (pre_mlp_layernorm): IdentityOp() |
| (mlp): MLP( |
| (linear_fc1): TELayerNormColumnParallelLinear(in_features=1920, out_features=9600, bias=False, TP=1) |
| (linear_fc2): TERowParallelLinear(in_features=4800, out_features=1920, bias=False, TP=1) |
| ) |
| ) |
| (22): TransformerLayer( |
| (input_layernorm): IdentityOp() |
| (self_attention): GatedDeltaNet( |
| (in_proj): TELayerNormColumnParallelLinear(in_features=1920, out_features=5184, bias=False, TP=1) |
| (conv1d): Conv1d(3072, 3072, kernel_size=(4,), stride=(1,), padding=(3,), groups=3072, bias=False) |
| (out_norm): RMSNorm() |
| (out_proj): TERowParallelLinear(in_features=2048, out_features=1920, bias=False, TP=1) |
| ) |
| (pre_cross_attn_layernorm): IdentityOp() |
| (cross_attention): IdentityOp() |
| (cross_attn_bda): IdentityFuncOp() |
| (pre_mlp_layernorm): IdentityOp() |
| (mlp): MLP( |
| (linear_fc1): TELayerNormColumnParallelLinear(in_features=1920, out_features=9600, bias=False, TP=1) |
| (linear_fc2): TERowParallelLinear(in_features=4800, out_features=1920, bias=False, TP=1) |
| ) |
| ) |
| (23): TransformerLayer( |
| (input_layernorm): IdentityOp() |
| (self_attention): SelfAttention( |
| (core_attention): TEDotProductAttention( |
| (flash_attention): FlashAttention() |
| (fused_attention): FusedAttention() |
| (unfused_attention): UnfusedDotProductAttention( |
| (scale_mask_softmax): FusedScaleMaskSoftmax() |
| (attention_dropout): Dropout(p=0.1, inplace=False) |
| ) |
| ) |
| (linear_proj): TERowParallelLinear(in_features=1920, out_features=1920, bias=False, TP=1) |
| (linear_qkv): TELayerNormColumnParallelLinear(in_features=1920, out_features=2688, bias=True, TP=1) |
| (q_layernorm): IdentityOp() |
| (k_layernorm): IdentityOp() |
| ) |
| (pre_cross_attn_layernorm): IdentityOp() |
| (cross_attention): IdentityOp() |
| (cross_attn_bda): IdentityFuncOp() |
| (pre_mlp_layernorm): IdentityOp() |
| (mlp): MLP( |
| (linear_fc1): TELayerNormColumnParallelLinear(in_features=1920, out_features=9600, bias=False, TP=1) |
| (linear_fc2): TERowParallelLinear(in_features=4800, out_features=1920, bias=False, TP=1) |
| ) |
| ) |
| (24-45): 22 x TransformerLayer( |
| (input_layernorm): IdentityOp() |
| (self_attention): GatedDeltaNet( |
| (in_proj): TELayerNormColumnParallelLinear(in_features=1920, out_features=5184, bias=False, TP=1) |
| (conv1d): Conv1d(3072, 3072, kernel_size=(4,), stride=(1,), padding=(3,), groups=3072, bias=False) |
| (out_norm): RMSNorm() |
| (out_proj): TERowParallelLinear(in_features=2048, out_features=1920, bias=False, TP=1) |
| ) |
| (pre_cross_attn_layernorm): IdentityOp() |
| (cross_attention): IdentityOp() |
| (cross_attn_bda): IdentityFuncOp() |
| (pre_mlp_layernorm): IdentityOp() |
| (mlp): MLP( |
| (linear_fc1): TELayerNormColumnParallelLinear(in_features=1920, out_features=9600, bias=False, TP=1) |
| (linear_fc2): TERowParallelLinear(in_features=4800, out_features=1920, bias=False, TP=1) |
| ) |
| ) |
| (46): TransformerLayer( |
| (input_layernorm): IdentityOp() |
| (self_attention): SelfAttention( |
| (core_attention): TEDotProductAttention( |
| (flash_attention): FlashAttention() |
| (fused_attention): FusedAttention() |
| (unfused_attention): UnfusedDotProductAttention( |
| (scale_mask_softmax): FusedScaleMaskSoftmax() |
| (attention_dropout): Dropout(p=0.1, inplace=False) |
| ) |
| ) |
| (linear_proj): TERowParallelLinear(in_features=1920, out_features=1920, bias=False, TP=1) |
| (linear_qkv): TELayerNormColumnParallelLinear(in_features=1920, out_features=2688, bias=True, TP=1) |
| (q_layernorm): IdentityOp() |
| (k_layernorm): IdentityOp() |
| ) |
| (pre_cross_attn_layernorm): IdentityOp() |
| (cross_attention): IdentityOp() |
| (cross_attn_bda): IdentityFuncOp() |
| (pre_mlp_layernorm): IdentityOp() |
| (mlp): MLP( |
| (linear_fc1): TELayerNormColumnParallelLinear(in_features=1920, out_features=9600, bias=False, TP=1) |
| (linear_fc2): TERowParallelLinear(in_features=4800, out_features=1920, bias=False, TP=1) |
| ) |
| ) |
| (47): TransformerLayer( |
| (input_layernorm): IdentityOp() |
| (self_attention): GatedDeltaNet( |
| (in_proj): TELayerNormColumnParallelLinear(in_features=1920, out_features=5184, bias=False, TP=1) |
| (conv1d): Conv1d(3072, 3072, kernel_size=(4,), stride=(1,), padding=(3,), groups=3072, bias=False) |
| (out_norm): RMSNorm() |
| (out_proj): TERowParallelLinear(in_features=2048, out_features=1920, bias=False, TP=1) |
| ) |
| (pre_cross_attn_layernorm): IdentityOp() |
| (cross_attention): IdentityOp() |
| (cross_attn_bda): IdentityFuncOp() |
| (pre_mlp_layernorm): IdentityOp() |
| (mlp): MLP( |
| (linear_fc1): TELayerNormColumnParallelLinear(in_features=1920, out_features=9600, bias=False, TP=1) |
| (linear_fc2): TERowParallelLinear(in_features=4800, out_features=1920, bias=False, TP=1) |
| ) |
| ) |
| (48-49): 2 x TransformerLayer( |
| (input_layernorm): IdentityOp() |
| (self_attention): SelfAttention( |
| (core_attention): TEDotProductAttention( |
| (flash_attention): FlashAttention() |
| (fused_attention): FusedAttention() |
| (unfused_attention): UnfusedDotProductAttention( |
| (scale_mask_softmax): FusedScaleMaskSoftmax() |
| (attention_dropout): Dropout(p=0.1, inplace=False) |
| ) |
| ) |
| (linear_proj): TERowParallelLinear(in_features=1920, out_features=1920, bias=False, TP=1) |
| (linear_qkv): TELayerNormColumnParallelLinear(in_features=1920, out_features=2688, bias=True, TP=1) |
| (q_layernorm): IdentityOp() |
| (k_layernorm): IdentityOp() |
| ) |
| (pre_cross_attn_layernorm): IdentityOp() |
| (cross_attention): IdentityOp() |
| (cross_attn_bda): IdentityFuncOp() |
| (pre_mlp_layernorm): IdentityOp() |
| (mlp): MLP( |
| (linear_fc1): TELayerNormColumnParallelLinear(in_features=1920, out_features=9600, bias=False, TP=1) |
| (linear_fc2): TERowParallelLinear(in_features=4800, out_features=1920, bias=False, TP=1) |
| ) |
| ) |
| (50-55): 6 x TransformerLayer( |
| (input_layernorm): IdentityOp() |
| (self_attention): GatedDeltaNet( |
| (in_proj): TELayerNormColumnParallelLinear(in_features=1920, out_features=5184, bias=False, TP=1) |
| (conv1d): Conv1d(3072, 3072, kernel_size=(4,), stride=(1,), padding=(3,), groups=3072, bias=False) |
| (out_norm): RMSNorm() |
| (out_proj): TERowParallelLinear(in_features=2048, out_features=1920, bias=False, TP=1) |
| ) |
| (pre_cross_attn_layernorm): IdentityOp() |
| (cross_attention): IdentityOp() |
| (cross_attn_bda): IdentityFuncOp() |
| (pre_mlp_layernorm): IdentityOp() |
| (mlp): MLP( |
| (linear_fc1): TELayerNormColumnParallelLinear(in_features=1920, out_features=9600, bias=False, TP=1) |
| (linear_fc2): TERowParallelLinear(in_features=4800, out_features=1920, bias=False, TP=1) |
| ) |
| ) |
| ) |
| (final_layernorm): RMSNorm() |
| ) |
| (output_layer): ColumnParallelLinear(in_features=1920, out_features=99000, bias=False, TP=1) |
| ) |
| /mnt/ssd/lvzhihao/PostTrain/YuLan-Pretrain/megatron/core/dist_checkpointing/strategies/common.py:89: UserWarning: Environment variable TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD detected, since the`weights_only` argument was not explicitly passed to `torch.load`, forcing weights_only=False. |
| return torch.load(load_path, map_location='cpu') |
| (TP, PP) mismatch after resume ((1, 1) vs (2, 1) from checkpoint): RNG state will be ignored |
| sharded_state_dict metadata loaded from the checkpoint: {'singleton_local_shards': True, 'distrib_optim_sharding_type': 'fully_reshardable', 'distrib_optim_fully_reshardable_mem_efficient': False, 'chained_optim_avoid_prefix': True} |
| Job sharding has changed: Rerun state will be ignored |
| loading distributed checkpoint from /mnt/hdd/lvzhihao/mcore_models/Dist-mathcode10b-s1randg-sch1-CPT-200b-stage3-r640k-GDN2.9b-A7-12_20_21_23_46_48_49-sl32768bs128lr2e5-2e5 at iteration 71525 |
| /mnt/ssd/lvzhihao/PostTrain/YuLan-Pretrain/megatron/core/dist_checkpointing/strategies/torch.py:956: FutureWarning: `load_state_dict` is deprecated and will be removed in future versions. Please use `load` instead. |
| checkpoint.load_state_dict( |
| checkpoint version 3.0 |
| successfully loaded checkpoint from /mnt/hdd/lvzhihao/mcore_models/Dist-mathcode10b-s1randg-sch1-CPT-200b-stage3-r640k-GDN2.9b-A7-12_20_21_23_46_48_49-sl32768bs128lr2e5-2e5 [ t 1/1, p 1/1 ] at iteration 71525 |
| INFO:root:Converting layer 0 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 1 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 2 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 3 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 4 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 5 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 6 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 7 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 8 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 9 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 10 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 11 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 12 is_gdn=False is_not_moe=True |
| INFO:root:[DEBUG] Layer 12: args.attention_output_gate=False |
| INFO:root:[DEBUG] set_gated_selfattn_state: args.attention_output_gate=False |
| INFO:root:[DEBUG] set_gated_selfattn_state: attention_output_gate=False, linear_layer=TELayerNormColumnParallelLinear |
| INFO:root:Converting layer 13 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 14 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 15 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 16 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 17 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 18 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 19 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 20 is_gdn=False is_not_moe=True |
| INFO:root:[DEBUG] Layer 20: args.attention_output_gate=False |
| INFO:root:[DEBUG] set_gated_selfattn_state: args.attention_output_gate=False |
| INFO:root:[DEBUG] set_gated_selfattn_state: attention_output_gate=False, linear_layer=TELayerNormColumnParallelLinear |
| INFO:root:Converting layer 21 is_gdn=False is_not_moe=True |
| INFO:root:[DEBUG] Layer 21: args.attention_output_gate=False |
| INFO:root:[DEBUG] set_gated_selfattn_state: args.attention_output_gate=False |
| INFO:root:[DEBUG] set_gated_selfattn_state: attention_output_gate=False, linear_layer=TELayerNormColumnParallelLinear |
| INFO:root:Converting layer 22 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 23 is_gdn=False is_not_moe=True |
| INFO:root:[DEBUG] Layer 23: args.attention_output_gate=False |
| INFO:root:[DEBUG] set_gated_selfattn_state: args.attention_output_gate=False |
| INFO:root:[DEBUG] set_gated_selfattn_state: attention_output_gate=False, linear_layer=TELayerNormColumnParallelLinear |
| INFO:root:Converting layer 24 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 25 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 26 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 27 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 28 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 29 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 30 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 31 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 32 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 33 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 34 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 35 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 36 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 37 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 38 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 39 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 40 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 41 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 42 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 43 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 44 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 45 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 46 is_gdn=False is_not_moe=True |
| INFO:root:[DEBUG] Layer 46: args.attention_output_gate=False |
| INFO:root:[DEBUG] set_gated_selfattn_state: args.attention_output_gate=False |
| INFO:root:[DEBUG] set_gated_selfattn_state: attention_output_gate=False, linear_layer=TELayerNormColumnParallelLinear |
| INFO:root:Converting layer 47 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 48 is_gdn=False is_not_moe=True |
| INFO:root:[DEBUG] Layer 48: args.attention_output_gate=False |
| INFO:root:[DEBUG] set_gated_selfattn_state: args.attention_output_gate=False |
| INFO:root:[DEBUG] set_gated_selfattn_state: attention_output_gate=False, linear_layer=TELayerNormColumnParallelLinear |
| INFO:root:Converting layer 49 is_gdn=False is_not_moe=True |
| INFO:root:[DEBUG] Layer 49: args.attention_output_gate=False |
| INFO:root:[DEBUG] set_gated_selfattn_state: args.attention_output_gate=False |
| INFO:root:[DEBUG] set_gated_selfattn_state: attention_output_gate=False, linear_layer=TELayerNormColumnParallelLinear |
| INFO:root:Converting layer 50 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 51 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 52 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 53 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 54 is_gdn=True is_not_moe=True |
| INFO:root:Converting layer 55 is_gdn=True is_not_moe=True |
| DEBUG:root:[RANK 0] 0 send op & 0 recv op. |
| INFO:root:[Iters 0 RANK 0] starts synchronizing parameters with other ranks... |
| INFO:root:[Iters 0 RANK 0] finishes synchronizing |
| [Iters 0 RANK 0] model.safetensors is saved. |
| DEBUG:root:[Iters 0 RANK 0] joined |
| Conversion finished in 70.92923641204834 seconds. |
| [rank0]:[W316 15:33:10.354704836 ProcessGroupNCCL.cpp:1538] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html |
|
|
