设置成功! 注意: 1. 仅限学术用途和加速访问github/huggingface,不承诺稳定性 2. 开启加速后对访问其他资源如pip源等会*更慢* [2026-06-27 18:01:05,220] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2026-06-27 18:01:07,944] [WARNING] [runner.py:215:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only. Detected VISIBLE_DEVICES=0 but ignoring it because one or several of --include/--exclude/--num_gpus/--num_nodes cl args were used. If you want to use CUDA_VISIBLE_DEVICES don't pass any of these arguments to deepspeed. [2026-06-27 18:01:07,944] [INFO] [runner.py:607:main] cmd = /root/miniconda3/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMF19 --master_addr=127.0.0.1 --master_port=22813 --enable_each_rank_log=None src/training/train.py --use_liger True --lora_enable True --vision_lora True --use_dora False --lora_namespan_exclude ["embed_tokens", "lm_head", "dino", "sam", "depth", "SD", "internvit", "pidinet", "siglip", "metaclip"] --lora_rank 16 --lora_alpha 32 --lora_dropout 0.01 --num_lora_modules -1 --model_id /root/autodl-tmp/covt_repro/covt_data/output/merged/stage1_merged --model_path /root/autodl-tmp/covt_repro/covt_data/output/merged/stage1_merged --anchor_model_id ["sam", "depth", "dino"] --data_path /root/autodl-tmp/covt_repro/covt_data/dataset/CoVT-Dataset --image_folder /root/autodl-tmp/covt_repro/covt_data/dataset/CoVT-Dataset --remove_unused_columns False --freeze_vision_tower True --freeze_llm True --tune_merger False --bf16 True --fp16 False --disable_flash_attn2 False --output_dir /root/autodl-tmp/covt_repro/covt_data/output/lora_stage234 --resume_from_checkpoint True --max_steps 10000 --per_device_train_batch_size 4 --gradient_accumulation_steps 2 --image_min_pixels 200704 --image_max_pixels 802816 --image_resized_width 448 --image_resized_height 448 --learning_rate 5e-5 --projection_layer_lr 1e-5 --weight_decay 0.1 --warmup_ratio 0.05 --lr_scheduler_type cosine --logging_steps 1 --tf32 True --gradient_checkpointing True --lazy_preprocess True --save_strategy steps --save_steps 1000 --dataloader_num_workers 0 --deepspeed /root/autodl-tmp/covt_repro/scripts/zero2_pro6000.json --report_to tensorboard --run_name sft_phase2 --training_stage full --stage_0_step 0 --stage_1_step 24000 --stage_2_step 48000 --vqa_only_stage 16000 [2026-06-27 18:01:09,158] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) [2026-06-27 18:01:11,746] [INFO] [launch.py:146:main] WORLD INFO DICT: {'localhost': [0]} [2026-06-27 18:01:11,746] [INFO] [launch.py:152:main] nnodes=1, num_local_procs=1, node_rank=0 [2026-06-27 18:01:11,746] [INFO] [launch.py:163:main] global_rank_mapping=defaultdict(, {'localhost': [0]}) [2026-06-27 18:01:11,746] [INFO] [launch.py:164:main] dist_world_size=1 [2026-06-27 18:01:11,746] [INFO] [launch.py:168:main] Setting CUDA_VISIBLE_DEVICES=0 [2026-06-27 18:01:11,747] [INFO] [launch.py:256:main] process 2010 spawned with command: ['/root/miniconda3/bin/python', '-u', 'src/training/train.py', '--local_rank=0', '--use_liger', 'True', '--lora_enable', 'True', '--vision_lora', 'True', '--use_dora', 'False', '--lora_namespan_exclude', '["embed_tokens", "lm_head", "dino", "sam", "depth", "SD", "internvit", "pidinet", "siglip", "metaclip"]', '--lora_rank', '16', '--lora_alpha', '32', '--lora_dropout', '0.01', '--num_lora_modules', '-1', '--model_id', '/root/autodl-tmp/covt_repro/covt_data/output/merged/stage1_merged', '--model_path', '/root/autodl-tmp/covt_repro/covt_data/output/merged/stage1_merged', '--anchor_model_id', '["sam", "depth", "dino"]', '--data_path', '/root/autodl-tmp/covt_repro/covt_data/dataset/CoVT-Dataset', '--image_folder', '/root/autodl-tmp/covt_repro/covt_data/dataset/CoVT-Dataset', '--remove_unused_columns', 'False', '--freeze_vision_tower', 'True', '--freeze_llm', 'True', '--tune_merger', 'False', '--bf16', 'True', '--fp16', 'False', '--disable_flash_attn2', 'False', '--output_dir', '/root/autodl-tmp/covt_repro/covt_data/output/lora_stage234', '--resume_from_checkpoint', 'True', '--max_steps', '10000', '--per_device_train_batch_size', '4', '--gradient_accumulation_steps', '2', '--image_min_pixels', '200704', '--image_max_pixels', '802816', '--image_resized_width', '448', '--image_resized_height', '448', '--learning_rate', '5e-5', '--projection_layer_lr', '1e-5', '--weight_decay', '0.1', '--warmup_ratio', '0.05', '--lr_scheduler_type', 'cosine', '--logging_steps', '1', '--tf32', 'True', '--gradient_checkpointing', 'True', '--lazy_preprocess', 'True', '--save_strategy', 'steps', '--save_steps', '1000', '--dataloader_num_workers', '0', '--deepspeed', '/root/autodl-tmp/covt_repro/scripts/zero2_pro6000.json', '--report_to', 'tensorboard', '--run_name', 'sft_phase2', '--training_stage', 'full', '--stage_0_step', '0', '--stage_1_step', '24000', '--stage_2_step', '48000', '--vqa_only_stage', '16000'] [2026-06-27 18:01:15,104] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect) xFormers not available xFormers not available /root/miniconda3/lib/python3.12/site-packages/diffusers/models/transformers/transformer_2d.py:34: FutureWarning: `Transformer2DModelOutput` is deprecated and will be removed in version 1.0.0. Importing `Transformer2DModelOutput` from `diffusers.models.transformer_2d` is deprecated and this will be removed in a future version. Please use `from diffusers.models.modeling_outputs import Transformer2DModelOutput`, instead. deprecate("Transformer2DModelOutput", "1.0.0", deprecation_message) [2026-06-27 18:01:16,952] [INFO] [comm.py:652:init_distributed] cdb=None [2026-06-27 18:01:16,952] [INFO] [comm.py:683:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with `model.to('cuda')`. Loading checkpoint shards: 0%| | 0/4 [00:00