post_strict_eval pipeline log

post_strict_eval.log

@@ -0,0 +1,141 @@
+[2026-06-29 11:15:00] ==========================================================
+[2026-06-29 11:15:00] POST-STRICT C4 + D1 EVAL PIPELINE START
+[2026-06-29 11:15:00] ==========================================================
+[2026-06-29 11:15:00] STAGE 0: wait for strict_continuation DONE marker
+[2026-06-29 11:15:00]   strict_continuation DONE marker present
+[2026-06-29 11:15:00] STAGE 1 (C4): merge strict ckpt-10000 + stage1_merged -> stage4_merged_strict
+/dev/md0        700G  548G  153G  79% /root/autodl-tmp
+/root/miniconda3/lib/python3.12/site-packages/transformers/models/auto/modeling_auto.py:2284: FutureWarning: The class `AutoModelForVision2Seq` is deprecated and will be removed in v5.0. Please use `AutoModelForImageTextToText` instead.
+  warnings.warn(
+`torch_dtype` is deprecated! Use `dtype` instead!
+Loading base: /root/autodl-tmp/covt_repro/covt_data/output/merged/stage1_merged
+
+Loading LoRA: /root/autodl-tmp/covt_repro/covt_data/output/lora_stage234/checkpoint-10000
+Merging...
+Saving to: /root/autodl-tmp/covt_repro/covt_data/output/merged/stage4_merged_strict
+[2026-06-29 11:16:58,296] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
+The image processor of type `Qwen2VLImageProcessor` is now loaded as a fast processor by default, even if the model checkpoint was saved with a slow processor. This is a breaking change and may produce slightly different outputs. To continue using the slow processor, instantiate this class with `use_fast=False`. Note that this behavior will be extended to all models in a future release.
+merge done
+[2026-06-29 11:17:30]   C4 merge OK
+total 16211332
+drwxr-xr-x 2 root root       4096 Jun 29 11:17 .
+drwxr-xr-x 4 root root         67 Jun 28 23:20 ..
+-rw-r--r-- 1 root root        605 Jun 29 11:17 added_tokens.json
+-rw-r--r-- 1 root root       1017 Jun 29 11:17 chat_template.jinja
+-rw-r--r-- 1 root root       1049 Jun 28 23:22 chat_template.json
+-rw-r--r-- 1 root root       3165 Jun 29 11:16 config.json
+-rw-r--r-- 1 root root        214 Jun 29 11:16 generation_config.json
+-rw-r--r-- 1 root root    1671853 Jun 29 11:17 merges.txt
+-rw------- 1 root root 4968243304 Jun 29 11:17 model-00001-of-00004.safetensors
+16G	/root/autodl-tmp/covt_repro/covt_data/output/merged/stage4_merged_strict
+[2026-06-29 11:17:30] STAGE 2 (D1-prep): VLMEvalKit install + CV-Bench prefetch
+[2026-06-29 11:17:48]   vlmeval importable ✓
+[2026-06-29 11:17:48]   prefetching CV-Bench dataset
+
+  cached nyu-visionx/CV-Bench: DatasetDict({
+    test: Dataset({
+        features: ['idx', 'type', 'task', 'image', 'question', 'choices', 'answer', 'prompt', 'filename', 'source', 'source_dataset', 'source_filename', 'target_class', 'target_size', 'bbox'],
+        num_rows: 2638
+    })
+})
+[2026-06-29 11:18:50] STAGE 3 (D1): run VLMEvalKit on stage4_merged_strict (CV-Bench-2D + CV-Bench-3D)
+[2026-06-29 11:18:50]   eval log -> /root/autodl-tmp/covt_repro/covt_data/output/eval_stage4_strict/run.log
+[2026-06-29 11:18:52] ERROR - misc.py: load_env - 215: Did not detect the .env file at /root/autodl-tmp/covt_repro/CoVT/VLMEvalKit/.env, failed to load. 
+[2026-06-29 11:18:57] ERROR - misc.py: load_env - 215: Did not detect the .env file at /root/autodl-tmp/covt_repro/CoVT/VLMEvalKit/.env, failed to load. 
+usage: run.py [-h] [--data DATA [DATA ...]] [--model MODEL [MODEL ...]]
+              [--config CONFIG] [--work-dir WORK_DIR]
+              [--mode {all,infer,eval}] [--api-nproc API_NPROC]
+              [--retry RETRY] [--judge-args JUDGE_ARGS] [--judge JUDGE]
+              [--verbose] [--ignore] [--reuse] [--reuse-aux REUSE_AUX]
+              [--use-vllm] [--use-verifier]
+run.py: error: argument --mode: invalid choice: 'infer_judge_score' (choose from 'all', 'infer', 'eval')
+[2026-06-29 11:18:58]   vlmeval rc=2
+[2026-06-29 11:18:58]   WARN: vlmeval exit code non-zero. Continuing to parse partial output.
+[2026-06-29 11:18:58] STAGE 4 (D2): parse scores + compare to paper-faithful gate
+score files candidates (0):
+
+=== headline tokens from run.log ===
+
+=== gate reference (manual verdict) ===
+  weighted >= 76.7   3D >= 81.0   Depth >= 82.5   Distance >= 79.0   2D >= 72.8
+[2026-06-29 11:18:58] STAGE 5: upload eval results + stage4_merged to Steven668866/CoVT-Phase2-3expert-Strict
+[2026-06-29 11:18:58]   S4 merged model
+[2026-06-29 11:18:58]   uploading stage4_merged_strict/added_tokens.json
+⚠️  Warning: 'huggingface-cli upload' is deprecated. Use 'hf upload' instead.
+https://hf-mirror.com/Steven668866/CoVT-Phase2-3expert-Strict/blob/main/stage4_merged_strict/added_tokens.json
+[2026-06-29 11:19:04]     verify: OK
+[2026-06-29 11:19:04]   uploading stage4_merged_strict/chat_template.jinja
+⚠️  Warning: 'huggingface-cli upload' is deprecated. Use 'hf upload' instead.
+https://hf-mirror.com/Steven668866/CoVT-Phase2-3expert-Strict/blob/main/stage4_merged_strict/chat_template.jinja
+[2026-06-29 11:19:09]     verify: OK
+[2026-06-29 11:19:09]   uploading stage4_merged_strict/chat_template.json
+⚠️  Warning: 'huggingface-cli upload' is deprecated. Use 'hf upload' instead.
+https://hf-mirror.com/Steven668866/CoVT-Phase2-3expert-Strict/blob/main/stage4_merged_strict/chat_template.json
+[2026-06-29 11:19:17]     verify: OK
+[2026-06-29 11:19:17]   uploading stage4_merged_strict/config.json
+⚠️  Warning: 'huggingface-cli upload' is deprecated. Use 'hf upload' instead.
+https://hf-mirror.com/Steven668866/CoVT-Phase2-3expert-Strict/blob/main/stage4_merged_strict/config.json
+[2026-06-29 11:19:26]     verify: OK
+[2026-06-29 11:19:26]   uploading stage4_merged_strict/generation_config.json
+⚠️  Warning: 'huggingface-cli upload' is deprecated. Use 'hf upload' instead.
+https://hf-mirror.com/Steven668866/CoVT-Phase2-3expert-Strict/blob/main/stage4_merged_strict/generation_config.json
+[2026-06-29 11:19:34]     verify: OK
+[2026-06-29 11:19:34]   uploading stage4_merged_strict/merges.txt
+⚠️  Warning: 'huggingface-cli upload' is deprecated. Use 'hf upload' instead.
+https://hf-mirror.com/Steven668866/CoVT-Phase2-3expert-Strict/blob/main/stage4_merged_strict/merges.txt
+[2026-06-29 11:19:44]     verify: OK
+[2026-06-29 11:19:44]   uploading stage4_merged_strict/model-00001-of-00004.safetensors
+
+
  ...0001-of-00004.safetensors: 100%|██████████| 4.97GB / 4.97GB            
+https://hf-mirror.com/Steven668866/CoVT-Phase2-3expert-Strict/blob/main/stage4_merged_strict/model-00001-of-00004.safetensors
+[2026-06-29 11:20:11]     verify: OK
+[2026-06-29 11:20:11]   uploading stage4_merged_strict/model-00002-of-00004.safetensors
+
+
  ...0002-of-00004.safetensors: 100%|██████████| 4.99GB / 4.99GB            
+https://hf-mirror.com/Steven668866/CoVT-Phase2-3expert-Strict/blob/main/stage4_merged_strict/model-00002-of-00004.safetensors
+[2026-06-29 11:20:36]     verify: OK
+[2026-06-29 11:20:36]   uploading stage4_merged_strict/model-00003-of-00004.safetensors
+
+
  ...0003-of-00004.safetensors: 100%|██████████| 4.93GB / 4.93GB            
+https://hf-mirror.com/Steven668866/CoVT-Phase2-3expert-Strict/blob/main/stage4_merged_strict/model-00003-of-00004.safetensors
+[2026-06-29 11:20:59]     verify: OK
+[2026-06-29 11:20:59]   uploading stage4_merged_strict/model-00004-of-00004.safetensors
+
+
  ...0004-of-00004.safetensors: 100%|██████████| 1.69GB / 1.69GB            
+https://hf-mirror.com/Steven668866/CoVT-Phase2-3expert-Strict/blob/main/stage4_merged_strict/model-00004-of-00004.safetensors
+[2026-06-29 11:21:18]     verify: OK
+[2026-06-29 11:21:18]   uploading stage4_merged_strict/model.safetensors.index.json
+⚠️  Warning: 'huggingface-cli upload' is deprecated. Use 'hf upload' instead.
+https://hf-mirror.com/Steven668866/CoVT-Phase2-3expert-Strict/blob/main/stage4_merged_strict/model.safetensors.index.json
+[2026-06-29 11:21:25]     verify: OK
+[2026-06-29 11:21:25]   uploading stage4_merged_strict/preprocessor_config.json
+⚠️  Warning: 'huggingface-cli upload' is deprecated. Use 'hf upload' instead.
+https://hf-mirror.com/Steven668866/CoVT-Phase2-3expert-Strict/blob/main/stage4_merged_strict/preprocessor_config.json
+[2026-06-29 11:21:32]     verify: OK
+[2026-06-29 11:21:32]   uploading stage4_merged_strict/special_tokens_map.json
+⚠️  Warning: 'huggingface-cli upload' is deprecated. Use 'hf upload' instead.
+https://hf-mirror.com/Steven668866/CoVT-Phase2-3expert-Strict/blob/main/stage4_merged_strict/special_tokens_map.json
+[2026-06-29 11:21:42]     verify: OK
+[2026-06-29 11:21:42]   uploading stage4_merged_strict/tokenizer.json
+
+
  ...ged_strict/tokenizer.json: 100%|██████████| 11.4MB / 11.4MB            
+https://hf-mirror.com/Steven668866/CoVT-Phase2-3expert-Strict/blob/main/stage4_merged_strict/tokenizer.json
+[2026-06-29 11:21:50]     verify: OK
+[2026-06-29 11:21:50]   uploading stage4_merged_strict/tokenizer_config.json
+⚠️  Warning: 'huggingface-cli upload' is deprecated. Use 'hf upload' instead.
+https://hf-mirror.com/Steven668866/CoVT-Phase2-3expert-Strict/blob/main/stage4_merged_strict/tokenizer_config.json
+[2026-06-29 11:21:58]     verify: OK
+[2026-06-29 11:21:58]   uploading stage4_merged_strict/video_preprocessor_config.json
+⚠️  Warning: 'huggingface-cli upload' is deprecated. Use 'hf upload' instead.
+https://hf-mirror.com/Steven668866/CoVT-Phase2-3expert-Strict/blob/main/stage4_merged_strict/video_preprocessor_config.json
+[2026-06-29 11:22:04]     verify: OK
+[2026-06-29 11:22:04]   uploading stage4_merged_strict/vocab.json
+⚠️  Warning: 'huggingface-cli upload' is deprecated. Use 'hf upload' instead.
+https://hf-mirror.com/Steven668866/CoVT-Phase2-3expert-Strict/blob/main/stage4_merged_strict/vocab.json
+[2026-06-29 11:22:18]     verify: OK
+[2026-06-29 11:22:18]   eval results
+[2026-06-29 11:22:18]   uploading eval_stage4_strict/run.log
+⚠️  Warning: 'huggingface-cli upload' is deprecated. Use 'hf upload' instead.
+https://hf-mirror.com/Steven668866/CoVT-Phase2-3expert-Strict/blob/main/eval_stage4_strict/run.log
+[2026-06-29 11:22:24]     verify: OK
+[2026-06-29 11:22:24]   uploading post_strict_eval.log