Junyi42 commited on
Commit
f7f2e2b
·
verified ·
1 Parent(s): 5c5f7b0

Upload checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins

Browse files
checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/wandb/offline-run-20260125_150523-checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins-run0/files/output.log CHANGED
@@ -1,3 +1,173 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  wandb: Detected [huggingface_hub.inference] in use.
2
  wandb: Use W&B Weave for improved LLM call tracing. Install Weave with `pip install weave` then add `import weave` to the top of your script.
3
  wandb: For more information, check out the docs at: https://weave-docs.wandb.ai/
@@ -1011,183 +1181,6 @@ wandb: For more information, check out the docs at: https://weave-docs.wandb.ai/
1011
  [2026-01-25 15:59:52] (step=0001000) Train Loss mse: 0.0000, Train Loss ce: 0.2671, Train Steps/Sec: 0.10,
1012
  [2026-01-25 15:59:55] (step=0001001) Train Loss mse: 0.0000, Train Loss ce: 0.2673, Train Steps/Sec: 0.37,
1013
  [2026-01-25 15:59:58] (step=0001002) Train Loss mse: 0.0000, Train Loss ce: 0.2456, Train Steps/Sec: 0.32,
1014
- FullyShardedDataParallel(
1015
- (_fsdp_wrapped_module): Bagel(
1016
- (language_model): Qwen2ForCausalLM(
1017
- (model): Qwen2Model(
1018
- (embed_tokens): Embedding(152064, 3584)
1019
- (layers): ModuleList(
1020
- (0-27): 28 x FullyShardedDataParallel(
1021
- (_fsdp_wrapped_module): CheckpointWrapper(
1022
- (_checkpoint_wrapped_module): Qwen2MoTDecoderLayer(
1023
- (self_attn): PackedAttentionMoT(
1024
- (q_proj): Linear(in_features=3584, out_features=3584, bias=True)
1025
- (k_proj): Linear(in_features=3584, out_features=512, bias=True)
1026
- (v_proj): Linear(in_features=3584, out_features=512, bias=True)
1027
- (o_proj): Linear(in_features=3584, out_features=3584, bias=False)
1028
- (q_norm): Qwen2RMSNorm((128,), eps=1e-06)
1029
- (k_norm): Qwen2RMSNorm((128,), eps=1e-06)
1030
- (q_norm_moe_gen): Qwen2RMSNorm((128,), eps=1e-06)
1031
- (k_norm_moe_gen): Qwen2RMSNorm((128,), eps=1e-06)
1032
- (q_proj_moe_gen): Linear(in_features=3584, out_features=3584, bias=True)
1033
- (k_proj_moe_gen): Linear(in_features=3584, out_features=512, bias=True)
1034
- (v_proj_moe_gen): Linear(in_features=3584, out_features=512, bias=True)
1035
- (o_proj_moe_gen): Linear(in_features=3584, out_features=3584, bias=False)
1036
- )
1037
- (mlp): Qwen2MLP(
1038
- (gate_proj): Linear(in_features=3584, out_features=18944, bias=False)
1039
- (up_proj): Linear(in_features=3584, out_features=18944, bias=False)
1040
- (down_proj): Linear(in_features=18944, out_features=3584, bias=False)
1041
- (act_fn): SiLU()
1042
- )
1043
- (mlp_moe_gen): Qwen2MLP(
1044
- (gate_proj): Linear(in_features=3584, out_features=18944, bias=False)
1045
- (up_proj): Linear(in_features=3584, out_features=18944, bias=False)
1046
- (down_proj): Linear(in_features=18944, out_features=3584, bias=False)
1047
- (act_fn): SiLU()
1048
- )
1049
- (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
1050
- (input_layernorm_moe_gen): Qwen2RMSNorm((3584,), eps=1e-06)
1051
- (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
1052
- (post_attention_layernorm_moe_gen): Qwen2RMSNorm((3584,), eps=1e-06)
1053
- )
1054
- )
1055
- )
1056
- )
1057
- (norm): Qwen2RMSNorm((3584,), eps=1e-06)
1058
- (norm_moe_gen): Qwen2RMSNorm((3584,), eps=1e-06)
1059
- (rotary_emb): Qwen2RotaryEmbedding()
1060
- )
1061
- (lm_head): Linear(in_features=3584, out_features=152064, bias=False)
1062
- )
1063
- (vit_model): SiglipVisionModel(
1064
- (vision_model): FullyShardedDataParallel(
1065
- (_fsdp_wrapped_module): SiglipVisionTransformer(
1066
- (embeddings): SiglipVisionEmbeddings(
1067
- (position_embedding): Embedding(4900, 1152)
1068
- (patch_embedding): Linear(in_features=588, out_features=1152, bias=True)
1069
- )
1070
- (encoder): SiglipEncoder(
1071
- (layers): ModuleList(
1072
- (0-25): 26 x FullyShardedDataParallel(
1073
- (_fsdp_wrapped_module): CheckpointWrapper(
1074
- (_checkpoint_wrapped_module): SiglipEncoderLayer(
1075
- (self_attn): SiglipFlashAttention2(
1076
- (k_proj): Linear(in_features=1152, out_features=1152, bias=True)
1077
- (v_proj): Linear(in_features=1152, out_features=1152, bias=True)
1078
- (q_proj): Linear(in_features=1152, out_features=1152, bias=True)
1079
- (out_proj): Linear(in_features=1152, out_features=1152, bias=True)
1080
- )
1081
- (layer_norm1): LayerNorm((1152,), eps=1e-06, elementwise_affine=True)
1082
- (mlp): SiglipMLP(
1083
- (activation_fn): PytorchGELUTanh()
1084
- (fc1): Linear(in_features=1152, out_features=4304, bias=True)
1085
- (fc2): Linear(in_features=4304, out_features=1152, bias=True)
1086
- )
1087
- (layer_norm2): LayerNorm((1152,), eps=1e-06, elementwise_affine=True)
1088
- )
1089
- )
1090
- )
1091
- )
1092
- )
1093
- (post_layernorm): LayerNorm((1152,), eps=1e-06, elementwise_affine=True)
1094
- )
1095
- )
1096
- )
1097
- (connector): FullyShardedDataParallel(
1098
- (_fsdp_wrapped_module): CheckpointWrapper(
1099
- (_checkpoint_wrapped_module): MLPconnector(
1100
- (activation_fn): PytorchGELUTanh()
1101
- (fc1): Linear(in_features=1152, out_features=3584, bias=True)
1102
- (fc2): Linear(in_features=3584, out_features=3584, bias=True)
1103
- )
1104
- )
1105
- )
1106
- (vit_pos_embed): FullyShardedDataParallel(
1107
- (_fsdp_wrapped_module): PositionEmbedding()
1108
- )
1109
- )
1110
- )
1111
- _flat_param True
1112
- language_model.model.layers.0._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1113
- language_model.model.layers.1._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1114
- language_model.model.layers.2._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1115
- language_model.model.layers.3._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1116
- language_model.model.layers.4._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1117
- language_model.model.layers.5._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1118
- language_model.model.layers.6._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1119
- language_model.model.layers.7._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1120
- language_model.model.layers.8._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1121
- language_model.model.layers.9._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1122
- language_model.model.layers.10._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1123
- language_model.model.layers.11._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1124
- language_model.model.layers.12._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1125
- language_model.model.layers.13._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1126
- language_model.model.layers.14._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1127
- language_model.model.layers.15._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1128
- language_model.model.layers.16._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1129
- language_model.model.layers.17._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1130
- language_model.model.layers.18._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1131
- language_model.model.layers.19._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1132
- language_model.model.layers.20._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1133
- language_model.model.layers.21._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1134
- language_model.model.layers.22._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1135
- language_model.model.layers.23._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1136
- language_model.model.layers.24._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1137
- language_model.model.layers.25._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1138
- language_model.model.layers.26._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1139
- language_model.model.layers.27._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1140
- vit_model.vision_model._fsdp_wrapped_module._flat_param True
1141
- vit_model.vision_model._fsdp_wrapped_module.encoder.layers.0._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1142
- vit_model.vision_model._fsdp_wrapped_module.encoder.layers.1._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1143
- vit_model.vision_model._fsdp_wrapped_module.encoder.layers.2._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1144
- vit_model.vision_model._fsdp_wrapped_module.encoder.layers.3._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1145
- vit_model.vision_model._fsdp_wrapped_module.encoder.layers.4._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1146
- vit_model.vision_model._fsdp_wrapped_module.encoder.layers.5._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1147
- vit_model.vision_model._fsdp_wrapped_module.encoder.layers.6._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1148
- vit_model.vision_model._fsdp_wrapped_module.encoder.layers.7._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1149
- vit_model.vision_model._fsdp_wrapped_module.encoder.layers.8._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1150
- vit_model.vision_model._fsdp_wrapped_module.encoder.layers.9._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1151
- vit_model.vision_model._fsdp_wrapped_module.encoder.layers.10._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1152
- vit_model.vision_model._fsdp_wrapped_module.encoder.layers.11._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1153
- vit_model.vision_model._fsdp_wrapped_module.encoder.layers.12._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1154
- vit_model.vision_model._fsdp_wrapped_module.encoder.layers.13._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1155
- vit_model.vision_model._fsdp_wrapped_module.encoder.layers.14._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1156
- vit_model.vision_model._fsdp_wrapped_module.encoder.layers.15._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1157
- vit_model.vision_model._fsdp_wrapped_module.encoder.layers.16._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1158
- vit_model.vision_model._fsdp_wrapped_module.encoder.layers.17._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1159
- vit_model.vision_model._fsdp_wrapped_module.encoder.layers.18._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1160
- vit_model.vision_model._fsdp_wrapped_module.encoder.layers.19._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1161
- vit_model.vision_model._fsdp_wrapped_module.encoder.layers.20._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1162
- vit_model.vision_model._fsdp_wrapped_module.encoder.layers.21._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1163
- vit_model.vision_model._fsdp_wrapped_module.encoder.layers.22._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1164
- vit_model.vision_model._fsdp_wrapped_module.encoder.layers.23._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1165
- vit_model.vision_model._fsdp_wrapped_module.encoder.layers.24._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1166
- vit_model.vision_model._fsdp_wrapped_module.encoder.layers.25._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1167
- connector._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
1168
- vit_pos_embed._fsdp_wrapped_module._flat_param False
1169
- Preparing Dataset vlm_gym_colorization_celoss_no_mse/vlm_gym_colorization_train
1170
- base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step0
1171
- Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
1172
- [eval debug] first 3 batch fingerprints:
1173
- fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
1174
- fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
1175
- fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
1176
- ce_avg: 0.8449353575706482, mse_avg: 0.0
1177
- base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step500
1178
- Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
1179
- [eval debug] first 3 batch fingerprints:
1180
- fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
1181
- fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
1182
- fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
1183
- ce_avg: 0.2859387993812561, mse_avg: 0.0
1184
- base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step1000
1185
- Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
1186
- [eval debug] first 3 batch fingerprints:
1187
- fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
1188
- fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
1189
- fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
1190
- ce_avg: 0.46788886189460754, mse_avg: 0.0
1191
  [2026-01-25 16:00:01] (step=0001003) Train Loss mse: 0.0000, Train Loss ce: 0.2735, Train Steps/Sec: 0.41,
1192
  [2026-01-25 16:00:03] (step=0001004) Train Loss mse: 0.0000, Train Loss ce: 0.2693, Train Steps/Sec: 0.40,
1193
  [2026-01-25 16:00:05] (step=0001005) Train Loss mse: 0.0000, Train Loss ce: 0.2883, Train Steps/Sec: 0.42,
@@ -1286,6 +1279,27 @@ ce_avg: 0.46788886189460754, mse_avg: 0.0
1286
  [2026-01-25 16:04:20] (step=0001098) Train Loss mse: 0.0000, Train Loss ce: 0.2518, Train Steps/Sec: 0.36,
1287
  [2026-01-25 16:04:23] (step=0001099) Train Loss mse: 0.0000, Train Loss ce: 0.2504, Train Steps/Sec: 0.36,
1288
  [2026-01-25 16:04:26] (step=0001100) Train Loss mse: 0.0000, Train Loss ce: 0.2578, Train Steps/Sec: 0.34,
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1289
  [2026-01-25 16:04:29] (step=0001101) Train Loss mse: 0.0000, Train Loss ce: 0.2687, Train Steps/Sec: 0.37,
1290
  [2026-01-25 16:04:32] (step=0001102) Train Loss mse: 0.0000, Train Loss ce: 0.2546, Train Steps/Sec: 0.33,
1291
  [2026-01-25 16:04:35] (step=0001103) Train Loss mse: 0.0000, Train Loss ce: 0.2656, Train Steps/Sec: 0.36,
@@ -2560,20 +2574,6 @@ ce_avg: 0.46788886189460754, mse_avg: 0.0
2560
  [2026-01-25 17:04:00] (step=0002372) Train Loss mse: 0.0000, Train Loss ce: 0.2340, Train Steps/Sec: 0.32,
2561
  [2026-01-25 17:04:02] (step=0002373) Train Loss mse: 0.0000, Train Loss ce: 0.2756, Train Steps/Sec: 0.40,
2562
  [2026-01-25 17:04:05] (step=0002374) Train Loss mse: 0.0000, Train Loss ce: 0.2349, Train Steps/Sec: 0.37,
2563
- base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step1500
2564
- Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
2565
- [eval debug] first 3 batch fingerprints:
2566
- fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
2567
- fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
2568
- fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
2569
- ce_avg: 0.6231057643890381, mse_avg: 0.0
2570
- base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step2000
2571
- Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
2572
- [eval debug] first 3 batch fingerprints:
2573
- fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
2574
- fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
2575
- fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
2576
- ce_avg: 0.6934272646903992, mse_avg: 0.0
2577
  [2026-01-25 17:04:08] (step=0002375) Train Loss mse: 0.0000, Train Loss ce: 0.2704, Train Steps/Sec: 0.35,
2578
  [2026-01-25 17:04:10] (step=0002376) Train Loss mse: 0.0000, Train Loss ce: 0.2496, Train Steps/Sec: 0.37,
2579
  [2026-01-25 17:04:13] (step=0002377) Train Loss mse: 0.0000, Train Loss ce: 0.2377, Train Steps/Sec: 0.34,
@@ -2713,6 +2713,20 @@ ce_avg: 0.6934272646903992, mse_avg: 0.0
2713
  [2026-01-25 17:10:35] (step=0002511) Train Loss mse: 0.0000, Train Loss ce: 0.2424, Train Steps/Sec: 0.45,
2714
  [2026-01-25 17:10:37] (step=0002512) Train Loss mse: 0.0000, Train Loss ce: 0.2441, Train Steps/Sec: 0.34,
2715
  [2026-01-25 17:10:40] (step=0002513) Train Loss mse: 0.0000, Train Loss ce: 0.2438, Train Steps/Sec: 0.34,
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2716
  [2026-01-25 17:10:43] (step=0002514) Train Loss mse: 0.0000, Train Loss ce: 0.2770, Train Steps/Sec: 0.37,
2717
  [2026-01-25 17:10:46] (step=0002515) Train Loss mse: 0.0000, Train Loss ce: 0.2555, Train Steps/Sec: 0.37,
2718
  [2026-01-25 17:10:49] (step=0002516) Train Loss mse: 0.0000, Train Loss ce: 0.2529, Train Steps/Sec: 0.37,
@@ -3572,41 +3586,6 @@ ce_avg: 0.6934272646903992, mse_avg: 0.0
3572
  [2026-01-25 17:50:54] (step=0003370) Train Loss mse: 0.0000, Train Loss ce: 0.2459, Train Steps/Sec: 0.38,
3573
  [2026-01-25 17:50:56] (step=0003371) Train Loss mse: 0.0000, Train Loss ce: 0.2480, Train Steps/Sec: 0.43,
3574
  [2026-01-25 17:50:59] (step=0003372) Train Loss mse: 0.0000, Train Loss ce: 0.2495, Train Steps/Sec: 0.36,
3575
- base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step2500
3576
- Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
3577
- [eval debug] first 3 batch fingerprints:
3578
- fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
3579
- fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
3580
- fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
3581
- ce_avg: 0.719638466835022, mse_avg: 0.0
3582
- base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step3000
3583
- Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
3584
- [eval debug] first 3 batch fingerprints:
3585
- fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
3586
- fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
3587
- fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
3588
- ce_avg: 0.7213400602340698, mse_avg: 0.0
3589
- base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step3500
3590
- Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
3591
- [eval debug] first 3 batch fingerprints:
3592
- fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
3593
- fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
3594
- fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
3595
- ce_avg: 0.6960676312446594, mse_avg: 0.0
3596
- base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step4000
3597
- Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
3598
- [eval debug] first 3 batch fingerprints:
3599
- fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
3600
- fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
3601
- fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
3602
- ce_avg: 0.7003546953201294, mse_avg: 0.0
3603
- base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step4500
3604
- Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
3605
- [eval debug] first 3 batch fingerprints:
3606
- fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
3607
- fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
3608
- fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
3609
- ce_avg: 0.6831608414649963, mse_avg: 0.0
3610
  [2026-01-25 17:51:01] (step=0003373) Train Loss mse: 0.0000, Train Loss ce: 0.2582, Train Steps/Sec: 0.42,
3611
  [2026-01-25 17:51:04] (step=0003374) Train Loss mse: 0.0000, Train Loss ce: 0.2335, Train Steps/Sec: 0.34,
3612
  [2026-01-25 17:51:07] (step=0003375) Train Loss mse: 0.0000, Train Loss ce: 0.2726, Train Steps/Sec: 0.36,
@@ -3716,6 +3695,27 @@ ce_avg: 0.6831608414649963, mse_avg: 0.0
3716
  [2026-01-25 17:55:59] (step=0003479) Train Loss mse: 0.0000, Train Loss ce: 0.2427, Train Steps/Sec: 0.32,
3717
  [2026-01-25 17:56:03] (step=0003480) Train Loss mse: 0.0000, Train Loss ce: 0.2365, Train Steps/Sec: 0.32,
3718
  [2026-01-25 17:56:05] (step=0003481) Train Loss mse: 0.0000, Train Loss ce: 0.2540, Train Steps/Sec: 0.36,
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3719
  [2026-01-25 17:56:08] (step=0003482) Train Loss mse: 0.0000, Train Loss ce: 0.2442, Train Steps/Sec: 0.38,
3720
  [2026-01-25 17:56:11] (step=0003483) Train Loss mse: 0.0000, Train Loss ce: 0.2501, Train Steps/Sec: 0.35,
3721
  [2026-01-25 17:56:14] (step=0003484) Train Loss mse: 0.0000, Train Loss ce: 0.2568, Train Steps/Sec: 0.37,
@@ -5046,13 +5046,6 @@ ce_avg: 0.6831608414649963, mse_avg: 0.0
5046
  [2026-01-25 18:58:34] (step=0004809) Train Loss mse: 0.0000, Train Loss ce: 0.2206, Train Steps/Sec: 0.36,
5047
  [2026-01-25 18:58:37] (step=0004810) Train Loss mse: 0.0000, Train Loss ce: 0.2408, Train Steps/Sec: 0.30,
5048
  [2026-01-25 18:58:40] (step=0004811) Train Loss mse: 0.0000, Train Loss ce: 0.2571, Train Steps/Sec: 0.35,
5049
- base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step5000
5050
- Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
5051
- [eval debug] first 3 batch fingerprints:
5052
- fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
5053
- fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
5054
- fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
5055
- ce_avg: 0.6794630289077759, mse_avg: 0.0
5056
  [2026-01-25 18:58:43] (step=0004812) Train Loss mse: 0.0000, Train Loss ce: 0.2358, Train Steps/Sec: 0.34,
5057
  [2026-01-25 18:58:46] (step=0004813) Train Loss mse: 0.0000, Train Loss ce: 0.2200, Train Steps/Sec: 0.36,
5058
  [2026-01-25 18:58:49] (step=0004814) Train Loss mse: 0.0000, Train Loss ce: 0.2575, Train Steps/Sec: 0.30,
@@ -5245,4 +5238,11 @@ ce_avg: 0.6794630289077759, mse_avg: 0.0
5245
  [2026-01-25 19:07:43] Saving checkpoint to /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/0005000.
5246
  /opt/conda/lib/python3.11/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
5247
  warnings.warn(
5248
- [2026-01-25 19:10:19] Done!
 
 
 
 
 
 
 
 
1
+ FullyShardedDataParallel(
2
+ (_fsdp_wrapped_module): Bagel(
3
+ (language_model): Qwen2ForCausalLM(
4
+ (model): Qwen2Model(
5
+ (embed_tokens): Embedding(152064, 3584)
6
+ (layers): ModuleList(
7
+ (0-27): 28 x FullyShardedDataParallel(
8
+ (_fsdp_wrapped_module): CheckpointWrapper(
9
+ (_checkpoint_wrapped_module): Qwen2MoTDecoderLayer(
10
+ (self_attn): PackedAttentionMoT(
11
+ (q_proj): Linear(in_features=3584, out_features=3584, bias=True)
12
+ (k_proj): Linear(in_features=3584, out_features=512, bias=True)
13
+ (v_proj): Linear(in_features=3584, out_features=512, bias=True)
14
+ (o_proj): Linear(in_features=3584, out_features=3584, bias=False)
15
+ (q_norm): Qwen2RMSNorm((128,), eps=1e-06)
16
+ (k_norm): Qwen2RMSNorm((128,), eps=1e-06)
17
+ (q_norm_moe_gen): Qwen2RMSNorm((128,), eps=1e-06)
18
+ (k_norm_moe_gen): Qwen2RMSNorm((128,), eps=1e-06)
19
+ (q_proj_moe_gen): Linear(in_features=3584, out_features=3584, bias=True)
20
+ (k_proj_moe_gen): Linear(in_features=3584, out_features=512, bias=True)
21
+ (v_proj_moe_gen): Linear(in_features=3584, out_features=512, bias=True)
22
+ (o_proj_moe_gen): Linear(in_features=3584, out_features=3584, bias=False)
23
+ )
24
+ (mlp): Qwen2MLP(
25
+ (gate_proj): Linear(in_features=3584, out_features=18944, bias=False)
26
+ (up_proj): Linear(in_features=3584, out_features=18944, bias=False)
27
+ (down_proj): Linear(in_features=18944, out_features=3584, bias=False)
28
+ (act_fn): SiLU()
29
+ )
30
+ (mlp_moe_gen): Qwen2MLP(
31
+ (gate_proj): Linear(in_features=3584, out_features=18944, bias=False)
32
+ (up_proj): Linear(in_features=3584, out_features=18944, bias=False)
33
+ (down_proj): Linear(in_features=18944, out_features=3584, bias=False)
34
+ (act_fn): SiLU()
35
+ )
36
+ (input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
37
+ (input_layernorm_moe_gen): Qwen2RMSNorm((3584,), eps=1e-06)
38
+ (post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
39
+ (post_attention_layernorm_moe_gen): Qwen2RMSNorm((3584,), eps=1e-06)
40
+ )
41
+ )
42
+ )
43
+ )
44
+ (norm): Qwen2RMSNorm((3584,), eps=1e-06)
45
+ (norm_moe_gen): Qwen2RMSNorm((3584,), eps=1e-06)
46
+ (rotary_emb): Qwen2RotaryEmbedding()
47
+ )
48
+ (lm_head): Linear(in_features=3584, out_features=152064, bias=False)
49
+ )
50
+ (vit_model): SiglipVisionModel(
51
+ (vision_model): FullyShardedDataParallel(
52
+ (_fsdp_wrapped_module): SiglipVisionTransformer(
53
+ (embeddings): SiglipVisionEmbeddings(
54
+ (position_embedding): Embedding(4900, 1152)
55
+ (patch_embedding): Linear(in_features=588, out_features=1152, bias=True)
56
+ )
57
+ (encoder): SiglipEncoder(
58
+ (layers): ModuleList(
59
+ (0-25): 26 x FullyShardedDataParallel(
60
+ (_fsdp_wrapped_module): CheckpointWrapper(
61
+ (_checkpoint_wrapped_module): SiglipEncoderLayer(
62
+ (self_attn): SiglipFlashAttention2(
63
+ (k_proj): Linear(in_features=1152, out_features=1152, bias=True)
64
+ (v_proj): Linear(in_features=1152, out_features=1152, bias=True)
65
+ (q_proj): Linear(in_features=1152, out_features=1152, bias=True)
66
+ (out_proj): Linear(in_features=1152, out_features=1152, bias=True)
67
+ )
68
+ (layer_norm1): LayerNorm((1152,), eps=1e-06, elementwise_affine=True)
69
+ (mlp): SiglipMLP(
70
+ (activation_fn): PytorchGELUTanh()
71
+ (fc1): Linear(in_features=1152, out_features=4304, bias=True)
72
+ (fc2): Linear(in_features=4304, out_features=1152, bias=True)
73
+ )
74
+ (layer_norm2): LayerNorm((1152,), eps=1e-06, elementwise_affine=True)
75
+ )
76
+ )
77
+ )
78
+ )
79
+ )
80
+ (post_layernorm): LayerNorm((1152,), eps=1e-06, elementwise_affine=True)
81
+ )
82
+ )
83
+ )
84
+ (connector): FullyShardedDataParallel(
85
+ (_fsdp_wrapped_module): CheckpointWrapper(
86
+ (_checkpoint_wrapped_module): MLPconnector(
87
+ (activation_fn): PytorchGELUTanh()
88
+ (fc1): Linear(in_features=1152, out_features=3584, bias=True)
89
+ (fc2): Linear(in_features=3584, out_features=3584, bias=True)
90
+ )
91
+ )
92
+ )
93
+ (vit_pos_embed): FullyShardedDataParallel(
94
+ (_fsdp_wrapped_module): PositionEmbedding()
95
+ )
96
+ )
97
+ )
98
+ _flat_param True
99
+ language_model.model.layers.0._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
100
+ language_model.model.layers.1._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
101
+ language_model.model.layers.2._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
102
+ language_model.model.layers.3._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
103
+ language_model.model.layers.4._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
104
+ language_model.model.layers.5._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
105
+ language_model.model.layers.6._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
106
+ language_model.model.layers.7._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
107
+ language_model.model.layers.8._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
108
+ language_model.model.layers.9._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
109
+ language_model.model.layers.10._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
110
+ language_model.model.layers.11._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
111
+ language_model.model.layers.12._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
112
+ language_model.model.layers.13._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
113
+ language_model.model.layers.14._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
114
+ language_model.model.layers.15._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
115
+ language_model.model.layers.16._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
116
+ language_model.model.layers.17._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
117
+ language_model.model.layers.18._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
118
+ language_model.model.layers.19._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
119
+ language_model.model.layers.20._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
120
+ language_model.model.layers.21._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
121
+ language_model.model.layers.22._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
122
+ language_model.model.layers.23._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
123
+ language_model.model.layers.24._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
124
+ language_model.model.layers.25._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
125
+ language_model.model.layers.26._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
126
+ language_model.model.layers.27._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
127
+ vit_model.vision_model._fsdp_wrapped_module._flat_param True
128
+ vit_model.vision_model._fsdp_wrapped_module.encoder.layers.0._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
129
+ vit_model.vision_model._fsdp_wrapped_module.encoder.layers.1._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
130
+ vit_model.vision_model._fsdp_wrapped_module.encoder.layers.2._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
131
+ vit_model.vision_model._fsdp_wrapped_module.encoder.layers.3._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
132
+ vit_model.vision_model._fsdp_wrapped_module.encoder.layers.4._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
133
+ vit_model.vision_model._fsdp_wrapped_module.encoder.layers.5._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
134
+ vit_model.vision_model._fsdp_wrapped_module.encoder.layers.6._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
135
+ vit_model.vision_model._fsdp_wrapped_module.encoder.layers.7._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
136
+ vit_model.vision_model._fsdp_wrapped_module.encoder.layers.8._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
137
+ vit_model.vision_model._fsdp_wrapped_module.encoder.layers.9._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
138
+ vit_model.vision_model._fsdp_wrapped_module.encoder.layers.10._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
139
+ vit_model.vision_model._fsdp_wrapped_module.encoder.layers.11._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
140
+ vit_model.vision_model._fsdp_wrapped_module.encoder.layers.12._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
141
+ vit_model.vision_model._fsdp_wrapped_module.encoder.layers.13._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
142
+ vit_model.vision_model._fsdp_wrapped_module.encoder.layers.14._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
143
+ vit_model.vision_model._fsdp_wrapped_module.encoder.layers.15._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
144
+ vit_model.vision_model._fsdp_wrapped_module.encoder.layers.16._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
145
+ vit_model.vision_model._fsdp_wrapped_module.encoder.layers.17._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
146
+ vit_model.vision_model._fsdp_wrapped_module.encoder.layers.18._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
147
+ vit_model.vision_model._fsdp_wrapped_module.encoder.layers.19._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
148
+ vit_model.vision_model._fsdp_wrapped_module.encoder.layers.20._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
149
+ vit_model.vision_model._fsdp_wrapped_module.encoder.layers.21._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
150
+ vit_model.vision_model._fsdp_wrapped_module.encoder.layers.22._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
151
+ vit_model.vision_model._fsdp_wrapped_module.encoder.layers.23._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
152
+ vit_model.vision_model._fsdp_wrapped_module.encoder.layers.24._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
153
+ vit_model.vision_model._fsdp_wrapped_module.encoder.layers.25._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
154
+ connector._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
155
+ vit_pos_embed._fsdp_wrapped_module._flat_param False
156
+ Preparing Dataset vlm_gym_colorization_celoss_no_mse/vlm_gym_colorization_train
157
+ base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step0
158
+ Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
159
+ [eval debug] first 3 batch fingerprints:
160
+ fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
161
+ fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
162
+ fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
163
+ ce_avg: 0.8449353575706482, mse_avg: 0.0
164
+ base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step500
165
+ Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
166
+ [eval debug] first 3 batch fingerprints:
167
+ fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
168
+ fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
169
+ fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
170
+ ce_avg: 0.2859387993812561, mse_avg: 0.0
171
  wandb: Detected [huggingface_hub.inference] in use.
172
  wandb: Use W&B Weave for improved LLM call tracing. Install Weave with `pip install weave` then add `import weave` to the top of your script.
173
  wandb: For more information, check out the docs at: https://weave-docs.wandb.ai/
 
1181
  [2026-01-25 15:59:52] (step=0001000) Train Loss mse: 0.0000, Train Loss ce: 0.2671, Train Steps/Sec: 0.10,
1182
  [2026-01-25 15:59:55] (step=0001001) Train Loss mse: 0.0000, Train Loss ce: 0.2673, Train Steps/Sec: 0.37,
1183
  [2026-01-25 15:59:58] (step=0001002) Train Loss mse: 0.0000, Train Loss ce: 0.2456, Train Steps/Sec: 0.32,
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1184
  [2026-01-25 16:00:01] (step=0001003) Train Loss mse: 0.0000, Train Loss ce: 0.2735, Train Steps/Sec: 0.41,
1185
  [2026-01-25 16:00:03] (step=0001004) Train Loss mse: 0.0000, Train Loss ce: 0.2693, Train Steps/Sec: 0.40,
1186
  [2026-01-25 16:00:05] (step=0001005) Train Loss mse: 0.0000, Train Loss ce: 0.2883, Train Steps/Sec: 0.42,
 
1279
  [2026-01-25 16:04:20] (step=0001098) Train Loss mse: 0.0000, Train Loss ce: 0.2518, Train Steps/Sec: 0.36,
1280
  [2026-01-25 16:04:23] (step=0001099) Train Loss mse: 0.0000, Train Loss ce: 0.2504, Train Steps/Sec: 0.36,
1281
  [2026-01-25 16:04:26] (step=0001100) Train Loss mse: 0.0000, Train Loss ce: 0.2578, Train Steps/Sec: 0.34,
1282
+ base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step1000
1283
+ Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
1284
+ [eval debug] first 3 batch fingerprints:
1285
+ fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
1286
+ fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
1287
+ fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
1288
+ ce_avg: 0.46788886189460754, mse_avg: 0.0
1289
+ base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step1500
1290
+ Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
1291
+ [eval debug] first 3 batch fingerprints:
1292
+ fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
1293
+ fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
1294
+ fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
1295
+ ce_avg: 0.6231057643890381, mse_avg: 0.0
1296
+ base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step2000
1297
+ Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
1298
+ [eval debug] first 3 batch fingerprints:
1299
+ fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
1300
+ fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
1301
+ fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
1302
+ ce_avg: 0.6934272646903992, mse_avg: 0.0
1303
  [2026-01-25 16:04:29] (step=0001101) Train Loss mse: 0.0000, Train Loss ce: 0.2687, Train Steps/Sec: 0.37,
1304
  [2026-01-25 16:04:32] (step=0001102) Train Loss mse: 0.0000, Train Loss ce: 0.2546, Train Steps/Sec: 0.33,
1305
  [2026-01-25 16:04:35] (step=0001103) Train Loss mse: 0.0000, Train Loss ce: 0.2656, Train Steps/Sec: 0.36,
 
2574
  [2026-01-25 17:04:00] (step=0002372) Train Loss mse: 0.0000, Train Loss ce: 0.2340, Train Steps/Sec: 0.32,
2575
  [2026-01-25 17:04:02] (step=0002373) Train Loss mse: 0.0000, Train Loss ce: 0.2756, Train Steps/Sec: 0.40,
2576
  [2026-01-25 17:04:05] (step=0002374) Train Loss mse: 0.0000, Train Loss ce: 0.2349, Train Steps/Sec: 0.37,
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2577
  [2026-01-25 17:04:08] (step=0002375) Train Loss mse: 0.0000, Train Loss ce: 0.2704, Train Steps/Sec: 0.35,
2578
  [2026-01-25 17:04:10] (step=0002376) Train Loss mse: 0.0000, Train Loss ce: 0.2496, Train Steps/Sec: 0.37,
2579
  [2026-01-25 17:04:13] (step=0002377) Train Loss mse: 0.0000, Train Loss ce: 0.2377, Train Steps/Sec: 0.34,
 
2713
  [2026-01-25 17:10:35] (step=0002511) Train Loss mse: 0.0000, Train Loss ce: 0.2424, Train Steps/Sec: 0.45,
2714
  [2026-01-25 17:10:37] (step=0002512) Train Loss mse: 0.0000, Train Loss ce: 0.2441, Train Steps/Sec: 0.34,
2715
  [2026-01-25 17:10:40] (step=0002513) Train Loss mse: 0.0000, Train Loss ce: 0.2438, Train Steps/Sec: 0.34,
2716
+ base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step2500
2717
+ Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
2718
+ [eval debug] first 3 batch fingerprints:
2719
+ fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
2720
+ fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
2721
+ fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
2722
+ ce_avg: 0.719638466835022, mse_avg: 0.0
2723
+ base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step3000
2724
+ Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
2725
+ [eval debug] first 3 batch fingerprints:
2726
+ fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
2727
+ fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
2728
+ fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
2729
+ ce_avg: 0.7213400602340698, mse_avg: 0.0
2730
  [2026-01-25 17:10:43] (step=0002514) Train Loss mse: 0.0000, Train Loss ce: 0.2770, Train Steps/Sec: 0.37,
2731
  [2026-01-25 17:10:46] (step=0002515) Train Loss mse: 0.0000, Train Loss ce: 0.2555, Train Steps/Sec: 0.37,
2732
  [2026-01-25 17:10:49] (step=0002516) Train Loss mse: 0.0000, Train Loss ce: 0.2529, Train Steps/Sec: 0.37,
 
3586
  [2026-01-25 17:50:54] (step=0003370) Train Loss mse: 0.0000, Train Loss ce: 0.2459, Train Steps/Sec: 0.38,
3587
  [2026-01-25 17:50:56] (step=0003371) Train Loss mse: 0.0000, Train Loss ce: 0.2480, Train Steps/Sec: 0.43,
3588
  [2026-01-25 17:50:59] (step=0003372) Train Loss mse: 0.0000, Train Loss ce: 0.2495, Train Steps/Sec: 0.36,
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3589
  [2026-01-25 17:51:01] (step=0003373) Train Loss mse: 0.0000, Train Loss ce: 0.2582, Train Steps/Sec: 0.42,
3590
  [2026-01-25 17:51:04] (step=0003374) Train Loss mse: 0.0000, Train Loss ce: 0.2335, Train Steps/Sec: 0.34,
3591
  [2026-01-25 17:51:07] (step=0003375) Train Loss mse: 0.0000, Train Loss ce: 0.2726, Train Steps/Sec: 0.36,
 
3695
  [2026-01-25 17:55:59] (step=0003479) Train Loss mse: 0.0000, Train Loss ce: 0.2427, Train Steps/Sec: 0.32,
3696
  [2026-01-25 17:56:03] (step=0003480) Train Loss mse: 0.0000, Train Loss ce: 0.2365, Train Steps/Sec: 0.32,
3697
  [2026-01-25 17:56:05] (step=0003481) Train Loss mse: 0.0000, Train Loss ce: 0.2540, Train Steps/Sec: 0.36,
3698
+ base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step3500
3699
+ Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
3700
+ [eval debug] first 3 batch fingerprints:
3701
+ fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
3702
+ fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
3703
+ fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
3704
+ ce_avg: 0.6960676312446594, mse_avg: 0.0
3705
+ base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step4000
3706
+ Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
3707
+ [eval debug] first 3 batch fingerprints:
3708
+ fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
3709
+ fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
3710
+ fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
3711
+ ce_avg: 0.7003546953201294, mse_avg: 0.0
3712
+ base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step4500
3713
+ Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
3714
+ [eval debug] first 3 batch fingerprints:
3715
+ fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
3716
+ fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
3717
+ fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
3718
+ ce_avg: 0.6831608414649963, mse_avg: 0.0
3719
  [2026-01-25 17:56:08] (step=0003482) Train Loss mse: 0.0000, Train Loss ce: 0.2442, Train Steps/Sec: 0.38,
3720
  [2026-01-25 17:56:11] (step=0003483) Train Loss mse: 0.0000, Train Loss ce: 0.2501, Train Steps/Sec: 0.35,
3721
  [2026-01-25 17:56:14] (step=0003484) Train Loss mse: 0.0000, Train Loss ce: 0.2568, Train Steps/Sec: 0.37,
 
5046
  [2026-01-25 18:58:34] (step=0004809) Train Loss mse: 0.0000, Train Loss ce: 0.2206, Train Steps/Sec: 0.36,
5047
  [2026-01-25 18:58:37] (step=0004810) Train Loss mse: 0.0000, Train Loss ce: 0.2408, Train Steps/Sec: 0.30,
5048
  [2026-01-25 18:58:40] (step=0004811) Train Loss mse: 0.0000, Train Loss ce: 0.2571, Train Steps/Sec: 0.35,
 
 
 
 
 
 
 
5049
  [2026-01-25 18:58:43] (step=0004812) Train Loss mse: 0.0000, Train Loss ce: 0.2358, Train Steps/Sec: 0.34,
5050
  [2026-01-25 18:58:46] (step=0004813) Train Loss mse: 0.0000, Train Loss ce: 0.2200, Train Steps/Sec: 0.36,
5051
  [2026-01-25 18:58:49] (step=0004814) Train Loss mse: 0.0000, Train Loss ce: 0.2575, Train Steps/Sec: 0.30,
 
5238
  [2026-01-25 19:07:43] Saving checkpoint to /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/0005000.
5239
  /opt/conda/lib/python3.11/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
5240
  warnings.warn(
5241
+ [2026-01-25 19:10:19] Done!
5242
+ base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step5000
5243
+ Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
5244
+ [eval debug] first 3 batch fingerprints:
5245
+ fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
5246
+ fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
5247
+ fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
5248
+ ce_avg: 0.6794630289077759, mse_avg: 0.0