Upload checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins
Browse files
checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/wandb/offline-run-20260125_150523-checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins-run0/files/output.log
CHANGED
|
@@ -1,3 +1,173 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
wandb: Detected [huggingface_hub.inference] in use.
|
| 2 |
wandb: Use W&B Weave for improved LLM call tracing. Install Weave with `pip install weave` then add `import weave` to the top of your script.
|
| 3 |
wandb: For more information, check out the docs at: https://weave-docs.wandb.ai/
|
|
@@ -1011,183 +1181,6 @@ wandb: For more information, check out the docs at: https://weave-docs.wandb.ai/
|
|
| 1011 |
[[34m2026-01-25 15:59:52[39m] (step=0001000) Train Loss mse: 0.0000, Train Loss ce: 0.2671, Train Steps/Sec: 0.10,
|
| 1012 |
[[34m2026-01-25 15:59:55[39m] (step=0001001) Train Loss mse: 0.0000, Train Loss ce: 0.2673, Train Steps/Sec: 0.37,
|
| 1013 |
[[34m2026-01-25 15:59:58[39m] (step=0001002) Train Loss mse: 0.0000, Train Loss ce: 0.2456, Train Steps/Sec: 0.32,
|
| 1014 |
-
FullyShardedDataParallel(
|
| 1015 |
-
(_fsdp_wrapped_module): Bagel(
|
| 1016 |
-
(language_model): Qwen2ForCausalLM(
|
| 1017 |
-
(model): Qwen2Model(
|
| 1018 |
-
(embed_tokens): Embedding(152064, 3584)
|
| 1019 |
-
(layers): ModuleList(
|
| 1020 |
-
(0-27): 28 x FullyShardedDataParallel(
|
| 1021 |
-
(_fsdp_wrapped_module): CheckpointWrapper(
|
| 1022 |
-
(_checkpoint_wrapped_module): Qwen2MoTDecoderLayer(
|
| 1023 |
-
(self_attn): PackedAttentionMoT(
|
| 1024 |
-
(q_proj): Linear(in_features=3584, out_features=3584, bias=True)
|
| 1025 |
-
(k_proj): Linear(in_features=3584, out_features=512, bias=True)
|
| 1026 |
-
(v_proj): Linear(in_features=3584, out_features=512, bias=True)
|
| 1027 |
-
(o_proj): Linear(in_features=3584, out_features=3584, bias=False)
|
| 1028 |
-
(q_norm): Qwen2RMSNorm((128,), eps=1e-06)
|
| 1029 |
-
(k_norm): Qwen2RMSNorm((128,), eps=1e-06)
|
| 1030 |
-
(q_norm_moe_gen): Qwen2RMSNorm((128,), eps=1e-06)
|
| 1031 |
-
(k_norm_moe_gen): Qwen2RMSNorm((128,), eps=1e-06)
|
| 1032 |
-
(q_proj_moe_gen): Linear(in_features=3584, out_features=3584, bias=True)
|
| 1033 |
-
(k_proj_moe_gen): Linear(in_features=3584, out_features=512, bias=True)
|
| 1034 |
-
(v_proj_moe_gen): Linear(in_features=3584, out_features=512, bias=True)
|
| 1035 |
-
(o_proj_moe_gen): Linear(in_features=3584, out_features=3584, bias=False)
|
| 1036 |
-
)
|
| 1037 |
-
(mlp): Qwen2MLP(
|
| 1038 |
-
(gate_proj): Linear(in_features=3584, out_features=18944, bias=False)
|
| 1039 |
-
(up_proj): Linear(in_features=3584, out_features=18944, bias=False)
|
| 1040 |
-
(down_proj): Linear(in_features=18944, out_features=3584, bias=False)
|
| 1041 |
-
(act_fn): SiLU()
|
| 1042 |
-
)
|
| 1043 |
-
(mlp_moe_gen): Qwen2MLP(
|
| 1044 |
-
(gate_proj): Linear(in_features=3584, out_features=18944, bias=False)
|
| 1045 |
-
(up_proj): Linear(in_features=3584, out_features=18944, bias=False)
|
| 1046 |
-
(down_proj): Linear(in_features=18944, out_features=3584, bias=False)
|
| 1047 |
-
(act_fn): SiLU()
|
| 1048 |
-
)
|
| 1049 |
-
(input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
|
| 1050 |
-
(input_layernorm_moe_gen): Qwen2RMSNorm((3584,), eps=1e-06)
|
| 1051 |
-
(post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
|
| 1052 |
-
(post_attention_layernorm_moe_gen): Qwen2RMSNorm((3584,), eps=1e-06)
|
| 1053 |
-
)
|
| 1054 |
-
)
|
| 1055 |
-
)
|
| 1056 |
-
)
|
| 1057 |
-
(norm): Qwen2RMSNorm((3584,), eps=1e-06)
|
| 1058 |
-
(norm_moe_gen): Qwen2RMSNorm((3584,), eps=1e-06)
|
| 1059 |
-
(rotary_emb): Qwen2RotaryEmbedding()
|
| 1060 |
-
)
|
| 1061 |
-
(lm_head): Linear(in_features=3584, out_features=152064, bias=False)
|
| 1062 |
-
)
|
| 1063 |
-
(vit_model): SiglipVisionModel(
|
| 1064 |
-
(vision_model): FullyShardedDataParallel(
|
| 1065 |
-
(_fsdp_wrapped_module): SiglipVisionTransformer(
|
| 1066 |
-
(embeddings): SiglipVisionEmbeddings(
|
| 1067 |
-
(position_embedding): Embedding(4900, 1152)
|
| 1068 |
-
(patch_embedding): Linear(in_features=588, out_features=1152, bias=True)
|
| 1069 |
-
)
|
| 1070 |
-
(encoder): SiglipEncoder(
|
| 1071 |
-
(layers): ModuleList(
|
| 1072 |
-
(0-25): 26 x FullyShardedDataParallel(
|
| 1073 |
-
(_fsdp_wrapped_module): CheckpointWrapper(
|
| 1074 |
-
(_checkpoint_wrapped_module): SiglipEncoderLayer(
|
| 1075 |
-
(self_attn): SiglipFlashAttention2(
|
| 1076 |
-
(k_proj): Linear(in_features=1152, out_features=1152, bias=True)
|
| 1077 |
-
(v_proj): Linear(in_features=1152, out_features=1152, bias=True)
|
| 1078 |
-
(q_proj): Linear(in_features=1152, out_features=1152, bias=True)
|
| 1079 |
-
(out_proj): Linear(in_features=1152, out_features=1152, bias=True)
|
| 1080 |
-
)
|
| 1081 |
-
(layer_norm1): LayerNorm((1152,), eps=1e-06, elementwise_affine=True)
|
| 1082 |
-
(mlp): SiglipMLP(
|
| 1083 |
-
(activation_fn): PytorchGELUTanh()
|
| 1084 |
-
(fc1): Linear(in_features=1152, out_features=4304, bias=True)
|
| 1085 |
-
(fc2): Linear(in_features=4304, out_features=1152, bias=True)
|
| 1086 |
-
)
|
| 1087 |
-
(layer_norm2): LayerNorm((1152,), eps=1e-06, elementwise_affine=True)
|
| 1088 |
-
)
|
| 1089 |
-
)
|
| 1090 |
-
)
|
| 1091 |
-
)
|
| 1092 |
-
)
|
| 1093 |
-
(post_layernorm): LayerNorm((1152,), eps=1e-06, elementwise_affine=True)
|
| 1094 |
-
)
|
| 1095 |
-
)
|
| 1096 |
-
)
|
| 1097 |
-
(connector): FullyShardedDataParallel(
|
| 1098 |
-
(_fsdp_wrapped_module): CheckpointWrapper(
|
| 1099 |
-
(_checkpoint_wrapped_module): MLPconnector(
|
| 1100 |
-
(activation_fn): PytorchGELUTanh()
|
| 1101 |
-
(fc1): Linear(in_features=1152, out_features=3584, bias=True)
|
| 1102 |
-
(fc2): Linear(in_features=3584, out_features=3584, bias=True)
|
| 1103 |
-
)
|
| 1104 |
-
)
|
| 1105 |
-
)
|
| 1106 |
-
(vit_pos_embed): FullyShardedDataParallel(
|
| 1107 |
-
(_fsdp_wrapped_module): PositionEmbedding()
|
| 1108 |
-
)
|
| 1109 |
-
)
|
| 1110 |
-
)
|
| 1111 |
-
_flat_param True
|
| 1112 |
-
language_model.model.layers.0._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1113 |
-
language_model.model.layers.1._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1114 |
-
language_model.model.layers.2._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1115 |
-
language_model.model.layers.3._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1116 |
-
language_model.model.layers.4._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1117 |
-
language_model.model.layers.5._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1118 |
-
language_model.model.layers.6._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1119 |
-
language_model.model.layers.7._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1120 |
-
language_model.model.layers.8._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1121 |
-
language_model.model.layers.9._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1122 |
-
language_model.model.layers.10._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1123 |
-
language_model.model.layers.11._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1124 |
-
language_model.model.layers.12._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1125 |
-
language_model.model.layers.13._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1126 |
-
language_model.model.layers.14._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1127 |
-
language_model.model.layers.15._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1128 |
-
language_model.model.layers.16._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1129 |
-
language_model.model.layers.17._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1130 |
-
language_model.model.layers.18._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1131 |
-
language_model.model.layers.19._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1132 |
-
language_model.model.layers.20._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1133 |
-
language_model.model.layers.21._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1134 |
-
language_model.model.layers.22._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1135 |
-
language_model.model.layers.23._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1136 |
-
language_model.model.layers.24._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1137 |
-
language_model.model.layers.25._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1138 |
-
language_model.model.layers.26._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1139 |
-
language_model.model.layers.27._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1140 |
-
vit_model.vision_model._fsdp_wrapped_module._flat_param True
|
| 1141 |
-
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.0._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1142 |
-
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.1._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1143 |
-
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.2._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1144 |
-
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.3._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1145 |
-
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.4._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1146 |
-
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.5._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1147 |
-
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.6._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1148 |
-
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.7._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1149 |
-
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.8._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1150 |
-
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.9._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1151 |
-
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.10._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1152 |
-
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.11._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1153 |
-
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.12._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1154 |
-
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.13._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1155 |
-
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.14._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1156 |
-
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.15._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1157 |
-
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.16._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1158 |
-
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.17._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1159 |
-
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.18._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1160 |
-
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.19._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1161 |
-
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.20._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1162 |
-
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.21._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1163 |
-
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.22._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1164 |
-
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.23._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1165 |
-
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.24._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1166 |
-
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.25._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1167 |
-
connector._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 1168 |
-
vit_pos_embed._fsdp_wrapped_module._flat_param False
|
| 1169 |
-
Preparing Dataset vlm_gym_colorization_celoss_no_mse/vlm_gym_colorization_train
|
| 1170 |
-
base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step0
|
| 1171 |
-
Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
|
| 1172 |
-
[eval debug] first 3 batch fingerprints:
|
| 1173 |
-
fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 1174 |
-
fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 1175 |
-
fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 1176 |
-
ce_avg: 0.8449353575706482, mse_avg: 0.0
|
| 1177 |
-
base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step500
|
| 1178 |
-
Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
|
| 1179 |
-
[eval debug] first 3 batch fingerprints:
|
| 1180 |
-
fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 1181 |
-
fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 1182 |
-
fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 1183 |
-
ce_avg: 0.2859387993812561, mse_avg: 0.0
|
| 1184 |
-
base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step1000
|
| 1185 |
-
Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
|
| 1186 |
-
[eval debug] first 3 batch fingerprints:
|
| 1187 |
-
fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 1188 |
-
fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 1189 |
-
fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 1190 |
-
ce_avg: 0.46788886189460754, mse_avg: 0.0
|
| 1191 |
[[34m2026-01-25 16:00:01[39m] (step=0001003) Train Loss mse: 0.0000, Train Loss ce: 0.2735, Train Steps/Sec: 0.41,
|
| 1192 |
[[34m2026-01-25 16:00:03[39m] (step=0001004) Train Loss mse: 0.0000, Train Loss ce: 0.2693, Train Steps/Sec: 0.40,
|
| 1193 |
[[34m2026-01-25 16:00:05[39m] (step=0001005) Train Loss mse: 0.0000, Train Loss ce: 0.2883, Train Steps/Sec: 0.42,
|
|
@@ -1286,6 +1279,27 @@ ce_avg: 0.46788886189460754, mse_avg: 0.0
|
|
| 1286 |
[[34m2026-01-25 16:04:20[39m] (step=0001098) Train Loss mse: 0.0000, Train Loss ce: 0.2518, Train Steps/Sec: 0.36,
|
| 1287 |
[[34m2026-01-25 16:04:23[39m] (step=0001099) Train Loss mse: 0.0000, Train Loss ce: 0.2504, Train Steps/Sec: 0.36,
|
| 1288 |
[[34m2026-01-25 16:04:26[39m] (step=0001100) Train Loss mse: 0.0000, Train Loss ce: 0.2578, Train Steps/Sec: 0.34,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1289 |
[[34m2026-01-25 16:04:29[39m] (step=0001101) Train Loss mse: 0.0000, Train Loss ce: 0.2687, Train Steps/Sec: 0.37,
|
| 1290 |
[[34m2026-01-25 16:04:32[39m] (step=0001102) Train Loss mse: 0.0000, Train Loss ce: 0.2546, Train Steps/Sec: 0.33,
|
| 1291 |
[[34m2026-01-25 16:04:35[39m] (step=0001103) Train Loss mse: 0.0000, Train Loss ce: 0.2656, Train Steps/Sec: 0.36,
|
|
@@ -2560,20 +2574,6 @@ ce_avg: 0.46788886189460754, mse_avg: 0.0
|
|
| 2560 |
[[34m2026-01-25 17:04:00[39m] (step=0002372) Train Loss mse: 0.0000, Train Loss ce: 0.2340, Train Steps/Sec: 0.32,
|
| 2561 |
[[34m2026-01-25 17:04:02[39m] (step=0002373) Train Loss mse: 0.0000, Train Loss ce: 0.2756, Train Steps/Sec: 0.40,
|
| 2562 |
[[34m2026-01-25 17:04:05[39m] (step=0002374) Train Loss mse: 0.0000, Train Loss ce: 0.2349, Train Steps/Sec: 0.37,
|
| 2563 |
-
base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step1500
|
| 2564 |
-
Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
|
| 2565 |
-
[eval debug] first 3 batch fingerprints:
|
| 2566 |
-
fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 2567 |
-
fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 2568 |
-
fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 2569 |
-
ce_avg: 0.6231057643890381, mse_avg: 0.0
|
| 2570 |
-
base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step2000
|
| 2571 |
-
Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
|
| 2572 |
-
[eval debug] first 3 batch fingerprints:
|
| 2573 |
-
fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 2574 |
-
fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 2575 |
-
fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 2576 |
-
ce_avg: 0.6934272646903992, mse_avg: 0.0
|
| 2577 |
[[34m2026-01-25 17:04:08[39m] (step=0002375) Train Loss mse: 0.0000, Train Loss ce: 0.2704, Train Steps/Sec: 0.35,
|
| 2578 |
[[34m2026-01-25 17:04:10[39m] (step=0002376) Train Loss mse: 0.0000, Train Loss ce: 0.2496, Train Steps/Sec: 0.37,
|
| 2579 |
[[34m2026-01-25 17:04:13[39m] (step=0002377) Train Loss mse: 0.0000, Train Loss ce: 0.2377, Train Steps/Sec: 0.34,
|
|
@@ -2713,6 +2713,20 @@ ce_avg: 0.6934272646903992, mse_avg: 0.0
|
|
| 2713 |
[[34m2026-01-25 17:10:35[39m] (step=0002511) Train Loss mse: 0.0000, Train Loss ce: 0.2424, Train Steps/Sec: 0.45,
|
| 2714 |
[[34m2026-01-25 17:10:37[39m] (step=0002512) Train Loss mse: 0.0000, Train Loss ce: 0.2441, Train Steps/Sec: 0.34,
|
| 2715 |
[[34m2026-01-25 17:10:40[39m] (step=0002513) Train Loss mse: 0.0000, Train Loss ce: 0.2438, Train Steps/Sec: 0.34,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2716 |
[[34m2026-01-25 17:10:43[39m] (step=0002514) Train Loss mse: 0.0000, Train Loss ce: 0.2770, Train Steps/Sec: 0.37,
|
| 2717 |
[[34m2026-01-25 17:10:46[39m] (step=0002515) Train Loss mse: 0.0000, Train Loss ce: 0.2555, Train Steps/Sec: 0.37,
|
| 2718 |
[[34m2026-01-25 17:10:49[39m] (step=0002516) Train Loss mse: 0.0000, Train Loss ce: 0.2529, Train Steps/Sec: 0.37,
|
|
@@ -3572,41 +3586,6 @@ ce_avg: 0.6934272646903992, mse_avg: 0.0
|
|
| 3572 |
[[34m2026-01-25 17:50:54[39m] (step=0003370) Train Loss mse: 0.0000, Train Loss ce: 0.2459, Train Steps/Sec: 0.38,
|
| 3573 |
[[34m2026-01-25 17:50:56[39m] (step=0003371) Train Loss mse: 0.0000, Train Loss ce: 0.2480, Train Steps/Sec: 0.43,
|
| 3574 |
[[34m2026-01-25 17:50:59[39m] (step=0003372) Train Loss mse: 0.0000, Train Loss ce: 0.2495, Train Steps/Sec: 0.36,
|
| 3575 |
-
base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step2500
|
| 3576 |
-
Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
|
| 3577 |
-
[eval debug] first 3 batch fingerprints:
|
| 3578 |
-
fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 3579 |
-
fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 3580 |
-
fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 3581 |
-
ce_avg: 0.719638466835022, mse_avg: 0.0
|
| 3582 |
-
base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step3000
|
| 3583 |
-
Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
|
| 3584 |
-
[eval debug] first 3 batch fingerprints:
|
| 3585 |
-
fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 3586 |
-
fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 3587 |
-
fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 3588 |
-
ce_avg: 0.7213400602340698, mse_avg: 0.0
|
| 3589 |
-
base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step3500
|
| 3590 |
-
Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
|
| 3591 |
-
[eval debug] first 3 batch fingerprints:
|
| 3592 |
-
fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 3593 |
-
fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 3594 |
-
fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 3595 |
-
ce_avg: 0.6960676312446594, mse_avg: 0.0
|
| 3596 |
-
base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step4000
|
| 3597 |
-
Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
|
| 3598 |
-
[eval debug] first 3 batch fingerprints:
|
| 3599 |
-
fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 3600 |
-
fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 3601 |
-
fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 3602 |
-
ce_avg: 0.7003546953201294, mse_avg: 0.0
|
| 3603 |
-
base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step4500
|
| 3604 |
-
Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
|
| 3605 |
-
[eval debug] first 3 batch fingerprints:
|
| 3606 |
-
fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 3607 |
-
fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 3608 |
-
fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 3609 |
-
ce_avg: 0.6831608414649963, mse_avg: 0.0
|
| 3610 |
[[34m2026-01-25 17:51:01[39m] (step=0003373) Train Loss mse: 0.0000, Train Loss ce: 0.2582, Train Steps/Sec: 0.42,
|
| 3611 |
[[34m2026-01-25 17:51:04[39m] (step=0003374) Train Loss mse: 0.0000, Train Loss ce: 0.2335, Train Steps/Sec: 0.34,
|
| 3612 |
[[34m2026-01-25 17:51:07[39m] (step=0003375) Train Loss mse: 0.0000, Train Loss ce: 0.2726, Train Steps/Sec: 0.36,
|
|
@@ -3716,6 +3695,27 @@ ce_avg: 0.6831608414649963, mse_avg: 0.0
|
|
| 3716 |
[[34m2026-01-25 17:55:59[39m] (step=0003479) Train Loss mse: 0.0000, Train Loss ce: 0.2427, Train Steps/Sec: 0.32,
|
| 3717 |
[[34m2026-01-25 17:56:03[39m] (step=0003480) Train Loss mse: 0.0000, Train Loss ce: 0.2365, Train Steps/Sec: 0.32,
|
| 3718 |
[[34m2026-01-25 17:56:05[39m] (step=0003481) Train Loss mse: 0.0000, Train Loss ce: 0.2540, Train Steps/Sec: 0.36,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3719 |
[[34m2026-01-25 17:56:08[39m] (step=0003482) Train Loss mse: 0.0000, Train Loss ce: 0.2442, Train Steps/Sec: 0.38,
|
| 3720 |
[[34m2026-01-25 17:56:11[39m] (step=0003483) Train Loss mse: 0.0000, Train Loss ce: 0.2501, Train Steps/Sec: 0.35,
|
| 3721 |
[[34m2026-01-25 17:56:14[39m] (step=0003484) Train Loss mse: 0.0000, Train Loss ce: 0.2568, Train Steps/Sec: 0.37,
|
|
@@ -5046,13 +5046,6 @@ ce_avg: 0.6831608414649963, mse_avg: 0.0
|
|
| 5046 |
[[34m2026-01-25 18:58:34[39m] (step=0004809) Train Loss mse: 0.0000, Train Loss ce: 0.2206, Train Steps/Sec: 0.36,
|
| 5047 |
[[34m2026-01-25 18:58:37[39m] (step=0004810) Train Loss mse: 0.0000, Train Loss ce: 0.2408, Train Steps/Sec: 0.30,
|
| 5048 |
[[34m2026-01-25 18:58:40[39m] (step=0004811) Train Loss mse: 0.0000, Train Loss ce: 0.2571, Train Steps/Sec: 0.35,
|
| 5049 |
-
base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step5000
|
| 5050 |
-
Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
|
| 5051 |
-
[eval debug] first 3 batch fingerprints:
|
| 5052 |
-
fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 5053 |
-
fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 5054 |
-
fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 5055 |
-
ce_avg: 0.6794630289077759, mse_avg: 0.0
|
| 5056 |
[[34m2026-01-25 18:58:43[39m] (step=0004812) Train Loss mse: 0.0000, Train Loss ce: 0.2358, Train Steps/Sec: 0.34,
|
| 5057 |
[[34m2026-01-25 18:58:46[39m] (step=0004813) Train Loss mse: 0.0000, Train Loss ce: 0.2200, Train Steps/Sec: 0.36,
|
| 5058 |
[[34m2026-01-25 18:58:49[39m] (step=0004814) Train Loss mse: 0.0000, Train Loss ce: 0.2575, Train Steps/Sec: 0.30,
|
|
@@ -5245,4 +5238,11 @@ ce_avg: 0.6794630289077759, mse_avg: 0.0
|
|
| 5245 |
[[34m2026-01-25 19:07:43[39m] Saving checkpoint to /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/0005000.
|
| 5246 |
/opt/conda/lib/python3.11/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
|
| 5247 |
warnings.warn(
|
| 5248 |
-
[[34m2026-01-25 19:10:19[39m] Done!
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
FullyShardedDataParallel(
|
| 2 |
+
(_fsdp_wrapped_module): Bagel(
|
| 3 |
+
(language_model): Qwen2ForCausalLM(
|
| 4 |
+
(model): Qwen2Model(
|
| 5 |
+
(embed_tokens): Embedding(152064, 3584)
|
| 6 |
+
(layers): ModuleList(
|
| 7 |
+
(0-27): 28 x FullyShardedDataParallel(
|
| 8 |
+
(_fsdp_wrapped_module): CheckpointWrapper(
|
| 9 |
+
(_checkpoint_wrapped_module): Qwen2MoTDecoderLayer(
|
| 10 |
+
(self_attn): PackedAttentionMoT(
|
| 11 |
+
(q_proj): Linear(in_features=3584, out_features=3584, bias=True)
|
| 12 |
+
(k_proj): Linear(in_features=3584, out_features=512, bias=True)
|
| 13 |
+
(v_proj): Linear(in_features=3584, out_features=512, bias=True)
|
| 14 |
+
(o_proj): Linear(in_features=3584, out_features=3584, bias=False)
|
| 15 |
+
(q_norm): Qwen2RMSNorm((128,), eps=1e-06)
|
| 16 |
+
(k_norm): Qwen2RMSNorm((128,), eps=1e-06)
|
| 17 |
+
(q_norm_moe_gen): Qwen2RMSNorm((128,), eps=1e-06)
|
| 18 |
+
(k_norm_moe_gen): Qwen2RMSNorm((128,), eps=1e-06)
|
| 19 |
+
(q_proj_moe_gen): Linear(in_features=3584, out_features=3584, bias=True)
|
| 20 |
+
(k_proj_moe_gen): Linear(in_features=3584, out_features=512, bias=True)
|
| 21 |
+
(v_proj_moe_gen): Linear(in_features=3584, out_features=512, bias=True)
|
| 22 |
+
(o_proj_moe_gen): Linear(in_features=3584, out_features=3584, bias=False)
|
| 23 |
+
)
|
| 24 |
+
(mlp): Qwen2MLP(
|
| 25 |
+
(gate_proj): Linear(in_features=3584, out_features=18944, bias=False)
|
| 26 |
+
(up_proj): Linear(in_features=3584, out_features=18944, bias=False)
|
| 27 |
+
(down_proj): Linear(in_features=18944, out_features=3584, bias=False)
|
| 28 |
+
(act_fn): SiLU()
|
| 29 |
+
)
|
| 30 |
+
(mlp_moe_gen): Qwen2MLP(
|
| 31 |
+
(gate_proj): Linear(in_features=3584, out_features=18944, bias=False)
|
| 32 |
+
(up_proj): Linear(in_features=3584, out_features=18944, bias=False)
|
| 33 |
+
(down_proj): Linear(in_features=18944, out_features=3584, bias=False)
|
| 34 |
+
(act_fn): SiLU()
|
| 35 |
+
)
|
| 36 |
+
(input_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
|
| 37 |
+
(input_layernorm_moe_gen): Qwen2RMSNorm((3584,), eps=1e-06)
|
| 38 |
+
(post_attention_layernorm): Qwen2RMSNorm((3584,), eps=1e-06)
|
| 39 |
+
(post_attention_layernorm_moe_gen): Qwen2RMSNorm((3584,), eps=1e-06)
|
| 40 |
+
)
|
| 41 |
+
)
|
| 42 |
+
)
|
| 43 |
+
)
|
| 44 |
+
(norm): Qwen2RMSNorm((3584,), eps=1e-06)
|
| 45 |
+
(norm_moe_gen): Qwen2RMSNorm((3584,), eps=1e-06)
|
| 46 |
+
(rotary_emb): Qwen2RotaryEmbedding()
|
| 47 |
+
)
|
| 48 |
+
(lm_head): Linear(in_features=3584, out_features=152064, bias=False)
|
| 49 |
+
)
|
| 50 |
+
(vit_model): SiglipVisionModel(
|
| 51 |
+
(vision_model): FullyShardedDataParallel(
|
| 52 |
+
(_fsdp_wrapped_module): SiglipVisionTransformer(
|
| 53 |
+
(embeddings): SiglipVisionEmbeddings(
|
| 54 |
+
(position_embedding): Embedding(4900, 1152)
|
| 55 |
+
(patch_embedding): Linear(in_features=588, out_features=1152, bias=True)
|
| 56 |
+
)
|
| 57 |
+
(encoder): SiglipEncoder(
|
| 58 |
+
(layers): ModuleList(
|
| 59 |
+
(0-25): 26 x FullyShardedDataParallel(
|
| 60 |
+
(_fsdp_wrapped_module): CheckpointWrapper(
|
| 61 |
+
(_checkpoint_wrapped_module): SiglipEncoderLayer(
|
| 62 |
+
(self_attn): SiglipFlashAttention2(
|
| 63 |
+
(k_proj): Linear(in_features=1152, out_features=1152, bias=True)
|
| 64 |
+
(v_proj): Linear(in_features=1152, out_features=1152, bias=True)
|
| 65 |
+
(q_proj): Linear(in_features=1152, out_features=1152, bias=True)
|
| 66 |
+
(out_proj): Linear(in_features=1152, out_features=1152, bias=True)
|
| 67 |
+
)
|
| 68 |
+
(layer_norm1): LayerNorm((1152,), eps=1e-06, elementwise_affine=True)
|
| 69 |
+
(mlp): SiglipMLP(
|
| 70 |
+
(activation_fn): PytorchGELUTanh()
|
| 71 |
+
(fc1): Linear(in_features=1152, out_features=4304, bias=True)
|
| 72 |
+
(fc2): Linear(in_features=4304, out_features=1152, bias=True)
|
| 73 |
+
)
|
| 74 |
+
(layer_norm2): LayerNorm((1152,), eps=1e-06, elementwise_affine=True)
|
| 75 |
+
)
|
| 76 |
+
)
|
| 77 |
+
)
|
| 78 |
+
)
|
| 79 |
+
)
|
| 80 |
+
(post_layernorm): LayerNorm((1152,), eps=1e-06, elementwise_affine=True)
|
| 81 |
+
)
|
| 82 |
+
)
|
| 83 |
+
)
|
| 84 |
+
(connector): FullyShardedDataParallel(
|
| 85 |
+
(_fsdp_wrapped_module): CheckpointWrapper(
|
| 86 |
+
(_checkpoint_wrapped_module): MLPconnector(
|
| 87 |
+
(activation_fn): PytorchGELUTanh()
|
| 88 |
+
(fc1): Linear(in_features=1152, out_features=3584, bias=True)
|
| 89 |
+
(fc2): Linear(in_features=3584, out_features=3584, bias=True)
|
| 90 |
+
)
|
| 91 |
+
)
|
| 92 |
+
)
|
| 93 |
+
(vit_pos_embed): FullyShardedDataParallel(
|
| 94 |
+
(_fsdp_wrapped_module): PositionEmbedding()
|
| 95 |
+
)
|
| 96 |
+
)
|
| 97 |
+
)
|
| 98 |
+
_flat_param True
|
| 99 |
+
language_model.model.layers.0._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 100 |
+
language_model.model.layers.1._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 101 |
+
language_model.model.layers.2._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 102 |
+
language_model.model.layers.3._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 103 |
+
language_model.model.layers.4._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 104 |
+
language_model.model.layers.5._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 105 |
+
language_model.model.layers.6._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 106 |
+
language_model.model.layers.7._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 107 |
+
language_model.model.layers.8._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 108 |
+
language_model.model.layers.9._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 109 |
+
language_model.model.layers.10._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 110 |
+
language_model.model.layers.11._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 111 |
+
language_model.model.layers.12._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 112 |
+
language_model.model.layers.13._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 113 |
+
language_model.model.layers.14._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 114 |
+
language_model.model.layers.15._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 115 |
+
language_model.model.layers.16._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 116 |
+
language_model.model.layers.17._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 117 |
+
language_model.model.layers.18._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 118 |
+
language_model.model.layers.19._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 119 |
+
language_model.model.layers.20._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 120 |
+
language_model.model.layers.21._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 121 |
+
language_model.model.layers.22._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 122 |
+
language_model.model.layers.23._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 123 |
+
language_model.model.layers.24._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 124 |
+
language_model.model.layers.25._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 125 |
+
language_model.model.layers.26._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 126 |
+
language_model.model.layers.27._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 127 |
+
vit_model.vision_model._fsdp_wrapped_module._flat_param True
|
| 128 |
+
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.0._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 129 |
+
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.1._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 130 |
+
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.2._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 131 |
+
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.3._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 132 |
+
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.4._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 133 |
+
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.5._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 134 |
+
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.6._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 135 |
+
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.7._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 136 |
+
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.8._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 137 |
+
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.9._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 138 |
+
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.10._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 139 |
+
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.11._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 140 |
+
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.12._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 141 |
+
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.13._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 142 |
+
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.14._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 143 |
+
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.15._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 144 |
+
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.16._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 145 |
+
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.17._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 146 |
+
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.18._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 147 |
+
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.19._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 148 |
+
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.20._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 149 |
+
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.21._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 150 |
+
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.22._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 151 |
+
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.23._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 152 |
+
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.24._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 153 |
+
vit_model.vision_model._fsdp_wrapped_module.encoder.layers.25._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 154 |
+
connector._fsdp_wrapped_module._checkpoint_wrapped_module._flat_param True
|
| 155 |
+
vit_pos_embed._fsdp_wrapped_module._flat_param False
|
| 156 |
+
Preparing Dataset vlm_gym_colorization_celoss_no_mse/vlm_gym_colorization_train
|
| 157 |
+
base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step0
|
| 158 |
+
Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
|
| 159 |
+
[eval debug] first 3 batch fingerprints:
|
| 160 |
+
fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 161 |
+
fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 162 |
+
fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 163 |
+
ce_avg: 0.8449353575706482, mse_avg: 0.0
|
| 164 |
+
base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step500
|
| 165 |
+
Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
|
| 166 |
+
[eval debug] first 3 batch fingerprints:
|
| 167 |
+
fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 168 |
+
fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 169 |
+
fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 170 |
+
ce_avg: 0.2859387993812561, mse_avg: 0.0
|
| 171 |
wandb: Detected [huggingface_hub.inference] in use.
|
| 172 |
wandb: Use W&B Weave for improved LLM call tracing. Install Weave with `pip install weave` then add `import weave` to the top of your script.
|
| 173 |
wandb: For more information, check out the docs at: https://weave-docs.wandb.ai/
|
|
|
|
| 1181 |
[[34m2026-01-25 15:59:52[39m] (step=0001000) Train Loss mse: 0.0000, Train Loss ce: 0.2671, Train Steps/Sec: 0.10,
|
| 1182 |
[[34m2026-01-25 15:59:55[39m] (step=0001001) Train Loss mse: 0.0000, Train Loss ce: 0.2673, Train Steps/Sec: 0.37,
|
| 1183 |
[[34m2026-01-25 15:59:58[39m] (step=0001002) Train Loss mse: 0.0000, Train Loss ce: 0.2456, Train Steps/Sec: 0.32,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1184 |
[[34m2026-01-25 16:00:01[39m] (step=0001003) Train Loss mse: 0.0000, Train Loss ce: 0.2735, Train Steps/Sec: 0.41,
|
| 1185 |
[[34m2026-01-25 16:00:03[39m] (step=0001004) Train Loss mse: 0.0000, Train Loss ce: 0.2693, Train Steps/Sec: 0.40,
|
| 1186 |
[[34m2026-01-25 16:00:05[39m] (step=0001005) Train Loss mse: 0.0000, Train Loss ce: 0.2883, Train Steps/Sec: 0.42,
|
|
|
|
| 1279 |
[[34m2026-01-25 16:04:20[39m] (step=0001098) Train Loss mse: 0.0000, Train Loss ce: 0.2518, Train Steps/Sec: 0.36,
|
| 1280 |
[[34m2026-01-25 16:04:23[39m] (step=0001099) Train Loss mse: 0.0000, Train Loss ce: 0.2504, Train Steps/Sec: 0.36,
|
| 1281 |
[[34m2026-01-25 16:04:26[39m] (step=0001100) Train Loss mse: 0.0000, Train Loss ce: 0.2578, Train Steps/Sec: 0.34,
|
| 1282 |
+
base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step1000
|
| 1283 |
+
Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
|
| 1284 |
+
[eval debug] first 3 batch fingerprints:
|
| 1285 |
+
fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 1286 |
+
fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 1287 |
+
fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 1288 |
+
ce_avg: 0.46788886189460754, mse_avg: 0.0
|
| 1289 |
+
base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step1500
|
| 1290 |
+
Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
|
| 1291 |
+
[eval debug] first 3 batch fingerprints:
|
| 1292 |
+
fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 1293 |
+
fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 1294 |
+
fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 1295 |
+
ce_avg: 0.6231057643890381, mse_avg: 0.0
|
| 1296 |
+
base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step2000
|
| 1297 |
+
Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
|
| 1298 |
+
[eval debug] first 3 batch fingerprints:
|
| 1299 |
+
fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 1300 |
+
fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 1301 |
+
fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 1302 |
+
ce_avg: 0.6934272646903992, mse_avg: 0.0
|
| 1303 |
[[34m2026-01-25 16:04:29[39m] (step=0001101) Train Loss mse: 0.0000, Train Loss ce: 0.2687, Train Steps/Sec: 0.37,
|
| 1304 |
[[34m2026-01-25 16:04:32[39m] (step=0001102) Train Loss mse: 0.0000, Train Loss ce: 0.2546, Train Steps/Sec: 0.33,
|
| 1305 |
[[34m2026-01-25 16:04:35[39m] (step=0001103) Train Loss mse: 0.0000, Train Loss ce: 0.2656, Train Steps/Sec: 0.36,
|
|
|
|
| 2574 |
[[34m2026-01-25 17:04:00[39m] (step=0002372) Train Loss mse: 0.0000, Train Loss ce: 0.2340, Train Steps/Sec: 0.32,
|
| 2575 |
[[34m2026-01-25 17:04:02[39m] (step=0002373) Train Loss mse: 0.0000, Train Loss ce: 0.2756, Train Steps/Sec: 0.40,
|
| 2576 |
[[34m2026-01-25 17:04:05[39m] (step=0002374) Train Loss mse: 0.0000, Train Loss ce: 0.2349, Train Steps/Sec: 0.37,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2577 |
[[34m2026-01-25 17:04:08[39m] (step=0002375) Train Loss mse: 0.0000, Train Loss ce: 0.2704, Train Steps/Sec: 0.35,
|
| 2578 |
[[34m2026-01-25 17:04:10[39m] (step=0002376) Train Loss mse: 0.0000, Train Loss ce: 0.2496, Train Steps/Sec: 0.37,
|
| 2579 |
[[34m2026-01-25 17:04:13[39m] (step=0002377) Train Loss mse: 0.0000, Train Loss ce: 0.2377, Train Steps/Sec: 0.34,
|
|
|
|
| 2713 |
[[34m2026-01-25 17:10:35[39m] (step=0002511) Train Loss mse: 0.0000, Train Loss ce: 0.2424, Train Steps/Sec: 0.45,
|
| 2714 |
[[34m2026-01-25 17:10:37[39m] (step=0002512) Train Loss mse: 0.0000, Train Loss ce: 0.2441, Train Steps/Sec: 0.34,
|
| 2715 |
[[34m2026-01-25 17:10:40[39m] (step=0002513) Train Loss mse: 0.0000, Train Loss ce: 0.2438, Train Steps/Sec: 0.34,
|
| 2716 |
+
base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step2500
|
| 2717 |
+
Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
|
| 2718 |
+
[eval debug] first 3 batch fingerprints:
|
| 2719 |
+
fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 2720 |
+
fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 2721 |
+
fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 2722 |
+
ce_avg: 0.719638466835022, mse_avg: 0.0
|
| 2723 |
+
base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step3000
|
| 2724 |
+
Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
|
| 2725 |
+
[eval debug] first 3 batch fingerprints:
|
| 2726 |
+
fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 2727 |
+
fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 2728 |
+
fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 2729 |
+
ce_avg: 0.7213400602340698, mse_avg: 0.0
|
| 2730 |
[[34m2026-01-25 17:10:43[39m] (step=0002514) Train Loss mse: 0.0000, Train Loss ce: 0.2770, Train Steps/Sec: 0.37,
|
| 2731 |
[[34m2026-01-25 17:10:46[39m] (step=0002515) Train Loss mse: 0.0000, Train Loss ce: 0.2555, Train Steps/Sec: 0.37,
|
| 2732 |
[[34m2026-01-25 17:10:49[39m] (step=0002516) Train Loss mse: 0.0000, Train Loss ce: 0.2529, Train Steps/Sec: 0.37,
|
|
|
|
| 3586 |
[[34m2026-01-25 17:50:54[39m] (step=0003370) Train Loss mse: 0.0000, Train Loss ce: 0.2459, Train Steps/Sec: 0.38,
|
| 3587 |
[[34m2026-01-25 17:50:56[39m] (step=0003371) Train Loss mse: 0.0000, Train Loss ce: 0.2480, Train Steps/Sec: 0.43,
|
| 3588 |
[[34m2026-01-25 17:50:59[39m] (step=0003372) Train Loss mse: 0.0000, Train Loss ce: 0.2495, Train Steps/Sec: 0.36,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3589 |
[[34m2026-01-25 17:51:01[39m] (step=0003373) Train Loss mse: 0.0000, Train Loss ce: 0.2582, Train Steps/Sec: 0.42,
|
| 3590 |
[[34m2026-01-25 17:51:04[39m] (step=0003374) Train Loss mse: 0.0000, Train Loss ce: 0.2335, Train Steps/Sec: 0.34,
|
| 3591 |
[[34m2026-01-25 17:51:07[39m] (step=0003375) Train Loss mse: 0.0000, Train Loss ce: 0.2726, Train Steps/Sec: 0.36,
|
|
|
|
| 3695 |
[[34m2026-01-25 17:55:59[39m] (step=0003479) Train Loss mse: 0.0000, Train Loss ce: 0.2427, Train Steps/Sec: 0.32,
|
| 3696 |
[[34m2026-01-25 17:56:03[39m] (step=0003480) Train Loss mse: 0.0000, Train Loss ce: 0.2365, Train Steps/Sec: 0.32,
|
| 3697 |
[[34m2026-01-25 17:56:05[39m] (step=0003481) Train Loss mse: 0.0000, Train Loss ce: 0.2540, Train Steps/Sec: 0.36,
|
| 3698 |
+
base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step3500
|
| 3699 |
+
Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
|
| 3700 |
+
[eval debug] first 3 batch fingerprints:
|
| 3701 |
+
fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 3702 |
+
fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 3703 |
+
fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 3704 |
+
ce_avg: 0.6960676312446594, mse_avg: 0.0
|
| 3705 |
+
base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step4000
|
| 3706 |
+
Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
|
| 3707 |
+
[eval debug] first 3 batch fingerprints:
|
| 3708 |
+
fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 3709 |
+
fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 3710 |
+
fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 3711 |
+
ce_avg: 0.7003546953201294, mse_avg: 0.0
|
| 3712 |
+
base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step4500
|
| 3713 |
+
Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
|
| 3714 |
+
[eval debug] first 3 batch fingerprints:
|
| 3715 |
+
fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 3716 |
+
fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 3717 |
+
fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 3718 |
+
ce_avg: 0.6831608414649963, mse_avg: 0.0
|
| 3719 |
[[34m2026-01-25 17:56:08[39m] (step=0003482) Train Loss mse: 0.0000, Train Loss ce: 0.2442, Train Steps/Sec: 0.38,
|
| 3720 |
[[34m2026-01-25 17:56:11[39m] (step=0003483) Train Loss mse: 0.0000, Train Loss ce: 0.2501, Train Steps/Sec: 0.35,
|
| 3721 |
[[34m2026-01-25 17:56:14[39m] (step=0003484) Train Loss mse: 0.0000, Train Loss ce: 0.2568, Train Steps/Sec: 0.37,
|
|
|
|
| 5046 |
[[34m2026-01-25 18:58:34[39m] (step=0004809) Train Loss mse: 0.0000, Train Loss ce: 0.2206, Train Steps/Sec: 0.36,
|
| 5047 |
[[34m2026-01-25 18:58:37[39m] (step=0004810) Train Loss mse: 0.0000, Train Loss ce: 0.2408, Train Steps/Sec: 0.30,
|
| 5048 |
[[34m2026-01-25 18:58:40[39m] (step=0004811) Train Loss mse: 0.0000, Train Loss ce: 0.2571, Train Steps/Sec: 0.35,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 5049 |
[[34m2026-01-25 18:58:43[39m] (step=0004812) Train Loss mse: 0.0000, Train Loss ce: 0.2358, Train Steps/Sec: 0.34,
|
| 5050 |
[[34m2026-01-25 18:58:46[39m] (step=0004813) Train Loss mse: 0.0000, Train Loss ce: 0.2200, Train Steps/Sec: 0.36,
|
| 5051 |
[[34m2026-01-25 18:58:49[39m] (step=0004814) Train Loss mse: 0.0000, Train Loss ce: 0.2575, Train Steps/Sec: 0.30,
|
|
|
|
| 5238 |
[[34m2026-01-25 19:07:43[39m] Saving checkpoint to /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/0005000.
|
| 5239 |
/opt/conda/lib/python3.11/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:690: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
|
| 5240 |
warnings.warn(
|
| 5241 |
+
[[34m2026-01-25 19:10:19[39m] Done!
|
| 5242 |
+
base_dir is /dev/shm/models/checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins/eval_used_rows, step_tag is checkpoints_vlm_gym_colorization_one_image_lr2e_5_ce_no_mse_ins_step5000
|
| 5243 |
+
Preparing Dataset vlm_gym_colorization_celoss_no_mse_evalonce/vlm_gym_colorization_val
|
| 5244 |
+
[eval debug] first 3 batch fingerprints:
|
| 5245 |
+
fp[0]: [{'data_indexes': [0], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 5246 |
+
fp[1]: [{'data_indexes': [8], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 5247 |
+
fp[2]: [{'data_indexes': [16], 'worker_id': 0, 'dataset_name': 'vlm_gym_colorization_celoss_no_mse_evalonce'}]
|
| 5248 |
+
ce_avg: 0.6794630289077759, mse_avg: 0.0
|