| 2026-06-24 14:06:37,942 [INFO] Seed: 42 |
| 2026-06-24 14:06:37,943 [INFO] Using CUDA: NVIDIA L40 |
| 2026-06-24 14:06:37,944 [INFO] Enabled TF32 matmul/cudnn for CUDA |
| 2026-06-24 14:06:47,930 [INFO] LLM2Seq Model Summary |
| ================================================== |
| Encoder: McGill-NLP/LLM2Vec-Sheared-LLaMA-mntp |
| Trainable: True |
| LoRA: True |
| Adaptor: gated_residual |
| Layer Fusion: True |
| Fusion Type: tokenwise |
| Salience Gate: True |
| EncStack: True |
| Global Tokens: 16 |
| Decoder: |
| Layers: 8 |
| Hidden Size: 1024 |
| Heads: 16 |
| FFN Size: 4096 |
| ================================================== |
| Total params: 1,558,708,737 |
| Trainable params: 278,821,377 |
| Frozen params: 1,279,887,360 |
| 2026-06-24 14:06:49,899 [INFO] Enabled gradient checkpointing for encoder |
| 2026-06-24 14:06:49,973 [INFO] Resuming from checkpoint: runs/llm2seq_phase1_warmup_vlsp/best.pt |
| 2026-06-24 14:06:50,658 [INFO] Checkpoint load allowed 554 missing keys for stage=phase2_lora_encoder_vlsp context=resume, prefixes=('encoder.',) |
| 2026-06-24 14:06:50,658 [INFO] Reset global step because training.reset_global_step_on_resume=true |
| 2026-06-24 14:06:50,658 [INFO] Reset best eval loss because training.reset_best_eval_loss_on_resume=true |
| 2026-06-24 14:06:50,658 [INFO] Resumed at step 0 |
| 2026-06-24 14:06:50,802 [INFO] Hugging Face uploads enabled: repo=BienKieu/llm2seq-wikilingua path=checkpoints/phase2_lora_encoder_vlsp |
| 2026-06-24 14:06:50,802 [INFO] Train file: llm2seq/data/processed/vlsp/train.jsonl (285 examples) |
| 2026-06-24 14:06:50,802 [INFO] Eval file: llm2seq/data/processed/vlsp/val.jsonl (15 examples) |
| 2026-06-24 14:06:50,803 [INFO] Starting training for 6 epochs (108 optimizer steps)... |
| 2026-06-24 14:06:50,803 [INFO] Batch size: 2 |
| 2026-06-24 14:06:50,803 [INFO] Gradient accumulation: 8 |
| 2026-06-24 14:06:50,803 [INFO] Effective batch size: 16 |
| 2026-06-24 14:06:50,803 [INFO] Steps per epoch: 18 |
| 2026-06-24 14:06:50,803 [INFO] Warmup steps: 5 |
| 2026-06-24 14:06:50,803 [INFO] Eval every: 18 optimizer steps |
| 2026-06-24 14:06:50,803 [INFO] Save every: 18 optimizer steps |
| 2026-06-24 14:06:50,803 [INFO] Mixed precision: fp16=False, bf16=True |
| 2026-06-24 14:06:50,803 [INFO] Initial optimizer LRs: adaptor=0.00e+00, decoder=0.00e+00, encoder=0.00e+00 |
| 2026-06-24 14:06:51,090 [WARNING] `use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`... |
| 2026-06-24 14:08:11,277 [INFO] Step 2/108 | Loss: 3.2700 | CE: 3.2700 | LR: 2.00e-05 | Epoch: 1/6 |
| 2026-06-24 14:09:22,289 [INFO] Step 4/108 | Loss: 3.0470 | CE: 3.0470 | LR: 4.00e-05 | Epoch: 1/6 |
| 2026-06-24 14:10:37,693 [INFO] Step 6/108 | Loss: 3.0143 | CE: 3.0143 | LR: 5.00e-05 | Epoch: 1/6 |
| 2026-06-24 14:11:52,377 [INFO] Step 8/108 | Loss: 3.0606 | CE: 3.0606 | LR: 4.99e-05 | Epoch: 1/6 |
| 2026-06-24 14:13:02,844 [INFO] Step 10/108 | Loss: 2.9571 | CE: 2.9571 | LR: 4.97e-05 | Epoch: 1/6 |
| 2026-06-24 14:14:17,782 [INFO] Step 12/108 | Loss: 2.9607 | CE: 2.9607 | LR: 4.94e-05 | Epoch: 1/6 |
| 2026-06-24 14:15:38,651 [INFO] Step 14/108 | Loss: 2.9912 | CE: 2.9912 | LR: 4.91e-05 | Epoch: 1/6 |
| 2026-06-24 14:16:53,542 [INFO] Step 16/108 | Loss: 2.9489 | CE: 2.9489 | LR: 4.86e-05 | Epoch: 1/6 |
| 2026-06-24 14:18:07,455 [INFO] Step 18/108 | Loss: 2.9488 | CE: 2.9488 | LR: 4.81e-05 | Epoch: 2/6 |
| 2026-06-24 14:18:15,582 [INFO] Step 18 | Eval Loss: 3.0409 |
| 2026-06-24 14:18:16,840 [INFO] Saved trainable-only checkpoint to runs/llm2seq_phase2_lora_encoder_vlsp/best.pt (519 model tensors, stores_base_encoder_weights=False) |
| 2026-06-24 14:18:17,787 [INFO] New best eval loss: 3.0409 |
| 2026-06-24 14:19:31,483 [INFO] Step 20/108 | Loss: 2.9404 | CE: 2.9404 | LR: 4.74e-05 | Epoch: 2/6 |
| 2026-06-24 14:20:48,001 [INFO] Step 22/108 | Loss: 2.9260 | CE: 2.9260 | LR: 4.67e-05 | Epoch: 2/6 |
| 2026-06-24 14:22:01,809 [INFO] Step 24/108 | Loss: 2.9450 | CE: 2.9450 | LR: 4.59e-05 | Epoch: 2/6 |
| 2026-06-24 14:23:14,301 [INFO] Step 26/108 | Loss: 2.9306 | CE: 2.9306 | LR: 4.50e-05 | Epoch: 2/6 |
| 2026-06-24 14:24:27,994 [INFO] Step 28/108 | Loss: 3.0252 | CE: 3.0252 | LR: 4.41e-05 | Epoch: 2/6 |
| 2026-06-24 14:25:45,581 [INFO] Step 30/108 | Loss: 2.9008 | CE: 2.9008 | LR: 4.31e-05 | Epoch: 2/6 |
| 2026-06-24 14:26:58,467 [INFO] Step 32/108 | Loss: 2.9021 | CE: 2.9021 | LR: 4.20e-05 | Epoch: 2/6 |
| 2026-06-24 14:28:08,114 [INFO] Step 34/108 | Loss: 2.9128 | CE: 2.9128 | LR: 4.08e-05 | Epoch: 2/6 |
| 2026-06-24 14:29:22,162 [INFO] Step 36/108 | Loss: 2.9144 | CE: 2.9144 | LR: 3.96e-05 | Epoch: 3/6 |
| 2026-06-24 14:29:30,213 [INFO] Step 36 | Eval Loss: 2.9889 |
| 2026-06-24 14:29:31,698 [INFO] Saved trainable-only checkpoint to runs/llm2seq_phase2_lora_encoder_vlsp/best.pt (519 model tensors, stores_base_encoder_weights=False) |
| 2026-06-24 14:29:32,564 [INFO] New best eval loss: 2.9889 |
| 2026-06-24 14:30:36,412 [INFO] Step 38/108 | Loss: 2.9718 | CE: 2.9718 | LR: 3.84e-05 | Epoch: 3/6 |
| 2026-06-24 14:31:53,450 [INFO] Step 40/108 | Loss: 2.8459 | CE: 2.8459 | LR: 3.71e-05 | Epoch: 3/6 |
| 2026-06-24 14:33:08,703 [INFO] Step 42/108 | Loss: 2.8913 | CE: 2.8913 | LR: 3.57e-05 | Epoch: 3/6 |
| 2026-06-24 14:34:21,487 [INFO] Step 44/108 | Loss: 2.8682 | CE: 2.8682 | LR: 3.43e-05 | Epoch: 3/6 |
| 2026-06-24 14:35:42,083 [INFO] Step 46/108 | Loss: 2.8798 | CE: 2.8798 | LR: 3.29e-05 | Epoch: 3/6 |
| 2026-06-24 14:36:55,269 [INFO] Step 48/108 | Loss: 2.8224 | CE: 2.8224 | LR: 3.14e-05 | Epoch: 3/6 |
| 2026-06-24 14:38:13,496 [INFO] Step 50/108 | Loss: 2.8738 | CE: 2.8738 | LR: 2.99e-05 | Epoch: 3/6 |
| 2026-06-24 14:39:27,025 [INFO] Step 52/108 | Loss: 2.9031 | CE: 2.9031 | LR: 2.84e-05 | Epoch: 3/6 |
| 2026-06-24 14:40:40,554 [INFO] Step 54/108 | Loss: 2.9082 | CE: 2.9082 | LR: 2.69e-05 | Epoch: 4/6 |
| 2026-06-24 14:40:48,984 [INFO] Step 54 | Eval Loss: 2.9596 |
| 2026-06-24 14:40:50,437 [INFO] Saved trainable-only checkpoint to runs/llm2seq_phase2_lora_encoder_vlsp/best.pt (519 model tensors, stores_base_encoder_weights=False) |
| 2026-06-24 14:40:51,281 [INFO] New best eval loss: 2.9596 |
| 2026-06-24 14:42:02,509 [INFO] Step 56/108 | Loss: 2.8164 | CE: 2.8164 | LR: 2.54e-05 | Epoch: 4/6 |
| 2026-06-24 14:43:15,080 [INFO] Step 58/108 | Loss: 2.8943 | CE: 2.8943 | LR: 2.39e-05 | Epoch: 4/6 |
| 2026-06-24 14:44:27,418 [INFO] Step 60/108 | Loss: 2.8097 | CE: 2.8097 | LR: 2.23e-05 | Epoch: 4/6 |
| 2026-06-24 14:45:37,348 [INFO] Step 62/108 | Loss: 2.8616 | CE: 2.8616 | LR: 2.08e-05 | Epoch: 4/6 |
| 2026-06-24 14:46:51,986 [INFO] Step 64/108 | Loss: 2.8705 | CE: 2.8705 | LR: 1.93e-05 | Epoch: 4/6 |
| 2026-06-24 14:48:00,950 [INFO] Step 66/108 | Loss: 2.8447 | CE: 2.8447 | LR: 1.79e-05 | Epoch: 4/6 |
| 2026-06-24 14:49:19,934 [INFO] Step 68/108 | Loss: 2.8467 | CE: 2.8467 | LR: 1.64e-05 | Epoch: 4/6 |
| 2026-06-24 14:50:34,696 [INFO] Step 70/108 | Loss: 2.8358 | CE: 2.8358 | LR: 1.50e-05 | Epoch: 4/6 |
| 2026-06-24 14:51:56,042 [INFO] Step 72/108 | Loss: 2.7892 | CE: 2.7892 | LR: 1.36e-05 | Epoch: 5/6 |
| 2026-06-24 14:52:04,190 [INFO] Step 72 | Eval Loss: 2.9376 |
| 2026-06-24 14:52:05,685 [INFO] Saved trainable-only checkpoint to runs/llm2seq_phase2_lora_encoder_vlsp/best.pt (519 model tensors, stores_base_encoder_weights=False) |
| 2026-06-24 14:52:06,548 [INFO] New best eval loss: 2.9376 |
| 2026-06-24 14:53:22,661 [INFO] Step 74/108 | Loss: 2.7908 | CE: 2.7908 | LR: 1.23e-05 | Epoch: 5/6 |
| 2026-06-24 14:54:26,080 [INFO] Step 76/108 | Loss: 2.8478 | CE: 2.8478 | LR: 1.10e-05 | Epoch: 5/6 |
| 2026-06-24 14:55:38,090 [INFO] Step 78/108 | Loss: 2.8309 | CE: 2.8309 | LR: 9.76e-06 | Epoch: 5/6 |
| 2026-06-24 14:56:57,562 [INFO] Step 80/108 | Loss: 2.7617 | CE: 2.7617 | LR: 8.58e-06 | Epoch: 5/6 |
| 2026-06-24 14:58:20,264 [INFO] Step 82/108 | Loss: 2.7724 | CE: 2.7724 | LR: 7.46e-06 | Epoch: 5/6 |
| 2026-06-24 14:59:34,508 [INFO] Step 84/108 | Loss: 2.8763 | CE: 2.8763 | LR: 6.40e-06 | Epoch: 5/6 |
| 2026-06-24 15:00:48,995 [INFO] Step 86/108 | Loss: 2.8617 | CE: 2.8617 | LR: 5.42e-06 | Epoch: 5/6 |
| 2026-06-24 15:02:00,373 [INFO] Step 88/108 | Loss: 2.8154 | CE: 2.8154 | LR: 5.00e-06 | Epoch: 5/6 |
| 2026-06-24 15:03:15,685 [INFO] Step 90/108 | Loss: 2.8582 | CE: 2.8582 | LR: 5.00e-06 | Epoch: 6/6 |
| 2026-06-24 15:03:23,831 [INFO] Step 90 | Eval Loss: 2.9265 |
| 2026-06-24 15:03:25,339 [INFO] Saved trainable-only checkpoint to runs/llm2seq_phase2_lora_encoder_vlsp/best.pt (519 model tensors, stores_base_encoder_weights=False) |
| 2026-06-24 15:03:26,189 [INFO] New best eval loss: 2.9265 |
| 2026-06-24 15:04:35,572 [INFO] Step 92/108 | Loss: 2.8329 | CE: 2.8329 | LR: 5.00e-06 | Epoch: 6/6 |
| 2026-06-24 15:05:52,443 [INFO] Step 94/108 | Loss: 2.8172 | CE: 2.8172 | LR: 5.00e-06 | Epoch: 6/6 |
| 2026-06-24 15:07:07,737 [INFO] Step 96/108 | Loss: 2.8165 | CE: 2.8165 | LR: 5.00e-06 | Epoch: 6/6 |
| 2026-06-24 15:08:30,961 [INFO] Step 98/108 | Loss: 2.7906 | CE: 2.7906 | LR: 5.00e-06 | Epoch: 6/6 |
| 2026-06-24 15:09:45,770 [INFO] Step 100/108 | Loss: 2.7602 | CE: 2.7602 | LR: 5.00e-06 | Epoch: 6/6 |
| 2026-06-24 15:11:01,349 [INFO] Step 102/108 | Loss: 2.7976 | CE: 2.7976 | LR: 5.00e-06 | Epoch: 6/6 |
| 2026-06-24 15:12:10,294 [INFO] Step 104/108 | Loss: 2.8255 | CE: 2.8255 | LR: 5.00e-06 | Epoch: 6/6 |
| 2026-06-24 15:13:20,827 [INFO] Step 106/108 | Loss: 2.8229 | CE: 2.8229 | LR: 5.00e-06 | Epoch: 6/6 |
| 2026-06-24 15:14:36,336 [INFO] Step 108/108 | Loss: 2.7861 | CE: 2.7861 | LR: 5.00e-06 | Epoch: 7/6 |
| 2026-06-24 15:14:44,484 [INFO] Step 108 | Eval Loss: 2.9224 |
| 2026-06-24 15:14:45,948 [INFO] Saved trainable-only checkpoint to runs/llm2seq_phase2_lora_encoder_vlsp/best.pt (519 model tensors, stores_base_encoder_weights=False) |
| 2026-06-24 15:14:46,808 [INFO] New best eval loss: 2.9224 |
| 2026-06-24 15:14:47,189 [INFO] Uploading to HF: runs/llm2seq_phase2_lora_encoder_vlsp/best.pt -> BienKieu/llm2seq-wikilingua/checkpoints/phase2_lora_encoder_vlsp/best.pt |
| 2026-06-24 15:15:04,465 [INFO] Uploading to HF: runs/llm2seq_phase2_lora_encoder_vlsp/config.yaml -> BienKieu/llm2seq-wikilingua/checkpoints/phase2_lora_encoder_vlsp/config.yaml |
| 2026-06-24 15:15:05,565 [INFO] Uploading to HF: runs/llm2seq_phase2_lora_encoder_vlsp/train.log -> BienKieu/llm2seq-wikilingua/checkpoints/phase2_lora_encoder_vlsp/train.log |
|
|
