| 2025-11-09 19:05:41,604 - INFO - |
| ╔══════════════════════════════════════════════════════════╗ |
| ║ T5 TRAINING CONFIGURATION ║ |
| ╚══════════════════════════════════════════════════════════╝ |
| Mode: FULL |
| Platform: vast |
| Repository: ranjan56cse/t5-base-xsum-lora |
| Epochs: 3 |
| Samples: ALL (204k) |
| Batch size: 16 |
| Gradient accum: 2 |
| Effective batch: 32 |
| Save every: 1000 steps |
| Expected time: ~8-10 hours |
|
|
| 2025-11-09 19:05:41,604 - INFO - Creating repository: ranjan56cse/t5-base-xsum-lora |
| 2025-11-09 19:05:41,807 - INFO - ✅ Repo: https://huggingface.co/ranjan56cse/t5-base-xsum-lora |
| 2025-11-09 19:05:41,807 - INFO - Loading google-t5/t5-base... |
| 2025-11-09 19:05:52,938 - INFO - ✅ Gradient checkpointing enabled |
| 2025-11-09 19:05:52,938 - INFO - Applying LoRA... |
| 2025-11-09 19:05:52,976 - INFO - Loading XSum dataset... |
| 2025-11-09 19:05:56,588 - INFO - ✅ Dataset: 204045 train, 11332 val |
| 2025-11-09 19:05:56,588 - INFO - Tokenizing... |
| 2025-11-09 19:08:02,802 - INFO - ✅ Tokenization complete |
| 2025-11-09 19:08:03,857 - INFO - ============================================================ |
| 2025-11-09 19:08:03,858 - INFO - 🚀 STARTING TRAINING (~8-10 hours) |
| 2025-11-09 19:08:03,859 - INFO - Effective batch size: 32 |
| 2025-11-09 19:08:03,859 - INFO - GPU: 0.84GB allocated, 0.92GB reserved |
| 2025-11-09 19:08:03,859 - INFO - System: 4.1% used (17.4GB / 503.7GB) |
| 2025-11-09 19:08:03,859 - INFO - ============================================================ |
| 2025-11-09 19:08:03,990 - INFO - ============================================================ |
| 2025-11-09 19:08:03,990 - INFO - 🚀 Training started |
| 2025-11-09 19:08:03,990 - INFO - Total steps: 19128 |
| 2025-11-09 19:08:03,990 - INFO - GPU: NVIDIA GeForce RTX 3090 |
| 2025-11-09 19:08:03,990 - INFO - GPU Memory: 0.84GB allocated, 0.92GB reserved |
| 2025-11-09 19:08:03,990 - INFO - System Memory: 4.1% used (17.4GB / 503.7GB) |
| 2025-11-09 19:08:03,991 - INFO - ============================================================ |
| 2025-11-09 19:08:51,266 - INFO - Step 50/19128 | Loss: 12.5022 | LR: 2.88e-05 | GPU: 0.87GB |
| 2025-11-09 19:09:38,077 - INFO - Step 100/19128 | Loss: 10.3469 | LR: 5.82e-05 | GPU: 0.87GB |
| 2025-11-09 19:10:24,938 - INFO - Step 150/19128 | Loss: 4.0200 | LR: 8.82e-05 | GPU: 0.87GB |
| 2025-11-09 19:11:11,674 - INFO - Step 200/19128 | Loss: 0.9201 | LR: 1.18e-04 | GPU: 0.87GB |
| 2025-11-09 19:11:58,405 - INFO - Step 250/19128 | Loss: 0.7357 | LR: 1.48e-04 | GPU: 0.87GB |
| 2025-11-09 19:12:45,152 - INFO - Step 300/19128 | Loss: 0.6602 | LR: 1.77e-04 | GPU: 0.87GB |
| 2025-11-09 19:13:31,815 - INFO - Step 350/19128 | Loss: 0.6121 | LR: 2.07e-04 | GPU: 0.87GB |
| 2025-11-09 19:14:18,499 - INFO - Step 400/19128 | Loss: 0.5817 | LR: 2.37e-04 | GPU: 0.87GB |
| 2025-11-09 19:15:05,185 - INFO - Step 450/19128 | Loss: 0.5916 | LR: 2.67e-04 | GPU: 0.87GB |
| 2025-11-09 19:15:51,879 - INFO - Step 500/19128 | Loss: 0.5675 | LR: 2.97e-04 | GPU: 0.87GB |
| 2025-11-09 19:16:38,691 - INFO - Step 550/19128 | Loss: 0.5700 | LR: 2.99e-04 | GPU: 0.87GB |
| 2025-11-09 19:17:25,546 - INFO - Step 600/19128 | Loss: 0.5610 | LR: 2.98e-04 | GPU: 0.87GB |
| 2025-11-09 19:18:12,459 - INFO - Step 650/19128 | Loss: 0.5669 | LR: 2.98e-04 | GPU: 0.87GB |
| 2025-11-09 19:18:59,163 - INFO - Step 700/19128 | Loss: 0.5659 | LR: 2.97e-04 | GPU: 0.87GB |
| 2025-11-09 19:19:45,942 - INFO - Step 750/19128 | Loss: 0.5673 | LR: 2.96e-04 | GPU: 0.87GB |
| 2025-11-09 19:20:32,786 - INFO - Step 800/19128 | Loss: 0.5619 | LR: 2.95e-04 | GPU: 0.87GB |
| 2025-11-09 19:21:19,739 - INFO - Step 850/19128 | Loss: 0.5719 | LR: 2.94e-04 | GPU: 0.87GB |
| 2025-11-09 19:22:06,708 - INFO - Step 900/19128 | Loss: 0.5576 | LR: 2.94e-04 | GPU: 0.87GB |
| 2025-11-09 19:22:53,641 - INFO - Step 950/19128 | Loss: 0.5567 | LR: 2.93e-04 | GPU: 0.87GB |
| 2025-11-09 19:23:40,445 - INFO - Step 1000/19128 | Loss: 0.5597 | LR: 2.92e-04 | GPU: 0.87GB |
| 2025-11-09 19:25:15,772 - INFO - Step 1000/19128 | Loss: 0.0000 | LR: 0.00e+00 | GPU: 0.87GB |
| 2025-11-09 19:25:15,772 - INFO - ============================================================ |
| 2025-11-09 19:25:15,772 - INFO - 📊 EVALUATION at step 1000 |
| 2025-11-09 19:25:15,772 - INFO - eval_loss: 0.5003 |
| 2025-11-09 19:25:15,772 - INFO - eval_runtime: 95.3235 |
| 2025-11-09 19:25:15,772 - INFO - eval_samples_per_second: 118.8790 |
| 2025-11-09 19:25:15,772 - INFO - eval_steps_per_second: 7.4380 |
| 2025-11-09 19:25:15,772 - INFO - epoch: 0.1600 |
| 2025-11-09 19:25:15,773 - INFO - gpu_memory_gb: 0.8662 |
| 2025-11-09 19:25:15,773 - INFO - system_memory_percent: 6.9000 |
| 2025-11-09 19:25:15,773 - INFO - ============================================================ |
| 2025-11-09 19:25:15,773 - INFO - 🏆 New best eval loss: 0.5003 |
| 2025-11-09 19:25:16,038 - INFO - ============================================================ |
| 2025-11-09 19:25:16,038 - INFO - 💾 Checkpoint 1: step 1000 |
| 2025-11-09 19:25:16,038 - INFO - GPU: 0.87GB allocated, 1.15GB reserved |
| 2025-11-09 19:25:16,038 - INFO - 📤 Uploading checkpoint-1000 to Hub... |
| 2025-11-09 19:25:20,110 - INFO - ✅ Checkpoint 1000 uploaded! |
| 2025-11-09 19:25:20,110 - INFO - 📂 https://huggingface.co/ranjan56cse/t5-base-xsum-lora |
| 2025-11-09 19:25:20,110 - INFO - ============================================================ |
| 2025-11-09 19:26:07,015 - INFO - Step 1050/19128 | Loss: 0.5565 | LR: 2.91e-04 | GPU: 0.87GB |
| 2025-11-09 19:26:53,807 - INFO - Step 1100/19128 | Loss: 0.5767 | LR: 2.91e-04 | GPU: 0.87GB |
| 2025-11-09 19:27:40,531 - INFO - Step 1150/19128 | Loss: 0.5620 | LR: 2.90e-04 | GPU: 0.87GB |
| 2025-11-09 19:28:27,359 - INFO - Step 1200/19128 | Loss: 0.5864 | LR: 2.89e-04 | GPU: 0.87GB |
| 2025-11-09 19:29:14,182 - INFO - Step 1250/19128 | Loss: 0.6260 | LR: 2.88e-04 | GPU: 0.87GB |
| 2025-11-09 19:30:01,074 - INFO - Step 1300/19128 | Loss: 0.7742 | LR: 2.87e-04 | GPU: 0.87GB |
| 2025-11-09 19:30:48,073 - INFO - Step 1350/19128 | Loss: 1.1101 | LR: 2.87e-04 | GPU: 0.87GB |
| 2025-11-09 19:31:34,986 - INFO - Step 1400/19128 | Loss: 1.3211 | LR: 2.86e-04 | GPU: 0.87GB |
| 2025-11-09 19:32:21,930 - INFO - Step 1450/19128 | Loss: 1.4130 | LR: 2.85e-04 | GPU: 0.87GB |
| 2025-11-09 19:33:08,830 - INFO - Step 1500/19128 | Loss: 1.4265 | LR: 2.84e-04 | GPU: 0.87GB |
| 2025-11-09 19:33:55,803 - INFO - Step 1550/19128 | Loss: 1.4700 | LR: 2.83e-04 | GPU: 0.87GB |
| 2025-11-09 19:34:42,910 - INFO - Step 1600/19128 | Loss: 1.4561 | LR: 2.83e-04 | GPU: 0.87GB |
| 2025-11-09 19:35:29,939 - INFO - Step 1650/19128 | Loss: 1.4693 | LR: 2.82e-04 | GPU: 0.87GB |
| 2025-11-09 19:36:16,685 - INFO - Step 1700/19128 | Loss: 1.4729 | LR: 2.81e-04 | GPU: 0.87GB |
| 2025-11-09 19:37:03,396 - INFO - Step 1750/19128 | Loss: 1.4599 | LR: 2.80e-04 | GPU: 0.87GB |
| 2025-11-09 19:37:50,039 - INFO - Step 1800/19128 | Loss: 1.4725 | LR: 2.79e-04 | GPU: 0.87GB |
| 2025-11-09 19:38:36,721 - INFO - Step 1850/19128 | Loss: 1.4503 | LR: 2.79e-04 | GPU: 0.87GB |
| 2025-11-09 19:39:23,367 - INFO - Step 1900/19128 | Loss: 1.4812 | LR: 2.78e-04 | GPU: 0.87GB |
| 2025-11-09 19:40:10,030 - INFO - Step 1950/19128 | Loss: 1.4761 | LR: 2.77e-04 | GPU: 0.87GB |
| 2025-11-09 19:40:56,713 - INFO - Step 2000/19128 | Loss: 1.4960 | LR: 2.76e-04 | GPU: 0.87GB |
| 2025-11-09 19:42:31,551 - INFO - Step 2000/19128 | Loss: 0.0000 | LR: 0.00e+00 | GPU: 0.87GB |
| 2025-11-09 19:42:31,551 - INFO - ============================================================ |
| 2025-11-09 19:42:31,551 - INFO - 📊 EVALUATION at step 2000 |
| 2025-11-09 19:42:31,551 - INFO - eval_loss: 1.2512 |
| 2025-11-09 19:42:31,551 - INFO - eval_runtime: 94.8348 |
| 2025-11-09 19:42:31,551 - INFO - eval_samples_per_second: 119.4920 |
| 2025-11-09 19:42:31,551 - INFO - eval_steps_per_second: 7.4760 |
| 2025-11-09 19:42:31,551 - INFO - epoch: 0.3100 |
| 2025-11-09 19:42:31,551 - INFO - gpu_memory_gb: 0.8662 |
| 2025-11-09 19:42:31,551 - INFO - system_memory_percent: 13.2000 |
| 2025-11-09 19:42:31,551 - INFO - ============================================================ |
| 2025-11-09 19:42:31,768 - INFO - ============================================================ |
| 2025-11-09 19:42:31,768 - INFO - 💾 Checkpoint 2: step 2000 |
| 2025-11-09 19:42:31,769 - INFO - GPU: 0.87GB allocated, 1.15GB reserved |
| 2025-11-09 19:42:31,769 - INFO - 📤 Uploading checkpoint-2000 to Hub... |
|
|
