Upload logs/train_codegen_20251130_092019.log with huggingface_hub
Browse files
logs/train_codegen_20251130_092019.log
ADDED
|
@@ -0,0 +1,92 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
2025-11-30 09:20:19 - train_codegen - INFO - Logging to: logs/codegen/train_codegen_20251130_092019.log
|
| 2 |
+
2025-11-30 09:20:19 - train_codegen - INFO - Monitor progress: tail -f logs/codegen/train_codegen_20251130_092019.log
|
| 3 |
+
2025-11-30 09:20:19 - train_codegen - INFO - ============================================================
|
| 4 |
+
2025-11-30 09:20:19 - train_codegen - INFO - CodeGen Training
|
| 5 |
+
2025-11-30 09:20:19 - train_codegen - INFO - ============================================================
|
| 6 |
+
2025-11-30 09:20:19 - train_codegen - INFO - Using CUDA device: 0
|
| 7 |
+
2025-11-30 09:20:19 - train_codegen - INFO - GPU: NVIDIA GeForce RTX 5090
|
| 8 |
+
2025-11-30 09:20:19 - train_codegen - INFO - Configuration:
|
| 9 |
+
2025-11-30 09:20:19 - train_codegen - INFO - model: Salesforce/codegen-350M-mono
|
| 10 |
+
2025-11-30 09:20:19 - train_codegen - INFO - data: datasets/csharp
|
| 11 |
+
2025-11-30 09:20:19 - train_codegen - INFO - output: model/checkpoints/run1-csharp-codegen
|
| 12 |
+
2025-11-30 09:20:19 - train_codegen - INFO - batch_size: 10
|
| 13 |
+
2025-11-30 09:20:19 - train_codegen - INFO - gradient_accumulation_steps: 4
|
| 14 |
+
2025-11-30 09:20:19 - train_codegen - INFO - effective_batch_size: 40
|
| 15 |
+
2025-11-30 09:20:19 - train_codegen - INFO - learning_rate: 5e-05
|
| 16 |
+
2025-11-30 09:20:19 - train_codegen - INFO - epochs: 5
|
| 17 |
+
2025-11-30 09:20:19 - train_codegen - INFO - max_length: 1024
|
| 18 |
+
2025-11-30 09:20:19 - train_codegen - INFO - max_steps: -1
|
| 19 |
+
2025-11-30 09:20:19 - train_codegen - INFO - fp16: True
|
| 20 |
+
2025-11-30 09:20:19 - train_codegen - INFO - gradient_checkpointing: True
|
| 21 |
+
2025-11-30 09:20:19 - train_codegen - INFO - seed: 42
|
| 22 |
+
2025-11-30 09:20:19 - train_codegen - INFO - Loading tokenizer and model: Salesforce/codegen-350M-mono
|
| 23 |
+
2025-11-30 09:20:33 - train_codegen - INFO - Loading model with gradient checkpointing enabled
|
| 24 |
+
2025-11-30 09:20:33 - train_codegen - INFO - Loading dataset...
|
| 25 |
+
2025-11-30 09:20:33 - train_codegen - INFO - Loading dataset from datasets/csharp
|
| 26 |
+
2025-11-30 09:20:35 - train_codegen - INFO - Train samples: 226616
|
| 27 |
+
2025-11-30 09:20:35 - train_codegen - INFO - Validation samples: 28327
|
| 28 |
+
2025-11-30 09:20:35 - train_codegen - INFO - ============================================================
|
| 29 |
+
2025-11-30 09:20:35 - train_codegen - INFO - Dataset Preprocessing
|
| 30 |
+
2025-11-30 09:20:35 - train_codegen - INFO - ============================================================
|
| 31 |
+
2025-11-30 09:20:35 - train_codegen - INFO - Preprocessing 226616 samples (optimized eager loading)...
|
| 32 |
+
2025-11-30 09:20:40 - train_codegen - INFO - Preprocessed 10000/226616 samples
|
| 33 |
+
2025-11-30 09:20:45 - train_codegen - INFO - Preprocessed 20000/226616 samples
|
| 34 |
+
2025-11-30 09:20:49 - train_codegen - INFO - Preprocessed 30000/226616 samples
|
| 35 |
+
2025-11-30 09:20:53 - train_codegen - INFO - Preprocessed 40000/226616 samples
|
| 36 |
+
2025-11-30 09:20:56 - train_codegen - INFO - Preprocessed 50000/226616 samples
|
| 37 |
+
2025-11-30 09:21:02 - train_codegen - INFO - Preprocessed 60000/226616 samples
|
| 38 |
+
2025-11-30 09:21:07 - train_codegen - INFO - Preprocessed 70000/226616 samples
|
| 39 |
+
2025-11-30 09:21:11 - train_codegen - INFO - Preprocessed 80000/226616 samples
|
| 40 |
+
2025-11-30 09:21:16 - train_codegen - INFO - Preprocessed 90000/226616 samples
|
| 41 |
+
2025-11-30 09:21:20 - train_codegen - INFO - Preprocessed 100000/226616 samples
|
| 42 |
+
2025-11-30 09:21:25 - train_codegen - INFO - Preprocessed 110000/226616 samples
|
| 43 |
+
2025-11-30 09:21:30 - train_codegen - INFO - Preprocessed 120000/226616 samples
|
| 44 |
+
2025-11-30 09:21:40 - train_codegen - INFO - Preprocessed 130000/226616 samples
|
| 45 |
+
2025-11-30 09:21:45 - train_codegen - INFO - Preprocessed 140000/226616 samples
|
| 46 |
+
2025-11-30 09:21:50 - train_codegen - INFO - Preprocessed 150000/226616 samples
|
| 47 |
+
2025-11-30 09:21:55 - train_codegen - INFO - Preprocessed 160000/226616 samples
|
| 48 |
+
2025-11-30 09:22:00 - train_codegen - INFO - Preprocessed 170000/226616 samples
|
| 49 |
+
2025-11-30 09:22:05 - train_codegen - INFO - Preprocessed 180000/226616 samples
|
| 50 |
+
2025-11-30 09:22:09 - train_codegen - INFO - Preprocessed 190000/226616 samples
|
| 51 |
+
2025-11-30 09:22:14 - train_codegen - INFO - Preprocessed 200000/226616 samples
|
| 52 |
+
2025-11-30 09:22:18 - train_codegen - INFO - Preprocessed 210000/226616 samples
|
| 53 |
+
2025-11-30 09:22:23 - train_codegen - INFO - Preprocessed 220000/226616 samples
|
| 54 |
+
2025-11-30 09:22:26 - train_codegen - INFO - Preprocessed 226616/226616 samples
|
| 55 |
+
2025-11-30 09:22:26 - train_codegen - INFO - Preprocessing complete: 226616 samples ready
|
| 56 |
+
2025-11-30 09:22:26 - train_codegen - INFO - Preprocessing 28327 samples (optimized eager loading)...
|
| 57 |
+
2025-11-30 09:22:33 - train_codegen - INFO - Preprocessed 10000/28327 samples
|
| 58 |
+
2025-11-30 09:22:38 - train_codegen - INFO - Preprocessed 20000/28327 samples
|
| 59 |
+
2025-11-30 09:22:42 - train_codegen - INFO - Preprocessed 28327/28327 samples
|
| 60 |
+
2025-11-30 09:22:42 - train_codegen - INFO - Preprocessing complete: 28327 samples ready
|
| 61 |
+
2025-11-30 09:22:42 - train_codegen - INFO - ============================================================
|
| 62 |
+
2025-11-30 09:22:42 - train_codegen - INFO - Training Arguments
|
| 63 |
+
2025-11-30 09:22:42 - train_codegen - INFO - ============================================================
|
| 64 |
+
2025-11-30 09:22:43 - train_codegen - INFO - Training log will be saved to: model/checkpoints/run1-csharp-codegen/training_log.csv
|
| 65 |
+
2025-11-30 09:22:43 - train_codegen - INFO - ============================================================
|
| 66 |
+
2025-11-30 09:22:43 - train_codegen - INFO - Training Strategy
|
| 67 |
+
2025-11-30 09:22:43 - train_codegen - INFO - ============================================================
|
| 68 |
+
2025-11-30 09:22:43 - train_codegen - INFO - Evaluation every 1000 steps (optimized for speed)
|
| 69 |
+
2025-11-30 09:22:43 - train_codegen - INFO - Eval batch size: 20 (2x train batch)
|
| 70 |
+
2025-11-30 09:22:43 - train_codegen - INFO - Eval accumulation steps: 4
|
| 71 |
+
2025-11-30 09:22:43 - train_codegen - INFO - Save checkpoint every 2000 steps
|
| 72 |
+
2025-11-30 09:22:43 - train_codegen - INFO - Gradient checkpointing: ENABLED (saves VRAM, slower training)
|
| 73 |
+
2025-11-30 09:22:43 - train_codegen - INFO - FP16 mixed precision enabled
|
| 74 |
+
2025-11-30 09:22:43 - train_codegen - INFO - Dynamic padding per batch (10-20x faster than max_length padding)
|
| 75 |
+
2025-11-30 09:22:43 - train_codegen - INFO - ============================================================
|
| 76 |
+
2025-11-30 09:22:43 - train_codegen - INFO - Starting Training
|
| 77 |
+
2025-11-30 09:22:43 - train_codegen - INFO - ============================================================
|
| 78 |
+
2025-11-30 09:22:43 - train_codegen - INFO - Total training samples: 226616
|
| 79 |
+
2025-11-30 09:22:43 - train_codegen - INFO - Total validation samples: 28327
|
| 80 |
+
2025-11-30 09:22:43 - train_codegen - INFO - Starting training from scratch
|
| 81 |
+
2025-12-01 11:09:58 - train_codegen - INFO - Training completed successfully
|
| 82 |
+
2025-12-01 11:09:58 - train_codegen - INFO - ============================================================
|
| 83 |
+
2025-12-01 11:09:58 - train_codegen - INFO - Saving Final Model
|
| 84 |
+
2025-12-01 11:09:58 - train_codegen - INFO - ============================================================
|
| 85 |
+
2025-12-01 11:10:00 - train_codegen - INFO - Model and tokenizer saved to model/checkpoints/run1-csharp-codegen
|
| 86 |
+
2025-12-01 11:10:00 - train_codegen - INFO - ============================================================
|
| 87 |
+
2025-12-01 11:10:00 - train_codegen - INFO - Training Summary
|
| 88 |
+
2025-12-01 11:10:00 - train_codegen - INFO - ============================================================
|
| 89 |
+
2025-12-01 11:10:00 - train_codegen - INFO - Total steps: 28325
|
| 90 |
+
2025-12-01 11:10:00 - train_codegen - INFO - Best model checkpoint: model/checkpoints/run1-csharp-codegen/checkpoint-22000
|
| 91 |
+
2025-12-01 11:10:00 - train_codegen - INFO - Best eval loss: 0.6229148507118225
|
| 92 |
+
2025-12-01 11:10:00 - train_codegen - INFO - Done.
|