File size: 1,411 Bytes
adc02fa | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 | # 📊 Training Status - Live Update
**Time:** 2026-06-23 10:00 UTC
---
## 🚀 Current Status
**Jobs Running:** Checking...
**Jobs Pending:** Checking...
**Checkpoints:** 4 models saved (37MB each)
---
## ✅ Confirmed Checkpoints
All Phase A5 horizons have saved models:
- ✅ H=4 checkpoint: 37MB
- ✅ H=8 checkpoint: 37MB
- ✅ H=12 checkpoint: 37MB
- ✅ H=16 checkpoint: 37MB
**This confirms all 4 horizon configs successfully trained!**
---
## 📋 How to Monitor Manually
Since `watch` has issues in this environment, use these commands:
**Check queue every minute:**
```bash
while true; do
clear
echo "=== $(date) ==="
echo ""
squeue -u $USER | grep dovla
echo ""
sleep 60
done
```
**Or simple one-time check:**
```bash
squeue -u $USER | grep dovla
```
**Check checkpoints:**
```bash
ls -lh /scratch/$USER/dovla/experiments/phase_a*/*/best.pt
```
---
## 💡 Recommendation
**Best approach:** Check status periodically (every 1-2 hours) instead of continuous watch:
```bash
# Create this as alias or script
check_dovla() {
echo "=== $(date) ==="
echo ""
echo "Running jobs:"
squeue -u $USER | grep dovla | grep " R " | wc -l
echo ""
echo "Pending jobs:"
squeue -u $USER | grep dovla | grep "PD" | wc -l
echo ""
echo "Checkpoints:"
ls /scratch/$USER/dovla/experiments/phase_a*/*/best.pt 2>/dev/null | wc -l
}
```
Let me check current status now:
|