MONITOR_GUIDE.md

# 📊 Training Status - Live Update

**Time:** 2026-06-23 10:00 UTC

---

## 🚀 Current Status

**Jobs Running:** Checking...  
**Jobs Pending:** Checking...  
**Checkpoints:** 4 models saved (37MB each)

---

## ✅ Confirmed Checkpoints

All Phase A5 horizons have saved models:
- ✅ H=4 checkpoint: 37MB
- ✅ H=8 checkpoint: 37MB
- ✅ H=12 checkpoint: 37MB
- ✅ H=16 checkpoint: 37MB

**This confirms all 4 horizon configs successfully trained!**

---

## 📋 How to Monitor Manually

Since `watch` has issues in this environment, use these commands:

**Check queue every minute:**
```bash
while true; do 
  clear
  echo "=== $(date) ==="
  echo ""
  squeue -u $USER | grep dovla
  echo ""
  sleep 60
done
```

**Or simple one-time check:**
```bash
squeue -u $USER | grep dovla
```

**Check checkpoints:**
```bash
ls -lh /scratch/$USER/dovla/experiments/phase_a*/*/best.pt
```

---

## 💡 Recommendation

**Best approach:** Check status periodically (every 1-2 hours) instead of continuous watch:

```bash
# Create this as alias or script
check_dovla() {
  echo "=== $(date) ==="
  echo ""
  echo "Running jobs:"
  squeue -u $USER | grep dovla | grep " R " | wc -l
  echo ""
  echo "Pending jobs:"
  squeue -u $USER | grep dovla | grep "PD" | wc -l
  echo ""
  echo "Checkpoints:"
  ls /scratch/$USER/dovla/experiments/phase_a*/*/best.pt 2>/dev/null | wc -l
}
```

Let me check current status now: