workspace/MONITOR_GUIDE.md

📊 Training Status - Live Update

Time: 2026-06-23 10:00 UTC

---

🚀 Current Status

Jobs Running: Checking...
Jobs Pending: Checking...
Checkpoints: 4 models saved (37MB each)

---

✅ Confirmed Checkpoints

All Phase A5 horizons have saved models:

• ✅ H=4 checkpoint: 37MB
• ✅ H=8 checkpoint: 37MB
• ✅ H=12 checkpoint: 37MB
• ✅ H=16 checkpoint: 37MB

This confirms all 4 horizon configs successfully trained!

---

📋 How to Monitor Manually

Since watch has issues in this environment, use these commands:

Check queue every minute:

while true; do 
  clear
  echo "=== $(date) ==="
  echo ""
  squeue -u $USER | grep dovla
  echo ""
  sleep 60
done

Or simple one-time check:

squeue -u $USER | grep dovla

Check checkpoints:

ls -lh /scratch/$USER/dovla/experiments/phase_a*/*/best.pt

---

💡 Recommendation

Best approach: Check status periodically (every 1-2 hours) instead of continuous watch:

# Create this as alias or script
check_dovla() {
  echo "=== $(date) ==="
  echo ""
  echo "Running jobs:"
  squeue -u $USER | grep dovla | grep " R " | wc -l
  echo ""
  echo "Pending jobs:"
  squeue -u $USER | grep dovla | grep "PD" | wc -l
  echo ""
  echo "Checkpoints:"
  ls /scratch/$USER/dovla/experiments/phase_a*/*/best.pt 2>/dev/null | wc -l
}

Let me check current status now: