docs/MONITORING.md

Service Monitoring & Auto-Recovery Guide

This guide explains how to ensure the Visa Photo Maker services (Frontend & Backend) remain online and automatically recover from crashes or stops.

1. Watchdog Script

A simple bash script scripts/watchdog.sh has been created to check the health of your services every minute.

• Location: scripts/watchdog.sh
• What it does:
  1. Checks if Port 13002 (Backend) is open.
  2. Checks if Port 13001 (Frontend) is open.
  3. If a port is closed, it triggers pm2 restart <app_name>.
  4. Logs all incidents to watchdog.log in the project root.

2. Setup Instructions

Step 1: Make Script Executable

Run this in your terminal:

chmod +x scripts/watchdog.sh

Step 2: Test the Script

Run it manually once to make sure it works (it should be silent if everything is OK):

./scripts/watchdog.sh

Step 3: Enable Auto-Run (Crontab)

To make it run every minute automatically:

1. Open your crontab editor:

   crontab -e
   

2. Add the following line at the bottom of the file: (Update the path if you moved the project)

   * * * * * /home/ubuntu/code/visa_photo_maker/scripts/watchdog.sh
   

3. Save and exit (usually Ctrl+O, Enter, Ctrl+X if using nano).

3. Viewing Logs

To see if the watchdog has restarted anything, check the log file:

tail -f watchdog.log

4. Other Protections Enabled

Besides this script, the following PM2 protections are already active:

1. Memory Limit: Backend restarts if it exceeds 2GB RAM.
2. Backoff Restart: If the app crashes repeatedly, PM2 waits 100ms+ before restarting to prevent CPU spikes.
3. 4 Workers: The backend runs 4 parallel instances to utilize the quad-core CPU.