docs/memory_monitoring.md

Monitoring Memory Usage in Production on Render

This document provides guidance on monitoring memory usage in production for the RAG application deployed on Render's free tier, which has a 512MB memory limit.

Integrated Memory Monitoring Tools

The application includes enhanced memory monitoring specifically optimized for Render deployments:

1. Memory Status Endpoint

The application exposes a dedicated endpoint for monitoring memory usage:

GET /memory/render-status

This endpoint returns detailed information about current memory usage, including:

• Current memory usage in MB
• Peak memory usage since startup
• Memory usage trends (5-minute and 1-hour)
• Current memory status (normal, warning, critical, emergency)
• Actions taken if memory thresholds were exceeded

Example response:

{
  "status": "success",
  "is_render": true,
  "memory_status": {
    "timestamp": "2023-10-25T14:32:15.123456",
    "memory_mb": 342.5,
    "peak_memory_mb": 398.2,
    "context": "api_request",
    "status": "warning",
    "action_taken": "light_cleanup",
    "memory_limit_mb": 512.0
  },
  "memory_trends": {
    "current_mb": 342.5,
    "peak_mb": 398.2,
    "samples_count": 356,
    "trend_5min_mb": 12.5,
    "trend_1hour_mb": -24.3
  },
  "render_limit_mb": 512
}

2. Detailed Diagnostics

For more detailed memory diagnostics, use:

GET /memory/diagnostics

This provides a deeper look at memory allocation and usage patterns.

3. Force Memory Cleanup

If you notice memory usage approaching critical levels, use diagnostics and consider scheduled maintenance windows for cleanup or service restarts. Manual force-clean endpoints were removed in favor of safer, observable operations.

Setting Up External Monitoring

Using Uptime Robot or Similar Services

1. Set up a monitor to check the /health endpoint every 5 minutes
2. Set up a separate monitor to check the /memory/render-status endpoint every 15 minutes

Automated Alerting

Configure alerts based on memory thresholds:

1. Warning Alert: When memory usage exceeds 400MB (78% of limit)
2. Critical Alert: When memory usage exceeds 450MB (88% of limit)

Monitoring Logs in Render Dashboard

1. Log into your Render dashboard
2. Navigate to the service logs
3. Filter for memory-related log messages:
   • [MEMORY CHECKPOINT]
   • [MEMORY MILESTONE]
   • Memory usage
   • WARNING: Memory usage
   • CRITICAL: Memory usage

Memory Usage Patterns to Watch For

Warning Signs

1. Steadily Increasing Memory: If memory trends show continuous growth
2. High Peak After Ingestion: Memory spikes above 450MB after document ingestion
3. Failure to Release Memory: Memory doesn't decrease after operations complete

Preventative Actions

1. Regular Cleanup: Schedule low-traffic time for calling /memory/force-clean
2. Batch Processing: For large document sets, ingest in smaller batches
3. Monitoring Before Bulk Operations: Check memory status before starting resource-intensive operations

Memory Optimization Features

The application includes several memory optimization features:

1. Automatic Thresholds: Memory is monitored against configured thresholds (400MB, 450MB, 480MB)
2. Progressive Cleanup: Different levels of cleanup based on severity
3. Request Circuit Breaker: Will reject new requests if memory is critically high
4. Memory Metrics Export: Memory metrics are saved to /tmp/render_metrics/ for later analysis

Troubleshooting Memory Issues

If you encounter persistent memory issues:

1. Review Logs: Check Render logs for memory checkpoints and milestones
2. Analyze Trends: Use the /memory/render-status endpoint to identify patterns
3. Check Operations Timing: High memory could correlate with specific operations
4. Adjust Configuration: Consider adjusting EMBEDDING_BATCH_SIZE or other parameters in config.py

Available Environment Variables

These environment variables can be configured in Render:

• MEMORY_DEBUG=1: Enable detailed memory diagnostics
• MEMORY_LOG_INTERVAL=10: Log memory usage every 10 seconds
• ENABLE_TRACEMALLOC=1: Enable tracemalloc for detailed memory allocation tracking
• RENDER=1: Enable Render-specific optimizations (automatically set on Render)