Setup and more markdown

README.md

@@ -13,6 +13,11 @@ Check out the configuration reference at https://huggingface.co/docs/hub/spaces-
 
 This is the backend for the Enflow application, which allows law enforcement agencies to build, maintain, and use specific workflows that automate tasks based on officers' daily logs.
 
+## Deployment Status
+
+The backend is currently deployed and running at:
+- **API Endpoint**: [https://huggingface.co/spaces/droov/enflow-api](https://huggingface.co/spaces/droov/enflow-api)
+
 ## Environment Setup
 
 The backend requires several environment variables to function properly. These can be set up in a `.env` file in the root of the backend directory.
@@ -32,10 +37,17 @@ Then edit the `.env` file with your actual credentials:
 - `CLOUDINARY_CLOUD_NAME`: Cloudinary cloud name for file storage
 - `CLOUDINARY_API_KEY`: Cloudinary API key
 - `CLOUDINARY_API_SECRET`: Cloudinary API secret
-- `REDIS_URL`: Redis URL for Celery task queue
+- `REDIS_HOST`: Hostname or IP address of your Redis server
+- `REDIS_PORT`: Redis port (default: 6379)
+- `REDIS_PASSWORD`: Password for Redis authentication
 - `OPENAI_API_KEY`: OpenAI API key for LLM processing
 - `FLASK_ENV`: Set to "development" or "production"
 
+Alternatively, you can set `REDIS_URL` directly as:
+```
+REDIS_URL=redis://:{password}@{host}:{port}/0
+```
+
 ### Important Security Notes
 
 - **Never commit the `.env` file to version control**
@@ -67,7 +79,7 @@ The API endpoints are organized by resource type:
 - `/api/logs`: Log upload and management
 - `/api/incidents`: Incident management
 
-For detailed API documentation, see the API documentation section.
+For detailed API documentation, see the `API_DOCUMENTATION.md` file.
 
 ## Setup Instructions
 
@@ -76,6 +88,7 @@ For detailed API documentation, see the API documentation section.
 - Python 3.10 or newer
 - Docker and Docker Compose (optional, for containerized deployment)
 - MongoDB (we're using MongoDB Atlas in the current setup)
+- Redis server (for Celery task queue)
 
 ### Environment Setup
 
@@ -93,6 +106,9 @@ For detailed API documentation, see the API documentation section.
    CLOUDINARY_API_KEY=your_cloudinary_api_key
    CLOUDINARY_API_SECRET=your_cloudinary_api_secret
    OPENAI_API_KEY=your_openai_api_key
+   REDIS_HOST=your_redis_host
+   REDIS_PORT=your_redis_port
+   REDIS_PASSWORD=your_redis_password
    FLASK_ENV=development
    ```
 
@@ -127,20 +143,15 @@ For detailed API documentation, see the API documentation section.
 
 ### HuggingFace Deployment
 
-1. Make sure the HuggingFace Space is set to use the Docker SDK in your Space settings.
-
-2. Update the environment variables in your HuggingFace Space settings:
-   - Go to the Space settings
-   - Add all the required environment variables (MONGO_URI, JWT_SECRET, etc.)
-
-3. Before deploying, you can test if the deployment will work:
-   ```
-   python test_hf_deployment.py
-   ```
+The backend is already deployed to HuggingFace Spaces at: [https://huggingface.co/spaces/droov/enflow-api](https://huggingface.co/spaces/droov/enflow-api)
 
-4. Push your changes to the HuggingFace repository.
+For detailed deployment instructions, see the `hugginface_setup.md` file.
 
-5. If you encounter permissions errors related to file creation, check the logs and ensure the app is configured to write to `/tmp` directory which should be writable.
+Key points for HuggingFace deployment:
+1. Set your Space to use the Docker SDK
+2. Add all environment variables in the Space settings
+3. Make sure the Redis configuration is properly set up
+4. The API will be available at your HuggingFace Space URL
 
 ## Project Structure
 
@@ -150,9 +161,13 @@ For detailed API documentation, see the API documentation section.
 - `controllers/` - Controller functions
 - `routes/` - API route definitions
 - `utils/` - Utility functions and middleware
+  - `celery_tasks.py` - Celery task definitions
+- `celery_config.py` - Celery configuration for task management
 - `setup_env.py` - Script to set up environment variables
 - `test_department.py` - Test script for department creation
 - `test_auth.py` - Test script for authentication
 - `test_hf_deployment.py` - Test script for HuggingFace deployment
 - `Dockerfile` - Docker configuration for containerized deployment
 - `docker-compose.yml` - Docker Compose configuration for local development
+- `API_DOCUMENTATION.md` - Detailed API documentation
+- `hugginface_setup.md` - Guide for deploying to HuggingFace Spaces

REDIS_SETUP.md

@@ -0,0 +1,85 @@
+# Redis Configuration for Enflow
+
+This document explains how to configure Redis for Enflow's background task processing with Celery.
+
+## Overview
+
+Enflow uses Redis as a message broker and result backend for Celery, which handles background tasks like:
+- Processing PDF logs with OCR
+- Running LLM analysis on log content
+- Generating incident forms
+
+## Configuration Options
+
+You can configure Redis connection in two ways:
+
+### Option 1: Using Individual Environment Variables
+
+Set these three environment variables:
+```
+REDIS_HOST=your-redis-hostname-or-ip
+REDIS_PORT=your-redis-port (default: 6379)
+REDIS_PASSWORD=your-redis-password
+```
+
+### Option 2: Using a Single REDIS_URL
+
+Alternatively, set a single `REDIS_URL` environment variable:
+```
+REDIS_URL=redis://:{password}@{host}:{port}/0
+```
+
+Example:
+```
+REDIS_URL=redis://:Nsdg@2314@2314@redis-12345.example.com:6379/0
+```
+
+## Configuration in Celery
+
+The configuration is handled in `celery_config.py`, which constructs the Redis URL from individual components if they're available, or uses the provided `REDIS_URL` directly.
+
+## Monitoring Celery Tasks
+
+You can monitor Celery tasks in several ways:
+
+1. **Celery Logs**: Run the Celery worker with higher verbosity:
+   ```
+   celery -A utils.celery_tasks.celery_app worker --loglevel=info
+   ```
+
+2. **Flower (Optional)**: You can install and run Flower for a web-based monitoring UI:
+   ```
+   pip install flower
+   celery -A utils.celery_tasks.celery_app flower
+   ```
+   Access the Flower dashboard at http://localhost:5555
+
+## Celery Task Configuration
+
+Task-specific configurations are defined in `celery_config.py`, including:
+
+- Rate limits for various tasks
+- Task timeouts
+- Queue routing
+- Serialization formats
+
+## Troubleshooting Redis Connection
+
+If you encounter issues connecting to Redis:
+
+1. Check if your Redis server is running
+2. Verify the host, port, and password are correct
+3. Check if any firewall is blocking the Redis port
+4. Ensure Redis is configured to accept remote connections (if connecting remotely)
+5. Try connecting using the redis-cli tool:
+   ```
+   redis-cli -h your-host -p your-port -a your-password ping
+   ```
+   You should receive "PONG" as the response if the connection is successful
+
+## Security Considerations
+
+1. Use a strong password for Redis authentication
+2. Don't expose your Redis server directly to the internet
+3. Consider using SSL/TLS if connecting to Redis over untrusted networks
+4. Avoid hardcoding Redis credentials in your application code 

env.example

@@ -9,8 +9,15 @@ CLOUDINARY_CLOUD_NAME=your_cloud_name
 CLOUDINARY_API_KEY=your_api_key
 CLOUDINARY_API_SECRET=your_api_secret
 
-# Redis URL for Celery
-REDIS_URL=redis://localhost:6379/0
+# Redis configuration
+# Option 1: Individual components (preferred for better flexibility)
+REDIS_HOST=your_redis_host
+REDIS_PORT=6379
+REDIS_PASSWORD=your_redis_password
+
+# Option 2: Full Redis URL (alternative)
+# If using this, you can comment out the individual components above
+REDIS_URL=redis://:{password}@{host}:{port}/0
 
 # OpenAI API key
 OPENAI_API_KEY=your_openai_api_key

setup_env.py

@@ -90,12 +90,50 @@ def main():
     elif 'CLOUDINARY_API_SECRET' not in env_vars:
         env_vars['CLOUDINARY_API_SECRET'] = "your_api_secret"
     
-    # Redis URL
-    redis_url = input(f"Redis URL [current: {env_vars.get('REDIS_URL', 'redis://localhost:6379/0')}]: ")
-    if redis_url:
-        env_vars['REDIS_URL'] = redis_url
-    elif 'REDIS_URL' not in env_vars:
-        env_vars['REDIS_URL'] = "redis://localhost:6379/0"
+    # Redis Configuration
+    print("\nRedis Configuration (for task queue):")
+    print("Option 1: Configure individual components (recommended)")
+    print("Option 2: Provide full Redis URL")
+    redis_option = input("Which option do you prefer? (1/2) [default: 1]: ")
+    
+    if redis_option == "2":
+        # Redis URL
+        redis_url = input(f"Redis URL [current: {env_vars.get('REDIS_URL', 'redis://localhost:6379/0')}]: ")
+        if redis_url:
+            env_vars['REDIS_URL'] = redis_url
+        elif 'REDIS_URL' not in env_vars:
+            env_vars['REDIS_URL'] = "redis://localhost:6379/0"
+        
+        # Remove individual components if they exist
+        for key in ['REDIS_HOST', 'REDIS_PORT', 'REDIS_PASSWORD']:
+            if key in env_vars:
+                del env_vars[key]
+    else:
+        # Redis Host
+        redis_host = input(f"Redis Host [current: {env_vars.get('REDIS_HOST', 'localhost')}]: ")
+        if redis_host:
+            env_vars['REDIS_HOST'] = redis_host
+        elif 'REDIS_HOST' not in env_vars:
+            env_vars['REDIS_HOST'] = "localhost"
+        
+        # Redis Port
+        redis_port = input(f"Redis Port [current: {env_vars.get('REDIS_PORT', '6379')}]: ")
+        if redis_port:
+            env_vars['REDIS_PORT'] = redis_port
+        elif 'REDIS_PORT' not in env_vars:
+            env_vars['REDIS_PORT'] = "6379"
+        
+        # Redis Password
+        redis_password = getpass.getpass(f"Redis Password [leave empty to keep current]: ")
+        if redis_password:
+            env_vars['REDIS_PASSWORD'] = redis_password
+        elif 'REDIS_PASSWORD' not in env_vars:
+            env_vars['REDIS_PASSWORD'] = ""
+            print("Warning: No Redis password set. This should only be used for local development.")
+        
+        # Remove full URL if it exists
+        if 'REDIS_URL' in env_vars:
+            del env_vars['REDIS_URL']
     
     # OpenAI API Key
     openai_key = getpass.getpass(f"OpenAI API Key [leave empty to keep current]: ")