Hugging Face's logo

Spaces:
kamau1
/
swiftops-backend
Sleeping

App Files Community
swiftops-backend / docs /research /cloudinary.md
kamau1's picture
kamau1
refactor: unify storage integrations and clean up dead services
d0a01ab
preview code
|
raw
history blame contribute delete
9.33 kB

Cloudinary Image Upload Integration Guide - SwiftOps Backend

Overview

This document outlines the integration of Cloudinary for image management in the SwiftOps backend. Our backend will handle image uploads from users, store them in organized Cloudinary folders, wait for the upload to complete and receive the URL, then update our Supabase database with the returned URL.

Architecture Flow 

User submits image → FastAPI receives file → Upload to Cloudinary → Receive URL → Update Supabase documents table

Understanding Cloudinary Folder Modes

Cloudinary accounts created after June 4th, 2024 use "dynamic folder mode" by default, which separates the folder organization (where files appear in the Media Library) from the public_id (the URL path used for delivery).

Key Concepts

Dynamic Folder Mode (Modern Approach)

  • The asset_folder parameter controls where files appear in the Cloudinary Media Library
  • The public_id parameter controls the URL path for accessing the file
  • These two can be different or the same, giving you flexibility
  • You can use public_id_prefix to prepend a path to the public_id with a forward slash for SEO purposes

Legacy Fixed Folder Mode (Older Accounts)

  • The folder parameter sets both the public ID path and the folder placement of the asset
  • Simpler but less flexible

Our Folder Structure

We will organize uploads into three main categories:

/swiftops/tickets/     - Work order photos, site images, equipment photos
/swiftops/users/       - User profile avatars and related images  
/swiftops/receipts/    - Payment receipts, expense documents, invoices

How Upload Parameters Work

For Dynamic Folder Mode Accounts (Post-June 2024)

When uploading a file, you should use:

  1. asset_folder - Sets where the file appears in your Cloudinary Media Library for organization
  2. public_id - Sets the unique identifier and URL path for the file
  3. use_filename - Boolean to use the original filename
  4. unique_filename - Boolean to generate unique names (prevents collisions)
  5. overwrite - Boolean to replace existing files with same public_id

For Legacy Fixed Folder Mode Accounts (Pre-June 2024)

Use the folder parameter which handles both organization and URL path.

Upload Response Structure

When you upload a file, Cloudinary returns a response containing the secure URL, public ID, format, dimensions, file size, and creation timestamp.

The key information we need for our database:

  • secure_url - The HTTPS URL to access the image (this goes to Supabase)
  • public_id - The unique identifier (for future updates/deletions)
  • format - File format (jpg, png, etc.)
  • bytes - File size
  • width / height - Image dimensions
  • created_at - Upload timestamp

Database Integration

Documents Table Structure

Our documents table in Supabase will store:

  • id - UUID primary key
  • entity_type - Type of entity (ticket, user, receipt)
  • entity_id - Foreign key to the related entity
  • document_url - The Cloudinary secure_url goes here
  • cloudinary_public_id - For managing the asset later
  • file_name - Original filename
  • file_type - MIME type
  • file_size - Size in bytes
  • uploaded_by - User who uploaded
  • created_at - Timestamp

Workflow Steps

  1. Receive file upload from user via FastAPI endpoint
  2. Validate file (check type, size limits, etc.)
  3. Upload to Cloudinary using appropriate folder:
    • Tickets → /swiftops/tickets/
    • Users → /swiftops/users/
    • Receipts → /swiftops/receipts/
  4. Wait for Cloudinary response (this is synchronous - the upload method waits)
  5. Extract the secure_url from response
  6. Insert record into documents table with:
    • The Cloudinary URL
    • Entity information (what this image is for)
    • File metadata
    • Upload details
  7. Return success response to user with the document ID

Error Handling Considerations

Upload Failures

If Cloudinary upload fails:

  • Return appropriate error to user
  • Log the error details
  • Do NOT create database record

Database Failures

If Supabase insert fails after successful upload:

  • Consider deleting the uploaded image from Cloudinary (cleanup)
  • Return error to user
  • Log for manual review

Partial Failures

  • Transaction-like behavior: Either both succeed or both fail
  • Prevents orphaned files in Cloudinary
  • Prevents broken URL references in database

Upload Presets

Upload presets can be configured in your Cloudinary dashboard to establish naming and storage conventions for specific types of assets.

Recommended Preset Configuration

For each asset type (tickets, users, receipts), create a preset with:

  • Signing mode: Signed (more secure for backend uploads)
  • Asset folder: Set to appropriate folder path
  • Generated public ID: Auto-generate an unguessable public ID value (prevents conflicts)
  • Overwrite: Based on your use case (usually OFF for tickets, ON for user avatars)

Benefits of using presets:

  • Centralized configuration
  • Consistent naming conventions
  • Easier to update behavior across all uploads
  • Can set transformation defaults (resize, format, quality)

File Size and Type Considerations

Size Limits

  • Files under 100 MB can use the standard upload method
  • Files larger than 100 MB require the upload_large method, which uploads in chunks for better reliability
  • Consider setting application-level limits (e.g., 10 MB for receipts, 5 MB for avatars)

Supported Formats

  • Cloudinary can upload any file type, not just images and videos
  • For our use case: JPEG, PNG, PDF (for receipts), WebP
  • Set resource_type="auto" to let Cloudinary detect the type

Security Considerations

API Credentials

  • Never expose your Cloudinary credentials in client-side code
  • Use environment variables for CLOUDINARY_CLOUD_NAME, CLOUDINARY_API_KEY, CLOUDINARY_API_SECRET
  • Backend-only uploads using signed requests

Access Control

  • All uploads go through authenticated FastAPI endpoints
  • Verify user permissions before allowing uploads
  • Validate file types and sizes server-side

URL Security

  • Cloudinary URLs are public by default (anyone with URL can access)
  • For sensitive documents, consider using Cloudinary's authenticated delivery
  • Store URLs in database with appropriate access control

Performance Optimization

Upload Speed

  • Cloudinary automatically optimizes upload routes
  • Files are uploaded to nearest data center
  • Response time typically 1-3 seconds for standard images

Delivery Optimization

  • Cloudinary provides automatic image optimization and transformation capabilities
  • URLs can include transformation parameters for on-the-fly resizing
  • Example: w_300,h_300,c_fill in URL for thumbnail generation
  • CDN delivery for fast global access

Monitoring and Maintenance

What to Track

  • Upload success/failure rates
  • Average upload times
  • Storage usage by folder
  • Failed uploads requiring retry

Cleanup Strategy

  • Implement soft deletes in database
  • Keep Cloudinary images even if database record is deleted (for audit trail)
  • Scheduled cleanup jobs for truly orphaned files
  • Use Admin API methods to list, search, and manage assets

Testing Considerations

Local Development

  • Use test/development folder structure: /swiftops-dev/tickets/
  • Separate Cloudinary account or folder for testing
  • Mock Cloudinary responses for unit tests

Staging Environment

  • Mirror production folder structure
  • Test with various file types and sizes
  • Verify error handling paths
  • Test concurrent uploads

Common Pitfalls to Avoid

  1. Not waiting for upload completion - The Python SDK upload method is synchronous and waits, but ensure you handle it properly
  2. Not storing public_id - You'll need this for updates or deletions
  3. Using wrong folder parameter - Check if your account is dynamic or fixed folder mode
  4. Not validating files - Always validate on backend, never trust client validation alone
  5. Forgetting error handling - Network issues, quota limits, invalid files all need handling
  6. Not setting resource limits - Can lead to quota exhaustion or large bills

Next Steps for Implementation

  1. Get Cloudinary credentials from dashboard (Settings → API Keys)
  2. Install Python SDK (pip install cloudinary)
  3. Configure environment variables in your .env file
  4. Create upload presets in Cloudinary dashboard for each folder
  5. Set up Supabase documents table with proper schema
  6. Implement upload endpoints in FastAPI
  7. Add error handling and validation
  8. Test thoroughly with various file types and error scenarios
  9. Deploy to staging and verify in production-like environment
  10. Monitor uploads and optimize based on real usage

Additional Resources

  • Cloudinary Upload Guide provides comprehensive details and examples of upload options
  • Python image and video upload documentation covers common usage patterns
  • Folder modes documentation explains the differences between dynamic and fixed folder modes

This document should be updated as implementation details are finalized and new requirements emerge.