Hugging Face's logo

Spaces:
findEthics
/
Atlas
Sleeping

App Files Community
Atlas / docs /reference /migration-guide.md
findEthics
Complete codebase cleanup and project structure validation
f0b765c
preview code
|
Raw
History Blame Contribute Delete
10.7 kB

User Authentication Migration Guide

Note: The user authentication migration has been completed and the migration scripts have been moved to the archive/ directory for historical reference. This documentation is maintained for reference purposes.

This directory contains scripts for migrating the Atlas API database to support user authentication while maintaining backward compatibility.

Overview

The user authentication feature adds optional user_id fields to existing collections (sessions, messages, search_analytics) and creates appropriate indexes for efficient user-specific queries.

Migration Scripts

1. Main Migration Script

File: archive/migrate_user_authentication.py (Archived - migration completed)

The primary migration script that handles the complete migration process.

# Run full migration
python archive/migrate_user_authentication.py migrate

# Check migration status
python archive/migrate_user_authentication.py status

# Create backup only
python archive/migrate_user_authentication.py backup

# Validate migration
python archive/migrate_user_authentication.py validate

# Rollback migration
python archive/migrate_user_authentication.py rollback

Features:

  • Automatic backup creation before migration
  • Safe addition of user_id fields to existing documents
  • Index creation for performance
  • Migration history tracking
  • Comprehensive validation
  • Rollback capability

2. Validation Script

File: archive/validate_user_migration.py (Archived - migration completed)

Comprehensive validation script to verify migration success.

# Basic validation
python archive/validate_user_migration.py

# Detailed validation with verbose output
python archive/validate_user_migration.py --detailed

# Validation with automatic issue fixing
python archive/validate_user_migration.py --detailed --fix-issues

Validation Tests:

  • Schema validation (user_id fields exist)
  • Index validation (proper indexes created)
  • Data integrity checks
  • Performance validation
  • Backward compatibility verification
  • Migration history validation

3. Rollback Script

File: archive/rollback_user_authentication.py (Archived - migration completed)

Dedicated rollback script to safely remove migration changes.

# Interactive rollback with confirmation
python archive/rollback_user_authentication.py

# Automated rollback (skip confirmation)
python archive/rollback_user_authentication.py --confirm

# Rollback with backup creation
python archive/rollback_user_authentication.py --backup-first --confirm

Features:

  • Pre-rollback backup creation
  • Safe removal of user_id fields
  • Index cleanup
  • Rollback verification
  • Data loss warnings

4. Index Management Script

File: scripts/deployment/create_user_indexes.py

Standalone script for managing user_id indexes.

# Create indexes
python scripts/deployment/create_user_indexes.py create

# Verify indexes exist
python scripts/deployment/create_user_indexes.py verify

# List all indexes
python scripts/deployment/create_user_indexes.py list

# Remove user_id indexes
python scripts/deployment/create_user_indexes.py rollback

Migration Process

Pre-Migration Checklist

  1. Environment Setup

    # Ensure environment variables are set
export MONGODB_URL="your_mongodb_connection_string"
export MONGODB_DATABASE="atlas_analytics"

# Install dependencies
pip install -r requirements.txt

  2. Database Connection Test

    python -c "
import asyncio
from analytics.database import test_connection
print('✅ Connected' if asyncio.run(test_connection()) else '❌ Failed')
"

  3. Current Data Backup (Recommended)

    python archive/migrate_user_authentication.py backup

Step-by-Step Migration

  1. Check Current Status

    python archive/migrate_user_authentication.py status

  2. Run Migration

    python archive/migrate_user_authentication.py migrate

  3. Validate Results

    python archive/validate_user_migration.py --detailed

  4. Test Application (Manual)

    • Test anonymous user requests (existing functionality)
    • Test authenticated user requests (new functionality)
    • Verify analytics dashboard works

Post-Migration Verification

  1. Schema Verification

    • All documents have user_id field (set to null for existing data)
    • New documents can have actual user_id values

  2. Index Verification

    • Sparse indexes on user_id fields
    • Compound indexes on (user_id, timestamp)

  3. Functionality Verification

    • Anonymous requests work unchanged
    • Authenticated requests store user_id
    • Analytics queries perform well

Rollback Process

When to Rollback

  • Migration validation fails
  • Application issues after migration
  • Performance problems
  • Need to revert to previous state

Rollback Steps

  1. Check Rollback Feasibility

    python archive/rollback_user_authentication.py
# Review warnings and data loss implications

  2. Create Pre-Rollback Backup (if user data exists)

    python archive/rollback_user_authentication.py --backup-first --confirm

  3. Verify Rollback Success

    python archive/validate_user_migration.py
# Should show no user_id fields or indexes

Safety Features

Backup System

  • Automatic backups before migration
  • Optional backups before rollback
  • JSON format for easy inspection
  • Stored in migration_backups/ and rollback_backups/

Migration History

  • All operations tracked in migration_history collection
  • Timestamps and status tracking
  • Error logging and details
  • Rollback event tracking

Data Integrity

  • Existing data preserved during migration
  • Null values for user_id in migrated documents
  • Validation of data consistency
  • Performance impact monitoring

Error Handling

  • Graceful failure handling
  • Detailed error logging
  • Partial migration recovery
  • Safe rollback procedures

Troubleshooting

Common Issues

  1. Database Connection Failed

    # Check environment variables
echo $MONGODB_URL
echo $MONGODB_DATABASE

# Test connection
python -c "import asyncio; from analytics.database import test_connection; print(asyncio.run(test_connection()))"

  2. Migration Partially Completed

    # Check status
python archive/migrate_user_authentication.py status

# Re-run migration (safe to run multiple times)
python archive/migrate_user_authentication.py migrate

  3. Index Creation Failed

    # Create indexes separately
python scripts/deployment/create_user_indexes.py create

# Verify indexes
python scripts/deployment/create_user_indexes.py verify

  4. Validation Failures

    # Run detailed validation
python archive/validate_user_migration.py --detailed

# Attempt automatic fixes
python archive/validate_user_migration.py --detailed --fix-issues

Recovery Procedures

  1. Restore from Backup

    # Manual restore from backup JSON file
# (Requires custom script based on backup structure)

  2. Partial Rollback

    # Remove only indexes
python scripts/deployment/create_user_indexes.py rollback

# Full rollback
python archive/rollback_user_authentication.py --confirm

  3. Re-run Migration

    # Safe to run multiple times
python archive/migrate_user_authentication.py migrate

Performance Considerations

Index Strategy

  • Sparse indexes: Handle null user_id values efficiently
  • Compound indexes: Optimize user history queries
  • Background creation: Minimize impact on running system

Query Performance

  • User-specific queries use indexes
  • Anonymous queries unaffected
  • Backward compatibility maintained

Storage Impact

  • Minimal storage overhead (one field per document)
  • Null values for existing anonymous data
  • Efficient index storage with sparse indexes

Security Considerations

User ID Handling

  • User IDs treated as identifiers, not authentication
  • No authorization logic based on user_id
  • Client-provided user_id values (validation only)

Data Privacy

  • User associations stored as provided
  • No automatic PII detection
  • Export functionality respects user filtering

Access Control

  • Database-level access controls unchanged
  • Application-level user validation required
  • Migration scripts require database admin access

Monitoring and Maintenance

Regular Checks 

# Weekly validation
python archive/validate_user_migration.py

# Index performance monitoring
python scripts/deployment/create_user_indexes.py list

# Migration history review
python archive/migrate_user_authentication.py status

Performance Monitoring

  • Monitor query execution times
  • Check index usage statistics
  • Review storage growth patterns

Backup Schedule

  • Regular database backups
  • Pre-migration backups for major changes
  • Retention policy for backup files

Support and Documentation

Log Files

  • Migration logs in application logs
  • Detailed error messages
  • Performance metrics

Documentation

  • API documentation updated for user_id parameter
  • Integration examples for frontend developers
  • Troubleshooting guides

Testing

  • Comprehensive test suite in tests/ directory
  • Integration tests for user authentication
  • Performance tests for user queries
  • Backward compatibility tests

Quick Reference

Essential Commands 

# Full migration workflow
python archive/migrate_user_authentication.py migrate
python archive/validate_user_migration.py --detailed

# Emergency rollback
python archive/rollback_user_authentication.py --confirm

# Status check
python archive/migrate_user_authentication.py status

File Structure 

├── archive/
│   ├── migrate_user_authentication.py    # Main migration script (archived)
│   ├── validate_user_migration.py        # Validation script (archived)
│   ├── rollback_user_authentication.py   # Rollback script (archived)
│   └── migration_backups/                # Migration backups (archived)
├── scripts/deployment/create_user_indexes.py # Index management (active)
├── docs/reference/migration-guide.md     # This documentation
└── rollback_backups/                     # Rollback backups

For additional support, refer to the main project documentation or contact the development team.