docs/reference/migration-guide.md

# User Authentication Migration Guide

> **Note:** The user authentication migration has been completed and the migration scripts have been moved to the `archive/` directory for historical reference. This documentation is maintained for reference purposes.

This directory contains scripts for migrating the Atlas API database to support user authentication while maintaining backward compatibility.

## Overview

The user authentication feature adds optional `user_id` fields to existing collections (`sessions`, `messages`, `search_analytics`) and creates appropriate indexes for efficient user-specific queries.

## Migration Scripts

### 1. Main Migration Script
**File:** `archive/migrate_user_authentication.py` *(Archived - migration completed)*

The primary migration script that handles the complete migration process.

```bash
# Run full migration
python archive/migrate_user_authentication.py migrate

# Check migration status
python archive/migrate_user_authentication.py status

# Create backup only
python archive/migrate_user_authentication.py backup

# Validate migration
python archive/migrate_user_authentication.py validate

# Rollback migration
python archive/migrate_user_authentication.py rollback
```

**Features:**
- Automatic backup creation before migration
- Safe addition of `user_id` fields to existing documents
- Index creation for performance
- Migration history tracking
- Comprehensive validation
- Rollback capability

### 2. Validation Script
**File:** `archive/validate_user_migration.py` *(Archived - migration completed)*

Comprehensive validation script to verify migration success.

```bash
# Basic validation
python archive/validate_user_migration.py

# Detailed validation with verbose output
python archive/validate_user_migration.py --detailed

# Validation with automatic issue fixing
python archive/validate_user_migration.py --detailed --fix-issues
```

**Validation Tests:**
- Schema validation (user_id fields exist)
- Index validation (proper indexes created)
- Data integrity checks
- Performance validation
- Backward compatibility verification
- Migration history validation

### 3. Rollback Script
**File:** `archive/rollback_user_authentication.py` *(Archived - migration completed)*

Dedicated rollback script to safely remove migration changes.

```bash
# Interactive rollback with confirmation
python archive/rollback_user_authentication.py

# Automated rollback (skip confirmation)
python archive/rollback_user_authentication.py --confirm

# Rollback with backup creation
python archive/rollback_user_authentication.py --backup-first --confirm
```

**Features:**
- Pre-rollback backup creation
- Safe removal of user_id fields
- Index cleanup
- Rollback verification
- Data loss warnings

### 4. Index Management Script
**File:** `scripts/deployment/create_user_indexes.py`

Standalone script for managing user_id indexes.

```bash
# Create indexes
python scripts/deployment/create_user_indexes.py create

# Verify indexes exist
python scripts/deployment/create_user_indexes.py verify

# List all indexes
python scripts/deployment/create_user_indexes.py list

# Remove user_id indexes
python scripts/deployment/create_user_indexes.py rollback
```

## Migration Process

### Pre-Migration Checklist

1. **Environment Setup**
   ```bash
   # Ensure environment variables are set
   export MONGODB_URL="your_mongodb_connection_string"
   export MONGODB_DATABASE="atlas_analytics"
   
   # Install dependencies
   pip install -r requirements.txt
   ```

2. **Database Connection Test**
   ```bash
   python -c "
   import asyncio
   from analytics.database import test_connection
   print('✅ Connected' if asyncio.run(test_connection()) else '❌ Failed')
   "
   ```

3. **Current Data Backup** (Recommended)
   ```bash
   python archive/migrate_user_authentication.py backup
   ```

### Step-by-Step Migration

1. **Check Current Status**
   ```bash
   python archive/migrate_user_authentication.py status
   ```

2. **Run Migration**
   ```bash
   python archive/migrate_user_authentication.py migrate
   ```

3. **Validate Results**
   ```bash
   python archive/validate_user_migration.py --detailed
   ```

4. **Test Application** (Manual)
   - Test anonymous user requests (existing functionality)
   - Test authenticated user requests (new functionality)
   - Verify analytics dashboard works

### Post-Migration Verification

1. **Schema Verification**
   - All documents have `user_id` field (set to `null` for existing data)
   - New documents can have actual user_id values

2. **Index Verification**
   - Sparse indexes on `user_id` fields
   - Compound indexes on `(user_id, timestamp)`

3. **Functionality Verification**
   - Anonymous requests work unchanged
   - Authenticated requests store user_id
   - Analytics queries perform well

## Rollback Process

### When to Rollback

- Migration validation fails
- Application issues after migration
- Performance problems
- Need to revert to previous state

### Rollback Steps

1. **Check Rollback Feasibility**
   ```bash
   python archive/rollback_user_authentication.py
   # Review warnings and data loss implications
   ```

2. **Create Pre-Rollback Backup** (if user data exists)
   ```bash
   python archive/rollback_user_authentication.py --backup-first --confirm
   ```

3. **Verify Rollback Success**
   ```bash
   python archive/validate_user_migration.py
   # Should show no user_id fields or indexes
   ```

## Safety Features

### Backup System
- Automatic backups before migration
- Optional backups before rollback
- JSON format for easy inspection
- Stored in `migration_backups/` and `rollback_backups/`

### Migration History
- All operations tracked in `migration_history` collection
- Timestamps and status tracking
- Error logging and details
- Rollback event tracking

### Data Integrity
- Existing data preserved during migration
- Null values for user_id in migrated documents
- Validation of data consistency
- Performance impact monitoring

### Error Handling
- Graceful failure handling
- Detailed error logging
- Partial migration recovery
- Safe rollback procedures

## Troubleshooting

### Common Issues

1. **Database Connection Failed**
   ```bash
   # Check environment variables
   echo $MONGODB_URL
   echo $MONGODB_DATABASE
   
   # Test connection
   python -c "import asyncio; from analytics.database import test_connection; print(asyncio.run(test_connection()))"
   ```

2. **Migration Partially Completed**
   ```bash
   # Check status
   python archive/migrate_user_authentication.py status
   
   # Re-run migration (safe to run multiple times)
   python archive/migrate_user_authentication.py migrate
   ```

3. **Index Creation Failed**
   ```bash
   # Create indexes separately
   python scripts/deployment/create_user_indexes.py create
   
   # Verify indexes
   python scripts/deployment/create_user_indexes.py verify
   ```

4. **Validation Failures**
   ```bash
   # Run detailed validation
   python archive/validate_user_migration.py --detailed
   
   # Attempt automatic fixes
   python archive/validate_user_migration.py --detailed --fix-issues
   ```

### Recovery Procedures

1. **Restore from Backup**
   ```bash
   # Manual restore from backup JSON file
   # (Requires custom script based on backup structure)
   ```

2. **Partial Rollback**
   ```bash
   # Remove only indexes
   python scripts/deployment/create_user_indexes.py rollback
   
   # Full rollback
   python archive/rollback_user_authentication.py --confirm
   ```

3. **Re-run Migration**
   ```bash
   # Safe to run multiple times
   python archive/migrate_user_authentication.py migrate
   ```

## Performance Considerations

### Index Strategy
- **Sparse indexes**: Handle null user_id values efficiently
- **Compound indexes**: Optimize user history queries
- **Background creation**: Minimize impact on running system

### Query Performance
- User-specific queries use indexes
- Anonymous queries unaffected
- Backward compatibility maintained

### Storage Impact
- Minimal storage overhead (one field per document)
- Null values for existing anonymous data
- Efficient index storage with sparse indexes

## Security Considerations

### User ID Handling
- User IDs treated as identifiers, not authentication
- No authorization logic based on user_id
- Client-provided user_id values (validation only)

### Data Privacy
- User associations stored as provided
- No automatic PII detection
- Export functionality respects user filtering

### Access Control
- Database-level access controls unchanged
- Application-level user validation required
- Migration scripts require database admin access

## Monitoring and Maintenance

### Regular Checks
```bash
# Weekly validation
python archive/validate_user_migration.py

# Index performance monitoring
python scripts/deployment/create_user_indexes.py list

# Migration history review
python archive/migrate_user_authentication.py status
```

### Performance Monitoring
- Monitor query execution times
- Check index usage statistics
- Review storage growth patterns

### Backup Schedule
- Regular database backups
- Pre-migration backups for major changes
- Retention policy for backup files

## Support and Documentation

### Log Files
- Migration logs in application logs
- Detailed error messages
- Performance metrics

### Documentation
- API documentation updated for user_id parameter
- Integration examples for frontend developers
- Troubleshooting guides

### Testing
- Comprehensive test suite in `tests/` directory
- Integration tests for user authentication
- Performance tests for user queries
- Backward compatibility tests

---

## Quick Reference

### Essential Commands
```bash
# Full migration workflow
python archive/migrate_user_authentication.py migrate
python archive/validate_user_migration.py --detailed

# Emergency rollback
python archive/rollback_user_authentication.py --confirm

# Status check
python archive/migrate_user_authentication.py status
```

### File Structure
```
├── archive/
│   ├── migrate_user_authentication.py    # Main migration script (archived)
│   ├── validate_user_migration.py        # Validation script (archived)
│   ├── rollback_user_authentication.py   # Rollback script (archived)
│   └── migration_backups/                # Migration backups (archived)
├── scripts/deployment/create_user_indexes.py # Index management (active)
├── docs/reference/migration-guide.md     # This documentation
└── rollback_backups/                     # Rollback backups
```

For additional support, refer to the main project documentation or contact the development team.