src/features/discovery/docs/troubleshooting.md

# Cloud Deployment Troubleshooting & Findings

## Overview

This document captures the key challenges, solutions, and learnings from troubleshooting the cloud deployment of the Vault AI video processing system. The findings are based on debugging the YouTube video processing pipeline in a Hugging Face Space deployment environment.

## Problem Analysis

### Initial Symptom
The system was failing to process YouTube videos with vague error messages. The process would appear to start but would either hang or return empty results without any meaningful error information.

### Root Cause Analysis

#### Major Finding: DNS Resolution Failure (ENOTFOUND)

**Issue:** The system was failing to resolve domain names (youtube.com and img.youtube.com) with the error:
```
[Errno -5] No address associated with hostname
```

**Root Cause:** 
The Hugging Face Space deployment environment was experiencing IPv6 network configuration issues. The free tier environment was having difficulty resolving DNS queries over IPv6, which caused failures when attempting to connect to YouTube APIs.

**Verification:**
- Detailed logs showed consistent DNS resolution failures
- The issue was reproducible in multiple deployment attempts
- Network connectivity tests confirmed the IPv6 configuration problem

## Solution Implemented

### Fix: Forcing IPv4 Connections

The solution involves forcing all network connections to use IPv4 instead of IPv6. This was implemented in two critical components:

#### 1. yt-dlp Configuration
Modified the yt-dlp command to include the `--force-ipv4` flag:

```javascript
// fly-server/server.js
const downloadVideo = async (videoUrl) => {
  return new Promise((resolve, reject) => {
    const tempDir = os.tmpdir();
    const filename = `video_${Date.now()}`;
    const outputPath = path.join(tempDir, `${filename}.%(ext)s`);

    // Force IPv4 for yt-dlp
    const command = `yt-dlp --force-ipv4 -o "${outputPath}" "${videoUrl}"`;
    exec(command, (error, stdout, stderr) => {
      if (error) {
        console.error('yt-dlp error:', stderr);
        reject(new Error(`yt-dlp failed: ${stderr}`));
        return;
      }
      
      // Find downloaded file
      const files = fs.readdirSync(tempDir);
      const videoFile = files.find(f => f.startsWith(filename));
      if (!videoFile) {
        reject(new Error('Video file not found after download'));
        return;
      }
      
      resolve(path.join(tempDir, videoFile));
    });
  });
};
```

#### 2. axios HTTP Client Configuration
Added IPv4 forcing configuration to the axios HTTP client:

```javascript
// fly-server/server.js
const axiosInstance = axios.create({
  timeout: 30000,
  headers: {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'
  },
  // Force IPv4 resolution
  httpAgent: new http.Agent({ family: 4 }),
  httpsAgent: new https.Agent({ family: 4 })
});
```

## Deployment Verification

### Testing Process
1. Applied the IPv4 forcing fix to the server.js file
2. Deployed the updated version to Hugging Face Space
3. Waited for the container to rebuild and start
4. Executed a manual trigger to test video processing
5. Verified results and monitored system behavior

### Results
- DNS resolution failures eliminated
- YouTube video downloads now succeed consistently
- Video processing pipeline works as expected
- System stability improved significantly

## Lessons Learned

### 1. Network Environment Considerations
Free tier cloud environments (like Hugging Face Spaces) often have limitations or configuration issues with IPv6 connectivity. Always test for both IPv4 and IPv6 compatibility when deploying network-dependent applications.

### 2. Logging and Debugging
Detailed error logging is critical for troubleshooting vague failures. The initial issue was challenging to diagnose because the system was not providing specific error information about the DNS resolution failure.

### 3. Fallback Mechanisms
Implementing robust fallback logic for network operations is essential. The system now includes:
- Multiple extraction methods (yt-dlp as primary, fallback HTTP methods)
- Detailed error logging for each failure scenario
- Retry mechanisms for transient network issues

### 4. Container Configuration
When running applications in containerized environments:
- Always test DNS resolution within the container
- Consider network restrictions and configuration options
- Monitor container logs for early detection of issues

## Ongoing Monitoring

The system now includes enhanced monitoring to detect and respond to network issues:
- Real-time logging of DNS resolution attempts
- Metrics for network connectivity success rates
- Alerting for repeated network failures
- Performance monitoring of video processing pipeline

## Conclusion

The cloud deployment issue was successfully resolved by forcing IPv4 connections. This fix addresses the underlying DNS resolution failure caused by IPv6 network configuration problems in the free tier Hugging Face Space environment.

**Key Takeaway:** Network connectivity issues in cloud environments often require specific configuration changes rather than code fixes. Understanding the deployment environment's network constraints is crucial for building reliable applications.