src/features/discovery/docs/troubleshooting.md

Cloud Deployment Troubleshooting & Findings

Overview

This document captures the key challenges, solutions, and learnings from troubleshooting the cloud deployment of the Vault AI video processing system. The findings are based on debugging the YouTube video processing pipeline in a Hugging Face Space deployment environment.

Problem Analysis

Initial Symptom

The system was failing to process YouTube videos with vague error messages. The process would appear to start but would either hang or return empty results without any meaningful error information.

Root Cause Analysis

Major Finding: DNS Resolution Failure (ENOTFOUND)

Issue: The system was failing to resolve domain names (youtube.com and img.youtube.com) with the error:

[Errno -5] No address associated with hostname

Root Cause: The Hugging Face Space deployment environment was experiencing IPv6 network configuration issues. The free tier environment was having difficulty resolving DNS queries over IPv6, which caused failures when attempting to connect to YouTube APIs.

Verification:

• Detailed logs showed consistent DNS resolution failures
• The issue was reproducible in multiple deployment attempts
• Network connectivity tests confirmed the IPv6 configuration problem

Solution Implemented

Fix: Forcing IPv4 Connections

The solution involves forcing all network connections to use IPv4 instead of IPv6. This was implemented in two critical components:

1. yt-dlp Configuration

Modified the yt-dlp command to include the --force-ipv4 flag:

// fly-server/server.js
const downloadVideo = async (videoUrl) => {
  return new Promise((resolve, reject) => {
    const tempDir = os.tmpdir();
    const filename = `video_${Date.now()}`;
    const outputPath = path.join(tempDir, `${filename}.%(ext)s`);

    // Force IPv4 for yt-dlp
    const command = `yt-dlp --force-ipv4 -o "${outputPath}" "${videoUrl}"`;
    exec(command, (error, stdout, stderr) => {
      if (error) {
        console.error('yt-dlp error:', stderr);
        reject(new Error(`yt-dlp failed: ${stderr}`));
        return;
      }
      
      // Find downloaded file
      const files = fs.readdirSync(tempDir);
      const videoFile = files.find(f => f.startsWith(filename));
      if (!videoFile) {
        reject(new Error('Video file not found after download'));
        return;
      }
      
      resolve(path.join(tempDir, videoFile));
    });
  });
};

2. axios HTTP Client Configuration

Added IPv4 forcing configuration to the axios HTTP client:

// fly-server/server.js
const axiosInstance = axios.create({
  timeout: 30000,
  headers: {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'
  },
  // Force IPv4 resolution
  httpAgent: new http.Agent({ family: 4 }),
  httpsAgent: new https.Agent({ family: 4 })
});

Deployment Verification

Testing Process

1. Applied the IPv4 forcing fix to the server.js file
2. Deployed the updated version to Hugging Face Space
3. Waited for the container to rebuild and start
4. Executed a manual trigger to test video processing
5. Verified results and monitored system behavior

Results

• DNS resolution failures eliminated
• YouTube video downloads now succeed consistently
• Video processing pipeline works as expected
• System stability improved significantly

Lessons Learned

1. Network Environment Considerations

Free tier cloud environments (like Hugging Face Spaces) often have limitations or configuration issues with IPv6 connectivity. Always test for both IPv4 and IPv6 compatibility when deploying network-dependent applications.

2. Logging and Debugging

Detailed error logging is critical for troubleshooting vague failures. The initial issue was challenging to diagnose because the system was not providing specific error information about the DNS resolution failure.

3. Fallback Mechanisms

Implementing robust fallback logic for network operations is essential. The system now includes:

• Multiple extraction methods (yt-dlp as primary, fallback HTTP methods)
• Detailed error logging for each failure scenario
• Retry mechanisms for transient network issues

4. Container Configuration

When running applications in containerized environments:

• Always test DNS resolution within the container
• Consider network restrictions and configuration options
• Monitor container logs for early detection of issues

Ongoing Monitoring

The system now includes enhanced monitoring to detect and respond to network issues:

• Real-time logging of DNS resolution attempts
• Metrics for network connectivity success rates
• Alerting for repeated network failures
• Performance monitoring of video processing pipeline

Conclusion

The cloud deployment issue was successfully resolved by forcing IPv4 connections. This fix addresses the underlying DNS resolution failure caused by IPv6 network configuration problems in the free tier Hugging Face Space environment.

Key Takeaway: Network connectivity issues in cloud environments often require specific configuration changes rather than code fixes. Understanding the deployment environment's network constraints is crucial for building reliable applications.