docs: Add troubleshooting guide for cloud deployment issues and solutions

README.md

@@ -22,6 +22,10 @@ Vault is an AI-powered platform that transforms YouTube video archives into shop
 - **Background Jobs**: Inngest for AI discovery pipeline
 - **Deployment**: Vercel (Frontend) + Railway (Workers)
 
+## Troubleshooting
+
+For detailed information about cloud deployment issues and solutions, see the [Troubleshooting Guide](src/features/discovery/docs/troubleshooting.md).
+
 ## Getting Started
 
 ### Prerequisites

src/features/discovery/docs/troubleshooting.md

@@ -0,0 +1,132 @@
+# Cloud Deployment Troubleshooting & Findings
+
+## Overview
+
+This document captures the key challenges, solutions, and learnings from troubleshooting the cloud deployment of the Vault AI video processing system. The findings are based on debugging the YouTube video processing pipeline in a Hugging Face Space deployment environment.
+
+## Problem Analysis
+
+### Initial Symptom
+The system was failing to process YouTube videos with vague error messages. The process would appear to start but would either hang or return empty results without any meaningful error information.
+
+### Root Cause Analysis
+
+#### Major Finding: DNS Resolution Failure (ENOTFOUND)
+
+**Issue:** The system was failing to resolve domain names (youtube.com and img.youtube.com) with the error:
+```
+[Errno -5] No address associated with hostname
+```
+
+**Root Cause:** 
+The Hugging Face Space deployment environment was experiencing IPv6 network configuration issues. The free tier environment was having difficulty resolving DNS queries over IPv6, which caused failures when attempting to connect to YouTube APIs.
+
+**Verification:**
+- Detailed logs showed consistent DNS resolution failures
+- The issue was reproducible in multiple deployment attempts
+- Network connectivity tests confirmed the IPv6 configuration problem
+
+## Solution Implemented
+
+### Fix: Forcing IPv4 Connections
+
+The solution involves forcing all network connections to use IPv4 instead of IPv6. This was implemented in two critical components:
+
+#### 1. yt-dlp Configuration
+Modified the yt-dlp command to include the `--force-ipv4` flag:
+
+```javascript
+// fly-server/server.js
+const downloadVideo = async (videoUrl) => {
+  return new Promise((resolve, reject) => {
+    const tempDir = os.tmpdir();
+    const filename = `video_${Date.now()}`;
+    const outputPath = path.join(tempDir, `${filename}.%(ext)s`);
+
+    // Force IPv4 for yt-dlp
+    const command = `yt-dlp --force-ipv4 -o "${outputPath}" "${videoUrl}"`;
+    exec(command, (error, stdout, stderr) => {
+      if (error) {
+        console.error('yt-dlp error:', stderr);
+        reject(new Error(`yt-dlp failed: ${stderr}`));
+        return;
+      }
+      
+      // Find downloaded file
+      const files = fs.readdirSync(tempDir);
+      const videoFile = files.find(f => f.startsWith(filename));
+      if (!videoFile) {
+        reject(new Error('Video file not found after download'));
+        return;
+      }
+      
+      resolve(path.join(tempDir, videoFile));
+    });
+  });
+};
+```
+
+#### 2. axios HTTP Client Configuration
+Added IPv4 forcing configuration to the axios HTTP client:
+
+```javascript
+// fly-server/server.js
+const axiosInstance = axios.create({
+  timeout: 30000,
+  headers: {
+    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'
+  },
+  // Force IPv4 resolution
+  httpAgent: new http.Agent({ family: 4 }),
+  httpsAgent: new https.Agent({ family: 4 })
+});
+```
+
+## Deployment Verification
+
+### Testing Process
+1. Applied the IPv4 forcing fix to the server.js file
+2. Deployed the updated version to Hugging Face Space
+3. Waited for the container to rebuild and start
+4. Executed a manual trigger to test video processing
+5. Verified results and monitored system behavior
+
+### Results
+- DNS resolution failures eliminated
+- YouTube video downloads now succeed consistently
+- Video processing pipeline works as expected
+- System stability improved significantly
+
+## Lessons Learned
+
+### 1. Network Environment Considerations
+Free tier cloud environments (like Hugging Face Spaces) often have limitations or configuration issues with IPv6 connectivity. Always test for both IPv4 and IPv6 compatibility when deploying network-dependent applications.
+
+### 2. Logging and Debugging
+Detailed error logging is critical for troubleshooting vague failures. The initial issue was challenging to diagnose because the system was not providing specific error information about the DNS resolution failure.
+
+### 3. Fallback Mechanisms
+Implementing robust fallback logic for network operations is essential. The system now includes:
+- Multiple extraction methods (yt-dlp as primary, fallback HTTP methods)
+- Detailed error logging for each failure scenario
+- Retry mechanisms for transient network issues
+
+### 4. Container Configuration
+When running applications in containerized environments:
+- Always test DNS resolution within the container
+- Consider network restrictions and configuration options
+- Monitor container logs for early detection of issues
+
+## Ongoing Monitoring
+
+The system now includes enhanced monitoring to detect and respond to network issues:
+- Real-time logging of DNS resolution attempts
+- Metrics for network connectivity success rates
+- Alerting for repeated network failures
+- Performance monitoring of video processing pipeline
+
+## Conclusion
+
+The cloud deployment issue was successfully resolved by forcing IPv4 connections. This fix addresses the underlying DNS resolution failure caused by IPv6 network configuration problems in the free tier Hugging Face Space environment.
+
+**Key Takeaway:** Network connectivity issues in cloud environments often require specific configuration changes rather than code fixes. Understanding the deployment environment's network constraints is crucial for building reliable applications.