Spaces:

abhisheksan
/

multiutility-server

Running

abhisheksan commited on 28 days ago

Commit

eb87c73

1 Parent(s): 2ce4373

Simplify: Remove proxy service, keep only audio file upload for HF Spaces

- Removed proxy_service directory and all proxy-related code
- Removed httpx dependency (no longer needed)
- Removed YOUTUBE_PROXY_URL configuration
- Simplified error messages to suggest audio upload endpoint
- Updated README with clear HF Spaces limitations
- Added HF_SPACES_GUIDE.md with detailed deployment instructions
- YouTube extraction endpoint remains but documented as self-hosted only
- Audio upload endpoint (/transcribe) works perfectly on HF Spaces

Files changed (11) hide show

HF_SPACES_GUIDE.md +442 -0
README.md +28 -70
app/apis/subtitles/service.py +4 -94
app/core/config.py +0 -3
poetry.lock +1 -48
proxy_service/Dockerfile +0 -35
proxy_service/README.md +0 -409
proxy_service/main.py +0 -216
proxy_service/render.yaml +0 -21
proxy_service/requirements.txt +0 -6
pyproject.toml +0 -1

HF_SPACES_GUIDE.md ADDED Viewed

	@@ -0,0 +1,442 @@

+# Hugging Face Spaces Deployment Guide
+## 🎯 Overview
+This guide explains how to deploy and use the Multi-Utility Server on Hugging Face Spaces, including limitations and workarounds.
+## 🚀 Quick Deployment
+### Step 1: Create a Space
+1. Go to [Hugging Face Spaces](https://huggingface.co/spaces)
+2. Click **"Create new Space"**
+3. Choose:
+   - **Space name:** Your choice
+   - **SDK:** Docker
+   - **Visibility:** Public or Private
+4. Click **"Create Space"**
+### Step 2: Configure Secrets
+1. Go to your Space's **Settings** → **Repository secrets**
+2. Add a new secret:
+   - **Name:** `API_KEYS`
+   - **Value:** `your-secure-api-key-here` (comma-separated for multiple keys)
+3. Save
+### Step 3: Push Code
+```bash
+# Clone your space
+git clone https://huggingface.co/spaces/YOUR_USERNAME/YOUR_SPACE_NAME
+cd YOUR_SPACE_NAME
+# Add this repository as a remote
+git remote add source https://github.com/YOUR_REPO/multiutility-server.git
+git pull source main
+# Push to HF Spaces
+git push origin main
+```
+Or connect your GitHub repository directly in Space settings.
+## 📊 Feature Availability on HF Spaces
+| Feature | Status | Endpoint |
+|---------|--------|----------|
+| **Text Embeddings** | ✅ Works | `POST /api/v1/embeddings/generate` |
+| **Audio File Transcription** | ✅ Works | `POST /api/v1/subtitles/transcribe` |
+| **YouTube Subtitle Extraction** | ❌ Blocked | `POST /api/v1/subtitles/extract` |
+| **Health Checks** | ✅ Works | `GET /health` |
+## ⚠️ Network Limitations
+### What's Blocked
+Hugging Face Spaces runs in a sandboxed environment that **blocks external internet access** for security reasons. This means:
+- ❌ Cannot download from YouTube directly
+- ❌ Cannot access external APIs
+- ❌ Cannot perform web scraping
+### What Works
+- ✅ File uploads from users
+- ✅ AI model inference (Whisper, embeddings)
+- ✅ Returning results to users
+- ✅ Internal HF services
+## 🎤 Audio Transcription Workflow
+Since YouTube downloads don't work on HF Spaces, use this workflow instead:
+### Option 1: User Downloads Audio Locally
+**Step 1:** User downloads audio using [yt-dlp](https://github.com/yt-dlp/yt-dlp)
+```bash
+# Install yt-dlp
+pip install yt-dlp
+# Download audio from YouTube
+yt-dlp -x --audio-format mp3 "https://www.youtube.com/watch?v=VIDEO_ID" -o audio.mp3
+```
+**Step 2:** User uploads audio to your HF Space
+```bash
+curl -X POST https://YOUR_SPACE.hf.space/api/v1/subtitles/transcribe \
+  -H "x-api-key: your-api-key" \
+  -F "file=@audio.mp3" \
+  -F "lang=en"
+```
+**Step 3:** Receive transcription
+```json
+{
+  "status": "success",
+  "language": "en",
+  "file_name": "audio.mp3",
+  "transcription": [
+    "First segment of transcribed text",
+    "Second segment of transcribed text",
+    "..."
+  ]
+}
+```
+### Option 2: Browser-Based Upload
+Create a simple HTML form for users:
+```html
+<!DOCTYPE html>
+<html>
+<body>
+  <h2>Audio Transcription</h2>
+  <form id="uploadForm">
+    <input type="file" id="audioFile" accept="audio/*" required>
+    <select id="language">
+      <option value="en">English</option>
+      <option value="es">Spanish</option>
+      <option value="fr">French</option>
+    </select>
+    <button type="submit">Transcribe</button>
+  </form>
+  <div id="result"></div>
+  <script>
+    document.getElementById('uploadForm').onsubmit = async (e) => {
+      e.preventDefault();
+      const formData = new FormData();
+      formData.append('file', document.getElementById('audioFile').files[0]);
+      formData.append('lang', document.getElementById('language').value);
+      const response = await fetch('https://YOUR_SPACE.hf.space/api/v1/subtitles/transcribe', {
+        method: 'POST',
+        headers: { 'x-api-key': 'your-api-key' },
+        body: formData
+      });
+      const result = await response.json();
+      document.getElementById('result').innerHTML =
+        '<pre>' + JSON.stringify(result, null, 2) + '</pre>';
+    };
+  </script>
+</body>
+</html>
+```
+## 📝 API Usage Examples
+### Text Embeddings (Works on HF Spaces)
+```python
+import requests
+url = "https://YOUR_SPACE.hf.space/api/v1/embeddings/generate"
+headers = {
+    "Content-Type": "application/json",
+    "x-api-key": "your-api-key"
+}
+data = {
+    "texts": [
+        "Hello, how are you?",
+        "Machine learning is fascinating"
+    ],
+    "normalize": True
+}
+response = requests.post(url, headers=headers, json=data)
+print(response.json())
+```
+### Audio File Transcription (Works on HF Spaces)
+```python
+import requests
+url = "https://YOUR_SPACE.hf.space/api/v1/subtitles/transcribe"
+headers = {"x-api-key": "your-api-key"}
+with open("audio.mp3", "rb") as audio_file:
+    files = {"file": audio_file}
+    data = {"lang": "en"}
+    response = requests.post(url, headers=headers, files=files, data=data)
+print(response.json())
+```
+### YouTube Extraction (Does NOT Work on HF Spaces)
+```python
+# ❌ This will fail on HF Spaces with network error
+import requests
+url = "https://YOUR_SPACE.hf.space/api/v1/subtitles/extract"
+headers = {
+    "Content-Type": "application/json",
+    "x-api-key": "your-api-key"
+}
+data = {
+    "url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ",
+    "lang": "en"
+}
+response = requests.post(url, headers=headers, json=data)
+# Error: Network connectivity issue
+```
+## 🔧 Configuration
+### Required Environment Variables
+Set these in HF Spaces **Repository secrets**:
+| Variable | Description | Example |
+|----------|-------------|---------|
+| `API_KEYS` | Comma-separated API keys | `key1,key2,key3` |
+### Optional Environment Variables
+| Variable | Description | Default |
+|----------|-------------|---------|
+| `CORS_ORIGINS` | Allowed origins | `*` |
+| `RATE_LIMIT_REQUESTS` | Requests per minute | `100` |
+| `LOG_LEVEL` | Logging level | `INFO` |
+| `WHISPER_MODEL` | Whisper model size | `base` |
+| `EMBEDDING_MODEL` | HuggingFace model | `mixedbread-ai/mxbai-embed-large-v1` |
+### Whisper Model Options
+| Model | Size | Speed | Accuracy |
+|-------|------|-------|----------|
+| `tiny` | 39 MB | Fastest | Lowest |
+| `base` | 74 MB | Fast | Good |
+| `small` | 244 MB | Medium | Better |
+| `medium` | 769 MB | Slow | Best |
+**Recommendation for HF Spaces:** Use `base` or `small` for good balance.
+## 🐛 Troubleshooting
+### Issue: Build fails with poetry.lock error
+**Error:**
+```
+The lock file might not be compatible with the current version of Poetry
+```
+**Solution:**
+```bash
+poetry lock
+git add poetry.lock
+git commit -m "Update poetry.lock"
+git push
+```
+### Issue: "Unauthorized" error
+**Error:**
+```json
+{"detail": "Unauthorized: Invalid or missing API key"}
+```
+**Solution:**
+- Verify `API_KEYS` secret is set in Space settings
+- Include `x-api-key` header in your requests
+- Check for typos in the API key
+### Issue: YouTube extraction fails
+**Error:**
+```json
+{
+  "status": "error",
+  "message": "Network connectivity issue: Unable to reach YouTube..."
+}
+```
+**Solution:**
+This is expected on HF Spaces. Use the audio upload endpoint instead:
+1. Download audio locally with yt-dlp
+2. Upload to `/api/v1/subtitles/transcribe`
+### Issue: Out of memory
+**Error:**
+```
+Container killed due to memory limit
+```
+**Solution:**
+- Use smaller Whisper model: `WHISPER_MODEL=tiny` or `WHISPER_MODEL=base`
+- Process shorter audio files
+- Consider upgrading to HF Spaces Pro (more RAM)
+### Issue: Slow transcription
+**Solution:**
+- Use smaller Whisper model (`tiny` or `base`)
+- Process shorter audio segments
+- Note: HF Spaces free tier uses CPU (no GPU)
+## 📈 Performance Tips
+### 1. Choose the Right Whisper Model
+```python
+# Fast but less accurate (good for testing)
+WHISPER_MODEL=tiny
+# Balanced (recommended for production)
+WHISPER_MODEL=base
+# Accurate but slow (only if you need high quality)
+WHISPER_MODEL=small
+```
+### 2. Optimize Audio Files
+```bash
+# Convert to optimal format before upload
+ffmpeg -i input.wav -ar 16000 -ac 1 -c:a libmp3lame output.mp3
+```
+### 3. Rate Limiting
+The server has rate limiting enabled:
+- Default: 100 requests per minute
+- Adjust via `RATE_LIMIT_REQUESTS` environment variable
+## 🔒 Security Best Practices
+### 1. Use Strong API Keys
+```bash
+# Generate secure API key
+openssl rand -base64 32
+```
+### 2. Rotate Keys Regularly
+Update `API_KEYS` in Space secrets monthly.
+### 3. Monitor Usage
+Check Space logs regularly:
+- Settings → Logs
+- Look for suspicious activity
+### 4. Use Private Spaces for Sensitive Data
+Consider making your Space private if handling sensitive content.
+## 💰 Cost Considerations
+### Free Tier
+- ✅ Unlimited inference
+- ✅ 16GB RAM
+- ✅ 2 vCPU
+- ⚠️ CPU-only (no GPU)
+- ⚠️ May sleep after inactivity
+### Spaces Pro ($5/month per Space)
+- ✅ Always-on
+- ✅ Better performance
+- ✅ More resources
+- ✅ Custom domains
+## 🎓 Best Practices
+### 1. Document the Workflow
+Add a README to your Space explaining:
+- How to download audio locally
+- How to use the upload endpoint
+- Supported audio formats
+### 2. Provide Examples
+Include example API calls and code snippets.
+### 3. Set Expectations
+Clearly state that YouTube direct extraction doesn't work on HF Spaces.
+### 4. Offer Alternatives
+Suggest self-hosted deployment for users who need YouTube extraction.
+## 🚀 Alternative Deployment
+If you need YouTube extraction, consider:
+### Self-Hosted Options
+1. **Docker on VPS** (DigitalOcean, Linode)
+   - Cost: $4-12/month
+   - Full control
+   - All features work
+2. **Cloud Platforms** (AWS, GCP, Azure)
+   - Scalable
+   - More expensive
+   - Enterprise-grade
+3. **Railway/Render**
+   - Easy deployment
+   - $5-20/month
+   - Good middle ground
+## 📚 Additional Resources
+- [Hugging Face Spaces Documentation](https://huggingface.co/docs/hub/spaces)
+- [yt-dlp Documentation](https://github.com/yt-dlp/yt-dlp)
+- [Whisper Model Information](https://github.com/openai/whisper)
+- [FastAPI Documentation](https://fastapi.tiangolo.com/)
+## 🆘 Support
+For issues:
+1. Check Space logs (Settings → Logs)
+2. Verify environment variables are set
+3. Test with simple requests first
+4. Check API key is correct
+5. Review this guide for common issues
+## ✅ Success Checklist
+After deployment, verify:
+- [ ] Space builds successfully
+- [ ] Health check works: `GET /health`
+- [ ] Embeddings endpoint works
+- [ ] Audio upload endpoint works
+- [ ] API key authentication works
+- [ ] Rate limiting is configured
+- [ ] Documentation is clear for users
+**Your HF Space is ready to use! 🎉**

README.md CHANGED Viewed

@@ -34,10 +34,7 @@ A centralized, extensible FastAPI server providing reusable APIs with robust aut
 | **Subtitles** | `POST /api/v1/subtitles/transcribe` | Transcribe uploaded audio with Whisper ✅ |
 | **Embeddings** | `POST /api/v1/embeddings/generate` | Generate text embeddings (1024-dim) |
-> ⚠️ **Note:** The YouTube extraction endpoint requires external network access. On **Hugging Face Spaces**, you can:
-> - ✅ Use the `/transcribe` endpoint (upload audio files)
-> - ✅ Deploy a [proxy service](#bypassing-hf-spaces-restrictions) to enable YouTube downloads
-> - ⚠️ Or use self-hosted deployment for direct access
 ## Quick Start
@@ -72,7 +69,6 @@ docker run -p 7860:7860 -e API_KEYS=your-key multiutility-server
 | `LOG_LEVEL` | Logging level | `INFO` |
 | `WHISPER_MODEL` | Whisper model size | `base` |
 | `EMBEDDING_MODEL` | HuggingFace model | `mixedbread-ai/mxbai-embed-large-v1` |
-| `YOUTUBE_PROXY_URL` | Proxy service URL (optional) | - |
 ## API Usage
@@ -88,7 +84,7 @@ curl -H "x-api-key: your-api-key" http://localhost:8000/api/v1/...
 #### Extract from YouTube URL
-> ⚠️ **Important:** This endpoint requires network access to YouTube. On HF Spaces, configure `YOUTUBE_PROXY_URL` to bypass restrictions (see [Proxy Setup](#bypassing-hf-spaces-restrictions)).
 ```bash
 curl -X POST http://localhost:8000/api/v1/subtitles/extract \
@@ -165,19 +161,22 @@ app/
 ### Hugging Face Spaces
-⚠️ **Network Limitation:** HF Spaces blocks external internet access. To enable YouTube downloads, deploy the included proxy service.
-**Working on HF Spaces:**
 - ✅ `/api/v1/subtitles/transcribe` - Upload audio files for transcription
 - ✅ `/api/v1/embeddings/generate` - Generate text embeddings
-- ⚠️ `/api/v1/subtitles/extract` - YouTube downloads (requires proxy service)
-**Basic Setup:**
 1. Create a Docker Space
 2. Set `API_KEYS` secret in Space settings
 3. Push repository
-**For YouTube extraction, see [Bypassing HF Spaces Restrictions](#bypassing-hf-spaces-restrictions) below.**
 ### Docker Compose
@@ -185,70 +184,29 @@ app/
 docker-compose up --build
 ```
-## Bypassing HF Spaces Restrictions
-### Problem
-Hugging Face Spaces blocks external network access, preventing YouTube downloads.
-### Solution: Proxy Service
-Deploy the included proxy service on a platform **with** internet access (Railway, Render, etc.) to act as an intermediary.
 ```
-HF Spaces → Proxy Service → YouTube → Proxy → HF Spaces → Whisper
-```
-### Quick Setup
-1. **Deploy Proxy Service** (choose one):
-   **Railway (Recommended):**
-   ```bash
-   cd proxy_service
-   railway login
-   railway init
-   railway up
-   railway domain  # Get your URL
-   ```
-   **Render.com:**
-   - Push `proxy_service/` to GitHub
-   - Create new Web Service on Render
-   - Connect repo, Render auto-detects configuration
-   **Docker (Self-hosted):**
-   ```bash
-   cd proxy_service
-   docker build -t youtube-proxy .
-   docker run -p 8080:8080 youtube-proxy
-   ```
-2. **Configure Main Server:**
-   ```bash
-   # In HF Spaces secrets or .env file
-   YOUTUBE_PROXY_URL=https://your-proxy.railway.app/download
-   ```
-3. **Test:**
-   ```bash
-   curl -X POST https://your-space.hf.space/api/v1/subtitles/extract \
-     -H "Content-Type: application/json" \
-     -H "x-api-key: your-key" \
-     -d '{"url": "https://youtube.com/watch?v=dQw4w9WgXcQ", "lang": "en"}'
-   ```
-### How It Works
-1. Main server tries direct YouTube download
-2. If blocked (network error), automatically falls back to proxy
-3. Proxy downloads audio and returns file
-4. Main server transcribes with Whisper
-See `proxy_service/README.md` for detailed deployment instructions and platform comparisons.
-### Free Deployment Options
-- **Railway:** 500 hours/month free
-- **Render:** Free tier with auto-sleep
-- **Fly.io:** 3 VMs free tier
-- **Google Cloud Run:** 2M requests/month free
 ## Development

 | **Subtitles** | `POST /api/v1/subtitles/transcribe` | Transcribe uploaded audio with Whisper ✅ |
 | **Embeddings** | `POST /api/v1/embeddings/generate` | Generate text embeddings (1024-dim) |
+> ⚠️ **Note on HF Spaces:** The YouTube extraction endpoint (`/extract`) requires external network access and will **not work on Hugging Face Spaces** due to platform restrictions. Instead, use the **audio file upload endpoint** (`/transcribe`) which works perfectly on HF Spaces. For YouTube extraction, use a self-hosted deployment.
 ## Quick Start
 | `LOG_LEVEL` | Logging level | `INFO` |
 | `WHISPER_MODEL` | Whisper model size | `base` |
 | `EMBEDDING_MODEL` | HuggingFace model | `mixedbread-ai/mxbai-embed-large-v1` |
 ## API Usage
 #### Extract from YouTube URL
+> ⚠️ **Important:** This endpoint requires network access to YouTube and will **not work on Hugging Face Spaces**. Use the audio file upload endpoint below instead, or deploy on a self-hosted environment.
 ```bash
 curl -X POST http://localhost:8000/api/v1/subtitles/extract \
 ### Hugging Face Spaces
+⚠️ **Network Limitation:** HF Spaces blocks external internet access, so YouTube downloads are not possible.
+**What works on HF Spaces:**
 - ✅ `/api/v1/subtitles/transcribe` - Upload audio files for transcription
 - ✅ `/api/v1/embeddings/generate` - Generate text embeddings
+- ❌ `/api/v1/subtitles/extract` - YouTube downloads (requires self-hosted deployment)
+**Setup:**
 1. Create a Docker Space
 2. Set `API_KEYS` secret in Space settings
 3. Push repository
+**Recommended workflow for subtitles:**
+1. Download audio locally using [yt-dlp](https://github.com/yt-dlp/yt-dlp): `yt-dlp -x --audio-format mp3 VIDEO_URL`
+2. Upload the audio file to `/api/v1/subtitles/transcribe` endpoint
+3. Receive transcription from Whisper
 ### Docker Compose
 docker-compose up --build
 ```
+## Alternative: Self-Hosted Deployment for YouTube Extraction
+If you need YouTube subtitle extraction, deploy the server on a platform with internet access:
+### Docker (VPS/Cloud VM)
+```bash
+docker build -t multiutility-server .
+docker run -p 7860:7860 -e API_KEYS=your-key multiutility-server
 ```
+### Cloud Platforms
+- **Railway:** Direct Docker deployment
+- **Render:** Connect GitHub repo, auto-deploy
+- **DigitalOcean:** Deploy on Droplet ($4-12/month)
+- **AWS/GCP/Azure:** Use ECS, Cloud Run, or App Service
+### Benefits of Self-Hosted
+- ✅ Direct YouTube access (no restrictions)
+- ✅ Full control over resources
+- ✅ No usage limits
+- ✅ All features work natively
 ## Development

app/apis/subtitles/service.py CHANGED Viewed

@@ -8,7 +8,6 @@ import threading
 from pathlib import Path
 from typing import TYPE_CHECKING, List, Optional, Tuple
-import httpx
 from cachetools import TTLCache
 from app.apis.subtitles.utils import extract_video_id
@@ -79,10 +78,7 @@ class SubtitleService:
             return SUBTITLE_CACHE[cache_key]
         with tempfile.TemporaryDirectory() as temp_dir:
-            # Try direct download first, fall back to proxy if available
-            audio_path = await self._download_audio_with_fallback(
-                url, temp_dir, video_id
-            )
             if not audio_path or not audio_path.exists():
                 raise SubtitleExtractionError("Failed to download audio from video")
@@ -96,51 +92,8 @@ class SubtitleService:
             SUBTITLE_CACHE[cache_key] = result
             return result
-    async def _download_audio_with_fallback(
-        self, url: str, temp_dir: str, video_id: str
-    ) -> Path:
-        """
-        Download audio with fallback to proxy service.
-        Tries direct yt-dlp download first. If that fails due to network restrictions
-        (e.g., on HF Spaces), falls back to proxy service if configured.
-        """
-        try:
-            # Try direct download first
-            return await self._download_audio(url, temp_dir, video_id)
-        except SubtitleExtractionError as e:
-            error_msg = str(e)
-            # Check if it's a network connectivity issue
-            if (
-                "Network connectivity issue" in error_msg
-                or "Failed to resolve" in error_msg
-            ):
-                # Try proxy service if configured
-                if settings.youtube_proxy_url:
-                    logger.info(
-                        f"Direct download failed, attempting proxy download via {settings.youtube_proxy_url}"
-                    )
-                    try:
-                        return await self._download_audio_via_proxy(
-                            url, temp_dir, video_id
-                        )
-                    except Exception as proxy_error:
-                        logger.error(f"Proxy download also failed: {proxy_error}")
-                        raise SubtitleExtractionError(
-                            f"Both direct and proxy downloads failed. Direct: {error_msg}. "
-                            f"Proxy: {str(proxy_error)}"
-                        )
-                else:
-                    logger.warning(
-                        "No proxy service configured, cannot bypass network restriction"
-                    )
-            # Re-raise original error if not a network issue or no proxy available
-            raise
     async def _download_audio(self, url: str, temp_dir: str, video_id: str) -> Path:
-        """Download audio from video URL using yt-dlp (direct method)."""
         cmd = [
             sys.executable,
             "-m",
@@ -178,7 +131,8 @@ class SubtitleService:
                     raise SubtitleExtractionError(
                         "Network connectivity issue: Unable to reach YouTube. "
                         "This service may be running in a sandboxed environment (e.g., Hugging Face Spaces) "
-                        "that blocks external internet access. Please use a self-hosted deployment for YouTube downloads."
                     )
                 if "Video unavailable" in error_msg or "Private video" in error_msg:
@@ -201,50 +155,6 @@ class SubtitleService:
         except asyncio.TimeoutError:
             raise DownloadTimeoutError("Timeout while downloading audio")
-    async def _download_audio_via_proxy(
-        self, url: str, temp_dir: str, video_id: str
-    ) -> Path:
-        """
-        Download audio via external proxy service.
-        The proxy service should accept POST requests with JSON body:
-        {"url": "youtube_url"} and return the audio file.
-        """
-        if not settings.youtube_proxy_url:
-            raise SubtitleExtractionError("Proxy URL not configured")
-        output_path = Path(temp_dir) / f"{video_id}.mp3"
-        logger.info(f"Requesting audio download from proxy: {url}")
-        try:
-            async with httpx.AsyncClient(timeout=self.timeout_download) as client:
-                response = await client.post(
-                    settings.youtube_proxy_url,
-                    json={"url": url, "format": "mp3"},
-                    follow_redirects=True,
-                )
-                if response.status_code != 200:
-                    error_msg = response.text[:200]
-                    raise SubtitleExtractionError(
-                        f"Proxy service returned status {response.status_code}: {error_msg}"
-                    )
-                # Save the downloaded audio
-                output_path.write_bytes(response.content)
-                if not output_path.exists() or output_path.stat().st_size == 0:
-                    raise SubtitleExtractionError("Proxy returned empty audio file")
-                logger.info(f"Audio downloaded via proxy: {output_path}")
-                return output_path
-        except httpx.TimeoutException:
-            raise DownloadTimeoutError("Timeout while downloading audio via proxy")
-        except httpx.RequestError as e:
-            raise SubtitleExtractionError(f"Proxy request failed: {str(e)}")
     async def _transcribe_audio(self, audio_path: Path, lang: str) -> List[str]:
         """Transcribe audio file using Whisper."""
         self._load_whisper_model()

 from pathlib import Path
 from typing import TYPE_CHECKING, List, Optional, Tuple
 from cachetools import TTLCache
 from app.apis.subtitles.utils import extract_video_id
             return SUBTITLE_CACHE[cache_key]
         with tempfile.TemporaryDirectory() as temp_dir:
+            audio_path = await self._download_audio(url, temp_dir, video_id)
             if not audio_path or not audio_path.exists():
                 raise SubtitleExtractionError("Failed to download audio from video")
             SUBTITLE_CACHE[cache_key] = result
             return result
     async def _download_audio(self, url: str, temp_dir: str, video_id: str) -> Path:
+        """Download audio from video URL using yt-dlp."""
         cmd = [
             sys.executable,
             "-m",
                     raise SubtitleExtractionError(
                         "Network connectivity issue: Unable to reach YouTube. "
                         "This service may be running in a sandboxed environment (e.g., Hugging Face Spaces) "
+                        "that blocks external internet access. Please use the audio file upload endpoint "
+                        "(/api/v1/subtitles/transcribe) instead, or use a self-hosted deployment."
                     )
                 if "Video unavailable" in error_msg or "Private video" in error_msg:
         except asyncio.TimeoutError:
             raise DownloadTimeoutError("Timeout while downloading audio")
     async def _transcribe_audio(self, audio_path: Path, lang: str) -> List[str]:
         """Transcribe audio file using Whisper."""
         self._load_whisper_model()

app/core/config.py CHANGED Viewed

@@ -31,9 +31,6 @@ class Settings(BaseSettings):
     # Embedding configuration
     embedding_model: str = "mixedbread-ai/mxbai-embed-large-v1"
-    # Proxy configuration for bypassing HF Spaces network restrictions
-    youtube_proxy_url: str = ""  # Optional proxy service URL for YouTube downloads
     # Server configuration
     host: str = "0.0.0.0"
     port: int = 8000

     # Embedding configuration
     embedding_model: str = "mixedbread-ai/mxbai-embed-large-v1"
     # Server configuration
     host: str = "0.0.0.0"
     port: int = 8000

poetry.lock CHANGED Viewed

@@ -672,28 +672,6 @@ files = [
 [package.extras]
 tests = ["pytest"]
-[[package]]
-name = "httpcore"
-version = "1.0.9"
-description = "A minimal low-level HTTP client."
-optional = false
-python-versions = ">=3.8"
-groups = ["main"]
-files = [
-    {file = "httpcore-1.0.9-py3-none-any.whl", hash = "sha256:2d400746a40668fc9dec9810239072b40b4484b640a8c38fd654a024c7a1bf55"},
-    {file = "httpcore-1.0.9.tar.gz", hash = "sha256:6e34463af53fd2ab5d807f399a9b45ea31c3dfa2276f15a2c3f00afff6e176e8"},
-]
-[package.dependencies]
-certifi = "*"
-h11 = ">=0.16"
-[package.extras]
-asyncio = ["anyio (>=4.0,<5.0)"]
-http2 = ["h2 (>=3,<5)"]
-socks = ["socksio (==1.*)"]
-trio = ["trio (>=0.22.0,<1.0)"]
 [[package]]
 name = "httptools"
 version = "0.6.4"
@@ -750,31 +728,6 @@ files = [
 [package.extras]
 test = ["Cython (>=0.29.24)"]
-[[package]]
-name = "httpx"
-version = "0.25.2"
-description = "The next generation HTTP client."
-optional = false
-python-versions = ">=3.8"
-groups = ["main"]
-files = [
-    {file = "httpx-0.25.2-py3-none-any.whl", hash = "sha256:a05d3d052d9b2dfce0e3896636467f8a5342fb2b902c819428e1ac65413ca118"},
-    {file = "httpx-0.25.2.tar.gz", hash = "sha256:8b8fcaa0c8ea7b05edd69a094e63a2094c4efcb48129fb757361bc423c0ad9e8"},
-]
-[package.dependencies]
-anyio = "*"
-certifi = "*"
-httpcore = "==1.*"
-idna = "*"
-sniffio = "*"
-[package.extras]
-brotli = ["brotli ; platform_python_implementation == \"CPython\"", "brotlicffi ; platform_python_implementation != \"CPython\""]
-cli = ["click (==8.*)", "pygments (==2.*)", "rich (>=10,<14)"]
-http2 = ["h2 (>=3,<5)"]
-socks = ["socksio (==1.*)"]
 [[package]]
 name = "huggingface-hub"
 version = "0.36.0"
@@ -3206,4 +3159,4 @@ test = ["pytest (>=8.1,<9.0)", "pytest-rerunfailures (>=14.0,<15.0)"]
 [metadata]
 lock-version = "2.1"
 python-versions = "^3.11"
-content-hash = "7ca1372ab8050eedee965f5b1059f004ace52d36c44c0f6f00e042a4d5a0b35e"

 [package.extras]
 tests = ["pytest"]
 [[package]]
 name = "httptools"
 version = "0.6.4"
 [package.extras]
 test = ["Cython (>=0.29.24)"]
 [[package]]
 name = "huggingface-hub"
 version = "0.36.0"
 [metadata]
 lock-version = "2.1"
 python-versions = "^3.11"
+content-hash = "ec39fc9067b87ef79eb93b123db27d3f8f462a61b46f0475263bd2a431f65fea"

proxy_service/Dockerfile DELETED Viewed

@@ -1,35 +0,0 @@
-# Dockerfile for YouTube Audio Proxy Service
-# Lightweight FastAPI service for downloading YouTube audio
-FROM python:3.11-slim
-# Set working directory
-WORKDIR /app
-# Install system dependencies for yt-dlp
-RUN apt-get update && apt-get install -y \
-    ffmpeg \
-    && rm -rf /var/lib/apt/lists/*
-# Copy requirements first for better caching
-COPY requirements.txt .
-# Install Python dependencies
-RUN pip install --no-cache-dir -r requirements.txt
-# Copy application code
-COPY main.py .
-# Expose port
-EXPOSE 8080
-# Set environment variables
-ENV PORT=8080
-ENV PYTHONUNBUFFERED=1
-# Health check
-HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
-    CMD python -c "import requests; requests.get('http://localhost:8080/health')"
-# Run the application
-CMD ["python", "main.py"]

proxy_service/README.md DELETED Viewed

@@ -1,409 +0,0 @@
-# YouTube Audio Proxy Service
-A lightweight FastAPI microservice that downloads YouTube audio files. Designed to bypass network restrictions in sandboxed environments like Hugging Face Spaces.
-## 🎯 Purpose
-Hugging Face Spaces and similar platforms block external internet access for security reasons. This proxy service runs on a platform **with** internet access and acts as an intermediary for YouTube downloads.
-## 🏗️ Architecture
-```
-┌─────────────────────┐         ┌──────────────────┐         ┌─────────────┐
-│  HF Spaces Server   │ ─────▶  │  Proxy Service   │ ─────▶  │  YouTube    │
-│  (No Internet)      │         │  (Has Internet)  │         │             │
-└─────────────────────┘         └──────────────────┘         └─────────────┘
-         │                               │
-         │                               │
-         ▼                               ▼
-   Transcribes                    Downloads Audio
-   with Whisper                   & Returns File
-```
-## 🚀 Quick Start
-### Local Testing
-```bash
-cd proxy_service
-# Install dependencies
-pip install -r requirements.txt
-# Run server
-python main.py
-```
-Server starts at `http://localhost:8080`
-### Test the Endpoint
-```bash
-curl -X POST http://localhost:8080/download \
-  -H "Content-Type: application/json" \
-  -d '{"url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ", "format": "mp3"}'
-```
-## 📦 Deployment Options
-### Option 1: Railway.app (Recommended - Free Tier Available)
-1. **Install Railway CLI:**
-   ```bash
-   npm install -g @railway/cli
-   ```
-2. **Login and deploy:**
-   ```bash
-   railway login
-   railway init
-   railway up
-   ```
-3. **Get your service URL:**
-   ```bash
-   railway domain
-   ```
-4. **Configure main server:**
-   ```bash
-   # In your main .env file
-   YOUTUBE_PROXY_URL=https://your-service.railway.app/download
-   ```
-**Pros:** Easy deployment, free tier, automatic HTTPS, good performance
-**Cons:** Free tier has usage limits
----
-### Option 2: Render.com (Free Tier Available)
-1. **Create a new Web Service** on [Render.com](https://render.com)
-2. **Connect your Git repository** or deploy manually
-3. **Configure:**
-   - Build Command: `pip install -r requirements.txt`
-   - Start Command: `python main.py`
-   - Or use the included `render.yaml` for automatic configuration
-4. **Copy the service URL** (e.g., `https://your-service.onrender.com`)
-5. **Update main server:**
-   ```bash
-   YOUTUBE_PROXY_URL=https://your-service.onrender.com/download
-   ```
-**Pros:** Free tier, simple setup, automatic SSL
-**Cons:** Free tier sleeps after inactivity (cold starts)
----
-### Option 3: Docker (Self-Hosted)
-```bash
-# Build image
-docker build -t youtube-proxy .
-# Run container
-docker run -p 8080:8080 youtube-proxy
-# Or use docker-compose
-docker-compose up -d
-```
-**docker-compose.yml example:**
-```yaml
-version: '3.8'
-services:
-  proxy:
-    build: .
-    ports:
-      - "8080:8080"
-    restart: unless-stopped
-    environment:
-      - PORT=8080
-```
-**Pros:** Full control, no usage limits
-**Cons:** Requires server infrastructure
----
-### Option 4: Fly.io (Free Tier Available)
-```bash
-# Install flyctl
-curl -L https://fly.io/install.sh | sh
-# Login and launch
-flyctl auth login
-flyctl launch
-# Deploy
-flyctl deploy
-```
-**Pros:** Good free tier, edge network, fast
-**Cons:** Requires credit card for verification
----
-### Option 5: AWS Lambda (Serverless)
-Use [Mangum](https://mangum.io/) to deploy FastAPI to AWS Lambda:
-```python
-# lambda_handler.py
-from mangum import Mangum
-from main import app
-handler = Mangum(app)
-```
-**Pros:** Scales automatically, pay-per-use
-**Cons:** More complex setup, cold starts
----
-### Option 6: Google Cloud Run (Free Tier)
-```bash
-# Build and deploy
-gcloud run deploy youtube-proxy \
-  --source . \
-  --platform managed \
-  --region us-central1 \
-  --allow-unauthenticated
-```
-**Pros:** Generous free tier, auto-scaling
-**Cons:** Requires Google Cloud account
-## 🔧 Configuration
-The proxy service accepts these environment variables:
-| Variable | Description | Default |
-|----------|-------------|---------|
-| `PORT` | Server port | `8080` |
-| `PYTHONUNBUFFERED` | Python output buffering | `1` |
-## 📡 API Endpoints
-### `POST /download`
-Download YouTube audio and return the file.
-**Request:**
-```json
-{
-  "url": "https://www.youtube.com/watch?v=VIDEO_ID",
-  "format": "mp3"
-}
-```
-**Supported formats:** `mp3`, `m4a`, `wav`, `opus`
-**Response:** Binary audio file
-**Status Codes:**
-- `200`: Success - returns audio file
-- `400`: Invalid request (bad URL or format)
-- `403`: Video is private
-- `404`: Video not found
-- `500`: Download failed
-- `504`: Download timeout
----
-### `GET /health`
-Health check endpoint.
-**Response:**
-```json
-{
-  "status": "healthy",
-  "service": "youtube-audio-proxy",
-  "yt_dlp_available": true
-}
-```
----
-### `GET /`
-Service information and usage instructions.
-## 🔗 Connecting to Main Server
-After deploying the proxy service:
-1. **Copy the service URL** (e.g., `https://your-proxy.railway.app`)
-2. **Update main server configuration:**
-   **Option A: Environment Variable**
-   ```bash
-   export YOUTUBE_PROXY_URL=https://your-proxy.railway.app/download
-   ```
-   **Option B: .env file**
-   ```env
-   YOUTUBE_PROXY_URL=https://your-proxy.railway.app/download
-   ```
-   **Option C: Docker**
-   ```bash
-   docker run -e YOUTUBE_PROXY_URL=https://your-proxy.railway.app/download ...
-   ```
-3. **Verify configuration:**
-   ```bash
-   # Test the subtitle extraction endpoint
-   curl -X POST https://your-hf-space.hf.space/api/v1/subtitles/extract \
-     -H "Content-Type: application/json" \
-     -H "x-api-key: your-key" \
-     -d '{"url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ", "lang": "en"}'
-   ```
-## 🔒 Security Considerations
-### Rate Limiting
-Consider adding rate limiting to prevent abuse:
-```python
-from slowapi import Limiter, _rate_limit_exceeded_handler
-from slowapi.util import get_remote_address
-limiter = Limiter(key_func=get_remote_address)
-app.state.limiter = limiter
-@app.post("/download")
-@limiter.limit("10/minute")
-async def download_audio(request: DownloadRequest):
-    ...
-```
-### Authentication
-Add API key authentication for production:
-```python
-from fastapi import Header, HTTPException
-async def verify_api_key(x_api_key: str = Header(...)):
-    if x_api_key not in VALID_API_KEYS:
-        raise HTTPException(status_code=401, detail="Invalid API key")
-```
-### CORS Configuration
-Update CORS settings for production:
-```python
-app.add_middleware(
-    CORSMiddleware,
-    allow_origins=["https://your-main-service.com"],  # Specific origins
-    allow_credentials=True,
-    allow_methods=["POST"],
-    allow_headers=["Content-Type"],
-)
-```
-## 📊 Monitoring
-### Health Checks
-All deployment platforms support health checks via `/health` endpoint.
-### Logging
-Add structured logging for monitoring:
-```python
-import logging
-logging.basicConfig(
-    level=logging.INFO,
-    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
-)
-logger = logging.getLogger(__name__)
-```
-## 🐛 Troubleshooting
-### "Video unavailable" error
-- Check if the video is private or region-restricted
-- Verify the URL is correct
-- Try the video on youtube.com directly
-### Timeout errors
-- Increase timeout in main server config: `YT_DLP_TIMEOUT_DOWNLOAD=180`
-- Check proxy service logs
-- Consider upgrading server resources
-### "yt-dlp not found" error
-- Ensure `yt-dlp` is in requirements.txt
-- Verify ffmpeg is installed (required for audio conversion)
-- Check Docker image includes system dependencies
-### Slow downloads
-- Upgrade proxy service plan for better resources
-- Use a region closer to your main service
-- Consider caching frequently requested videos
-## 💰 Cost Estimates
-### Free Tier Options
-- **Railway:** 500 hours/month free, then $5/month
-- **Render:** 750 hours/month free, sleep after 15min inactivity
-- **Fly.io:** 3 shared-cpu VMs free
-- **Google Cloud Run:** 2 million requests/month free
-### Paid Options
-- **Railway:** $5-20/month for consistent uptime
-- **AWS Lambda:** ~$0.20 per 1 million requests
-- **DigitalOcean:** $4/month for basic droplet
-## 🎓 How It Works
-1. **Main server** receives subtitle extraction request
-2. **Main server** tries direct YouTube download via `yt-dlp`
-3. **If blocked** (network error), falls back to proxy service
-4. **Proxy service** downloads audio using `yt-dlp` (has internet access)
-5. **Proxy service** returns audio file bytes
-6. **Main server** saves audio to temp directory
-7. **Main server** transcribes audio with Whisper
-8. Returns subtitles to user
-## 🔄 Updates
-Keep yt-dlp updated for best compatibility:
-```bash
-pip install --upgrade yt-dlp
-```
-## 📝 License
-Same as main project (MIT License)
-## 🤝 Contributing
-To improve this proxy service:
-1. Add caching for frequently requested videos
-2. Implement video quality selection
-3. Add support for playlists
-4. Improve error handling and logging
-5. Add metrics and analytics
-## 📚 Additional Resources
-- [yt-dlp Documentation](https://github.com/yt-dlp/yt-dlp)
-- [FastAPI Documentation](https://fastapi.tiangolo.com/)
-- [Railway Documentation](https://docs.railway.app/)
-- [Render Documentation](https://render.com/docs)

proxy_service/main.py DELETED Viewed

@@ -1,216 +0,0 @@
-"""
-YouTube Audio Download Proxy Service
-A simple FastAPI service that downloads YouTube audio and returns it.
-Deploy this on platforms with internet access (Vercel, Railway, Render, etc.)
-to bypass Hugging Face Spaces network restrictions.
-Usage:
-    uvicorn main:app --host 0.0.0.0 --port 8080
-Deployment:
-    - Vercel: Use vercel.json configuration
-    - Railway: Direct deployment
-    - Render: Use render.yaml configuration
-    - Docker: Standard FastAPI Docker setup
-"""
-import asyncio
-import os
-import sys
-import tempfile
-from pathlib import Path
-from typing import Optional
-from fastapi import FastAPI, HTTPException
-from fastapi.middleware.cors import CORSMiddleware
-from fastapi.responses import FileResponse, JSONResponse
-from pydantic import BaseModel, HttpUrl, field_validator
-# Initialize FastAPI app
-app = FastAPI(
-    title="YouTube Audio Proxy Service",
-    description="Proxy service for downloading YouTube audio in restricted environments",
-    version="1.0.0",
-)
-# Configure CORS - allow all origins for proxy service
-app.add_middleware(
-    CORSMiddleware,
-    allow_origins=["*"],
-    allow_credentials=True,
-    allow_methods=["*"],
-    allow_headers=["*"],
-)
-class DownloadRequest(BaseModel):
-    """Request model for audio download."""
-    url: HttpUrl
-    format: str = "mp3"
-    @field_validator("url")
-    @classmethod
-    def validate_youtube_url(cls, v: HttpUrl) -> HttpUrl:
-        """Validate that the URL is a YouTube URL."""
-        url_str = str(v)
-        if not any(domain in url_str for domain in ["youtube.com", "youtu.be"]):
-            raise ValueError("URL must be a valid YouTube URL")
-        return v
-    @field_validator("format")
-    @classmethod
-    def validate_format(cls, v: str) -> str:
-        """Validate audio format."""
-        allowed_formats = {"mp3", "m4a", "wav", "opus"}
-        if v.lower() not in allowed_formats:
-            raise ValueError(f"Format must be one of {allowed_formats}, got '{v}'")
-        return v.lower()
-@app.get("/")
-async def root():
-    """Root endpoint with service information."""
-    return {
-        "service": "YouTube Audio Proxy",
-        "version": "1.0.0",
-        "status": "operational",
-        "endpoints": {
-            "download": "POST /download",
-            "health": "GET /health",
-        },
-        "usage": {
-            "method": "POST",
-            "url": "/download",
-            "body": {
-                "url": "https://www.youtube.com/watch?v=VIDEO_ID",
-                "format": "mp3",
-            },
-        },
-    }
-@app.get("/health")
-async def health_check():
-    """Health check endpoint."""
-    return {
-        "status": "healthy",
-        "service": "youtube-audio-proxy",
-        "yt_dlp_available": True,
-    }
-@app.post("/download")
-async def download_audio(request: DownloadRequest):
-    """
-    Download YouTube audio and return the file.
-    Args:
-        request: Contains YouTube URL and desired audio format
-    Returns:
-        Audio file in requested format
-    """
-    temp_dir = None
-    try:
-        # Create temporary directory
-        temp_dir = tempfile.mkdtemp()
-        output_template = str(Path(temp_dir) / f"audio.%(ext)s")
-        # Build yt-dlp command
-        cmd = [
-            sys.executable,
-            "-m",
-            "yt_dlp",
-            "--extract-audio",
-            "--audio-format",
-            request.format,
-            "--audio-quality",
-            "5",
-            "--no-warnings",
-            "--no-playlist",
-            "--output",
-            output_template,
-            str(request.url),
-        ]
-        # Execute download
-        process = await asyncio.create_subprocess_exec(
-            *cmd,
-            stdout=asyncio.subprocess.PIPE,
-            stderr=asyncio.subprocess.PIPE,
-        )
-        stdout, stderr = await asyncio.wait_for(process.communicate(), timeout=120)
-        if process.returncode != 0:
-            error_msg = stderr.decode("utf-8", errors="ignore")
-            # Parse common errors
-            if "Video unavailable" in error_msg:
-                raise HTTPException(
-                    status_code=404, detail="Video not found or unavailable"
-                )
-            elif "Private video" in error_msg:
-                raise HTTPException(status_code=403, detail="Video is private")
-            else:
-                raise HTTPException(
-                    status_code=500,
-                    detail=f"Download failed: {error_msg[:200]}",
-                )
-        # Find downloaded file
-        audio_files = list(Path(temp_dir).glob(f"audio.*"))
-        if not audio_files:
-            raise HTTPException(
-                status_code=500, detail="Audio file not found after download"
-            )
-        audio_file = audio_files[0]
-        # Return the audio file
-        return FileResponse(
-            path=str(audio_file),
-            media_type=f"audio/{request.format}",
-            filename=f"audio.{request.format}",
-            background=None,  # Don't delete yet
-        )
-    except asyncio.TimeoutError:
-        raise HTTPException(status_code=504, detail="Download timeout (exceeded 120s)")
-    except HTTPException:
-        raise
-    except Exception as e:
-        raise HTTPException(status_code=500, detail=f"Unexpected error: {str(e)}")
-    finally:
-        # Cleanup will happen automatically when temp dir is garbage collected
-        # For production, consider implementing proper cleanup
-        pass
-@app.exception_handler(Exception)
-async def global_exception_handler(request, exc):
-    """Global exception handler."""
-    return JSONResponse(
-        status_code=500,
-        content={
-            "status": "error",
-            "message": str(exc),
-            "detail": "An unexpected error occurred",
-        },
-    )
-if __name__ == "__main__":
-    import uvicorn
-    port = int(os.environ.get("PORT", 8080))
-    uvicorn.run(
-        "main:app",
-        host="0.0.0.0",
-        port=port,
-        reload=False,
-    )

proxy_service/render.yaml DELETED Viewed

@@ -1,21 +0,0 @@
-# Render.com deployment configuration for YouTube Audio Proxy Service
-# This service provides YouTube audio downloads for restricted environments
-services:
-  - type: web
-    name: youtube-audio-proxy
-    env: docker
-    dockerfilePath: ./Dockerfile
-    plan: free
-    region: oregon
-    healthCheckPath: /health
-    envVars:
-      - key: PORT
-        value: 8080
-      - key: PYTHONUNBUFFERED
-        value: 1
-    autoDeploy: true
-    disk:
-      name: temp-storage
-      mountPath: /tmp
-      sizeGB: 1

proxy_service/requirements.txt DELETED Viewed

@@ -1,6 +0,0 @@
-fastapi==0.104.1
-uvicorn[standard]==0.24.0
-pydantic==2.5.0
-pydantic-settings==2.1.0
-yt-dlp==2023.11.16
-httpx==0.25.2

pyproject.toml CHANGED Viewed

@@ -20,7 +20,6 @@ cachetools = "^5.3.0"
 sentence-transformers = "^2.2.2"
 torch = "^2.0.0"
 faster-whisper = "^1.0.0"
-httpx = "^0.25.2"
 [tool.poetry.group.dev.dependencies]
 pytest = "^7.4.3"

 sentence-transformers = "^2.2.2"
 torch = "^2.0.0"
 faster-whisper = "^1.0.0"
 [tool.poetry.group.dev.dependencies]
 pytest = "^7.4.3"