Spaces:

abhisheksan
/

multiutility-server

Running

App Files Files Community

abhisheksan commited on Jan 14

Commit

e125769

1 Parent(s): bc9b12e

Remove extra documentation files

Browse files

Files changed (2) hide show

HF_SPACES_GUIDE.md +0 -442
SOLUTION.md +0 -176

HF_SPACES_GUIDE.md DELETED Viewed

@@ -1,442 +0,0 @@
-# Hugging Face Spaces Deployment Guide
-## 🎯 Overview
-This guide explains how to deploy and use the Multi-Utility Server on Hugging Face Spaces, including limitations and workarounds.
-## 🚀 Quick Deployment
-### Step 1: Create a Space
-1. Go to [Hugging Face Spaces](https://huggingface.co/spaces)
-2. Click **"Create new Space"**
-3. Choose:
-   - **Space name:** Your choice
-   - **SDK:** Docker
-   - **Visibility:** Public or Private
-4. Click **"Create Space"**
-### Step 2: Configure Secrets
-1. Go to your Space's **Settings** → **Repository secrets**
-2. Add a new secret:
-   - **Name:** `API_KEYS`
-   - **Value:** `your-secure-api-key-here` (comma-separated for multiple keys)
-3. Save
-### Step 3: Push Code
-```bash
-# Clone your space
-git clone https://huggingface.co/spaces/YOUR_USERNAME/YOUR_SPACE_NAME
-cd YOUR_SPACE_NAME
-# Add this repository as a remote
-git remote add source https://github.com/YOUR_REPO/multiutility-server.git
-git pull source main
-# Push to HF Spaces
-git push origin main
-```
-Or connect your GitHub repository directly in Space settings.
-## 📊 Feature Availability on HF Spaces
-| Feature | Status | Endpoint |
-|---------|--------|----------|
-| **Text Embeddings** | ✅ Works | `POST /api/v1/embeddings/generate` |
-| **Audio File Transcription** | ✅ Works | `POST /api/v1/subtitles/transcribe` |
-| **YouTube Subtitle Extraction** | ❌ Blocked | `POST /api/v1/subtitles/extract` |
-| **Health Checks** | ✅ Works | `GET /health` |
-## ⚠️ Network Limitations
-### What's Blocked
-Hugging Face Spaces runs in a sandboxed environment that **blocks external internet access** for security reasons. This means:
-- ❌ Cannot download from YouTube directly
-- ❌ Cannot access external APIs
-- ❌ Cannot perform web scraping
-### What Works
-- ✅ File uploads from users
-- ✅ AI model inference (Whisper, embeddings)
-- ✅ Returning results to users
-- ✅ Internal HF services
-## 🎤 Audio Transcription Workflow
-Since YouTube downloads don't work on HF Spaces, use this workflow instead:
-### Option 1: User Downloads Audio Locally
-**Step 1:** User downloads audio using [yt-dlp](https://github.com/yt-dlp/yt-dlp)
-```bash
-# Install yt-dlp
-pip install yt-dlp
-# Download audio from YouTube
-yt-dlp -x --audio-format mp3 "https://www.youtube.com/watch?v=VIDEO_ID" -o audio.mp3
-```
-**Step 2:** User uploads audio to your HF Space
-```bash
-curl -X POST https://YOUR_SPACE.hf.space/api/v1/subtitles/transcribe \
-  -H "x-api-key: your-api-key" \
-  -F "file=@audio.mp3" \
-  -F "lang=en"
-```
-**Step 3:** Receive transcription
-```json
-{
-  "status": "success",
-  "language": "en",
-  "file_name": "audio.mp3",
-  "transcription": [
-    "First segment of transcribed text",
-    "Second segment of transcribed text",
-    "..."
-  ]
-}
-```
-### Option 2: Browser-Based Upload
-Create a simple HTML form for users:
-```html
-<!DOCTYPE html>
-<html>
-<body>
-  <h2>Audio Transcription</h2>
-  <form id="uploadForm">
-    <input type="file" id="audioFile" accept="audio/*" required>
-    <select id="language">
-      <option value="en">English</option>
-      <option value="es">Spanish</option>
-      <option value="fr">French</option>
-    </select>
-    <button type="submit">Transcribe</button>
-  </form>
-  <div id="result"></div>
-  <script>
-    document.getElementById('uploadForm').onsubmit = async (e) => {
-      e.preventDefault();
-      const formData = new FormData();
-      formData.append('file', document.getElementById('audioFile').files[0]);
-      formData.append('lang', document.getElementById('language').value);
-      const response = await fetch('https://YOUR_SPACE.hf.space/api/v1/subtitles/transcribe', {
-        method: 'POST',
-        headers: { 'x-api-key': 'your-api-key' },
-        body: formData
-      });
-      const result = await response.json();
-      document.getElementById('result').innerHTML =
-        '<pre>' + JSON.stringify(result, null, 2) + '</pre>';
-    };
-  </script>
-</body>
-</html>
-```
-## 📝 API Usage Examples
-### Text Embeddings (Works on HF Spaces)
-```python
-import requests
-url = "https://YOUR_SPACE.hf.space/api/v1/embeddings/generate"
-headers = {
-    "Content-Type": "application/json",
-    "x-api-key": "your-api-key"
-}
-data = {
-    "texts": [
-        "Hello, how are you?",
-        "Machine learning is fascinating"
-    ],
-    "normalize": True
-}
-response = requests.post(url, headers=headers, json=data)
-print(response.json())
-```
-### Audio File Transcription (Works on HF Spaces)
-```python
-import requests
-url = "https://YOUR_SPACE.hf.space/api/v1/subtitles/transcribe"
-headers = {"x-api-key": "your-api-key"}
-with open("audio.mp3", "rb") as audio_file:
-    files = {"file": audio_file}
-    data = {"lang": "en"}
-    response = requests.post(url, headers=headers, files=files, data=data)
-print(response.json())
-```
-### YouTube Extraction (Does NOT Work on HF Spaces)
-```python
-# ❌ This will fail on HF Spaces with network error
-import requests
-url = "https://YOUR_SPACE.hf.space/api/v1/subtitles/extract"
-headers = {
-    "Content-Type": "application/json",
-    "x-api-key": "your-api-key"
-}
-data = {
-    "url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ",
-    "lang": "en"
-}
-response = requests.post(url, headers=headers, json=data)
-# Error: Network connectivity issue
-```
-## 🔧 Configuration
-### Required Environment Variables
-Set these in HF Spaces **Repository secrets**:
-| Variable | Description | Example |
-|----------|-------------|---------|
-| `API_KEYS` | Comma-separated API keys | `key1,key2,key3` |
-### Optional Environment Variables
-| Variable | Description | Default |
-|----------|-------------|---------|
-| `CORS_ORIGINS` | Allowed origins | `*` |
-| `RATE_LIMIT_REQUESTS` | Requests per minute | `100` |
-| `LOG_LEVEL` | Logging level | `INFO` |
-| `WHISPER_MODEL` | Whisper model size | `base` |
-| `EMBEDDING_MODEL` | HuggingFace model | `mixedbread-ai/mxbai-embed-large-v1` |
-### Whisper Model Options
-| Model | Size | Speed | Accuracy |
-|-------|------|-------|----------|
-| `tiny` | 39 MB | Fastest | Lowest |
-| `base` | 74 MB | Fast | Good |
-| `small` | 244 MB | Medium | Better |
-| `medium` | 769 MB | Slow | Best |
-**Recommendation for HF Spaces:** Use `base` or `small` for good balance.
-## 🐛 Troubleshooting
-### Issue: Build fails with poetry.lock error
-**Error:**
-```
-The lock file might not be compatible with the current version of Poetry
-```
-**Solution:**
-```bash
-poetry lock
-git add poetry.lock
-git commit -m "Update poetry.lock"
-git push
-```
-### Issue: "Unauthorized" error
-**Error:**
-```json
-{"detail": "Unauthorized: Invalid or missing API key"}
-```
-**Solution:**
-- Verify `API_KEYS` secret is set in Space settings
-- Include `x-api-key` header in your requests
-- Check for typos in the API key
-### Issue: YouTube extraction fails
-**Error:**
-```json
-{
-  "status": "error",
-  "message": "Network connectivity issue: Unable to reach YouTube..."
-}
-```
-**Solution:**
-This is expected on HF Spaces. Use the audio upload endpoint instead:
-1. Download audio locally with yt-dlp
-2. Upload to `/api/v1/subtitles/transcribe`
-### Issue: Out of memory
-**Error:**
-```
-Container killed due to memory limit
-```
-**Solution:**
-- Use smaller Whisper model: `WHISPER_MODEL=tiny` or `WHISPER_MODEL=base`
-- Process shorter audio files
-- Consider upgrading to HF Spaces Pro (more RAM)
-### Issue: Slow transcription
-**Solution:**
-- Use smaller Whisper model (`tiny` or `base`)
-- Process shorter audio segments
-- Note: HF Spaces free tier uses CPU (no GPU)
-## 📈 Performance Tips
-### 1. Choose the Right Whisper Model
-```python
-# Fast but less accurate (good for testing)
-WHISPER_MODEL=tiny
-# Balanced (recommended for production)
-WHISPER_MODEL=base
-# Accurate but slow (only if you need high quality)
-WHISPER_MODEL=small
-```
-### 2. Optimize Audio Files
-```bash
-# Convert to optimal format before upload
-ffmpeg -i input.wav -ar 16000 -ac 1 -c:a libmp3lame output.mp3
-```
-### 3. Rate Limiting
-The server has rate limiting enabled:
-- Default: 100 requests per minute
-- Adjust via `RATE_LIMIT_REQUESTS` environment variable
-## 🔒 Security Best Practices
-### 1. Use Strong API Keys
-```bash
-# Generate secure API key
-openssl rand -base64 32
-```
-### 2. Rotate Keys Regularly
-Update `API_KEYS` in Space secrets monthly.
-### 3. Monitor Usage
-Check Space logs regularly:
-- Settings → Logs
-- Look for suspicious activity
-### 4. Use Private Spaces for Sensitive Data
-Consider making your Space private if handling sensitive content.
-## 💰 Cost Considerations
-### Free Tier
-- ✅ Unlimited inference
-- ✅ 16GB RAM
-- ✅ 2 vCPU
-- ⚠️ CPU-only (no GPU)
-- ⚠️ May sleep after inactivity
-### Spaces Pro ($5/month per Space)
-- ✅ Always-on
-- ✅ Better performance
-- ✅ More resources
-- ✅ Custom domains
-## 🎓 Best Practices
-### 1. Document the Workflow
-Add a README to your Space explaining:
-- How to download audio locally
-- How to use the upload endpoint
-- Supported audio formats
-### 2. Provide Examples
-Include example API calls and code snippets.
-### 3. Set Expectations
-Clearly state that YouTube direct extraction doesn't work on HF Spaces.
-### 4. Offer Alternatives
-Suggest self-hosted deployment for users who need YouTube extraction.
-## 🚀 Alternative Deployment
-If you need YouTube extraction, consider:
-### Self-Hosted Options
-1. **Docker on VPS** (DigitalOcean, Linode)
-   - Cost: $4-12/month
-   - Full control
-   - All features work
-2. **Cloud Platforms** (AWS, GCP, Azure)
-   - Scalable
-   - More expensive
-   - Enterprise-grade
-3. **Railway/Render**
-   - Easy deployment
-   - $5-20/month
-   - Good middle ground
-## 📚 Additional Resources
-- [Hugging Face Spaces Documentation](https://huggingface.co/docs/hub/spaces)
-- [yt-dlp Documentation](https://github.com/yt-dlp/yt-dlp)
-- [Whisper Model Information](https://github.com/openai/whisper)
-- [FastAPI Documentation](https://fastapi.tiangolo.com/)
-## 🆘 Support
-For issues:
-1. Check Space logs (Settings → Logs)
-2. Verify environment variables are set
-3. Test with simple requests first
-4. Check API key is correct
-5. Review this guide for common issues
-## ✅ Success Checklist
-After deployment, verify:
-- [ ] Space builds successfully
-- [ ] Health check works: `GET /health`
-- [ ] Embeddings endpoint works
-- [ ] Audio upload endpoint works
-- [ ] API key authentication works
-- [ ] Rate limiting is configured
-- [ ] Documentation is clear for users
-**Your HF Space is ready to use! 🎉**

SOLUTION.md DELETED Viewed

@@ -1,176 +0,0 @@
-# HF Spaces Solution - Simple Summary
-## ❌ Problem
-YouTube subtitle extraction fails on Hugging Face Spaces with network error:
-```
-Failed to resolve 'www.youtube.com' - No address associated with hostname
-```
-**Why?** HF Spaces blocks all external internet access for security.
----
-## ✅ Solution: Audio File Upload
-Instead of downloading from YouTube, users upload audio files directly.
-### What Works on HF Spaces
-| Endpoint | Status | Description |
-|----------|--------|-------------|
-| `POST /api/v1/subtitles/transcribe` | ✅ Works | Upload audio, get transcription |
-| `POST /api/v1/embeddings/generate` | ✅ Works | Generate text embeddings |
-| `POST /api/v1/subtitles/extract` | ❌ Blocked | YouTube downloads (self-hosted only) |
----
-## 🚀 Usage
-### Step 1: Download Audio Locally
-```bash
-# User downloads audio on their machine
-pip install yt-dlp
-yt-dlp -x --audio-format mp3 "https://youtube.com/watch?v=VIDEO_ID" -o audio.mp3
-```
-### Step 2: Upload to HF Spaces
-```bash
-curl -X POST https://your-space.hf.space/api/v1/subtitles/transcribe \
-  -H "x-api-key: your-key" \
-  -F "file=@audio.mp3" \
-  -F "lang=en"
-```
-### Step 3: Get Transcription
-```json
-{
-  "status": "success",
-  "language": "en",
-  "file_name": "audio.mp3",
-  "transcription": [
-    "Transcribed text segment 1",
-    "Transcribed text segment 2",
-    "..."
-  ]
-}
-```
----
-## 📝 Python Example
-```python
-import requests
-# Upload audio file for transcription
-url = "https://your-space.hf.space/api/v1/subtitles/transcribe"
-headers = {"x-api-key": "your-api-key"}
-with open("audio.mp3", "rb") as f:
-    files = {"file": f}
-    data = {"lang": "en"}
-    response = requests.post(url, headers=headers, files=files, data=data)
-result = response.json()
-print(result["transcription"])
-```
----
-## 🎯 Key Points
-1. **YouTube extraction doesn't work on HF Spaces** - this is by design, not a bug
-2. **Audio upload DOES work** - users download audio locally, then upload
-3. **Embeddings work fine** - no network access needed
-4. **For YouTube extraction** - use self-hosted deployment (Docker, VPS, cloud)
----
-## 📦 Supported Audio Formats
-- ✅ MP3
-- ✅ WAV
-- ✅ M4A
-- ✅ FLAC
-- ✅ OGG
-- ✅ WEBM
----
-## 🔧 HF Spaces Configuration
-### Required Secret
-```
-API_KEYS=your-secure-api-key
-```
-### Optional Variables
-```
-WHISPER_MODEL=base          # tiny, base, small, medium
-LOG_LEVEL=INFO              # DEBUG, INFO, WARNING, ERROR
-RATE_LIMIT_REQUESTS=100     # Requests per minute
-```
----
-## 🐛 Common Issues
-### "Unauthorized" Error
-**Fix:** Add `API_KEYS` secret in HF Spaces settings
-### "Network connectivity issue"
-**Fix:** This is expected - use audio upload endpoint instead
-### "Out of memory"
-**Fix:** Use smaller Whisper model: `WHISPER_MODEL=tiny` or `WHISPER_MODEL=base`
----
-## 🎓 Recommended Workflow
-### For End Users:
-1. Download YouTube audio using yt-dlp
-2. Upload audio file to your HF Space
-3. Receive transcription
-### For Developers:
-1. Deploy on HF Spaces (free hosting)
-2. Document the two-step process
-3. Or deploy self-hosted for direct YouTube access
----
-## 🚀 Self-Hosted Alternative
-If you need direct YouTube extraction:
-```bash
-# Deploy with Docker
-docker build -t multiutility-server .
-docker run -p 7860:7860 -e API_KEYS=your-key multiutility-server
-```
-Then YouTube extraction works natively (no upload needed).
----
-## ✅ Summary
-**What Changed:**
-- ✅ Added audio file upload endpoint (`/transcribe`)
-- ✅ Works perfectly on HF Spaces
-- ✅ No external dependencies needed
-- ❌ Removed proxy service (too complex)
-- ❌ YouTube endpoint doesn't work on HF Spaces (as expected)
-**Recommendation:**
-- Deploy on HF Spaces for free hosting
-- Use audio upload for transcription
-- Document the workflow for users
-- Or use self-hosted for YouTube extraction
-**Current Status:** ✅ Ready to deploy and use!
----
-For detailed guide, see: `HF_SPACES_GUIDE.md`