Spaces:
Sleeping
Sleeping
File size: 4,385 Bytes
e2c1f2e |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 |
# Distributed Media Encoder Service
A distributed video encoding service with a modern dark-themed UI, supporting multiple encoding instances for high-performance video processing.
## Features
- Modern dark-themed web interface with drag-and-drop
- Distributed video encoding across multiple workers
- Multiple quality outputs (480p, 720p, 1080p)
- Real-time encoding progress tracking
- Single-port access through Python proxy
- Temporary directory support for restricted environments
## System Architecture
- **Web Interface**: Modern dark theme UI for video upload and monitoring
- **Task Queue**: Redis-based distributed task queue
- **Workers**: Multiple Celery workers for distributed encoding
- **Storage**: Temporary directory based file storage
- **Proxy**: Python-based reverse proxy for single-port access
## Prerequisites
- Python 3.8+
- FFmpeg
- Redis
## Quick Start
1. Install dependencies:
```bash
pip install -r requirements.txt
```
2. Start the service:
```bash
python3 start_service.py --port 5000 --tmp-dir /tmp/encoder
```
The service will be available at `http://localhost:5000`
## Distributed Setup
### Main Instance
```bash
python3 start_service.py --port 5000 --tmp-dir /tmp/encoder
```
### Additional Worker Instances
```bash
# Worker 1
python3 start_service.py --worker-name encoder_worker_1 --tmp-dir /tmp/encoder_1
# Worker 2
python3 start_service.py --worker-name encoder_worker_2 --tmp-dir /tmp/encoder_2
# Add more workers as needed
```
## Command Line Options
- `--port`: Port number for the web interface (default: 5000)
- `--tmp-dir`: Temporary directory for storage (default: /tmp/encoder)
- `--worker-name`: Custom name for worker instance
## Directory Structure
```
/tmp/encoder/
├── uploads/ # Uploaded video files
├── encoded/ # Encoded video files
├── redis.conf # Redis configuration
├── redis.rdb # Redis database
└── redis.log # Redis logs
```
## Monitoring
### View Service Logs
```bash
tail -f /tmp/encoder/redis.log # Redis logs
```
### Check Worker Status
```python
from app.services.encoder_service import celery
celery.control.inspect().active() # View active tasks
celery.control.inspect().registered() # View registered workers
```
## Scaling
The service can be scaled by adding more worker instances. Each worker should:
1. Have FFmpeg installed
2. Connect to the same Redis instance
3. Have access to the shared storage directory
### Example Scaling Setup
```bash
# Main instance (handles web UI and coordination)
python3 start_service.py --port 5000 --tmp-dir /shared/tmp/encoder
# Worker instances (handle encoding tasks)
python3 start_service.py --worker-name encoder_worker_1 --tmp-dir /shared/tmp/encoder
python3 start_service.py --worker-name encoder_worker_2 --tmp-dir /shared/tmp/encoder
```
## Performance Considerations
1. **Storage Performance**:
- Use fast storage for /tmp directory
- Consider local SSD for better I/O performance
2. **Network Bandwidth**:
- Ensure sufficient bandwidth between workers
- Consider network topology when distributing workers
3. **Resource Allocation**:
- Monitor CPU usage across workers
- Balance memory usage for concurrent encoding
## Troubleshooting
1. **Service Won't Start**:
```bash
# Check if Redis is already running
ps aux | grep redis
# Check port availability
netstat -tulpn | grep 5000
```
2. **Encoding Fails**:
- Verify FFmpeg installation
- Check temporary directory permissions
- Verify Redis connectivity
3. **Worker Connection Issues**:
- Check Redis connection string
- Verify network connectivity
- Check firewall settings
## Security Notes
1. **Temporary Directory**:
- Regularly clean up old files
- Set appropriate permissions
```bash
chmod 700 /tmp/encoder
```
2. **Redis Security**:
- Use strong passwords in production
- Configure proper network restrictions
3. **File Validation**:
- Service validates video file types
- Implements size restrictions
- Sanitizes filenames
## API Endpoints
- `POST /api/upload`: Upload video file
- `GET /api/status/<job_id>`: Get encoding status
- `GET /api/video/<job_id>/<quality>`: Stream encoded video
- `GET /api/jobs`: List all encoding jobs
## Contributing
[Contribution Guidelines]
## License
[Your License Here]
|