PLAN.md

ShortSmith v2 - Implementation Plan

Overview

Build a Hugging Face Space that extracts "hype" moments from videos with optional person-specific filtering.

Project Structure

shortsmith-v2/
├── app.py                    # Gradio UI (Hugging Face interface)
├── requirements.txt          # Dependencies
├── config.py                 # Configuration and constants
├── utils/
│   ├── __init__.py
│   ├── logger.py             # Centralized logging
│   └── helpers.py            # Utility functions
├── core/
│   ├── __init__.py
│   ├── video_processor.py    # FFmpeg video/audio extraction
│   ├── scene_detector.py     # PySceneDetect integration
│   ├── frame_sampler.py      # Hierarchical sampling logic
│   └── clip_extractor.py     # Final clip cutting
├── models/
│   ├── __init__.py
│   ├── visual_analyzer.py    # Qwen2-VL integration
│   ├── audio_analyzer.py     # Wav2Vec 2.0 + Librosa
│   ├── face_recognizer.py    # InsightFace (SCRFD + ArcFace)
│   ├── body_recognizer.py    # OSNet for body recognition
│   ├── motion_detector.py    # RAFT optical flow
│   └── tracker.py            # ByteTrack integration
├── scoring/
│   ├── __init__.py
│   ├── hype_scorer.py        # Hype scoring logic
│   └── domain_presets.py     # Domain-specific weights
└── pipeline/
    ├── __init__.py
    └── orchestrator.py       # Main pipeline coordinator

Implementation Phases

Phase 1: Core Infrastructure

1. config.py - Configuration management

   • Model paths, thresholds, domain presets
   • HuggingFace API key handling

2. utils/logger.py - Centralized logging

   • File and console handlers
   • Different log levels per module
   • Timing decorators for performance tracking

3. utils/helpers.py - Common utilities

   • File validation
   • Temporary file management
   • Error formatting

Phase 2: Video Processing Layer

4. core/video_processor.py - FFmpeg operations

   • Extract frames at specified FPS
   • Extract audio track
   • Get video metadata (duration, resolution, fps)
   • Cut clips at timestamps

5. core/scene_detector.py - Scene boundary detection

   • PySceneDetect integration
   • Content-aware detection
   • Return scene timestamps

6. core/frame_sampler.py - Hierarchical sampling

   • First pass: 1 frame per 5-10 seconds
   • Second pass: Dense sampling on candidates
   • Dynamic FPS based on motion

Phase 3: AI Models

7. models/visual_analyzer.py - Qwen2-VL-2B

   • Load quantized model
   • Process frame batches
   • Extract visual embeddings/scores

8. models/audio_analyzer.py - Audio analysis

   • Librosa for basic features (RMS, spectral flux, centroid)
   • Optional Wav2Vec 2.0 for advanced understanding
   • Return audio hype signals per segment

9. models/face_recognizer.py - Face detection/recognition

   • InsightFace SCRFD for detection
   • ArcFace for embeddings
   • Reference image matching

10. models/body_recognizer.py - Body recognition

    • OSNet for full-body embeddings
    • Handle non-frontal views

11. models/motion_detector.py - Motion analysis

    • RAFT optical flow
    • Motion magnitude scoring

12. models/tracker.py - Multi-object tracking

    • ByteTrack integration
    • Maintain identity across frames

Phase 4: Scoring & Selection

13. scoring/domain_presets.py - Domain configurations

    • Sports, Vlogs, Music, Podcasts presets
    • Custom weight definitions

14. scoring/hype_scorer.py - Hype calculation

    • Combine visual + audio scores
    • Apply domain weights
    • Normalize and rank segments

Phase 5: Pipeline & UI

15. pipeline/orchestrator.py - Main coordinator

    • Coordinate all components
    • Handle errors gracefully
    • Progress reporting

16. app.py - Gradio interface

    • Video upload
    • API key input (secure)
    • Prompt/instructions input
    • Domain selection
    • Reference image upload (for person filtering)
    • Progress bar
    • Output video gallery

Key Design Decisions

Error Handling Strategy

• Each module has try/except with specific exception types
• Errors bubble up with context
• Pipeline continues with degraded functionality when possible
• User-friendly error messages in UI

Logging Strategy

• DEBUG: Model loading, frame processing details
• INFO: Pipeline stages, timing, results
• WARNING: Fallback triggers, degraded mode
• ERROR: Failures with stack traces

Memory Management

• Process frames in batches
• Clear GPU memory between stages
• Use generators where possible
• Temporary file cleanup

HuggingFace Space Considerations

• Use gr.State for session data
• Respect ZeroGPU limits (if using)
• Cache models in /tmp or HF cache
• Handle timeouts gracefully

API Key Usage

The API key input is for future extensibility (e.g., external services). For MVP, all processing is local using open-weight models.

Gradio UI Layout

┌─────────────────────────────────────────────────────────────┐
│  ShortSmith v2 - AI Video Highlight Extractor               │
├─────────────────────────────────────────────────────────────┤
│  ┌─────────────────────┐  ┌─────────────────────────────┐   │
│  │ Upload Video        │  │ Settings                    │   │
│  │ [Drop zone]         │  │ Domain: [Dropdown]          │   │
│  │                     │  │ Clip Duration: [Slider]     │   │
│  └─────────────────────┘  │ Num Clips: [Slider]         │   │
│                           │ API Key: [Password field]   │   │
│  ┌─────────────────────┐  └─────────────────────────────┘   │
│  │ Reference Image     │                                    │
│  │ (Optional)          │  ┌─────────────────────────────┐   │
│  │ [Drop zone]         │  │ Additional Instructions     │   │
│  └─────────────────────┘  │ [Textbox]                   │   │
│                           └─────────────────────────────┘   │
├─────────────────────────────────────────────────────────────┤
│  [🚀 Extract Highlights]                                    │
├─────────────────────────────────────────────────────────────┤
│  Progress: [████████████░░░░░░░░] 60%                       │
│  Status: Analyzing audio...                                 │
├─────────────────────────────────────────────────────────────┤
│  Results                                                    │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐                    │
│  │ Clip 1   │ │ Clip 2   │ │ Clip 3   │                    │
│  │ [Video]  │ │ [Video]  │ │ [Video]  │                    │
│  │ Score:85 │ │ Score:78 │ │ Score:72 │                    │
│  └──────────┘ └──────────┘ └──────────┘                    │
│  [Download All]                                             │
└─────────────────────────────────────────────────────────────┘

Dependencies (requirements.txt)

gradio>=4.0.0
torch>=2.0.0
transformers>=4.35.0
accelerate
bitsandbytes
qwen-vl-utils
librosa>=0.10.0
soundfile
insightface
onnxruntime-gpu
opencv-python-headless
scenedetect[opencv]
numpy
pillow
tqdm
ffmpeg-python

Implementation Order

1. config.py, utils/ (foundation)
2. core/video_processor.py (essential)
3. models/audio_analyzer.py (simpler, Librosa first)
4. core/scene_detector.py
5. core/frame_sampler.py
6. scoring/ modules
7. models/visual_analyzer.py (Qwen2-VL)
8. models/face_recognizer.py, body_recognizer.py
9. models/tracker.py, motion_detector.py
10. pipeline/orchestrator.py
11. app.py (Gradio UI)

Notes

• Start with Librosa-only audio (MVP), add Wav2Vec later
• Face/body recognition is optional (triggered by reference image)
• Motion detection can be skipped in MVP for speed
• ByteTrack only needed when person filtering is enabled