Hugging Face's logo

Spaces:
Adicodecrafter
/
Multiverse-AI-Studio
Running

App Files Community
Multiverse-AI-Studio / README.md
3v324v23's picture
3v324v23
Make demo deployable on Hugging Face Spaces + add local/Groq/demo onboarding
29e37f2
preview code
|
Raw
History Blame Contribute Delete
13.4 kB
metadata 
title: Multiverse AI Studio
emoji: 🌌
colorFrom: indigo
colorTo: red
sdk: docker
app_port: 7860
pinned: false

Multiverse AI Studio

A portfolio-grade, full-stack generative AI application that chains five Hugging Face models into a unified, coherent multimedia pipeline. A single human prompt is expanded into specialized instructions to generate a base visual scene, estimate its 3D depth geometry, synthesize an ambient background soundscape, and render a cinematic video anchoring the visual track.

Designed with a strict focus on system architecture, event-loop safety, memory management, and smooth user experience.

🚦 Choosing your mode

Multiverse AI Studio runs the same five-stage pipeline (prompt → image → depth → audio → video) in three configurations. Pick one in the in-app welcome chooser when you open the Space, or set it yourself via environment variables.

Mode Where models run Keys needed Hardware Guide
Local (full models) On your machine via transformers/diffusers None (no gated models) NVIDIA GPU (8GB+ VRAM) Run locally
Hugging Face Space (cloud/demo) Images via Pollinations, LLM via Groq, depth on CPU Optional GROQ_API_KEY None (runs in the Space) Deploy to HF Spaces
Demo Same as the Space, no key None None Click “Just demo” in the welcome modal
  • Local = maximum quality and privacy, but needs a GPU and downloads weights.
  • Space / Groq = real images with zero setup; bring a free Groq key for real LLM prompt expansion (BYOK), or skip and still get real images + local depth + mock audio/video.
  • Demo = instant, no configuration.

🏗️ System Architecture

1. Model Pipeline Flow

Every stage in the pipeline consumes something meaningful from the previous step. The visual assets (depth and video) are conditioned on the generated image, ensuring visual cohesion.

graph TD
    A[User Prompt] -->|POST /api/generate| B(Prompt Expansion LLM)
    B -->|image_prompt| C(Image Generation Service)
    B -->|audio_prompt| D(Audio Generation Service)
    C -->|RGB Image| E(Depth Estimation Service)
    C -->|RGB Image| F(Video Generation Service)
    E -->|Depth Map| F
    D -->|Audio Track| G[Final Coherent Scene]
    F -->|MP4 Stream| G
    
    style C fill:#4f46e5,stroke:#333,stroke-width:2px,color:#fff
    style E fill:#ec4899,stroke:#333,stroke-width:2px,color:#fff
    style F fill:#db2777,stroke:#333,stroke-width:2px,color:#fff

2. Event-Loop & Threading Execution Flow

ML model inference is CPU/GPU-bound and blocking. To prevent blocking FastAPI's main asynchronous event loop, all inferences are offloaded to a background thread pool.

Browser               FastAPI Route             Background Worker             Job Store
  │                         │                           │                         │
  │── POST /generate ──────>│                           │                         │
  │   (prompt payload)      │── Create Job (UUID) ───────────────────────────────>│ (QUEUED)
  │                         │── Schedule run_pipeline ──>│                        │
  │<── Return job_id ───────│                           │                         │
  │                         │                           │── Update Stage ────────>│ (EXPANDING...)
  │                         │                           │── Run Prompt LLM        │
  │                         │                           │── Run Image Gen         │
  │                         │                           │── Update Asset (img) ──>│ (Image URL)
  │                         │                           │── Run Depth Est         │
  │                         │                           │── Update Asset (depth) ─>│ (Depth URL)
  │                         │                           │── Run Audio/Video       │
  │                         │                           │── Update final state ──>│ (COMPLETED)
  │                         │                           │                         │
  │── GET /result/{id} ────>│────────────────────────────────────────────────────>│
  │<── Returns assets ──────│<────────────────────────────────────────────────────│

⚡ Key Features

  • Progressive Polling & Rendering: The frontend polls /api/result/{job_id}. Completed assets (like the base image) are rendered on the screen immediately while downstream stages are still processing.
  • Granular Memory/VRAM Management: Chaining 5 heavy models sequentially can cause VRAM Out-of-Memory (OOM) crashes. Each model wrapper implements a strict cleanup() method that deletes pipeline instances, runs garbage collection (gc.collect()), and flushes PyTorch's CUDA memory cache (torch.cuda.empty_cache()) before loading the next stage.
  • Stage Error Isolation: Wrap-around try/except boundaries guarantee that a single failed stage (e.g., depth map or audio timeout) does not crash the entire pipeline. The server flags a PARTIAL_FAILURE and delivers all other successfully compiled assets.
  • Interactive CSS Depth Slider: Features a custom swipable comparison slider built using CSS clip-path polygon slicing for smooth 60fps comparisons between the visual base image and its calculated depth map.
  • Custom Media Players: Custom glassmorphic React components for audio and video playback, including a dynamic pulsing audio waveform visualizer.

⚙️ Project Setup

Prerequisites

  • Python 3.10+
  • Node.js 18+
  • Hugging Face Access Token (for gated model downloads and Inference API)

1. Environment Configuration

Create a .env file in the project root:

HF_TOKEN=your_huggingface_access_token_here
MOCK_INFERENCE=False
FORCE_CPU_INFERENCE=False

Environment Variables Explained:

  • HF_TOKEN: Your Hugging Face user access token (required for querying the cloud image generation API and downloading gated local models).
  • MOCK_INFERENCE:
    • True (Default DX): Bypasses all local and cloud model execution, returning mock visual, audio, and video assets in 1 second. Useful for testing UI components on any computer.
    • False (Hybrid Production): Connects to the cloud and local machine learning models for real generation.
  • FORCE_CPU_INFERENCE:
    • False (Safe Fallback): If running MOCK_INFERENCE=False on a CPU-only machine, the backend will run cloud image generation and local depth maps, but will automatically bypass the heavy MusicGen and i2vgen-xl local models to prevent RAM exhaustion.
    • True (Force CPU): Forces the backend to download, load, and execute the full MusicGen and Video models locally on your CPU. Warning: MusicGen takes 2–5 minutes, and i2vgen-xl takes 20–45 minutes on CPU.

🚀 GPU & Production Execution Setup

If you want to run the complete, real local PyTorch pipelines (audio and video) on a GPU-enabled developer machine:

  1. Install CUDA-enabled PyTorch: Ensure your virtual environment is using a GPU-compiled version of PyTorch:
    pip install torch --index-url https://download.pytorch.org/whl/cu121
  2. Configure environment: Open your .env file and set:
    HF_TOKEN=your_real_huggingface_token
MOCK_INFERENCE=False
FORCE_CPU_INFERENCE=False
  3. Launch: Run uvicorn. The backend will automatically detect the GPU (cuda), log the status, and run the real local Hugging Face and Diffusers pipelines at maximum speed.

2. Backend Installation 

# Navigate to the backend directory
cd backend

# Create a virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Start the development server
python -m uvicorn main:app --host 127.0.0.1 --port 8000 --reload

The backend health check is available at http://127.0.0.1:8000/api/health.

3. Frontend Installation 

# Navigate to the frontend directory
cd frontend

# Install packages
npm install

# Start the Vite React development server
npm run dev

Open http://localhost:3000 to access the Multiverse AI Studio interface.

🤗 Deploy to Hugging Face Spaces (Zero-Config Demo)

This repository is configured as a Docker SDK Space (see the sdk: docker / app_port: 7860 header at the top of this file and the Dockerfile). The FastAPI backend and the React frontend are built into a single container; the API is served under /api and the UI under /.

What works with NO keys at all

The default profile is groq_cloud, which uses Pollinations.ai for image generation (completely free, no API key) and runs Depth-Anything-V2-Small locally on CPU. Audio and video fall back to procedural mock assets. So a freshly deployed Space produces real images + real depth maps even if you set no secrets. The LLM prompt-expansion stage gracefully degrades to a local template when no key is present, so the pipeline never crashes.

Recommended Space settings

  1. Create a new Space → SDK: Docker.
  2. In Settings → Secrets and variables → Environment variables, set:
    INFERENCE_PROFILE=groq_cloud
MOCK_INFERENCE=False
FORCE_CPU_INFERENCE=False
  3. (Optional, enables real LLM prompt expansion) add GROQ_API_KEY (free from console.groq.com). Visitors can also paste their own Groq key in the welcome modal (BYOK) — that uses their quota.
  4. Push the repo (the Dockerfile builds the frontend and installs the Python deps automatically).
  5. Open the Space. Enter a prompt → it runs the pipeline and polls for results.

The default MOCK_INFERENCE is False (see backend/config.py) so a clean deploy runs the real hybrid pipeline. Set MOCK_INFERENCE=True only if you want instant fake assets for UI testing.

🖥️ Run locally (full models, no cloud)

Run the entire pipeline on your own hardware — no API keys and no per-request cloud calls. Set INFERENCE_PROFILE=local_gpu and the backend downloads and runs each model locally with transformers/diffusers.

Prerequisites

  • Python 3.10+
  • An NVIDIA GPU with ~8GB+ VRAM recommended
  • CUDA-enabled PyTorch
  • (The first run downloads model weights; afterwards everything runs offline.)

1. Install CUDA PyTorch 

pip install torch --index-url https://download.pytorch.org/whl/cu121

2. Install dependencies 

cd backend
python -m venv venv
source venv/bin/activate   # Windows: venv\Scripts\activate
pip install -r requirements.txt

3. Select local models 

export INFERENCE_PROFILE=local_gpu   # Windows (PowerShell): $env:INFERENCE_PROFILE="local_gpu"

This profile runs every stage locally:

  • Prompt expansion: mistralai/Mistral-7B-Instruct-v0.2 (local LLM)
  • Image: a local Stable Diffusion 1.5 checkpoint
  • Depth: depth-anything/Depth-Anything-V2-Small-hf (runs fine on CPU)
  • Audio: facebook/musicgen-small
  • Video: ali-vilab/i2vgen-xl

Optional: set HF_TOKEN only if you swap in a gated model — none of the defaults are gated.

4. Run

Build the frontend once, then start the server (it serves both the API on /api and the UI on /):

cd frontend && npm install && npm run build && cd ..
python -m uvicorn backend.main:app --host 0.0.0.0 --port 8000

Open http://localhost:8000.

Hardware notes

  • Depth runs comfortably on CPU.
  • Image and Audio run on a 6–8GB GPU.
  • Video (i2vgen-xl, ~10GB+) needs more VRAM. On a CPU-only machine the server automatically bypasses audio/video to mock output unless you set FORCE_CPU_INFERENCE=True (slow / OOM risk).
  • Models cache in ~/.cache/huggingface; after the first download the pipeline works fully offline.

📈 Engineering Decisions & Tradeoffs

For a detailed analysis of our engineering decisions (such as choosing client polling over WebSockets, utilising an in-memory job store instead of Celery/Redis, and enforcing the BaseModel wrapper abstraction), please refer to the dedicated [Tradeoffs and Decisions Report](file:///c:\AI Native founder\AI_Engineering\Projects\Multiverse_AI_Studio\docs\decisions\tradeoffs.md).

🚀 Future Roadmap

  • Server-Sent Events (SSE): Migrate the progressive rendering polling system to standard Server-Sent Events to push updates in real-time without client request overhead.
  • Persistent Database storage: Replace the volatile in-memory dictionary with SQLite or PostgreSQL to keep user history across restarts.
  • Muxed Video Audio: Integrate system FFmpeg binaries to merge (mux) the Stage 4 ambient soundscape directly into the Stage 5 MP4 video container.