Spaces:

BiasLab2025
/

contextual-communication-demo

Sleeping

App Files Files Community

contextual-communication-demo / API.md

Raheeb Hassan

Add aggressiveness parameter for bandwidth savings strategy in video processing

760687a 29 days ago

preview code

raw

history blame contribute delete

33.8 kB

A newer version of the Gradio SDK is available: 6.6.0

Upgrade

API Documentation

This document describes the Gradio API endpoints exposed by the ROI-VAE image and video compression application. The API allows programmatic access to segmentation, compression, detection, and full pipeline processing for both images and videos.

Live Demo: https://biaslab2025-contextual-communication-demo.hf.space

Quick Start
Important Notes
Image API Endpoints
Video API Endpoints
Streaming Video API Endpoints
- /stream_process_video
- /stream_compress_video
Class Reference
Error Handling
GPU Quota Handling
cURL Examples
Example Scripts

Quick Start

Installation

pip install gradio_client

Image Processing

from gradio_client import Client, handle_file

# Connect to the API
client = Client("https://biaslab2025-contextual-communication-demo.hf.space")
# Or local: client = Client("http://localhost:7860")

# Full pipeline: segment → compress → detect
compressed, mask, bpp, ratio, coverage, detections_json = client.predict(
    handle_file("path/to/image.jpg"),
    "car, person",      # segmentation prompt
    "sam3",             # segmentation method
    4,                  # quality level (1-5)
    0.3,                # sigma (background compression)
    True,               # run detection
    "yolo",             # detection method
    "",                 # detection classes (empty for closed-vocab)
    api_name="/process"
)

print(f"Compression: {bpp:.4f} bpp ({ratio:.2f}x)")

Video Processing

from gradio_client import Client, handle_file
import json

client = Client("http://localhost:7860")

# Full pipeline with static settings
output_video, stats_json = client.predict(
    handle_file("path/to/video.mp4"),
    "person, car",      # segmentation classes
    "sam3",             # segmentation method
    "static",           # mode: "static" or "dynamic"
    4,                  # quality level (1-5)
    0.3,                # sigma
    15.0,               # output FPS
    500,                # bandwidth (dynamic mode)
    5,                  # min_fps (dynamic mode)
    30,                 # max_fps (dynamic mode)
    False,              # run detection
    "yolo",             # detection method
    None,               # mask_file_path (optional)
    api_name="/process_video"
)

stats = json.loads(stats_json)
print(f"Compressed video: {output_video}")
print(f"Total frames: {stats['total_frames']}")

Important Notes

File Handling

Always wrap file paths with handle_file() when using gradio_client:

from gradio_client import handle_file

# ✅ Correct
client.predict(handle_file("image.jpg"), ...)

# ❌ Incorrect - will fail with validation error
client.predict("image.jpg", ...)

Detection Output Format

All detection endpoints return JSON strings with this structure:

import json

detections = json.loads(detections_json)
# Each detection has:
# - label: str (class name)
# - score: float (confidence 0-1)
# - bbox_xyxy: list[float] (bounding box [x1, y1, x2, y2])

Open-Vocabulary Detectors

The following detectors require a classes parameter:

yolo_world - YOLO-World
grounding_dino - Grounding DINO

Closed-vocabulary detectors (yolo, detr, faster_rcnn, etc.) use pretrained COCO classes and ignore the classes parameter.

Image API Endpoints

1. `/segment` - Generate ROI Mask

Segments an image to create a Region of Interest (ROI) mask.

Parameters:

Parameter	Type	Default	Description
`image`	Image	required	Input image file
`prompt`	str	`"object"`	Comma-separated classes or natural language prompt
`method`	str	`"sam3"`	Segmentation method (see methods)
`return_overlay`	bool	`False`	If `True`, returns image with ROI highlighted instead of mask

Returns:

Output	Type	Description
`result_image`	Image	Grayscale mask OR image with ROI overlay (if `return_overlay=True`)
`roi_coverage`	float	Fraction of image covered by ROI (0.0-1.0)
`classes_used`	str	JSON list of classes/prompts used

Example:

# Get binary mask (default)
mask, coverage, classes = client.predict(
    handle_file("car_scene.jpg"),
    "car, road",
    "sam3",
    False,  # return_overlay
    api_name="/segment"
)
print(f"ROI covers {coverage*100:.2f}% of image")

# Get image with ROI highlighted
highlighted, coverage, classes = client.predict(
    handle_file("car_scene.jpg"),
    "car, road",
    "sam3",
    True,   # return_overlay=True
    api_name="/segment"
)

2. `/compress` - Compress Image

Compresses an image using TIC VAE, optionally with an ROI mask for variable quality.

Parameters:

Parameter	Type	Default	Description
`image`	Image	required	Input image file
`mask_image`	Image	`None`	ROI mask (white=ROI, black=background)
`quality`	int	`4`	Quality level 1-5
`sigma`	float	`0.3`	Background preservation (0.01-1.0)

Quality Levels:

Level	Lambda	Description
1	0.0035	Smallest file
2	0.013	Smaller file
3	0.025	Balanced
4	0.0483	Higher quality (default)
5	0.0932	Best quality

Returns:

Output	Type	Description
`compressed_image`	Image	Compressed output image
`bpp`	float	Bits per pixel
`compression_ratio`	float	Compression ratio (24/bpp)

Example:

# Compress without mask (uniform quality)
compressed, bpp, ratio = client.predict(
    handle_file("image.jpg"),
    None,   # no mask
    4,      # quality
    0.3,    # sigma (ignored without mask)
    api_name="/compress"
)

# Compress with ROI mask
mask, _, _ = client.predict(handle_file("image.jpg"), "person", "yolo", False, api_name="/segment")

compressed, bpp, ratio = client.predict(
    handle_file("image.jpg"),
    handle_file(mask),
    4,
    0.2,    # aggressive background compression
    api_name="/compress"
)

3. `/detect` - Object Detection

Runs object detection on an image and returns detection results as JSON.

Parameters:

Parameter	Type	Default	Description
`image`	Image	required	Input image file
`method`	str	`"yolo"`	Detection method (see methods)
`classes`	str	`""`	Comma-separated classes (required for open-vocab detectors)
`confidence`	float	`0.25`	Confidence threshold (0.0-1.0)

Returns:

Output	Type	Description
`detections_json`	str	JSON string of detection results

Example - Closed-Vocabulary:

import json

# YOLO detection (COCO classes)
dets_json = client.predict(
    handle_file("street_scene.jpg"),
    "yolo",
    "",      # no classes needed
    0.25,
    api_name="/detect"
)

detections = json.loads(dets_json)
for det in detections:
    print(f"{det['label']}: {det['score']:.2f}")

Example - Open-Vocabulary:

# YOLO-World with custom classes
dets_json = client.predict(
    handle_file("image.jpg"),
    "yolo_world",
    "hat, backpack, umbrella",  # custom classes required
    0.25,
    api_name="/detect"
)

3.1. `/detect_overlay` - Detection with Visualization

Runs object detection and returns the image with bounding boxes drawn.

Parameters:

Parameter	Type	Default	Description
`image`	Image	required	Input image file
`method`	str	`"yolo"`	Detection method (see methods)
`classes`	str	`""`	Comma-separated classes (required for open-vocab detectors)
`confidence`	float	`0.25`	Confidence threshold (0.0-1.0)

Returns:

Output	Type	Description
`result_image`	Image	Image with detection bounding boxes
`detections_json`	str	JSON string of detection results

Example:

import json

# Get image with detection boxes
result_img, dets_json = client.predict(
    handle_file("street_scene.jpg"),
    "yolo",
    "",
    0.25,
    api_name="/detect_overlay"
)

# result_img is a file path to the image with boxes drawn
print(f"Image with boxes: {result_img}")
detections = json.loads(dets_json)

4. `/process` - Full Image Pipeline

Runs the complete pipeline: segmentation → compression → optional detection.

Parameters:

Parameter	Type	Default	Description
`image`	Image	required	Input image file
`prompt`	str	`"object"`	Segmentation prompt/classes
`segmentation_method`	str	`"sam3"`	ROI segmentation method
`quality`	int	`4`	Compression quality (1-5)
`sigma`	float	`0.3`	Background preservation (0.01-1.0)
`run_detection`	bool	`False`	Whether to run detection on output
`detection_method`	str	`"yolo"`	Detector to use
`detection_classes`	str	`""`	Classes for open-vocab detectors

Returns:

Output	Type	Description
`compressed_image`	Image	Compressed output image
`mask_image`	Image	Generated ROI mask
`bpp`	float	Bits per pixel
`compression_ratio`	float	Compression ratio
`roi_coverage`	float	ROI coverage percentage (0-1)
`detections_json`	str	JSON detections (empty list if `run_detection=False`)

Example:

import json

compressed, mask, bpp, ratio, coverage, dets_json = client.predict(
    handle_file("street.jpg"),
    "car, person, road",
    "sam3",
    4,
    0.3,
    True,   # run detection
    "yolo",
    "",
    api_name="/process"
)

print(f"ROI Coverage: {coverage*100:.2f}%")
print(f"Compression: {bpp:.4f} bpp ({ratio:.2f}x)")
print(f"Detections: {len(json.loads(dets_json))}")

Video API Endpoints

1. `/segment_video` - Segment Video

Segments a video to find ROI regions, returning either a mask file or overlay video.

Parameters:

Parameter	Type	Default	Description
`video_path`	Video	required	Input video file
`prompt`	str	`"object"`	Comma-separated classes or natural language prompt
`method`	str	`"sam3"`	Segmentation method
`return_overlay`	bool	`False`	If `True`, returns video with ROI highlighted
`output_fps`	float	`15.0`	Output framerate (max 30)

Returns:

Output	Type	Description
`result_path`	File/Video	Mask file (NPZ) OR video with ROI overlay
`stats_json`	str	JSON with frame count, coverage, and classes

Example:

import json

# Get mask file for reuse in compression
mask_file, stats_json = client.predict(
    handle_file("video.mp4"),
    "person, car",
    "sam3",
    False,   # return masks file
    15.0,    # fps
    api_name="/segment_video"
)

stats = json.loads(stats_json)
print(f"Processed {stats['total_frames']} frames")
print(f"Avg ROI coverage: {stats['avg_roi_coverage']*100:.2f}%")

# Get video with ROI overlay for visualization
overlay_video, _ = client.predict(
    handle_file("video.mp4"),
    "person, car",
    "sam3",
    True,    # return overlay video
    15.0,
    api_name="/segment_video"
)

2. `/compress_video` - Compress Video

Compresses a video with optional ROI mask preservation.

Parameters:

Parameter	Type	Default	Description
`video_path`	Video	required	Input video file
`mask_file_path`	str	`None`	Path to pre-computed masks (from `/segment_video`)
`quality`	int	`4`	Quality level (1-5)
`sigma`	float	`0.3`	Background preservation (0.01-1.0)
`output_fps`	float	`15.0`	Target output framerate

Returns:

Output	Type	Description
`compressed_video`	Video	Compressed output video
`stats_json`	str	JSON with compression statistics

Example:

import json

# First, segment to get masks
mask_file, _ = client.predict(
    handle_file("video.mp4"), "person", "sam3", False, 15.0,
    api_name="/segment_video"
)

# Then compress with cached masks (3-5x faster!)
compressed, stats_json = client.predict(
    handle_file("video.mp4"),
    mask_file,   # reuse masks
    4,           # quality
    0.3,         # sigma
    15.0,        # fps
    api_name="/compress_video"
)

stats = json.loads(stats_json)
print(f"Compression ratio: {stats['compression_ratio']}x")
print(f"Total size: {stats['total_size_kb']} KB")

3. `/detect_video` - Video Detection

Runs object detection on each frame of a video.

Parameters:

Parameter	Type	Default	Description
`video_path`	Video	required	Input video file
`method`	str	`"yolo"`	Detection method
`classes`	str	`""`	Comma-separated classes (required for open-vocab)
`confidence`	float	`0.25`	Confidence threshold (0.0-1.0)
`return_overlay`	bool	`False`	If `True`, returns video with detection boxes
`output_fps`	float	`15.0`	Output framerate (max 30)

Returns:

Output	Type	Description
`result_video`	Video	Video with detection boxes (if `return_overlay=True`), None otherwise
`detections_json`	str	JSON with per-frame detections

Example:

import json

# Get per-frame detections JSON
_, dets_json = client.predict(
    handle_file("video.mp4"),
    "yolo",
    "",
    0.25,
    False,   # return JSON only
    15.0,
    api_name="/detect_video"
)

data = json.loads(dets_json)
print(f"Total detections: {data['total_detections']}")
print(f"Avg per frame: {data['avg_detections_per_frame']}")

# Get video with detection overlays
det_video, _ = client.predict(
    handle_file("video.mp4"),
    "yolo",
    "",
    0.25,
    True,    # return overlay video
    15.0,
    api_name="/detect_video"
)

4. `/process_video` - Full Video Pipeline

Processes a video with ROI-based compression (segment → compress), with optional detection.

Parameters:

Parameter	Type	Default	Description
`video_path`	Video	required	Input video file
`prompt`	str	`"object"`	Segmentation prompt/classes
`segmentation_method`	str	`"sam3"`	ROI segmentation method
`mode`	str	`"static"`	`"static"` or `"dynamic"`
`quality`	int	`4`	Quality level 1-5 (static mode)
`sigma`	float	`0.3`	Background preservation (static mode)
`output_fps`	float	`15.0`	Target framerate (static mode)
`bandwidth_kbps`	float	`500.0`	Target bandwidth (dynamic mode)
`min_fps`	float	`5.0`	Minimum framerate (dynamic mode)
`max_fps`	float	`30.0`	Maximum framerate (dynamic mode)
`aggressiveness`	float	`0.5`	Bandwidth savings strategy (dynamic mode): `0.0` = use full bandwidth (high FPS always), `0.5` = moderate savings, `1.0` = maximum savings (aggressive FPS reduction for low motion)
`run_detection`	bool	`False`	Whether to run detection/tracking
`detection_method`	str	`"yolo"`	Detector to use
`mask_file_path`	str	`None`	Path to pre-computed masks (skips segmentation)

Returns:

Output	Type	Description
`output_video`	Video	Compressed video
`stats_json`	str	JSON with detailed statistics

Example - Static Mode:

import json

output, stats_json = client.predict(
    handle_file("video.mp4"),
    "person, car",
    "sam3",
    "static",
    4, 0.3, 15.0,          # static: quality, sigma, fps
    500, 5, 30,            # dynamic: bandwidth, min_fps, max_fps (ignored)
    False, "yolo", None,
    api_name="/process_video"
)

stats = json.loads(stats_json)
print(f"Processed {stats['total_frames']} frames")

Example - Dynamic Mode:

output, stats_json = client.predict(
    handle_file("video.mp4"),
    "person",
    "yolo",
    "dynamic",
    4, 0.3, 15.0,          # static settings (ignored)
    750,                    # target bandwidth 750 kbps
    8,                      # min FPS
    30,                     # max FPS
    True, "yolo", None,
    api_name="/process_video"
)

Streaming Video API Endpoints

The streaming API provides HLS-style chunk-by-chunk delivery for real-time video processing. Unlike the buffered endpoints above, these endpoints yield chunks progressively as they're produced, enabling:

Real-time streaming to frontend
Lower latency (first chunks available immediately)
Memory efficient (no buffering entire video)
Backwards compatible (existing endpoints remain unchanged)

⚡ Real-Time Behavior

Yes, this is true streaming! Chunks are yielded immediately after compression:

Video frames are extracted and accumulated into ~1 second chunks (15-30 frames)
Each chunk is segmented and compressed using batch processing
Chunk is yielded immediately - no waiting for subsequent chunks
Frontend receives and can display frames right away

First chunk latency: ~1.5-4 seconds (depending on models)
Subsequent chunks: Streamed continuously as they're ready

The "chunk" granularity (vs frame-by-frame) is for efficiency - batch processing 15-30 frames at once is much faster than processing individually.

1. `/stream_process_video` - Full Streaming Pipeline

Streams compressed video chunks with segmentation and optional detection.

Parameters:

Same as /process_video, plus:
- frame_format (str, default: "jpeg"): Frame encoding format ("jpeg" or "png")
- frame_quality (int, default: 85): JPEG quality 1-95 (ignored for PNG)
- max_resolution (int, default: 720): Maximum height in pixels (e.g., 360, 480, 720, 1080). Video is resized before processing for faster performance. Lower values = faster processing.

Note: The aggressiveness parameter (0.0-1.0) controls bandwidth savings strategy in dynamic mode - higher values aggressively reduce FPS during low-motion scenes for maximum bandwidth efficiency, while lower values maintain high FPS to use available bandwidth.

Yields: JSON strings, each containing one chunk:

{
  "chunk_index": 0,
  "frames": ["base64_encoded_jpeg_1", "base64_encoded_jpeg_2", ...],
  "timestamps": [0.0, 0.033, 0.066, ...],
  "fps": 15.0,
  "stats": {
    "avg_bpp": 0.256,
    "estimated_bytes": 32768,
    "quality_level": 4,
    "sigma": 0.3
  }
}

Final message:

{"status": "complete"}

Example (Python):

from gradio_client import Client, handle_file
import json
import base64
from PIL import Image
from io import BytesIO

client = Client("http://localhost:7860")

# Get generator of chunks
chunk_stream = client.submit(
    handle_file("video.mp4"),
    "person, car",          # prompt
    "sam3",                 # segmentation_method
    "static",               # mode
    4,                      # quality
    0.3,                    # sigma
    15.0,                   # output_fps
    500.0,                  # bandwidth_kbps (dynamic mode)
    5.0,                    # min_fps
    30.0,                   # max_fps
    None,                   # mask_file_path
    "jpeg",                 # frame_format
    85,                     # frame_quality
    360,                    # max_resolution (360p for speed)
    api_name="/stream_process_video"
)

# Process chunks as they arrive
all_frames = []
for chunk_json in chunk_stream:
    chunk = json.loads(chunk_json)
    
    if "status" in chunk and chunk["status"] == "complete":
        print("Streaming complete!")
        break
    
    if "error" in chunk:
        print(f"Error: {chunk['error']}")
        break
    
    # Decode frames from base64
    for frame_b64 in chunk["frames"]:
        frame_bytes = base64.b64decode(frame_b64)
        frame = Image.open(BytesIO(frame_bytes))
        all_frames.append(frame)
    
    # Print progress
    print(f"Chunk {chunk['chunk_index']}: "
          f"{len(chunk['frames'])} frames @ {chunk['fps']} FPS, "
          f"BPP: {chunk['stats']['avg_bpp']:.3f}")

print(f"Total frames received: {len(all_frames)}")

Example (JavaScript/TypeScript):

async function streamVideo(videoFile: File) {
  const client = await Client.connect("http://localhost:7860");
  
  const chunks: VideoChunk[] = [];
  
  // Start streaming
  const stream = client.submit("/stream_process_video", [
    videoFile,
    "person, car",    // prompt
    "sam3",           // method
    "static",         // mode
    4,                // quality
    0.3,              // sigma
    15.0,             // fps
    500, 5, 30,       // dynamic settings
    null,             // mask_file
    "jpeg",           // format
    85,               // quality
    360               // max_resolution (360p for speed)
  ]);
  
  // Process chunks as they arrive
  for await (const chunkJson of stream) {
    const chunk = JSON.parse(chunkJson);
    
    if (chunk.status === "complete") {
      console.log("✅ Stream complete");
      break;
    }
    
    if (chunk.error) {
      console.error("❌ Error:", chunk.error);
      break;
    }
    
    // Decode frames
    const frames = chunk.frames.map((b64: string) => {
      const blob = base64ToBlob(b64, "image/jpeg");
      return URL.createObjectURL(blob);
    });
    
    chunks.push({
      index: chunk.chunk_index,
      frames: frames,
      timestamps: chunk.timestamps,
      fps: chunk.fps,
      stats: chunk.stats
    });
    
    console.log(`📦 Chunk ${chunk.chunk_index}: ${frames.length} frames`);
    
    // Display first frame of chunk immediately
    displayFrame(frames[0]);
  }
  
  return chunks;
}

function base64ToBlob(base64: string, mimeType: string): Blob {
  const byteString = atob(base64);
  const arrayBuffer = new ArrayBuffer(byteString.length);
  const uint8Array = new Uint8Array(arrayBuffer);
  for (let i = 0; i < byteString.length; i++) {
    uint8Array[i] = byteString.charCodeAt(i);
  }
  return new Blob([uint8Array], { type: mimeType });
}

2. `/stream_compress_video` - Simplified Streaming Compression

Simpler streaming endpoint without segmentation configuration (use with pre-computed masks).

Parameters:

video_path (str): Input video file
mask_file_path (str, optional): Pre-computed mask file from /segment_video
quality (int, default: 4): Quality level 1-5
sigma (float, default: 0.3): Background preservation 0.01-1.0
output_fps (float, default: 15.0): Target framerate
frame_format (str, default: "jpeg"): Frame encoding
frame_quality (int, default: 85): JPEG quality

Yields: Same format as /stream_process_video

Example:

from gradio_client import Client, handle_file
import json

client = Client("http://localhost:7860")

# Pre-segment video once
mask_file, _ = client.predict(
    handle_file("video.mp4"),
    "person, car",
    "sam3",
    False,  # return mask file
    15.0,
    api_name="/segment_video"
)

# Stream compression with cached masks
chunk_stream = client.submit(
    handle_file("video.mp4"),
    mask_file,     # reuse masks
    4,             # quality
    0.3,           # sigma
    15.0,          # fps
    "jpeg",        # format
    85,            # quality
    api_name="/stream_compress_video"
)

for chunk_json in chunk_stream:
    chunk = json.loads(chunk_json)
    if "status" in chunk:
        break
    print(f"Chunk {chunk['chunk_index']}: {len(chunk['frames'])} frames")

Benefits of Streaming API

Lower Latency: First chunks available in ~1 second (vs buffering entire video)
Memory Efficient: Process frames incrementally, no need to buffer
Real-time Display: Show frames to user as they're compressed
Progress Updates: Monitor compression progress chunk-by-chunk
Bandwidth Adaptive: Works with dynamic mode for adaptive streaming

Chunk Structure

Each chunk contains:

chunk_index: Sequential number (0, 1, 2, ...)
frames: List of base64-encoded images (typically 15-30 frames per chunk)
timestamps: Frame timestamps in seconds since video start
fps: Effective framerate for this chunk
stats: Compression statistics
- avg_bpp: Average bits per pixel
- estimated_bytes: Chunk size estimate
- quality_level: TIC model quality (1-5)
- sigma: Background compression factor
- motion (dynamic mode only): Motion analysis metrics

Backwards Compatibility

All existing API endpoints (/process_video, /compress_video, etc.) remain unchanged and continue to work as before. The streaming endpoints are additive - they don't modify existing behavior.

Class Reference

Segmentation Methods

Method	Description	Classes
`sam3`	Prompt-based (natural language)	Any text prompt
`yolo`	YOLO instance segmentation	80 COCO classes
`segformer`	Cityscapes semantic segmentation	19 classes
`mask2former`	Swin-based panoptic/semantic	133 COCO / 150 ADE20K
`maskrcnn`	ResNet50-FPN instance segmentation	80 COCO classes

Detection Methods

Closed-Vocabulary (COCO pretrained):

Method	Description
`yolo`	Ultralytics YOLO
`detr`	Facebook DETR
`faster_rcnn`	Faster R-CNN
`retinanet`	RetinaNet
`fcos`	FCOS
`ssd`	SSD300

Open-Vocabulary (requires classes parameter):

Method	Description
`yolo_world`	YOLO-World
`grounding_dino`	Grounding DINO

COCO Classes (80)

person, bicycle, car, motorcycle, airplane, bus, train, truck, boat,
traffic light, fire hydrant, stop sign, parking meter, bench, bird, cat,
dog, horse, sheep, cow, elephant, bear, zebra, giraffe, backpack, umbrella,
handbag, tie, suitcase, frisbee, skis, snowboard, sports ball, kite,
baseball bat, baseball glove, skateboard, surfboard, tennis racket, bottle,
wine glass, cup, fork, knife, spoon, bowl, banana, apple, sandwich, orange,
broccoli, carrot, hot dog, pizza, donut, cake, chair, couch, potted plant,
bed, dining table, toilet, tv, laptop, mouse, remote, keyboard, cell phone,
microwave, oven, toaster, sink, refrigerator, book, clock, vase, scissors,
teddy bear, hair drier, toothbrush

Cityscapes Classes (19)

road, sidewalk, building, wall, fence, pole, traffic light, traffic sign,
vegetation, terrain, sky, person, rider, car, truck, bus, train, motorcycle,
bicycle

Error Handling

try:
    result = client.predict(
        handle_file("image.jpg"),
        ...,
        api_name="/endpoint"
    )
except Exception as e:
    print(f"API Error: {e}")

Common Errors:

Error	Cause	Solution
Validation error for ImageData	Missing `handle_file()`	Wrap file paths with `handle_file()`
File does not exist	Invalid path	Check file path is correct
Empty detection classes	Open-vocab detector without classes	Provide classes for `yolo_world`, `grounding_dino`
GPU quota exceeded	HF Spaces limit	Wait and retry (see below)

Handling GPU Quota on HF Spaces

When using Hugging Face Spaces with ZeroGPU, you may encounter quota limits:

You have exceeded your GPU quota (60s requested vs. 0s left). Try again in 0:05:30

Automatic Retry with Backoff

import time
import re

def extract_wait_time(error_msg):
    """Extract wait time from GPU quota error message."""
    match = re.search(r'Try again in (\d+):(\d+)(?::(\d+))?', error_msg)
    if match:
        if match.group(3):  # HH:MM:SS
            return int(match.group(1)) * 3600 + int(match.group(2)) * 60 + int(match.group(3))
        else:  # MM:SS
            return int(match.group(1)) * 60 + int(match.group(2))
    return 60

def call_with_retry(client, *args, api_name, max_retries=5):
    """Call API with exponential backoff retry."""
    delay = 10
    
    for attempt in range(max_retries):
        try:
            return client.predict(*args, api_name=api_name)
        except Exception as e:
            error_msg = str(e)
            if "exceeded your GPU quota" in error_msg:
                wait_time = extract_wait_time(error_msg)
                actual_delay = max(delay, wait_time + 5)
                print(f"⏳ GPU quota exhausted. Waiting {actual_delay}s... (attempt {attempt + 1})")
                time.sleep(actual_delay)
                delay *= 2
            else:
                raise
    raise Exception("Max retries reached")

# Usage
result = call_with_retry(
    client,
    handle_file("image.jpg"),
    "car", "sam3", False, 4, 0.3, False, "yolo", "",
    api_name="/process"
)

Using with cURL

Upload File First

# Upload image
FILE_URL=$(curl -s -X POST http://localhost:7860/upload \
  -F "files=@image.jpg" | \
  python3 -c "import sys, json; print(json.load(sys.stdin)[0])")

Call Endpoints

# Segment
curl -X POST http://localhost:7860/api/segment \
  -H "Content-Type: application/json" \
  -d "{\"data\": [\"$FILE_URL\", \"car, person\", \"sam3\", false]}"

# Compress (no mask)
curl -X POST http://localhost:7860/api/compress \
  -H "Content-Type: application/json" \
  -d "{\"data\": [\"$FILE_URL\", null, 4, 0.3]}"

# Detect
curl -X POST http://localhost:7860/api/detect \
  -H "Content-Type: application/json" \
  -d "{\"data\": [\"$FILE_URL\", \"yolo\", \"\", 0.25, false]}"

# Full pipeline
curl -X POST http://localhost:7860/api/process \
  -H "Content-Type: application/json" \
  -d "{\"data\": [\"$FILE_URL\", \"car, person\", \"sam3\", 4, 0.3, true, \"yolo\", \"\"]}"

Example Scripts

Batch Image Processing

from gradio_client import Client, handle_file
from pathlib import Path

client = Client("http://localhost:7860")
output_dir = Path("compressed_output")
output_dir.mkdir(exist_ok=True)

for img_path in Path("images").glob("*.jpg"):
    print(f"Processing {img_path.name}...")
    
    compressed, mask, bpp, ratio, coverage, _ = client.predict(
        handle_file(str(img_path)),
        "car, person",
        "sam3",
        4, 0.3,
        False, "", "",
        api_name="/process"
    )
    
    # Save compressed image
    output_path = output_dir / f"compressed_{img_path.name}"
    with open(output_path, "wb") as f:
        f.write(open(compressed, "rb").read())
    
    print(f"  BPP: {bpp:.4f}, Ratio: {ratio:.2f}x, ROI: {coverage*100:.2f}%")

Video Processing with Mask Caching

from gradio_client import Client, handle_file
import json

client = Client("http://localhost:7860")
video_path = "input_video.mp4"

# Step 1: Segment video (one-time cost)
mask_file, seg_stats = client.predict(
    handle_file(video_path),
    "person, car",
    "sam3",
    False,  # return mask file
    15.0,
    api_name="/segment_video"
)
print(f"Segmented video, masks saved to: {mask_file}")

# Step 2: Compress with different settings, reusing masks
for quality in [3, 4, 5]:
    compressed, comp_stats = client.predict(
        handle_file(video_path),
        mask_file,   # reuse cached masks
        quality,
        0.3,
        15.0,
        api_name="/compress_video"
    )
    stats = json.loads(comp_stats)
    print(f"Quality {quality}: {stats['compression_ratio']}x compression")

Detection Comparison (Original vs Compressed)

from gradio_client import Client, handle_file
import json

client = Client("http://localhost:7860")
image = "street_scene.jpg"

# Detect on original
_, dets_orig = client.predict(
    handle_file(image), "yolo", "", 0.25, False,
    api_name="/detect"
)
orig_count = len(json.loads(dets_orig))
print(f"Original: {orig_count} detections")

# Compress and detect
compressed, _, bpp, ratio, _, dets_comp = client.predict(
    handle_file(image),
    "car, person, road",
    "sam3",
    4, 0.3,
    True, "yolo", "",
    api_name="/process"
)
comp_count = len(json.loads(dets_comp))

retention = comp_count / orig_count * 100 if orig_count else 0
print(f"Compressed ({ratio:.2f}x): {comp_count} detections")
print(f"Detection retention: {retention:.1f}%")

Additional Resources

Web UI: Visit http://localhost:7860 for interactive interface
GitHub: See repository for source code and examples
Model Checkpoints: Available in checkpoints/ directory
Test Images: Sample images in data/images/ directory

API Documentation

Table of Contents

Quick Start

Installation

Image Processing

Video Processing

Important Notes

File Handling

Detection Output Format

Open-Vocabulary Detectors

Image API Endpoints

1. /segment - Generate ROI Mask

2. /compress - Compress Image

3. /detect - Object Detection

3.1. /detect_overlay - Detection with Visualization

4. /process - Full Image Pipeline

Video API Endpoints

1. /segment_video - Segment Video

2. /compress_video - Compress Video

3. /detect_video - Video Detection

4. /process_video - Full Video Pipeline

Streaming Video API Endpoints

⚡ Real-Time Behavior

1. /stream_process_video - Full Streaming Pipeline

2. /stream_compress_video - Simplified Streaming Compression

Benefits of Streaming API

Chunk Structure

Backwards Compatibility

Class Reference

Segmentation Methods

Detection Methods

COCO Classes (80)

Cityscapes Classes (19)

Error Handling

Handling GPU Quota on HF Spaces

Automatic Retry with Backoff

Using with cURL

Upload File First

Call Endpoints

Example Scripts

Batch Image Processing

Video Processing with Mask Caching

Detection Comparison (Original vs Compressed)

Additional Resources

1. `/segment` - Generate ROI Mask

2. `/compress` - Compress Image

3. `/detect` - Object Detection

3.1. `/detect_overlay` - Detection with Visualization

4. `/process` - Full Image Pipeline

1. `/segment_video` - Segment Video

2. `/compress_video` - Compress Video

3. `/detect_video` - Video Detection

4. `/process_video` - Full Video Pipeline

1. `/stream_process_video` - Full Streaming Pipeline

2. `/stream_compress_video` - Simplified Streaming Compression