Image to Video (No Audio)

input_primary: image

output_primary: video

processType: synthesis

motionType: general

metadata_maturity_level: mature

metadata_common_use_cases:
Static image animation
Product visualization
Concept previsualization

metadata_platforms:
Replicate
FAL AI
Stability AI

metadata_example_models:
Stable Video Diffusion
AnimateDiff

output_modality: video

operation_type: creation


Image to Video (Ambient Audio)

output_audio_type: ambient

audioGeneration: synthesized

audioPrompting: text-based

audioCharacteristics: background environmental atmospheric

metadata_maturity_level: emerging

metadata_common_use_cases:
Scene ambiance creation
Marketplace atmosphere
Environmental storytelling

metadata_platforms:
FAL AI
Experimental


Image to Video (Lip Sync from Text)

output_audio_type: speech

audioGeneration: text-to-speech

lipSyncMethod: generated-from-text

motionType: facial

metadata_common_use_cases:
Avatar creation
Character animation from portrait
Marketing personalization

metadata_platforms:
Replicate
FAL AI
HeyGen

metadata_example_models: Wav2Lip SadTalker DreamTalk


Image to Video (Lip Sync from Audio)

audioGeneration: reference-based

audioPrompting: audio-reference

lipSyncMethod: audio-driven

metadata_common_use_cases:
Voice cloning with video
Dubbing and localization
Podcast video generation

metadata_platforms:
Replicate
FAL AI


name: Image to Video (Lip Sync with LoRA Character)

input_secondary: text lora-model

characterReference: lora

metadata_maturity_level: experimental

metadata_common_use_cases:
Consistent character animation
Brand mascot videos
Personalized avatars


Text to Video (No Audio)

input_primary: text

metadata_common_use_cases:
Concept visualization
Storyboarding
Creative exploration

metadata_platforms:
Replicate
FAL AI
RunwayML

metadata_example_models: ModelScope ZeroScope


Text to Video (With Audio)

output_audio_type: synchronized

metadata_common_use_cases:
Complete scene generation
Multimedia storytelling


Audio to Video

input_primary: audio

output_audio_type: original

motionType: audio-reactive

metadata_common_use_cases:
Music visualization
Audio-reactive art
Podcast video generation


Image + Audio to Video

motionType: audio-driven

metadata_common_use_cases:
Audio-driven animation
Dance video generation
Music-driven motion


Text + Image to Video

guidanceType: text-and-visual

motionType: guided

metadata_common_use_cases:
Guided video generation
Controlled animation
Reference-based video creation


3D Model to Video

processType: rendering

motionType: camera-path

metadata_common_use_cases:
3D visualization
Product rendering
Architectural visualization

metadata_platforms:
Blender
Unreal Engine
Unity


Music to Video

input_primary: music

output_audio_type: music

metadata_common_use_cases:
Music video generation
Lyric videos
Album visualizers

metadata_platforms: Replicate Experimental


Text to Video with Music

metadata_common_use_cases:
Promotional videos
Social media content
Advertisement creation


Image to Video with Music

audioCharacteristics: melodic rhythmic

metadata_common_use_cases:
Photo slideshow creation
Social media posts
Memory videos


Video Upscaling

input_primary: video

processType: enhancement

audioHandling: passthrough

metadata_common_use_cases:
Resolution enhancement
Quality improvement
Restoration

metadata_platforms: Topaz Replicate

metadata_example_models:
Topaz Video AI
Real-ESRGAN Video


Video + Text to Video

processType: transformation

metadata_common_use_cases:
Video-guided generation
Motion transfer
Style-guided video

metadata_platforms: Replicate RunwayML


Video to Video (No Audio)

metadata_common_use_cases:
Video style transfer
Video editing
Motion manipulation

operation_type: editing


Video to Video (Preserve Audio)

metadata_common_use_cases: Content transformation maintaining soundtrack


Video Inpainting

processType: inpainting

modification: selective-editing

metadata_common_use_cases:
Object removal
Background replacement
Video cleanup

metadata_platforms: Replicate RunwayML Adobe


Video Enhancement

modification: enhancement

metadata_common_use_cases:
Quality improvement
Noise reduction
Color grading
Stabilization

metadata_platforms: Topaz Adobe Replicate

metadata_example_models:
Topaz Video AI
DAIN
Real-ESRGAN


Video Frame Interpolation

metadata_common_use_cases:
Frame rate increase
Slow motion creation
Smooth motion

metadata_example_models: RIFE DAIN Flowframes


Video Colorization

metadata_common_use_cases:
Black and white restoration
Historical footage colorization
Archival restoration

metadata_platforms: Replicate DeOldify

metadata_example_models:
DeOldify
Video Colorization


Video Face Swap

metadata_common_use_cases:
Face replacement
Character substitution
Visual effects

metadata_platforms: Replicate DeepFaceLab

metadata_example_models: DeepFaceLab Roop FaceSwap


Video Relighting

metadata_common_use_cases:
Lighting adjustment
Time of day change
Mood alteration


Video Segmentation

metadata_common_use_cases:
Background removal
Object isolation
Green screen replacement

metadata_platforms: Replicate Runway Unscreen

metadata_example_models:
Segment Anything Video
XMem
Cutout.pro


Text to Audio

output_primary: audio

output_audio_type: general

audioType: general

audioCategories: speech sound-effects music ambient

metadata_common_use_cases:
Sound effect generation
Voiceover creation
Audio asset production

metadata_platforms: Replicate ElevenLabs AudioCraft

metadata_example_models: AudioGen MusicGen

output_modality: audio


Text to Speech

audioType: speech

metadata_common_use_cases:
Narration
Accessibility
Voice assistants

metadata_platforms:
ElevenLabs
Google Cloud
Azure
AWS

metadata_example_models:
ElevenLabs
Google WaveNet
Azure Neural TTS


Text to Music

audioType: music

metadata_common_use_cases:
Background music generation
Musical composition
Soundtrack creation

metadata_platforms:
Replicate
Stability AI

metadata_example_models:
MusicGen
Stable Audio


Text to Speech (Voice Cloning)

metadata_common_use_cases:
Custom voice synthesis
Personalized narration
Voice preservation

metadata_platforms: ElevenLabs Replicate PlayHT

metadata_example_models: ElevenLabs XTTS Bark


Text to Sound Effects

output_audio_type: sound-effects

audioType: sound-effects

metadata_common_use_cases:
SFX generation
Foley creation
Game audio

metadata_platforms: Replicate AudioCraft

metadata_example_models: AudioGen AudioLDM


Image to Audio

metadata_common_use_cases:
Image sonification
Scene audio generation
Accessibility


Music Style Transfer

metadata_common_use_cases:
Genre transformation
Instrument swap
Musical reimagining


Video to Audio Extraction

metadata_common_use_cases:
Audio extraction
Soundtrack isolation
Voice extraction

metadata_platforms:
FFmpeg
Standard tools


Humming to Music

metadata_common_use_cases:
Melody to full track
Musical idea development
Composition assistance


Lyrics to Music

metadata_common_use_cases:
Song generation
Music composition
Vocal track creation

metadata_platforms: Suno Udio Replicate

metadata_example_models: Suno Udio MusicGen


Audio to Audio (Inpainting)

metadata_common_use_cases:
Audio editing
Sound design
Audio restoration


Music to Music (Inpainting)

audioSubtype: music

metadata_common_use_cases:
Music editing
Compositional modifications
Arrangement changes

parent: audio-to-audio-inpainting

note: Music inpainting is a specialized subset of audio inpainting


Audio Enhancement

metadata_common_use_cases:
Noise reduction
Quality improvement
Audio cleanup

metadata_platforms: Adobe iZotope Replicate

metadata_example_models:
Adobe Podcast
Krisp
Denoiser


Audio Restoration

modification: restoration

metadata_common_use_cases:
Historical recording restoration
Audio artifact removal
Damaged audio repair

metadata_platforms: iZotope Adobe Accusonus

metadata_example_models:
iZotope RX
Adobe Audition


Voice Conversion

modification: transformation

metadata_common_use_cases:
Voice swapping
Dubbing
Voice translation

metadata_platforms: ElevenLabs Respeecher Replicate

metadata_example_models: RVC So-VITS-SVC Respeecher


Music Stem Separation

metadata_common_use_cases:
Vocal isolation
Instrument extraction
Remixing
Karaoke creation

metadata_platforms: Spleeter Demucs Replicate

metadata_example_models:
Demucs
Spleeter
Ultimate Vocal Remover


Audio Speed/Pitch Modification

metadata_common_use_cases:
Tempo adjustment
Pitch shifting
Time stretching

metadata_platforms:
Standard tools
Adobe

metadata_example_models:
Rubber Band
Paulstretch


AI Music Mastering

metadata_common_use_cases:
Automated mastering
Mix enhancement
Final polish

metadata_platforms: LANDR iZotope eMastered

metadata_example_models:
LANDR
iZotope Ozone


Spatial Audio Conversion

metadata_common_use_cases:
Stereo to spatial
Binaural conversion
3D audio creation

metadata_platforms: Dolby Sony Experimental

metadata_example_models:
Dolby Atmos
Sony 360 Reality Audio


Text to Image

output_primary: image

generationType: synthesis

metadata_common_use_cases:
Concept art generation
Product mockups
Marketing assets

metadata_platforms:
Replicate
Stability AI
Midjourney
DALL-E

metadata_example_models:
Stable Diffusion
DALL-E 3
Midjourney

output_modality: image


Text + Image to Image

metadata_common_use_cases:
Image-guided generation
Style reference
Composition guidance

metadata_platforms:
Replicate
Stability AI
Midjourney

metadata_example_models:
Stable Diffusion with ControlNet
DALL-E 3
Midjourney


Image Upscaling

metadata_common_use_cases:
Resolution enhancement
Quality improvement
Detail enhancement

metadata_platforms:
Topaz
Replicate
Stability AI

metadata_example_models:
Real-ESRGAN
Topaz Gigapixel
SUPIR


Video to Image (Frame Extraction)

metadata_common_use_cases:
Frame extraction
Thumbnail generation
Video analysis


3D to Image (Rendering)

metadata_common_use_cases:
Product rendering
3D visualization
Architectural rendering

metadata_platforms:
Blender
Unreal Engine
Unity


Audio to Image (Visualization)

metadata_common_use_cases:
Album art generation
Sound visualization
Music imagery


Sketch to Image

metadata_common_use_cases:
Sketch refinement
Concept development
Design exploration


Image to Image

transformationTypes: style-transfer enhancement editing inpainting

metadata_common_use_cases:
Image editing
Style transfer
Image enhancement
Object removal/addition

metadata_example_models:
Stable Diffusion img2img
ControlNet


Image Inpainting

metadata_common_use_cases:
Object removal
Background extension
Image repair
Content-aware fill

metadata_platforms:
Replicate
Adobe
Stability AI

metadata_example_models:
Stable Diffusion Inpainting
LaMa
Adobe Firefly


Image Outpainting

metadata_common_use_cases:
Canvas extension
Image expansion
Background generation

metadata_platforms:
Replicate
DALL-E
Stability AI

metadata_example_models:
Stable Diffusion Outpainting
DALL-E Outpainting


Image Style Transfer

input_secondary: image text

metadata_common_use_cases:
Artistic style application
Photo stylization
Creative filters

metadata_platforms:
Replicate
Stability AI

metadata_example_models:
StyleGAN
Neural Style Transfer
InstantStyle


Image Colorization

metadata_common_use_cases:
Black and white colorization
Historical photo restoration
Photo enhancement

metadata_platforms: Replicate DeOldify

metadata_example_models:
DeOldify
Colorful Image Colorization


Image Enhancement

metadata_common_use_cases:
Quality improvement
Noise reduction
Sharpening
Dynamic range enhancement

metadata_platforms: Topaz Adobe Replicate

metadata_example_models:
Topaz Photo AI
Adobe Enhance


Image Restoration

metadata_common_use_cases:
Old photo restoration
Damaged image repair
Artifact removal

metadata_platforms: Replicate Remini

metadata_example_models: GFPGAN CodeFormer Remini


Background Removal

metadata_common_use_cases:
Background removal
Subject isolation
Product photography

metadata_platforms: Remove.bg Adobe Replicate

metadata_example_models: U2-Net RMBG SAM


Image Relighting

metadata_common_use_cases:
Lighting adjustment
Portrait relighting
Scene mood change

metadata_platforms: Experimental Adobe


Face Swap

metadata_common_use_cases:
Face replacement
Identity swap
Portrait editing

metadata_platforms: Replicate FaceSwap

metadata_example_models: InsightFace SimSwap Roop


Depth Map Generation

metadata_common_use_cases:
Depth estimation
3D reconstruction prep
Spatial understanding

metadata_platforms: Replicate HuggingFace

metadata_example_models: Depth-Anything MiDaS ZoeDepth


Image Segmentation

metadata_common_use_cases:
Object isolation
Semantic segmentation
Masking

metadata_platforms: Replicate Meta

metadata_example_models:
Segment Anything (SAM)
Semantic Segment Anything


Audio to Text (Transcription)

output_primary: text

metadata_common_use_cases:
Speech transcription
Meeting notes
Subtitling
Accessibility

metadata_platforms:
OpenAI
AssemblyAI
Deepgram
Google Cloud

metadata_example_models:
Whisper
AssemblyAI
Deepgram Nova

output_modality: text


Image to Text (Captioning)

metadata_common_use_cases:
Image description
Alt text generation
Scene understanding
Accessibility

metadata_platforms:
OpenAI
Google Cloud
HuggingFace

metadata_example_models:
GPT-4 Vision
BLIP
LLaVA
Gemini Vision


Image to Text (OCR)

metadata_common_use_cases:
Text extraction
Document digitization
Receipt scanning
Data entry automation

metadata_platforms:
Google Cloud
AWS
Azure
Tesseract

metadata_example_models:
Google Cloud Vision
AWS Textract
Tesseract
EasyOCR


Video to Text (Transcription)

metadata_common_use_cases:
Video subtitling
Content indexing
Meeting transcription
Accessibility

metadata_platforms: OpenAI AssemblyAI YouTube Rev

metadata_example_models:
Whisper
AssemblyAI
Google Speech-to-Text


Video to Text (Captioning/Description)

metadata_common_use_cases:
Video description
Content summarization
Scene understanding
Accessibility

metadata_platforms: OpenAI Google Experimental

metadata_example_models:
GPT-4 Vision
Gemini Video
Video-LLaMA


Visual Question Answering

guidanceType: multimodal

metadata_common_use_cases:
Image Q&A
Visual information retrieval
Educational applications
Accessibility

metadata_platforms: OpenAI Anthropic Google

metadata_example_models:
GPT-4 Vision
Claude
Gemini Vision


3D Model to Text (Description)

metadata_common_use_cases:
3D model description
Asset cataloging
Model understanding


Music to Text (Transcription)

metadata_common_use_cases:
Music notation
Sheet music generation
MIDI to score

metadata_platforms: Experimental AnthemScore

metadata_example_models:
AnthemScore
Audio to MIDI


Audio to Text (Speaker Diarization)

metadata_common_use_cases:
Multi-speaker transcription
Meeting notes with speakers
Interview transcription

metadata_platforms: AssemblyAI Deepgram Pyannote

metadata_example_models:
Pyannote
AssemblyAI
Whisper + Diarization


Text Translation

metadata_common_use_cases:
Language translation
Localization
Multilingual content

metadata_platforms:
Google Translate
DeepL
OpenAI

metadata_example_models:
Google Translate
DeepL
GPT-4
NLLB


Text Summarization

metadata_common_use_cases:
Document summarization
Content condensation
Abstract generation

metadata_platforms: OpenAI Anthropic HuggingFace

metadata_example_models: GPT-4 Claude BART


Text Paraphrasing/Rewriting

metadata_common_use_cases:
Content rewriting
Style adjustment
Tone modification

metadata_platforms: OpenAI Anthropic QuillBot

metadata_example_models: GPT-4 Claude QuillBot


Grammar & Spelling Correction

metadata_common_use_cases:
Proofreading
Error correction
Writing improvement

metadata_platforms: Grammarly LanguageTool OpenAI

metadata_example_models: Grammarly LanguageTool GPT-4


Text to 3D Model

metadata_common_use_cases:
3D asset generation
Rapid prototyping
Game asset creation

metadata_platforms: Replicate Meshy 3DFY

metadata_example_models: Point-E Shap-E DreamFusion


Image to 3D Model

metadata_common_use_cases:
3D reconstruction
Object digitization
Asset creation from photos

metadata_platforms:
Replicate
Meshy
Luma AI


Multi-Image to 3D Model

metadata_common_use_cases:
Photogrammetry
3D scanning
Object reconstruction

metadata_platforms:
Luma AI
Polycam
Reality Capture

metadata_example_models:
NeRF
Gaussian Splatting
Photogrammetry


Video to 3D Model

metadata_common_use_cases:
Video-based reconstruction
Motion to 3D
Scene capture

metadata_platforms:
Luma AI
Polycam

metadata_example_models:
NeRF
Gaussian Splatting


Text + Image to 3D Model

metadata_common_use_cases:
Guided 3D generation
Controlled asset creation
Reference-based modeling


3D Model Optimization

metadata_common_use_cases:
Polygon reduction
LOD generation
Performance optimization

metadata_platforms: Blender Maya Simplygon


3D Model Texturing

metadata_common_use_cases:
Texture generation
Material application
PBR material creation

metadata_platforms:
Experimental
Adobe Substance


3D Model Rigging

metadata_common_use_cases:
Auto-rigging
Skeleton generation
Animation preparation

metadata_platforms: Mixamo AccuRIG


3D Style Transfer

input_secondary: text image

metadata_common_use_cases:
Artistic 3D styling
Model transformation
Creative effects