README.md

metadata

title: MODUS
emoji: 🎨
colorFrom: indigo
colorTo: purple
sdk: gradio
sdk_version: 5.12.0
app_file: app.py
pinned: false
short_description: MODUS any-to-any multimodal demo (16 aligned modalities)

MODUS — any-to-any multimodal demo

Three-tab Gradio demo for the MODUS 16-modality any-to-any model:

1. Any-to-Any — one condition modality (image or caption) → any set of target modalities, shown in a gallery (4M-style).
2. Chained — condition → intermediate (bridge) → target, showing both.
3. Representation Analysis — RGB→Depth/Normal conditioned on {ViT only, VAE only, ViT+VAE}, side by side.

Setup (Space Settings → Variables and secrets)

• Hardware: ZeroGPU.
• HF_TOKEN (secret): a read token with access to the gated weights repo mqye/modus-16mod-stage3.
• Optional MODUS_WEIGHTS_REPO: defaults to mqye/modus-16mod-stage3.

The weights (bf16, 30GB) + VAE + config + tokenizer are pulled once at startup via snapshot_download; the model is built on CPU (5min) and moved to the ZeroGPU GPU per request. Inference config is baked in: modality config conf/modalities/instruction_16mod_stage2.yaml, use_instruction off for the generic modalities (seg/det/grounding keep their own instructions).