README.md

---
title: MODUS
emoji: 🎨
colorFrom: indigo
colorTo: purple
sdk: gradio
sdk_version: 5.12.0
app_file: app.py
pinned: false
short_description: MODUS any-to-any multimodal demo (16 aligned modalities)
---

# MODUS — any-to-any multimodal demo

Three-tab Gradio demo for the MODUS 16-modality any-to-any model:

1. **Any-to-Any** — one condition modality (image or caption) → any set of target
   modalities, shown in a gallery (4M-style).
2. **Chained** — condition → intermediate (bridge) → target, showing both.
3. **Representation Analysis** — RGB→Depth/Normal conditioned on
   {ViT only, VAE only, ViT+VAE}, side by side.

## Setup (Space Settings → Variables and secrets)
- **Hardware:** ZeroGPU.
- **`HF_TOKEN`** (secret): a *read* token with access to the gated weights repo
  `mqye/modus-16mod-stage3`.
- Optional **`MODUS_WEIGHTS_REPO`**: defaults to `mqye/modus-16mod-stage3`.

The weights (bf16, ~30GB) + VAE + config + tokenizer are pulled once at startup
via `snapshot_download`; the model is built on CPU (~5min) and moved to the
ZeroGPU GPU per request. Inference config is baked in: modality config
`conf/modalities/instruction_16mod_stage2.yaml`, `use_instruction` off for the
generic modalities (seg/det/grounding keep their own instructions).