| title: Multimodal Coherence AI | |
| emoji: "\U0001f3a8" | |
| colorFrom: purple | |
| colorTo: pink | |
| sdk: streamlit | |
| sdk_version: "1.41.0" | |
| app_file: app.py | |
| pinned: false | |
| license: mit | |
| short_description: Coherent text + image + audio with MSCI | |
| # Multimodal Coherence AI | |
| Generate semantically coherent **text + image + audio** bundles and evaluate | |
| cross-modal alignment using the **Multimodal Semantic Coherence Index (MSCI)**. | |
| ## How it works | |
| 1. **Text** — generated via HF Inference API | |
| 2. **Image** — retrieved from a curated index using CLIP (ViT-B/32) embeddings | |
| 3. **Audio** — retrieved from a curated index using CLAP (HTSAT-unfused) embeddings | |
| 4. **MSCI** — computed as `0.45 * cos_sim(text, image) + 0.45 * cos_sim(text, audio)` | |
| ## Research | |
| This demo accompanies a study evaluating multimodal semantic coherence across | |
| three research questions: | |
| - **RQ1**: Is MSCI sensitive to controlled semantic perturbations? (Supported, d > 2.0) | |
| - **RQ2**: Does structured planning improve cross-modal alignment? (Not supported) | |
| - **RQ3**: Does MSCI correlate with human coherence judgments? (Supported, rho = 0.379) | |