README.md

---
language:
- en
tags:
- vision-language
- mdetr
- xai
license: mit
model_index:
- name: mdetr-gridvqa-pure
  task: visual-question-answering
- name: mdetr-gridvqa-spurious
  task: visual-question-answering
---

# GridVQA-X Models

This repository contains two paired reference models, **M_pure** and **M_spur**, built on identical transformer architectures (**MDETR**). These models, coupled with their corresponding datasets, together form a diagnostic framework to evaluate if Multimodal Explainable AI (MxAI) methods genuinely capture cross-modal synergy or simply report shallow feature correlations.

## Model Descriptions

### 1. M_pure (The Faithful Spatial Reasoner)
*   **Training Distribution:** Trained exclusively on the D_pure dataset.
*   **Behavioral Dynamics:** Trained via explanation-guided dynamics using a two-phase optimization. Phase 1 forces explicit visual-textual token alignment via L1 and generalized IoU losses. Phase 2 handles Question Answering using class-frequency weighted cross-entropy to completely eliminate answer prior biases. 
*   **Capabilities:** Successfully internalizes true causal spatial-relational synergy, achieving robust accuracy across both clean and heavily distractor-crowded grids.

### 2. M_spur (The Shortcut / Bag-of-Words Model)
*   **Training Distribution:** Trained exclusively on the D_spur dataset.
*   **Behavioral Dynamics:** Structurally forced to rely on cross-modal shortcuts during training. It skips relational spatial geometry entirely and maps keywords directly to target visual volume.
*   **Capabilities:** Achieves perfect accuracy (1.000) on its native spurious distribution, but fails catastrophically when evaluated on D_pure multi-hop queries.

## Intended Diagnostic Use
These models are released explicitly to stress-test vision-language explainability algorithms (e.g., DIME, MultiSHAP, MultiViz, EMAP, InterSHAP):
*   **The Litmus Test:** A faithful explainer *must* output completely different attribution heatmaps or synergy scalars for M_pure and M_spur on the same input question.
*   **The Reality Check:** If your explainer highlights identical spatial regions for both models, it suffers from "model blindness" or is simply behaving as a superficial object detector.

## Performance Benchmark Metrics

| Evaluation Metric | M_pure on D_pure | M_spur on D_spur | M_spur on D_pure |
| :--- | :---: | :---: | :---: |
| **Global Accuracy** | >99% | 100% | **Catastrophic Failure** (8%-14% on multi-hop) |
| **Causal Pathway** | True Spatial Relations | Bag-of-Words Shortcut | Unimodal Feature Collapse |