Multimodal Generation: Research Notes
Status
Working note / experiment plan. No completed benchmark results are claimed here.
1. Scope and motivation
This document is a research sketch about controllable multimodal generation and prompt sensitivity. The central question is whether the proposed change improves the target behavior under a matched training and evaluation budget. The note deliberately separates hypotheses from observations so that future results can be added without rewriting the rationale.
2. Context
Research on multimodal generation often mixes improvements from architecture, data scale, preprocessing, and compute. A useful comparison therefore needs controlled baselines and explicit reporting of resource use. For this topic, the main confound is that automatic metrics may miss hallucination, bias, and prompt brittleness.
3. Working hypothesis
A focused change to the representation or interaction mechanism may improve task accuracy without increasing deployment cost disproportionately. The hypothesis should be rejected if gains disappear after matching parameter count, data exposure, or tuning budget.
4. Proposed approach
The first implementation should keep modality-specific preprocessing simple, project inputs into a shared representation space, and isolate the new component behind a small interface. Baselines should include a comparable model without the component and a stronger off-the-shelf reference. Any optimization should be applied to all systems, not only the proposed one.
5. Evaluation plan
| Dataset | Role | Primary measure |
|---|---|---|
| Conceptual Captions | primary evaluation | task accuracy |
| VQAv2 | transfer / robustness | CLIPScore |
| Flickr30k | transfer / robustness | human factuality rating |
Planned comparisons include a matched-capacity baseline, an ablation that removes the proposed component, and an out-of-domain transfer check. Default training values for the first controlled run are learning rate 0.0001, batch size 48, and 3 independent seeds. These are planning values, not claims about a finished experiment.
6. Reproducibility checklist
- Fix preprocessing before tuning.
- Report mean and standard deviation across seeds.
- Keep a held-out error-analysis split.
- Record wall-clock time and peak memory.
7. Failure modes and responsible use
The analysis should report subgroup and category-level failures instead of relying only on a single aggregate score. Particular attention is needed because automatic metrics may miss hallucination, bias, and prompt brittleness. No production use is recommended without task-specific validation, data review, and an assessment of privacy and bias.
8. Open questions
- Can a simpler baseline recover the same gain with more careful tuning?
- How sensitive is the conclusion to preprocessing and random seed?
- Which gain survives when the compute budget is matched?
References
[1] Sharma et al., Conceptual Captions, 2018. [2] Hessel et al., CLIPScore, 2021. [3] Li et al., BLIP-2, 2023.