Multimodal models fail to detect implicit harmful content from image-text composition
In brief
- Multimodal models struggle to detect harm arising from image-text composition.
- MuPHI benchmark exposes weaknesses in cross-modal reasoning for VLMs.
- CHIL method enhances generalization and robustness to counterfactual interventions.
A harmless image and caption can become harmful only when combined. This study introduces MuPHI, a diagnostic benchmark to assess models' ability to reason about such compositional harm mechanisms. It highlights major gaps in multimodal reasoning and proposes CHIL, a method that significantly improves model generalization and robustness to counterfactuals.
This summary gives you the essentials. The full article, its figures and quotes are on cs.AI updates on arXiv.org.
Read the original articleAI-generated summary of the source, checked by rules. Report an error