Multimodal Adversarial Defense for Vision-Language Models by Leveraging One-To-Many Relationships

TL;DR

• VLMs are defended against image and text attacks: Multimodal adversarial training addresses attacks on both input modalities.
• One-to-many pairs capture image-text ambiguity: MAT+ uses multiple valid images or descriptions instead of fixed one-to-one supervision.
• Multimodal defense reaches state-of-the-art performance: Improves robustness across vision-language tasks under multimodal attacks.

Presentation Video

Presentation Poster

Open poster

Futa Waseda

Project Assistant Professor