Quality Text, Robust Vision: The Role of Language in Enhancing Visual Robustness of Vision-Language Models
TL;DR
- High-quality captions guide adversarial fine-tuning: QT-AFT uses rich image descriptions rather than class labels alone.
- Richer language supports broader visual robustness: Captions guide attacks away from diverse image semantics and reduce narrow class overfitting.
- Robustness and accuracy improve on 16 datasets: Evaluates zero-shot transfer across diverse downstream classification tasks.
Type
Publication
ACM Multimedia 2025
Presentation Video
Presentation Slides
