Understanding Sensitivity of Differential Attention through the Lens of Adversarial Robustness

Abstract

We investigate how Differential Attention (DA) affects adversarial vulnerability in vision transformers and CLIP models. While DA suppresses redundant context via a subtractive structure, we show it paradoxically increases adversarial sensitivity through negative gradient alignment. Our analysis reveals a fundamental trade-off: DA improves discriminative focus on clean inputs but increases adversarial vulnerability.

Futa Waseda

Project Assistant Professor | Trustworthy AI: Robustness, Reliability, and VLM Defense