Understanding Sensitivity of Differential Attention through the Lens of Adversarial Robustness

TL;DR

• Differential Attention increases adversarial sensitivity: Analyzes vulnerability in vision transformers and CLIP models.
• Opposing gradients explain perturbation amplification: The subtractive attention structure creates negative gradient alignment.
• Selective attention trades off with robustness: Better focus on clean inputs does not imply stronger adversarial resistance.

Futa Waseda

Project Assistant Professor