Adversarial robustness

Adversarial robustness is a model's ability to keep behaving as intended when an attacker deliberately chooses inputs to make it fail. It is always defined relative to a threat model: what the attacker may change, by how much, and with what knowledge of the model.

Formal definition

For a classifier $f$ with loss $L$, the classic setting allows the attacker to add a perturbation $\delta$ of bounded size to an input $x$ with label $y$. Adversarial training solves the min-max problem of Madry et al. (2017):

where $\theta$ are the model parameters and $ϵ$ is the perturbation budget. The inner maximisation is the attack; the outer minimisation trains the model to resist the worst case. The headline metric, often called adversarial accuracy, is accuracy under the strongest attack found within the budget.

Worked example

Szegedy et al. (2013) and Goodfellow et al. (2014) showed that image classifiers could be flipped by perturbations invisible to people. The fast gradient sign method computes $\delta =ϵ\cdot \text{sign}({\nabla }_{x}L)$: one gradient step that moves each pixel by $ϵ$ in the direction that most increases the loss. A model with 95% clean accuracy can drop to near 0% under such attacks if it was not trained to resist them.

Robustness for language models

For LLMs the input is discrete text and the "budget" is harder to define, since any string is allowed. Robustness then means resisting jailbreaks, prompt injections and paraphrase attacks while staying helpful. Optimisation attacks such as GCG transfer the gradient idea to tokens. Evaluation practice borrows two lessons from vision: test against adaptive attacks designed for the specific defence, and report results at a stated attack budget.

Why it matters

Average-case accuracy says little about behaviour under attack, and deployed systems face attackers. A defence evaluated only against yesterday's attacks often fails against an adaptive one; the adversarial ML literature has many examples of defences broken soon after publication. Claims about robustness are only meaningful with a threat model, an attack budget and an attack success rate with uncertainty.

Related terms

• Attack success rate (ASR)
• Jailbreak
• Prompt injection
• AI red teaming
• MITRE ATLAS

Sources

1. Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, Adrian Vladu. Towards Deep Learning Models Resistant to Adversarial Attacks, 2017. arxiv.org
2. Ian J. Goodfellow, Jonathon Shlens, Christian Szegedy. Explaining and Harnessing Adversarial Examples, 2014. arxiv.org
3. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, Rob Fergus. Intriguing properties of neural networks, 2013. arxiv.org