Adversarial Robustness: Attacks, Defenses, and the Robustness Trade-off

Adversarial examples are inputs deliberately modified to cause a model error under specified constraints. Some small image perturbations change predictions while preserving human-recognizable content, but neither imperceptibility nor attack success follows from dimension alone. This article develops bounded input attacks, an adversarial training step, and the distinction between testing robustness and certifying it.

The threat model comes first

A threat model specifies the allowed input changes, input domain, attacker access and query budget, and objective. An untargeted attack seeks any label different from the true label; a targeted attack seeks a chosen wrong label. The code below uses untargeted white-box cross-entropy maximization on deterministic models. It is an instructional implementation, not a complete evaluation suite.

A common image benchmark uses \(\ell_\infty\) budget \(\epsilon=8/255\) for pixels scaled to [0,1]: every channel value may move by at most about 0.0314. This bound does not guarantee invisibility or preserved semantics. White-box access includes model internals; black-box settings can expose scores, labels, or transfer from another model. If the model standardizes inputs, apply normalization inside its forward path so the attack budget remains in raw pixel units, or explicitly transform per-channel bounds.

Convolution and pooling are introduced in CNN Fundamentals: From Convolution to Image Classification.

FGSM: one step

A first-order expansion gives \(\mathcal L(x+\delta)\approx\mathcal L(x)+\nabla_x\mathcal L^\top\delta\). Under an \(\ell_\infty\) bound, maximizing this linear term chooses each coordinate’s gradient sign. The gradient direction itself is the steepest direction under an \(\ell_2\) step constraint; the choice of norm matters:

\[x_{adv}=x+\epsilon\cdot\text{sign}\left(\nabla_x\mathcal{L}(f(x),y)\right)\]

Before clipping, coordinates with nonzero gradient move by \(\epsilon\); zero-gradient coordinates stay fixed. Clipping to the valid [0,1] range can reduce the actual change. This maximizes a local linear approximation, not generally the nonlinear loss over the entire feasible region.

import math
from contextlib import contextmanager
import torch, torch.nn as nn, torch.nn.functional as F

@contextmanager
def evaluation(model):
    flags = [(module, module.training) for module in model.modules()]
    try:
        model.eval()
        yield
    finally:
        for module, flag in flags:
            module.training = flag

def check_attack(x, y, eps):
    if not math.isfinite(eps) or eps < 0:
        raise ValueError("eps must be finite and nonnegative")
    if x.ndim < 2 or len(x) == 0 or not x.is_floating_point() or not torch.isfinite(x).all():
        raise ValueError("Expected a finite floating-point batch")
    if not ((x >= 0) & (x <= 1)).all() or y.shape != (len(x),) or y.dtype != torch.long or y.device != x.device:
        raise ValueError("Expected inputs in [0,1] and colocated class indices")

def fgsm(model, x, y, eps=8/255):
    check_attack(x,y,eps)
    with evaluation(model), torch.enable_grad():
        candidate = x.detach().clone().requires_grad_(True)
        loss = F.cross_entropy(model(candidate),y)
        grad, = torch.autograd.grad(loss,candidate)
        return (candidate+eps*grad.sign()).clamp(0,1).detach()

# A fixed two-feature classifier, not an image benchmark.
toy = nn.Linear(2,2)
with torch.no_grad():
    toy.weight.copy_(torch.tensor([[-5.,-5.],[5.,5.]]))
    toy.bias.copy_(torch.tensor([5.,-5.]))
x = torch.tensor([[.55,.55],[.45,.45],[.9,.9],[.1,.1]])
y = torch.tensor([1,0,1,0])
attacked = fgsm(toy,x,y,eps=.1)
print("clean correct", int((toy(x).argmax(1)==y).sum()))
print("FGSM correct", int((toy(attacked).argmax(1)==y).sum()))
print("within budget", bool(((attacked-x).abs() <= .100001).all()))
# clean correct 4
# FGSM correct 2
# within budget True

The attack differentiates with respect to the input and leaves parameter gradients untouched. The helper temporarily uses evaluation mode and restores all training flags. It supports a deterministic, differentiable, batch-independent classifier whose output is a batch of logits. It enables gradients inside an ordinary no_grad context, but should not be called inside inference_mode. The four toy rows make the budget and label changes easy to inspect: all four are initially correct, and FGSM changes two labels at epsilon=0.1.

PGD: iterative search with projection

The commonly named PGD attack performs gradient ascent on loss, then projects back into the intersection of the perturbation box and the valid input range. Random starts and restarts explore different paths without guaranteeing a global maximum. Retain candidates per example across steps: the final iterate can be weaker than an earlier one.

def pgd(model,x,y,eps=8/255,alpha=2/255,steps=10,restarts=2,prefer_success=True):
    check_attack(x,y,eps)
    if not math.isfinite(alpha) or alpha <= 0 or not isinstance(steps,int) or steps < 1 or not isinstance(restarts,int) or restarts < 1:
        raise ValueError("Positive step size, steps and restarts required")
    original = x.detach()
    lower, upper = (original-eps).clamp(0,1), (original+eps).clamp(0,1)
    with evaluation(model), torch.enable_grad():
        with torch.no_grad():
            logits = model(original)
            best = original.clone()
            best_loss = F.cross_entropy(logits,y,reduction="none")
            best_success = logits.argmax(1) != y
        for restart in range(restarts):
            candidate = original.clone() if restart == 0 else lower+torch.rand_like(original)*(upper-lower)
            for step in range(steps+1):
                candidate = candidate.detach().requires_grad_(True)
                logits = model(candidate)
                losses = F.cross_entropy(logits,y,reduction="none")
                success = logits.argmax(1) != y
                with torch.no_grad():
                    better = losses > best_loss
                    if prefer_success:
                        better = (success & ~best_success) | ((success == best_success) & better)
                    best[better] = candidate[better]
                    best_loss[better] = losses[better]
                    best_success[better] = success[better]
                if step < steps:
                    grad, = torch.autograd.grad(losses.sum(),candidate)
                    candidate = torch.maximum(torch.minimum(candidate.detach()+alpha*grad.sign(),upper),lower)
        return best.detach()

torch.manual_seed(0)
adv = pgd(toy,x,y,eps=.1,alpha=.025,steps=10,restarts=3)
print("PGD correct", int((toy(adv).argmax(1)==y).sum()))
print("PGD bounds valid", bool((((adv-x).abs()<=.100001)&(adv>=0)&(adv<=1)).all()))
clean_ok = toy(x).argmax(1)==y
survives = clean_ok & (toy(attacked).argmax(1)==y) & (toy(adv).argmax(1)==y)
print("survives both attacks", int(survives.sum()), "of", len(y))
# PGD correct 2
# PGD bounds valid True
# survives both attacks 2 of 4

For evaluation, this helper preserves any successful candidate it finds and then favors higher loss among candidates of the same success status. The first restart begins at the clean input. Report clean accuracy and the fraction correct after the union of tested attacks, counting clean errors as non-robust. Attack success among initially correct examples uses a different denominator. Empirical attack-survival accuracy is an upper bound on true worst-case robust accuracy for that dataset: a failed search does not rule out an unseen adversarial example. State steps, restarts, loss, preprocessing, model mode, and randomness handling. Adaptive attacks and suites such as AutoAttack provide additional evidence, not a proof.

Adversarial training

Adversarial training is one widely studied approach: update model parameters using approximately worst-case inputs. The inner optimization is normally a search, not an exact solution. For image inputs, the feasible set also includes x+delta in [0,1]:

\[\min_\theta\;\mathbb{E}_{(x,y)}\left[\max_{\lVert\delta\rVert_\infty\le\epsilon,\;x+\delta\in[0,1]^d}\mathcal{L}(f_\theta(x+\delta),y)\right]\]

def adversarial_train_step(model,x,y,optimizer,eps=8/255,steps=7):
    flags = [(module,module.training) for module in model.modules()]
    try:
        x_adv = pgd(model,x,y,eps=eps,steps=steps,prefer_success=False)
        model.train()
        optimizer.zero_grad(set_to_none=True)
        loss = F.cross_entropy(model(x_adv),y)
        loss.backward()
        optimizer.step()
        return loss.item()
    finally:
        for module,flag in flags:
            module.training = flag

from copy import deepcopy
learner = deepcopy(toy)
optimizer = torch.optim.SGD(learner.parameters(),lr=.01)
before = [p.detach().clone() for p in learner.parameters()]
loss = adversarial_train_step(learner,x,y,optimizer,eps=.1,steps=7)
print("finite training loss", math.isfinite(loss))
print("parameters updated", any(not torch.equal(a,b) for a,b in zip(before,learner.parameters())))
# finite training loss True
# parameters updated True

The training helper maximizes retained loss rather than prioritizing misclassification, then performs one parameter update with the attacked inputs detached. The attack uses evaluation behavior; the outer update uses training behavior. This is an explicit BatchNorm/dropout policy, not a universal requirement. Other adversarial training methods choose different modes or statistics. Attack-time forward/input-gradient passes add work, but their cost is not identical to a full parameter update, so measure runtime rather than asserting an exact k+1 multiplier. One toy update does not establish improved robustness.

The robustness-accuracy trade-off

Robust and clean accuracy depend on architecture, training data, objective, attack budget, and evaluation procedure. Adversarial training can reduce clean accuracy, but it is not guaranteed to do so in every comparison. Report matched experiments or a clearly identified benchmark entry instead of treating a fixed pair of percentages as universal.

Constructed distributions show that optimal clean and adversarial accuracy can conflict. This demonstrates a possible fundamental trade-off, not an unavoidable loss of a fixed size on every dataset. Optimization, available data, and the threat model can also affect observed gaps.

One explanation involves predictive features that are easy to alter within the attack budget. That account is useful for some settings; it does not establish that every adversarial failure has the same cause or that every high-dimensional classifier is vulnerable on every input.

When an attack overestimates robustness

Gradient masking occurs when gradients become a poor guide for the attack, for example because of nondifferentiable preprocessing or uncontrolled randomness. Such changes can also alter the classifier’s decision boundary; masking does not require that boundary to remain unchanged. The concern is that an ineffective optimizer may report high attack-survival accuracy without strong robustness evidence.

Use attack-strength comparisons as diagnostic clues and investigate their causes. They are not necessary-and-sufficient tests for masking:

• If FGSM succeeds where PGD fails, inspect step sizes, losses, starts, and candidate retention.
• If a black-box method outperforms a particular gradient attack, investigate whether the gradient implementation or objective is ineffective.
• Check stronger budgets and more search effort, but recognize that finite searches need not improve monotonically and can legitimately saturate.
• Compare random, transfer, and gradient candidates under the same perturbation and validity constraints.

True worst-case robust accuracy cannot increase when the allowed perturbation sets are nested. A particular attack’s measured accuracy can, because the search changes. Keeping the union of found adversarial examples avoids discarding prior successes. A nonzero limiting robust accuracy is not automatically masking: a constant classifier, for example, cannot be attacked into changing its label on examples of that class. Standardized suites are useful baselines, and unusual preprocessing or stochastic defenses may require defense-specific adaptive evaluation.

Certified defenses

A certificate establishes that a specified classifier’s prediction is unchanged throughout a stated region. Randomized smoothing defines a new classifier by the most probable class under Gaussian input noise. Finite Monte Carlo voting alone is not a certificate: it needs a confidence bound. If a selected class has probability lower bound \(p_L>1/2\), a conservative Gaussian smoothing certificate has \(\ell_2\) radius \(R=\sigma\Phi^{-1}(p_L)\), where \(\Phi^{-1}\) is the standard normal quantile. The code selects a class using independent pilot draws, estimates its probability with new draws, and abstains if the bound is too low.

from scipy.stats import beta, norm

@torch.no_grad()
def certify_smoothed(model,x,sigma=.2,n0=100,n=2000,failure_prob=.001,batch_size=128):
    if sigma <= 0 or not math.isfinite(sigma) or not 0 < failure_prob < 1 or min(n0,n,batch_size) < 1:
        raise ValueError("Invalid smoothing parameters")
    def sample_votes(count):
        votes = []
        for start in range(0,count,batch_size):
            size = min(batch_size,count-start)
            noise = torch.randn((size,)+tuple(x.shape),device=x.device,dtype=x.dtype)
            votes.append(model(x.unsqueeze(0)+sigma*noise).argmax(1))
        return torch.cat(votes)
    with evaluation(model):
        pilot = sample_votes(n0)
        selected = int(torch.bincount(pilot).argmax())
        successes = int((sample_votes(n)==selected).sum())
    lower = 0. if successes == 0 else float(beta.ppf(failure_prob,successes,n-successes+1))
    if lower <= .5:
        return None, 0., lower
    return selected, float(sigma*norm.ppf(lower)), lower

torch.manual_seed(0)
label, radius, lower = certify_smoothed(toy,torch.tensor([.8,.8]))
print("smoothed class", label)
print(f"probability lower bound {lower:.4f} certified L2 radius {radius:.4f}")
# smoothed class 1
# probability lower bound 0.9759 certified L2 radius 0.3950

The one-sided Clopper–Pearson bound gives a Monte Carlo confidence statement at the specified failure probability for this input; it is not the probability that its label is correct. The certificate applies to the ideal smoothed classifier inside the stated radius, not directly to the original network or every future finite-vote prediction. For the toy point, the run gives a probability lower bound of 0.9759 and radius 0.3950; no image-model certificate is measured here. This implementation assumes a deterministic, batch-independent base classifier defined on Gaussian-perturbed inputs and does not clamp the noise. Other preprocessing must be incorporated into a fixed base function covered by the analysis. Certified accuracy additionally requires a correct class and a sufficient radius. Certificate size and compute depend on the method and data; a general ranking against empirical L-infinity accuracy is not meaningful.

Beyond \(\ell_p\) balls

An \(\ell_\infty\) budget captures one attack family. Physical patches, rotations, crops, recompression, and other changes need their own constraints and evaluations. There are limited norm-containment implications: an \(\ell_2\) ball of radius epsilon lies inside the \(\ell_\infty\) ball of the same radius. A valid certificate for the latter therefore covers the former, but survival of a finite attack search is not that certificate.

Prompt injection, jailbreak attempts, data poisoning, and backdoors involve different attacker controls and objectives. Prompt injection can introduce untrusted instructions through retrieved content; jailbreak attempts need not use that route. Poisoning changes training information, while a backdoor can be implanted through training data or model modification and activated by a trigger. None is fully characterized by the pixel budget used here.

Alignment and preference optimization are covered in Fine-Tuning and Alignment: SFT, LoRA, RLHF, and DPO.

What to do in practice

Choose defenses according to plausible attacker access, failure consequences, and operational constraints. Clean-data quality, uncertainty checks, and monitoring remain relevant, but do not substitute for adversarial evaluation when deliberate manipulation is in scope.

Model hardening can be combined with query controls, input checks, escalation, and monitoring. Their effectiveness depends on the attack and system, so assess them under the same operational scenario instead of assuming a universal cost-effectiveness ranking.

Monitoring and deployment are covered in Model Compression and Deployment: Distillation, Pruning, and Serving.

Exercises

1. FGSM against a linear model. Derive and verify the exact logit change an \(\ell_\infty\) perturbation of size \(\epsilon\) produces on a linear classifier, and evaluate it at \(d = 784\) with \(\epsilon = 8/255\). Explain how the weight scaling and pixel bounds affect the change.

You should get: an exact unclipped shift of epsilon times the L1 weight norm, with a smaller shift after clipping.

import numpy as np
rng = np.random.default_rng(0)
d, eps = 784, 8/255
w = rng.normal(0,1/np.sqrt(d),d)
x = rng.uniform(0,1,d)
# Increase a binary logit; this is the loss-ascent direction for y=0.
delta = eps*np.sign(w)
unclipped = x+delta
clipped = np.clip(unclipped,0,1)
print(f"unclipped shift {w @ (unclipped-x):.4f}")
print(f"epsilon times L1 norm {eps*np.abs(w).sum():.4f}")
print(f"clipped shift {w @ (clipped-x):.4f}")
print("pixel bounds respected", bool(((clipped>=0)&(clipped<=1)).all()))
# unclipped shift 0.7080
# epsilon times L1 norm 0.7080
# clipped shift 0.6959
# pixel bounds respected True

2. Robust accuracy has a price. Given illustrative percentages for a standard and an adversarially trained model, compute the clean accuracy given up per point of robust accuracy gained. State whether this trade-off is fundamental.

You should get: eight clean-accuracy percentage points exchanged for 53 attack-evaluated points in the hypothetical comparison.

std_clean, std_robust = 95.0, 0.0        # illustrative baseline, not a published benchmark
at_clean, at_robust = 87.0, 53.0         # illustrative candidate under a matched evaluation
print(f"clean accuracy given up {std_clean - at_clean:.1f} points")
print(f"robust accuracy gained {at_robust - std_robust:.1f} points")
print(f"ratio {(std_clean - at_clean) / (at_robust - std_robust):.3f}"
      f" clean points per robust point")
# clean accuracy given up 8.0 points
# robust accuracy gained 53.0 points
# ratio 0.151 clean points per robust point

3. Detect gradient masking. List the checks that distinguish a genuinely robust model from one that merely obscures its gradients, and say what each would show on a broken defense.

You should get: diagnostic clues that require follow-up, with matched budgets and retained attack successes.

References

Cohen et al., Certified Adversarial Robustness via Randomized Smoothing; Athalye et al., Obfuscated Gradients Give a False Sense of Security; Croce and Hein, Reliable Evaluation of Adversarial Robustness.