Transactions on Machine Learning Research, 2026
Accepted to NeurIPS 2026 in the MLRC track

[Re] Boosting the Visual Interpretability of CLIP via Adversarial Fine-Tuning

Anton Nuzhdin, Andrei Gruzitski, Balázs Egyed, Aleksandr Raudvee
University of Amsterdam. All authors contributed equally.

CLIP is a widely used model that links images and text, and many AI systems build on it. It can tell you a photo contains an umbrella. Ask it which pixels made it say so and the answer is noise. A paper at ICLR 2025 claimed that a short round of extra training fixes this. We retrained the models and rebuilt the evaluations to check, then added a way to tell the model where to look.

Where does the model look?

A saliency map marks the pixels an answer depends on most. For the original CLIP, Simple Gradient maps are a lattice of speckles over the whole picture that barely change with the prompt. After adversarial fine-tuning, or AFT, they concentrate on the objects in the photo.

Pick a photo and a prompt. The AFT maps come from our retrained model.

What adversarial fine-tuning does

AFT works in two steps for each ImageNet photo. First, it finds the tiny change to the pixels, at most 4/255 per colour channel, that confuses the model the most. Then it trains the model to see the altered photo the same way the original, untouched CLIP sees the clean one. It needs no labels or captions.

The original authors showed why this helps. To first order, the training penalises the input gradient with a Huber-type norm that pushes most of its entries to zero, and that gradient is a smoothed saliency map.

Adversarial fine-tuning: a perturbed image through the trainable encoder must land near the frozen encoder's embedding of the clean image. clean image worst-case nudge at most 4/255 per pixel encoder being trained starts as a copy of CLIP original CLIP encoder frozen keep these two close
Only the red encoder learns. The dashed one stays frozen as the reference.

Four claims, re-tested

The original authors released their training code but not their evaluation code. So we retrained the models and rebuilt every evaluation from the descriptions in their paper. Our retrained models closely match the ones they released, but our scores differ from the numbers printed in their paper. The most likely reason is that the paper leaves out some settings, such as the threshold that turns a map into a mask. We tried a range of values for these settings. The scores moved, but the ranking of the models stayed the same. That's why the verdicts below judge whether each effect goes the same way, not whether the numbers match.

  1. Holds in directionSimple Gradient maps.

    Saliency maps get sharper

    The pointing game asks whether a map's hottest pixel lands on the object. For Simple Gradient maps on ImageNet-S, AFT raises it from 25.3 to 56.6 percent.

  2. Holds in trendCounts depend on an unwritten threshold.

    More neurons respond to a nameable concept

    Network Dissection counts how many of the 768 neurons in the encoder's last layer line up with one labelled concept. AFT raises the count from 160 to 390. The original paper reports 42 to 159. The count hinges on a threshold it doesn't state, which moves plain CLIP alone between 64 and 572, but the ordering does not flip.

  3. Holds in directionSkin lesions and a new task.

    The gains carry to other domains and other tasks

    On skin-lesion images the model never saw in training, the Simple Gradient pointing game rises from 0.107 to 0.428. For a six-class classifier trained on its features, it rises from 0.140 to 0.333.

  4. Holds on natural imagesSatellite imagery falls to chance.

    Accuracy is mostly kept, robustness is gained

    For the model trained at ε = 1/255, clean zero-shot accuracy drops by at most 3 points on CIFAR-10, STL-10 and Imagenette, in line with the original paper's 2.84. Satellite images are the exception, where every AFT model falls to chance. In return, where plain CLIP scores zero under attack, the models trained at ε = 4/255 keep about two thirds of their clean accuracy even under the stronger attack.

    Zero-shot accuracy averaged over five benchmarks, clean and under attack at two budgets.

Our extension: telling the model where to look

AFT never says where the saliency should go. We added a training term that does. It measures the share of the saliency map that falls inside the object's outline, and the Energy Pointing Game loss is one minus that share.

Ltotal= Ladv+ ws (1− 1B ∑i=1B ∑p∈Mi S^i(p) )

Ŝi is the normalised saliency map of image i, Mi its object mask, ws the weight of the new term. Setting ws = 0 gives plain AFT back.

The term needs object outlines, so we train on COCO. It takes the gradient of a gradient, so each step runs 1.27 times as long. Applied from the start, training doesn't converge. As a 2,000-step stage on top of a finished AFT model it works, and its weight becomes a dial.

00.20.40.6
Localisation on ImageNet-S as the weight rises. The control at 0 gets the same COCO training without the new term.

Against the control, ws = 0.2 costs 1.3 points of average zero-shot accuracy clean and 1.0 and 0.5 points under attack. At 0.6 the three drops are 3.3 to 3.9 points.

Grad-ECLIP, a saliency method the loss never sees, improves at every step too, which points to a change in the encoder itself. On skin-lesion images the Simple Gradient pointing game rises from 0.428 with AFT to 0.820 at ws = 0.6. In theory the term shifts the model's sensitivity from the background to the object, and a background-only attack agrees. Plain AFT reacts 1.25 to 1.29 times more to background pixels than to object pixels. The new term brings that ratio down to 0.68 at the highest weight, though part of that drop may come from the COCO training itself.

Deletion and insertion tests, and a test of whether the map follows the prompt, don't improve consistently with the weight. The clearest gain is in foreground alignment.

What to take away

The original paper's main claims hold in direction. The exact localisation scores and detector counts hinge on settings the original paper never states, and our code releases the rebuilt evaluations with those settings configurable. When object outlines are available, one extra training term moves saliency onto the object further than AFT alone.

The main training run used 28.7 kWh, about 7.7 kg CO₂e. The extension added 14.9 kWh.

Cite
@article{nuzhdin2026reboosting,
  title   = {[Re] Boosting the Visual Interpretability of CLIP via Adversarial Fine-Tuning},
  author  = {Nuzhdin, Anton and Gruzitski, Andrei and Egyed, Bal{\'a}zs and Raudvee, Aleksandr},
  journal = {Transactions on Machine Learning Research},
  year    = {2026},
  url     = {https://openreview.net/forum?id=uPXRBZfkYy}
}