Some decisions we make with our eyes 👀, but what about VLMs? Do they have structured, exploitable visual preferences that we can discover systematically before adversarial actors do?
In our new paper, we propose a new optimization method for this and show substantial effects on VLMs’ decisions.