The web is littered with images, once created for human consumption and now increasingly interpreted by agents using vision-language models (VLMs). These agents make visual decisions at scale, deciding what to click, recommend, or buy. Yet, we know little about the structure of their visual preferences. We introduce a framework for studying this by placing VLMs in controlled image-based choice tasks and systematically perturbing their inputs. Our key idea is to treat the agent's decision function as a latent visual utility that can be inferred through revealed preference: choices between systematically edited images. Starting from common images, such as product photos, we propose methods for visual prompt optimization, adapting text optimization methods to iteratively propose and apply visually plausible modifications using an image generation model (such as in composition, lighting, or background). We then evaluate which edits increase selection probability. Through large-scale experiments on frontier VLMs, we demonstrate that optimized edits significantly shift choice probabilities in head-to-head comparisons. We develop an automatic interpretability pipeline to explain these preferences, identifying consistent visual themes that drive selection. We argue that this approach offers a practical and efficient way to surface visual vulnerabilities, safety concerns that might otherwise be discovered implicitly in the wild, supporting more proactive auditing and governance of image-based AI agents.

翻译：互联网充斥着大量图像，这些图像最初为人类消费而创建，如今正日益被使用视觉-语言模型（VLMs）的智能体所解读。这些智能体大规模地做出视觉决策，决定点击、推荐或购买什么内容。然而，我们对它们的视觉偏好结构知之甚少。为此，我们引入了一个研究框架，将VLMs置于受控的基于图像的选择任务中，并系统性地扰动其输入。我们的核心思想是将智能体的决策函数视为一种潜在的视觉效用，可以通过显示性偏好来推断：即通过对经过系统性编辑的图像进行选择来揭示偏好。从常见图像（如产品照片）出发，我们提出了视觉提示优化的方法，将文本优化技术适配为迭代式地提出并应用视觉上合理的修改（例如在构图、光照或背景方面），利用图像生成模型来实现。随后，我们评估哪些编辑能提高选择概率。通过对前沿VLMs进行大规模实验，我们证明在直接比较中，优化后的编辑能显著改变选择概率。我们开发了一个自动可解释性流程来解释这些偏好，识别出驱动选择的一致视觉主题。我们认为，这种方法提供了一种实用且高效的途径来揭示视觉漏洞和安全隐患——这些隐患若在现实世界中被动发现可能代价高昂，从而支持对基于图像的人工智能智能体进行更主动的审计与治理。