This paper aims to advance our understanding of how Visual Language Models (VLMs) handle privacy-sensitive information, a crucial concern as these technologies become integral to everyday life. To this end, we introduce a new benchmark PrivBench, which contains images from 8 sensitive categories such as passports, or fingerprints. We evaluate 10 state-of-the-art VLMs on this benchmark and observe a generally limited understanding of privacy, highlighting a significant area for model improvement. Based on this we introduce PrivTune, a new instruction-tuning dataset aimed at equipping VLMs with knowledge about visual privacy. By tuning two pretrained VLMs, TinyLLaVa and MiniGPT-v2, on this small dataset, we achieve strong gains in their ability to recognize sensitive content, outperforming even GPT4-V. At the same time, we show that privacy-tuning only minimally affects the VLMs performance on standard benchmarks such as VQA. Overall, this paper lays out a crucial challenge for making VLMs effective in handling real-world data safely and provides a simple recipe that takes the first step towards building privacy-aware VLMs.

翻译：本文旨在深化对视觉语言模型（VLM）如何处理隐私敏感信息的理解，随着这些技术日益融入日常生活，这一问题至关重要。为此，我们引入了一个新的基准测试集PrivBench，其中包含来自护照、指纹等8个敏感类别的图像。我们在此基准上评估了10个最先进的VLM，观察到模型对隐私的理解普遍有限，这突显了一个重要的模型改进领域。基于此，我们引入了PrivTune，一个新的指令微调数据集，旨在为VLM提供关于视觉隐私的知识。通过在两个预训练的VLM（TinyLLaVa和MiniGPT-v2）上使用这个小数据集进行微调，我们在其识别敏感内容的能力上取得了显著提升，甚至超越了GPT4-V。同时，我们表明隐私微调仅对VLM在VQA等标准基准测试上的性能产生极小影响。总体而言，本文为实现VLM安全有效地处理现实世界数据提出了一个关键挑战，并提供了一种简单的方案，为构建隐私感知的VLM迈出了第一步。