Large Vision-Language Models (LVLMs) exhibit impressive potential across various tasks but also face significant privacy risks, limiting their practical applications. Current researches on privacy assessment for LVLMs is limited in scope, with gaps in both assessment dimensions and privacy categories. To bridge this gap, we propose Multi-PA, a comprehensive benchmark for evaluating the privacy preservation capabilities of LVLMs in terms of privacy awareness and leakage. Privacy awareness measures the model's ability to recognize the privacy sensitivity of input data, while privacy leakage assesses the risk of the model unintentionally disclosing privacy information in its output. We design a range of sub-tasks to thoroughly evaluate the model's privacy protection offered by LVLMs. Multi-PA covers 26 categories of personal privacy, 15 categories of trade secrets, and 18 categories of state secrets, totaling 31,962 samples. Based on Multi-PA, we evaluate the privacy preservation capabilities of 21 open-source and 2 closed-source LVLMs. Our results reveal that current LVLMs generally pose a high risk of facilitating privacy breaches, with vulnerabilities varying across personal privacy, trade secret, and state secret.

翻译：大型视觉语言模型（LVLMs）在各种任务中展现出巨大潜力，但也面临显著的隐私风险，这限制了其实际应用。当前针对LVLMs隐私评估的研究在评估维度和隐私类别方面均存在局限。为填补这一空白，我们提出Multi-PA——一个从隐私感知与隐私泄露两个维度综合评估LVLMs隐私保护能力的基准框架。隐私感知衡量模型识别输入数据隐私敏感性的能力，隐私泄露则评估模型在输出中无意披露隐私信息的风险。我们设计了一系列子任务以全面评估LVLMs提供的隐私保护能力。Multi-PA涵盖26类个人隐私、15类商业秘密和18类国家秘密，共计31,962个样本。基于Multi-PA，我们对21个开源和2个闭源LVLMs进行了隐私保护能力评估。结果表明，当前LVLMs普遍存在较高的隐私泄露风险，且在个人隐私、商业秘密和国家秘密方面的脆弱性存在差异。