As language-model agents evolve from passive chatbots into proactive assistants that handle personal data, evaluating their adherence to social norms becomes increasingly critical, often through the lens of Contextual Integrity (CI). However, existing CI benchmarks are largely text-centric and primarily emphasize negative refusal scenarios, overlooking multimodal privacy risks and the fundamental trade-off between privacy and utility. In this paper, we introduce MPCI-Bench, the first Multimodal Pairwise Contextual Integrity benchmark for evaluating privacy behavior in agentic settings. MPCI-Bench consists of paired positive and negative instances derived from the same visual source and instantiated across three tiers: normative Seed judgments, context-rich Story reasoning, and executable agent action Traces. Data quality is ensured through a Tri-Principle Iterative Refinement pipeline. Evaluations of state-of-the-art multimodal models reveal systematic failures to balance privacy and utility and a pronounced modality leakage gap, where sensitive visual information is leaked more frequently than textual information. We will open-source MPCI-Bench to facilitate future research on agentic CI.

翻译：随着语言模型智能体从被动的聊天机器人演变为能够处理个人数据的主动助手，评估其对社会规范的遵循变得日益关键，而上下文完整性（CI）常被用作评估视角。然而，现有的CI基准主要局限于文本模态，且过度强调负面拒绝场景，忽视了多模态隐私风险以及隐私与效用的基本权衡。本文提出了MPCI-Bench，这是首个用于评估智能体场景下隐私行为的多模态成对上下文完整性基准。MPCI-Bench包含源自同一视觉源的正负配对实例，并通过三个层级进行实例化：规范性的种子判断、上下文丰富的故事推理以及可执行的智能体行动轨迹。数据质量通过一项三原则迭代精炼流程予以保证。对前沿多模态模型的评估揭示了其在平衡隐私与效用方面存在系统性缺陷，并存在显著的模态泄露差距，即敏感视觉信息比文本信息更频繁地被泄露。我们将开源MPCI-Bench，以促进未来关于智能体上下文完整性的研究。