Privacy policies provide individuals with information about their rights and how their personal information is handled. Natural language understanding (NLU) technologies can support individuals and practitioners to understand better privacy practices described in lengthy and complex documents. However, existing efforts that use NLU technologies are limited by processing the language in a way exclusive to a single task focusing on certain privacy practices. To this end, we introduce the Privacy Policy Language Understanding Evaluation (PLUE) benchmark, a multi-task benchmark for evaluating the privacy policy language understanding across various tasks. We also collect a large corpus of privacy policies to enable privacy policy domain-specific language model pre-training. We evaluate several generic pre-trained language models and continue pre-training them on the collected corpus. We demonstrate that domain-specific continual pre-training offers performance improvements across all tasks.

翻译：隐私策略为个人提供关于其权利及个人信息处理方式的信息。自然语言理解技术能够帮助个人及从业人员更好地理解冗长复杂文档中描述的隐私实践。然而，现有应用自然语言理解技术的研究受限于以单一任务方式处理语言，仅聚焦于特定隐私实践。为此，我们提出隐私策略语言理解评估基准——PLUE基准，这是一个面向隐私策略语言理解的多任务评估基准。我们还收集了大规模隐私策略语料库，用于支持隐私策略领域的语言模型预训练。我们对多个通用预训练语言模型进行了评估，并在所收集语料库上对其进行了持续预训练。实验证明，领域特定的持续预训练能够显著提升所有任务的性能表现。