Machine learning based classifiers that take a privacy policy as the input and predict relevant concepts are useful in different applications such as (semi-)automated compliance analysis against requirements of the EU GDPR. In all past studies, such classifiers produce a concept label per segment (e.g., sentence or paragraph) and their performances were evaluated by using a dataset of labeled segments without considering the privacy policy they belong to. However, such an approach could overestimate the performance in real-world settings, where all segments in a new privacy policy are supposed to be unseen. Additionally, we also observed other research gaps, including the lack of a more complete GDPR taxonomy and the less consideration of hierarchical information in privacy policies. To fill such research gaps, we developed a more complete GDPR taxonomy, created the first corpus of labeled privacy policies with hierarchical information, and conducted the most comprehensive performance evaluation of GDPR concept classifiers for privacy policies. Our work leads to multiple novel findings, including the confirmed inappropriateness of splitting training and test sets at the segment level, the benefits of considering hierarchical information, and the limitations of the "one size fits all" approach, and the significance of testing cross-corpus generalizability.

翻译：基于机器学习的分类器以隐私政策文本作为输入并预测相关概念，在（半）自动化合规分析（如针对欧盟《通用数据保护条例》要求）等不同应用中具有重要价值。过往所有研究中，此类分类器均针对每个文本片段（如句子或段落）生成概念标签，其性能评估均基于已标注片段的独立数据集完成，而未考虑片段所属的原始隐私政策整体。然而，这种方法可能在实际应用场景中高估模型性能，因为现实场景中新隐私政策的所有文本片段理应均为未见过的新数据。此外，我们还观察到其他研究空白，包括缺乏更完整的GDPR分类体系，以及对隐私政策中层级结构信息的考量不足。为填补这些研究空白，我们构建了更完整的GDPR分类体系，创建了首个包含层级标注信息的隐私政策标注语料库，并开展了迄今最全面的隐私政策GDPR概念分类器性能评估。本研究获得多项创新发现，包括：证实了在片段层面划分训练集与测试集的不合理性，验证了利用层级信息的有效性，揭示了"一刀切"方法的局限性，并论证了跨语料库泛化能力测试的重要性。