Language models such as mBERT, XLM-R, and BLOOM aim to achieve multilingual generalization or compression to facilitate transfer to a large number of (potentially unseen) languages. However, these models should ideally also be private, linguistically fair, and transparent, by relating their predictions to training data. Can these requirements be simultaneously satisfied? We show that multilingual compression and linguistic fairness are compatible with differential privacy, but that differential privacy is at odds with training data influence sparsity, an objective for transparency. We further present a series of experiments on two common NLP tasks and evaluate multilingual compression and training data influence sparsity under different privacy guarantees, exploring these trade-offs in more detail. Our results suggest that we need to develop ways to jointly optimize for these objectives in order to find practical trade-offs.

翻译：诸如mBERT、XLM-R和BLOOM等语言模型旨在实现多语言泛化或压缩，以促进向大量（潜在未见过的）语言进行迁移。然而，这些模型理想情况下还应通过将预测与训练数据相关联，具备隐私性、语言公平性和透明性。这些需求能否同时满足？我们证明多语言压缩与语言公平性与差分隐私兼容，但差分隐私与训练数据影响稀疏性（透明性的一个目标）存在矛盾。我们进一步在两项常见自然语言处理任务上开展系列实验，在不同隐私保障下评估多语言压缩与训练数据影响稀疏性，深入探讨这些权衡关系。研究结果表明，我们需要开发联合优化这些目标的方法，以找到实用的折衷方案。