With the exponential growth in data volume and the emergence of data-intensive applications, particularly in the field of machine learning, concerns related to resource utilization, privacy, and fairness have become paramount. This paper focuses on the textual domain of data and addresses challenges regarding encoding sentences to their optimized representations through the lens of information-theory. In particular, we use empirical estimates of mutual information, using the Donsker-Varadhan definition of Kullback-Leibler divergence. Our approach leverages this estimation to train an information-theoretic sentence embedding, called TexShape, for (task-based) data compression or for filtering out sensitive information, enhancing privacy and fairness. In this study, we employ a benchmark language model for initial text representation, complemented by neural networks for information-theoretic compression and mutual information estimations. Our experiments demonstrate significant advancements in preserving maximal targeted information and minimal sensitive information over adverse compression ratios, in terms of predictive accuracy of downstream models that are trained using the compressed data.

翻译：摘要：随着数据量的指数级增长以及数据密集型应用（尤其在机器学习领域）的兴起，与资源利用、隐私保护和公平性相关的关切已变得至关重要。本文聚焦文本数据领域，从信息论视角应对句子优化编码表征的挑战。具体而言，我们采用基于Donsker-Varadhan定义的Kullback-Leibler散度来对互信息进行经验估计。该方法利用此估计训练一种名为TexShape的信息论句子嵌入，用于（任务导向的）数据压缩或过滤敏感信息，从而增强隐私保护与公平性。本研究采用基准语言模型进行初始文本表征，并辅以神经网络实现信息论压缩与互信息估计。实验结果表明，在不利压缩比条件下，该方法在保留最大目标信息与最小化敏感信息方面取得了显著进展，这体现在使用压缩数据训练的下游模型的预测准确性上。