Information theoretic quantities play a central role in machine learning. The recent surge in the complexity of data and models has increased the demand for accurate estimation of these quantities. However, as the dimension grows the estimation presents significant challenges, with existing methods struggling already in relatively low dimensions. To address this issue, in this work, we introduce $\texttt{REMEDI}$ for efficient and accurate estimation of differential entropy, a fundamental information theoretic quantity. The approach combines the minimization of the cross-entropy for simple, adaptive base models and the estimation of their deviation, in terms of the relative entropy, from the data density. Our approach demonstrates improvement across a broad spectrum of estimation tasks, encompassing entropy estimation on both synthetic and natural data. Further, we extend important theoretical consistency results to a more generalized setting required by our approach. We illustrate how the framework can be naturally extended to information theoretic supervised learning models, with a specific focus on the Information Bottleneck approach. It is demonstrated that the method delivers better accuracy compared to the existing methods in Information Bottleneck. In addition, we explore a natural connection between $\texttt{REMEDI}$ and generative modeling using rejection sampling and Langevin dynamics.

翻译：信息论量在机器学习中扮演着核心角色。近年来数据和模型复杂性的激增提高了对这些量进行精确估计的需求。然而，随着维度的增长，估计面临巨大挑战，现有方法即使在相对较低的维度中也难以应对。为解决这一问题，本文引入$\texttt{REMEDI}$，用于高效且精确地估计微分熵——一种基础的信息论量。该方法结合了对简单、自适应基模型的交叉熵最小化，以及通过相对熵估计这些基模型与数据密度之间的偏差。我们的方法在广泛的估计任务中展现出改进，涵盖合成数据和自然数据上的熵估计。此外，我们将重要的理论一致性结果推广到适用于我们方法的更一般化场景。我们展示了该框架如何自然地扩展到信息论监督学习模型，并特别关注信息瓶颈方法。实验表明，在信息瓶颈中，该方法比现有方法具有更高的准确性。同时，我们探索了$\texttt{REMEDI}$与使用拒绝采样和朗之万动力学的生成建模之间的自然联系。