Total correlation (TC) is a fundamental concept in information theory that measures statistical dependency among multiple random variables. Recently, TC has shown noticeable effectiveness as a regularizer in many learning tasks, where the correlation among multiple latent embeddings requires to be jointly minimized or maximized. However, calculating precise TC values is challenging, especially when the closed-form distributions of embedding variables are unknown. In this paper, we introduce a unified framework to estimate total correlation values with sample-based mutual information (MI) estimators. More specifically, we discover a relation between TC and MI and propose two types of calculation paths (tree-like and line-like) to decompose TC into MI terms. With each MI term being bounded, the TC values can be successfully estimated. Further, we provide theoretical analyses concerning the statistical consistency of the proposed TC estimators. Experiments are presented on both synthetic and real-world scenarios, where our estimators demonstrate effectiveness in all TC estimation, minimization, and maximization tasks. The code is available at https://github.com/Linear95/TC-estimation.

翻译：总相关性（TC）是信息论中衡量多个随机变量间统计依赖性的基本概念。近年来，TC作为正则化项在诸多学习任务中展现出显著效果，其中多组潜在嵌入间的相关性需要被联合最小化或最大化。然而，精确计算TC值颇具挑战性，尤其是在嵌入变量的闭式分布未知的情况下。本文提出统一框架，通过基于样本的互信息（MI）估计器估算总相关性数值。具体而言，我们发现了TC与MI之间的关联，并提出了两种计算路径（树状路径与线状路径），将TC分解为多个MI项。通过约束每个MI项，TC值可被成功估计。此外，我们针对所提TC估计器的统计一致性给出了理论分析。在合成数据与真实场景中的实验表明，我们的估计器在TC估算、最小化及最大化任务中均展现出有效性。代码开源在 https://github.com/Linear95/TC-estimation。