The information bottleneck (IB) method offers an attractive framework for understanding representation learning, however its applications are often limited by its computational intractability. Analytical characterization of the IB method is not only of practical interest, but it can also lead to new insights into learning phenomena. Here we consider a generalized IB problem, in which the mutual information in the original IB method is replaced by correlation measures based on Renyi and Jeffreys divergences. We derive an exact analytical IB solution for the case of Gaussian correlated variables. Our analysis reveals a series of structural transitions, similar to those previously observed in the original IB case. We find further that although solving the original, Renyi and Jeffreys IB problems yields different representations in general, the structural transitions occur at the same critical tradeoff parameters, and the Renyi and Jeffreys IB solutions perform well under the original IB objective. Our results suggest that formulating the IB method with alternative correlation measures could offer a strategy for obtaining an approximate solution to the original IB problem.

翻译：信息瓶颈（IB）方法为理解表示学习提供了一个富有吸引力的框架，但其应用常受限于计算复杂性。对IB方法的解析刻画不仅具有实际意义，还能为学习现象带来新见解。本文考虑了一个广义IB问题，其中原始IB方法中的互信息被基于Renyi散度和Jeffreys散度的相关性度量所替代。我们推导了高斯相关变量情形下的精确解析IB解。分析揭示了一系列结构相变现象，与原始IB情形中先前观察到的现象类似。进一步发现，尽管求解原始IB、Renyi IB和Jeffreys IB问题通常会产生不同的表示，但这些结构相变发生在相同的临界权衡参数处，且Renyi IB和Jeffreys IB解在原始IB目标下表现良好。我们的结果表明，采用替代相关性度量构建IB方法可作为获取原始IB问题近似解的一种策略。