We present a novel information-theoretic framework, termed as TURBO, designed to systematically analyse and generalise auto-encoding methods. We start by examining the principles of information bottleneck and bottleneck-based networks in the auto-encoding setting and identifying their inherent limitations, which become more prominent for data with multiple relevant, physics-related representations. The TURBO framework is then introduced, providing a comprehensive derivation of its core concept consisting of the maximisation of mutual information between various data representations expressed in two directions reflecting the information flows. We illustrate that numerous prevalent neural network models are encompassed within this framework. The paper underscores the insufficiency of the information bottleneck concept in elucidating all such models, thereby establishing TURBO as a preferable theoretical reference. The introduction of TURBO contributes to a richer understanding of data representation and the structure of neural network models, enabling more efficient and versatile applications.

翻译：我们提出了一种新颖的信息论框架，称为TURBO，旨在系统性地分析并泛化自编码方法。首先，我们考察了信息瓶颈和基于瓶颈网络在自编码设置中的原理，并识别出其固有局限性——这些局限在处理具有多种相关物理表示的数据时尤为突出。随后，我们引入TURBO框架，详尽推导了其核心概念：最大化反映信息流两个方向的数据表示间的互信息。我们阐明众多主流神经网络模型均被涵盖于此框架之中。本文强调了信息瓶颈概念在解释所有这些模型方面的不足，从而确立TURBO作为更优选的理论参考。TURBO的引入有助于更深刻地理解数据表示与神经网络模型结构，助力实现更高效、更通用的应用。