The Information Bottleneck (IB) principle offers a compelling theoretical framework to understand how neural networks (NNs) learn. However, its practical utility has been constrained by unresolved theoretical ambiguities and significant challenges in accurate estimation. In this paper, we present a \textit{Generalized Information Bottleneck (GIB)} framework that reformulates the original IB principle through the lens of synergy, i.e., the information obtainable only through joint processing of features. We provide theoretical and empirical evidence demonstrating that synergistic functions achieve superior generalization compared to their non-synergistic counterparts. Building on these foundations we re-formulate the IB using a computable definition of synergy based on the average interaction information (II) of each feature with those remaining. We demonstrate that the original IB objective is upper bounded by our GIB in the case of perfect estimation, ensuring compatibility with existing IB theory while addressing its limitations. Our experimental results demonstrate that GIB consistently exhibits compression phases across a wide range of architectures (including those with \textit{ReLU} activations where the standard IB fails), while yielding interpretable dynamics in both CNNs and Transformers and aligning more closely with our understanding of adversarial robustness.

翻译：信息瓶颈（IB）原理为理解神经网络（NNs）的学习机制提供了一个引人注目的理论框架。然而，其实际应用一直受限于未解决的理论模糊性以及准确估计方面的重大挑战。本文提出了一种\textit{广义信息瓶颈（GIB）}框架，该框架通过协同性（即仅通过对特征进行联合处理才能获得的信息）的视角重新阐述了原始IB原理。我们提供了理论和实证证据，证明与非协同性函数相比，协同性函数能实现更优的泛化性能。基于这些基础，我们利用基于每个特征与其余特征的平均交互信息（II）的可计算协同性定义，重新构建了IB。我们证明，在完美估计的情况下，原始IB目标函数受我们的GIB上界约束，从而确保了与现有IB理论的兼容性，同时解决了其局限性。我们的实验结果表明，GIB在多种架构（包括标准IB失效的\textit{ReLU}激活函数架构）中均能一致地表现出压缩阶段，同时在CNN和Transformer中产生可解释的动态特性，并且更贴近我们对于对抗鲁棒性的理解。