Information Bottlenecks (IBs) learn representations that generalize to unseen data by information compression. However, existing IBs are practically unable to guarantee generalization in real-world scenarios due to the vacuous generalization bound. The recent PAC-Bayes IB uses information complexity instead of information compression to establish a connection with the mutual information generalization bound. However, it requires the computation of expensive second-order curvature, which hinders its practical application. In this paper, we establish the connection between the recognizability of representations and the recent functional conditional mutual information (f-CMI) generalization bound, which is significantly easier to estimate. On this basis we propose a Recognizable Information Bottleneck (RIB) which regularizes the recognizability of representations through a recognizability critic optimized by density ratio matching under the Bregman divergence. Extensive experiments on several commonly used datasets demonstrate the effectiveness of the proposed method in regularizing the model and estimating the generalization gap.

翻译：信息瓶颈（IB）通过信息压缩学习能够泛化到未见数据的表征。然而，由于泛化界空洞问题，现有IB在实际场景中无法保证泛化能力。最近的PAC-Bayes IB利用信息复杂度而非信息压缩建立与互信息泛化界的关联，但该方法需要计算昂贵的二阶曲率，阻碍了其实际应用。本文建立了表征可识别性与近期提出的函数条件互信息（f-CMI）泛化界之间的联系，后者显著更易于估计。基于此，我们提出可识别信息瓶颈（RIB），通过基于布雷格曼散度下密度比匹配优化的可识别性判别器，对表征的可识别性进行正则化。在多个常用数据集上的大量实验表明，所提方法在正则化模型和估计泛化差距方面具有有效性。