The information bottleneck (IB) approach is popular to improve the generalization, robustness and explainability of deep neural networks. Essentially, it aims to find a minimum sufficient representation $\mathbf{t}$ by striking a trade-off between a compression term $I(\mathbf{x};\mathbf{t})$ and a prediction term $I(y;\mathbf{t})$, where $I(\cdot;\cdot)$ refers to the mutual information (MI). MI is for the IB for the most part expressed in terms of the Kullback-Leibler (KL) divergence, which in the regression case corresponds to prediction based on mean squared error (MSE) loss with Gaussian assumption and compression approximated by variational inference. In this paper, we study the IB principle for the regression problem and develop a new way to parameterize the IB with deep neural networks by exploiting favorable properties of the Cauchy-Schwarz (CS) divergence. By doing so, we move away from MSE-based regression and ease estimation by avoiding variational approximations or distributional assumptions. We investigate the improved generalization ability of our proposed CS-IB and demonstrate strong adversarial robustness guarantees. We demonstrate its superior performance on six real-world regression tasks over other popular deep IB approaches. We additionally observe that the solutions discovered by CS-IB always achieve the best trade-off between prediction accuracy and compression ratio in the information plane. The code is available at \url{https://github.com/SJYuCNEL/Cauchy-Schwarz-Information-Bottleneck}.

翻译：信息瓶颈（IB）方法是提升深度神经网络泛化性、鲁棒性和可解释性的流行方法。其核心旨在通过权衡压缩项 $I(\mathbf{x};\mathbf{t})$ 和预测项 $I(y;\mathbf{t})$ 来寻找最小充分表征 $\mathbf{t}$，其中 $I(\cdot;\cdot)$ 表示互信息。在IB中，互信息通常以Kullback-Leibler散度表达，在回归情形下对应基于高斯假设的均方误差损失进行预测，并通过变分推断近似压缩项。本文研究了回归问题的IB原理，利用柯西-施瓦茨散度的有利性质，提出了一种用深度神经网络参数化IB的新方法。由此，我们摆脱了基于均方误差的回归范式，通过避免变分近似或分布假设简化了估计过程。我们探究了所提CS-IB的泛化能力提升，并展示了其强大的对抗鲁棒性保证。在六个真实回归任务上，我们证明了该方法相较于其他主流深度IB方法的优越性能。此外，我们观察到CS-IB发现的解在信息平面上始终能达到预测精度与压缩比的最佳权衡。代码见 \url{https://github.com/SJYuCNEL/Cauchy-Schwarz-Information-Bottleneck}。