Encoding only the task-related information from the raw data, \ie, disentangled representation learning, can greatly contribute to the robustness and generalizability of models. Although significant advances have been made by regularizing the information in representations with information theory, two major challenges remain: 1) the representation compression inevitably leads to performance drop; 2) the disentanglement constraints on representations are in complicated optimization. To these issues, we introduce Bayesian networks with transmitted information to formulate the interaction among input and representations during disentanglement. Building upon this framework, we propose \textbf{DisTIB} (\textbf{T}ransmitted \textbf{I}nformation \textbf{B}ottleneck for \textbf{Dis}entangled representation learning), a novel objective that navigates the balance between information compression and preservation. We employ variational inference to derive a tractable estimation for DisTIB. This estimation can be simply optimized via standard gradient descent with a reparameterization trick. Moreover, we theoretically prove that DisTIB can achieve optimal disentanglement, underscoring its superior efficacy. To solidify our claims, we conduct extensive experiments on various downstream tasks to demonstrate the appealing efficacy of DisTIB and validate our theoretical analyses.

翻译：仅从原始数据中编码任务相关信息，即分离表示学习，能够显著提升模型的鲁棒性与泛化能力。尽管通过信息论约束表示中的信息已取得显著进展，但仍存在两大挑战：1）表示压缩不可避免地导致性能下降；2）表示上的分离约束存在复杂的优化问题。针对这些问题，我们引入带有传输信息的贝叶斯网络，以形式化分离过程中输入与表示之间的交互关系。在此框架基础上，我们提出\textbf{DisTIB}（用于\textbf{分离}表示学习的\textbf{传输信息}瓶颈），这是一种能平衡信息压缩与保留的新型目标函数。我们采用变分推断推导出DisTIB的可处理估计量，该估计量可通过重参数化技巧配合标准梯度下降法进行简单优化。此外，我们理论上证明了DisTIB能够实现最优分离，彰显其卓越效能。为验证上述论断，我们在多种下游任务上开展广泛实验，不仅展示了DisTIB的显著效果，还验证了我们的理论分析。