Variational dimensionality reduction methods are known for their high accuracy, generative abilities, and robustness. We introduce a framework to unify many existing variational methods and design new ones. The framework is based on an interpretation of the multivariate information bottleneck, in which an encoder graph, specifying what information to compress, is traded-off against a decoder graph, specifying a generative model. Using this framework, we rederive existing dimensionality reduction methods including the deep variational information bottleneck and variational auto-encoders. The framework naturally introduces a trade-off parameter extending the deep variational CCA (DVCCA) family of algorithms to beta-DVCCA. We derive a new method, the deep variational symmetric informational bottleneck (DVSIB), which simultaneously compresses two variables to preserve information between their compressed representations. We implement these algorithms and evaluate their ability to produce shared low dimensional latent spaces on Noisy MNIST dataset. We show that algorithms that are better matched to the structure of the data (in our case, beta-DVCCA and DVSIB) produce better latent spaces as measured by classification accuracy, dimensionality of the latent variables, and sample efficiency. We believe that this framework can be used to unify other multi-view representation learning algorithms and to derive and implement novel problem-specific loss functions.

翻译：变分降维方法以其高精度、生成能力及鲁棒性而著称。我们提出一个统一现有多种变分方法并设计新方法的框架。该框架基于多变量信息瓶颈的诠释，其中编码器图（指定需压缩的信息）与解码器图（指定生成模型）之间存在权衡。利用该框架，我们重新推导了现有降维方法，包括深度变分信息瓶颈和变分自编码器。该框架自然地引入一个权衡参数，将深度变分CCA（DVCCA）算法族扩展至β-DVCCA。我们推导出一种新方法——深度变分对称信息瓶颈（DVSIB），该方法同步压缩两个变量，以保留其压缩表示之间的信息。我们在含噪MNIST数据集上实现这些算法，并评估其生成共享低维潜在空间的能力。实验表明，与数据结构更匹配的算法（本文中为β-DVCCA和DVSIB）能产生更优的潜在空间，具体体现在分类精度、潜在变量维度及样本效率上。我们相信该框架可用于统一其他多视角表示学习算法，并推导及实现面向特定问题的新型损失函数。