Deep InfoMax (DIM) is a well-established method for self-supervised representation learning (SSRL) based on maximization of the mutual information between the input and the output of a deep neural network encoder. Despite the DIM and contrastive SSRL in general being well-explored, the task of learning representations conforming to a specific distribution (i.e., distribution matching, DM) is still under-addressed. Motivated by the importance of DM to several downstream tasks (including generative modeling, disentanglement, outliers detection and other), we enhance DIM to enable automatic matching of learned representations to a selected prior distribution. To achieve this, we propose injecting an independent noise into the normalized outputs of the encoder, while keeping the same InfoMax training objective. We show that such modification allows for learning uniformly and normally distributed representations, as well as representations of other absolutely continuous distributions. Our approach is tested on various downstream tasks. The results indicate a moderate trade-off between the performance on the downstream tasks and quality of DM.

翻译：深度信息最大化（DIM）是一种基于最大化深度神经网络编码器输入与输出间互信息的成熟自监督表示学习方法。尽管DIM及对比式自监督表示学习已得到充分研究，但学习符合特定分布的表示（即分布匹配，DM）这一任务仍未得到充分解决。鉴于分布匹配在多项下游任务（包括生成建模、解耦表示、异常检测等）中的重要性，我们对DIM进行改进，使其能够自动将学习到的表示匹配至选定的先验分布。为实现这一目标，我们提出在编码器归一化输出中注入独立噪声，同时保持相同的信息最大化训练目标。我们证明这种改进能够学习均匀分布、正态分布的表示，以及其他绝对连续分布的表示。我们在多种下游任务上测试了该方法。结果表明，在下游任务性能与分布匹配质量之间存在适度的权衡关系。