Most works studying representation learning focus only on classification and neglect regression. Yet, the learning objectives and therefore the representation topologies of the two tasks are fundamentally different: classification targets class separation, leading to disconnected representations, whereas regression requires ordinality with respect to the target, leading to continuous representations. We thus wonder how the effectiveness of a regression representation is influenced by its topology, with evaluation based on the Information Bottleneck (IB) principle. The IB principle is an important framework that provides principles for learning effectiveness representations. We establish two connections between it and the topology of regression representations. The first connection reveals that a lower intrinsic dimension of the feature space implies a reduced complexity of the representation Z. This complexity can be quantified as the conditional entropy of Z on the target space Y and serves as an upper bound on the generalization error. The second connection suggests learning a feature space that is topologically similar to the target space will better align with the IB principle. Based on these two connections, we introduce PH-Reg, a regularizer specific to regression that matches the intrinsic dimension and topology of the feature space with the target space. Experiments on synthetic and real-world regression tasks demonstrate the benefits of PH-Reg.

翻译：大多数表示学习研究仅关注分类问题而忽视了回归任务。然而，这两个任务的学习目标及表示拓扑存在本质差异：分类追求类别分离，产生非连通的表示；而回归需保持与目标的序数关系，形成连续表示。因此，我们探究回归表示的有效性如何受其拓扑影响，并基于信息瓶颈（IB）原则进行评估。IB原则是衡量有效表示学习的重要框架。我们建立了该原则与回归表示拓扑的两类关联：第一类关联揭示特征空间的内在维度越低，表示Z的复杂度越小——该复杂度可量化为Z关于目标空间Y的条件熵，并作为泛化误差的上界；第二类关联表明，学习与目标空间拓扑相似的特征空间能更好地契合IB原则。基于这两类关联，我们提出PH-Reg——一种专用于回归的正则化方法，使特征空间的内在维度和拓扑与目标空间对齐。在合成数据与真实回归任务上的实验证明了PH-Reg的有效性。