Most works studying representation learning focus only on classification and neglect regression. Yet, the learning objectives and therefore the representation topologies of the two tasks are fundamentally different: classification targets class separation, leading to disconnected representations, whereas regression requires ordinality with respect to the target, leading to continuous representations. We thus wonder how the effectiveness of a regression representation is influenced by its topology, with evaluation based on the Information Bottleneck (IB) principle. The IB principle is an important framework that provides principles for learning effectiveness representations. We establish two connections between it and the topology of regression representations. The first connection reveals that a lower intrinsic dimension of the feature space implies a reduced complexity of the representation Z. This complexity can be quantified as the conditional entropy of Z on the target space Y and serves as an upper bound on the generalization error. The second connection suggests learning a feature space that is topologically similar to the target space will better align with the IB principle. Based on these two connections, we introduce PH-Reg, a regularizer specific to regression that matches the intrinsic dimension and topology of the feature space with the target space. Experiments on synthetic and real-world regression tasks demonstrate the benefits of PH-Reg.

翻译：大多数表示学习研究仅关注分类任务而忽视了回归任务。然而，这两个任务的学习目标及由此产生的表示拓扑结构存在根本差异：分类目标要求类别分离，导致不连续的表示；而回归则需要目标值的有序性，从而形成连续表示。因此，我们探究回归表示的有效性如何受其拓扑结构影响，并基于信息瓶颈（Information Bottleneck, IB）原则进行评估。IB原则为学习有效表示提供了重要理论框架。我们建立了回归表示拓扑与IB原则之间的两种关联：第一重关联揭示特征空间的内在维度越低，表示Z的复杂度越小。该复杂度可通过Z关于目标空间Y的条件熵量化，并构成泛化误差的上界。第二重关联表明，学习与目标空间拓扑相似的特征空间能更好地符合IB原则。基于这两种关联，我们提出PH-Reg——一种专用于回归的正则化方法，使特征空间的内在维度和拓扑结构与目标空间相匹配。在合成数据和真实回归任务上的实验验证了PH-Reg的有效性。