Representation learning for images has been advanced by recent progress in more complex neural models such as the Vision Transformers and new learning theories such as the structural causal models. However, these models mainly rely on the classification loss to implicitly regularize the class-level data distributions, and they may face difficulties when handling classes with diverse visual patterns. We argue that the incorporation of the structural information between data samples may improve this situation. To achieve this goal, this paper presents a framework termed \textbf{C}lass-level Structural Relation Modeling and Smoothing for Visual Representation Learning (CSRMS), which includes the Class-level Relation Modelling, Class-aware Graph Sampling, and Relational Graph-Guided Representation Learning modules to model a relational graph of the entire dataset and perform class-aware smoothing and regularization operations to alleviate the issue of intra-class visual diversity and inter-class similarity. Specifically, the Class-level Relation Modelling module uses a clustering algorithm to learn the data distributions in the feature space and identify three types of class-level sample relations for the training set; Class-aware Graph Sampling module extends typical training batch construction process with three strategies to sample dataset-level sub-graphs; and Relational Graph-Guided Representation Learning module employs a graph convolution network with knowledge-guided smoothing operations to ease the projection from different visual patterns to the same class. Experiments demonstrate the effectiveness of structured knowledge modelling for enhanced representation learning and show that CSRMS can be incorporated with any state-of-the-art visual representation learning models for performance gains. The source codes and demos have been released at https://github.com/czt117/CSRMS.

翻译：图像表示学习得益于视觉Transformer等复杂神经模型及结构因果模型等新学习理论的进展。然而，这些模型主要依赖分类损失隐式约束类别级数据分布，在处理具有多样化视觉模式的类别时可能面临困难。我们认为，引入数据样本间的结构信息可改善这一状况。为此，本文提出**类别级结构关系建模与平滑**框架（CSRMS），包含类别级关系建模、类别感知图采样和关系图引导表示学习模块，通过构建整个数据集的关系图并执行类别感知的平滑与正则化操作，缓解类内视觉多样性与类间相似性问题。具体而言，类别级关系建模模块利用聚类算法学习特征空间中的数据分布，为训练集识别三类类别级样本关系；类别感知图采样模块通过三种策略扩展典型批次构建流程，对数据集级子图进行采样；关系图引导表示学习模块采用图卷积网络结合知识引导的平滑操作，简化不同视觉模式到同一类别的投影过程。实验表明，结构化知识建模能有效增强表示学习，且CSRMS可与任何先进视觉表示学习模型集成以提升性能。源代码与演示已发布至https://github.com/czt117/CSRMS。