In recent years, deep models have achieved remarkable success in many vision tasks. Unfortunately, their performance largely depends on intensive training samples. In contrast, human beings typically perform hybrid learning, e.g., spontaneously integrating structured knowledge for cross-domain recognition or on a much smaller amount of data samples for few-shot learning. Thus it is very attractive to extend hybrid learning for the computer vision tasks by seamlessly integrating structured knowledge with data samples to achieve more effective representation learning. However, such a hybrid learning approach remains a great challenge due to the huge gap between the structured knowledge and the deep features (learned from data samples) on both dimensions and knowledge granularity. In this paper, a novel Epistemic Graph Layer (EGLayer) is developed to enable hybrid learning, such that the information can be exchanged more effectively between the deep features and a structured knowledge graph. Our EGLayer is composed of three major parts: (a) a local graph module to establish a local prototypical graph through the learned deep features, i.e., aligning the deep features with the structured knowledge graph at the same granularity; (b) a query aggregation model to aggregate useful information from the local graphs, and using such representations to compute their similarity with global node embeddings for final prediction; and (c) a novel correlation loss function to constrain the linear consistency between the local and global adjacency matrices.

翻译：近年来，深度模型在众多视觉任务中取得了显著成功。然而，其性能高度依赖于大量训练样本。相比之下，人类通常能进行混合学习，例如自发整合结构化知识进行跨域识别，或利用少量数据样本进行小样本学习。因此，通过将结构化知识与数据样本无缝融合以提升视觉任务中的表示学习效率，具有重要的研究价值。然而，由于结构化知识与深度特征（从数据样本中习得）在维度和知识粒度上存在巨大差异，这种混合学习方法仍面临重大挑战。本文提出一种新型知识图谱层（Epistemic Graph Layer, EGLayer），通过促进深度特征与结构化知识图谱之间的信息高效交互实现混合学习。该层由三部分构成：（a）局部图谱模块：基于学到的深度特征构建局部原型图谱，即将深度特征与结构化知识图谱对齐至相同粒度；（b）查询聚合模型：从局部图谱中聚合有效信息，并利用这些表示计算其与全局节点嵌入的相似度以完成最终预测；（c）新型相关性损失函数：约束局部邻接矩阵与全局邻接矩阵之间的线性一致性。