Visual object recognition systems need to generalize from a set of 2D training views to novel views. The question of how the human visual system can generalize to novel views has been studied and modeled in psychology, computer vision, and neuroscience. Modern deep learning architectures for object recognition generalize well to novel views, but the mechanisms are not well understood. In this paper, we characterize the ability of common deep learning architectures to generalize to novel views. We formulate this as a supervised classification task where labels correspond to unique 3D objects and examples correspond to 2D views of the objects at different 3D orientations. We consider three common models of generalization to novel views: (i) full 3D generalization, (ii) pure 2D matching, and (iii) matching based on a linear combination of views. We find that deep models generalize well to novel views, but they do so in a way that differs from all these existing models. Extrapolation to views beyond the range covered by views in the training set is limited, and extrapolation to novel rotation axes is even more limited, implying that the networks do not infer full 3D structure, nor use linear interpolation. Yet, generalization is far superior to pure 2D matching. These findings help with designing datasets with 2D views required to achieve 3D generalization. Code to reproduce our experiments is publicly available: https://github.com/shoaibahmed/investigating_3d_generalization.git

翻译：视觉物体识别系统需要从一组二维训练视图泛化到新视图。人类视觉系统如何泛化到新视图的问题已在心理学、计算机视觉和神经科学中被研究并建模。用于物体识别的现代深度学习架构能很好地泛化到新视图，但其机制尚未被充分理解。本文中，我们刻画了常见深度学习架构泛化到新视图的能力。我们将此形式化为一个监督分类任务，其中标签对应唯一的3D物体，样本对应物体在不同3D姿态下的2D视图。我们考虑了三种常见的新视图泛化模型：(i) 完全3D泛化，(ii) 纯2D匹配，以及(iii) 基于视图线性组合的匹配。我们发现深度模型能很好地泛化到新视图，但其方式与所有现有模型不同。外推到训练集中视图范围之外的视图的能力有限，而外推到新旋转轴的能力更有限，这表明网络既未推断出完整的3D结构，也未使用线性插值。然而，其泛化能力远优于纯2D匹配。这些发现有助于设计实现3D泛化所需的2D视图数据集。重现我们实验的代码已公开：https://github.com/shoaibahmed/investigating_3d_generalization.git