In this paper, we study the inductive biases in convolutional neural networks (CNNs), which are believed to be vital drivers behind CNNs' exceptional performance on vision-like tasks. We first analyze the universality of CNNs, i.e., the ability to approximate continuous functions. We prove that a depth of $\mathcal{O}(\log d)$ is sufficient for achieving universality, where $d$ is the input dimension. This is a significant improvement over existing results that required a depth of $\Omega(d)$. We also prove that learning sparse functions with CNNs needs only $\tilde{\mathcal{O}}(\log^2d)$ samples, indicating that deep CNNs can efficiently capture long-range sparse correlations. Note that all these are achieved through a novel combination of increased network depth and the utilization of multichanneling and downsampling. Lastly, we study the inductive biases of weight sharing and locality through the lens of symmetry. To separate two biases, we introduce locally-connected networks (LCNs), which can be viewed as CNNs without weight sharing. Specifically, we compare the performance of CNNs, LCNs, and fully-connected networks (FCNs) on a simple regression task. We prove that LCNs require ${\Omega}(d)$ samples while CNNs need only $\tilde{\mathcal{O}}(\log^2d)$ samples, which highlights the cruciality of weight sharing. We also prove that FCNs require $\Omega(d^2)$ samples while LCNs need only $\tilde{\mathcal{O}}(d)$ samples, demonstrating the importance of locality. These provable separations quantify the difference between the two biases, and our major observation behind is that weight sharing and locality break different symmetries in the learning process.

翻译：本文研究了卷积神经网络（CNN）中的归纳偏置，这一特性被认为是CNN在视觉类任务上表现卓越的关键驱动力。我们首先分析了CNN的普适性，即逼近连续函数的能力。我们证明深度为$\mathcal{O}(\log d)$即可实现普适性，其中$d$为输入维度。这一结果相较于现有需深度为$\Omega(d)$的研究成果取得了显著改进。我们还证明，使用CNN学习稀疏函数仅需$\tilde{\mathcal{O}}(\log^2d)$个样本，表明深度CNN能高效捕获长程稀疏相关性。值得注意的是，这些成果均通过增加网络深度、利用多通道及降采样的新型组合方式实现。最后，我们从对称性视角研究了权值共享与局部性的归纳偏置。为分离这两种偏置，我们引入了局部连接网络（LCN），该结构可视为不含权值共享的CNN。具体而言，我们在简单回归任务上比较了CNN、LCN与全连接网络（FCN）的性能。我们证明LCN需要${\Omega}(d)$个样本，而CNN仅需$\tilde{\mathcal{O}}(\log^2d)$个样本，这凸显了权值共享的关键作用。同时证明FCN需要$\Omega(d^2)$个样本而LCN仅需$\tilde{\mathcal{O}}(d)$个样本，展示了局部性的重要性。这些可证明的分离量化了两种偏置的差异，其核心发现在于：权值共享与局部性打破了学习过程中不同的对称性。