Recent works indicate that convolutional neural networks (CNN) need large receptive fields (RF) to compete with visual transformers and their attention mechanism. In CNNs, RFs can simply be enlarged by increasing the convolution kernel sizes. Yet the number of trainable parameters, which scales quadratically with the kernel's size in the 2D case, rapidly becomes prohibitive, and the training is notoriously difficult. This paper presents a new method to increase the RF size without increasing the number of parameters. The dilated convolution (DC) has already been proposed for the same purpose. DC can be seen as a convolution with a kernel that contains only a few non-zero elements placed on a regular grid. Here we present a new version of the DC in which the spacings between the non-zero elements, or equivalently their positions, are no longer fixed but learnable via backpropagation thanks to an interpolation technique. We call this method "Dilated Convolution with Learnable Spacings" (DCLS) and generalize it to the n-dimensional convolution case. However, our main focus here will be on the 2D case. We first tried our approach on ResNet50: we drop-in replaced the standard convolutions with DCLS ones, which increased the accuracy of ImageNet1k classification at iso-parameters, but at the expense of the throughput. Next, we used the recent ConvNeXt state-of-the-art convolutional architecture and drop-in replaced the depthwise convolutions with DCLS ones. This not only increased the accuracy of ImageNet1k classification but also of typical downstream and robustness tasks, again at iso-parameters but this time with negligible cost on throughput, as ConvNeXt uses separable convolutions. Conversely, classic DC led to poor performance with both ResNet50 and ConvNeXt. The code of the method is available at: https://github.com/K-H-Ismail/Dilated-Convolution-with-Learnable-Spacings-PyTorch.

翻译：近期研究表明，卷积神经网络（CNN）需要较大的感受野（RF）才能与视觉Transformer及其注意力机制相匹敌。在CNN中，通过增大卷积核尺寸即可简单扩大感受野。然而，二维情况下可训练参数数量随卷积核尺寸平方增长，导致参数规模迅速变得不可接受，且训练难度众所周知地大。本文提出一种在不增加参数数量的前提下扩大感受野的新方法。空洞卷积（DC）此前已被提出用于实现相同目标，其本质是卷积核中仅包含按规则网格分布的少量非零元素。本文提出一种新型空洞卷积，其中非零元素之间的间隔（即其位置）不再固定，而是通过插值技术实现可反向传播学习。我们将该方法命名为"可学习间隔的空洞卷积"（DCLS），并将其推广至n维卷积场景。但本文主要聚焦于二维情况。我们首先在ResNet50上测试该方法：将标准卷积替换为DCLS后，在等参数量条件下提升了ImageNet1k分类精度，但以牺牲吞吐量为代价。接着，我们采用最新的ConvNeXt卷积架构，将深度可分离卷积替换为DCLS版本。这不仅在等参数量条件下提升了ImageNet1k分类精度，还显著改善了典型下游任务与鲁棒性任务的效果，且由于ConvNeXt采用可分离卷积，吞吐量代价可忽略不计。相反，经典DC在ResNet50和ConvNeXt上的表现均不理想。该方法代码已开源：https://github.com/K-H-Ismail/Dilated-Convolution-with-Learnable-Spacings-PyTorch