Convolution is the most expensive operation among neural network operations, thus its performance is critical to the overall performance of neural networks. Commonly used convolution approaches, including general matrix multiplication (GEMM)-based convolution and direct convolution, rely on im2col for data transformation or do not use data transformation at all, respectively. However, the im2col data transformation can lead to at least 2$\times$ memory footprint compared to not using data transformation at all, thus limiting the size of neural network models running on memory-limited systems. Meanwhile, not using data transformation usually performs poorly due to nonconsecutive memory access although it consumes less memory. To solve those problems, we propose a new memory-efficient data transformation algorithm, called im2win. This algorithm refactorizes a row of square or rectangle dot product windows of the input image and flattens unique elements within these windows into a row in the output tensor, which enables consecutive memory access and data reuse, and thus greatly reduces the memory overhead. Furthermore, we propose a high-performance im2win-based convolution algorithm with various optimizations, including vectorization, loop reordering, etc. Our experimental results show that our algorithm reduces the memory overhead by average to 41.6% compared to the PyTorch's convolution implementation based on im2col, and achieves average to 3.6$\times$ and 5.3$\times$ speedup in performance compared to the im2col-based convolution and not using data transformation, respectively.

翻译：卷积是神经网络操作中计算代价最高的操作，因此其性能对神经网络的整体性能至关重要。常用的卷积方法，包括基于通用矩阵乘法（GEMM）的卷积和直接卷积，分别依赖im2col进行数据变换或不使用任何数据变换。然而，与完全不使用数据变换相比，im2col数据变换可能导致至少2倍的内存占用，从而限制了在内存受限系统上运行的神经网络模型的规模。同时，不使用数据变换虽然消耗更少内存，但由于非连续内存访问，通常性能较差。为解决这些问题，我们提出了一种新的内存高效数据变换算法，称为im2win。该算法将输入图像的方形或矩形点积窗口的每一行进行重构，并将这些窗口内的唯一元素展平到输出张量的一行中，从而实现连续内存访问和数据重用，大幅降低内存开销。此外，我们提出了一种基于im2win的高性能卷积算法，并进行了包括向量化、循环重排等多种优化。实验结果表明，与PyTorch基于im2col的卷积实现相比，我们的算法平均降低了41.6%的内存开销，并且在性能上相比基于im2col的卷积和不使用数据变换分别实现了平均3.6倍和5.3倍的加速。