In the past few years, channel-wise and spatial-wise attention blocks have been widely adopted as supplementary modules in deep neural networks, enhancing network representational abilities while introducing low complexity. Most attention modules follow a squeeze-and-excitation paradigm. However, to design such attention modules, requires a substantial amount of experiments and computational resources. Neural Architecture Search (NAS), meanwhile, is able to automate the design of neural networks and spares the numerous experiments required for an optimal architecture. This motivates us to design a search architecture that can automatically find near-optimal attention modules through NAS. We propose SASE, a Searching Architecture for Squeeze and Excitation operations, to form a plug-and-play attention block by searching within certain search space. The search space is separated into 4 different sets, each corresponds to the squeeze or excitation operation along the channel or spatial dimension. Additionally, the search sets include not only existing attention blocks but also other operations that have not been utilized in attention mechanisms before. To the best of our knowledge, SASE is the first attempt to subdivide the attention search space and search for architectures beyond currently known attention modules. The searched attention module is tested with extensive experiments across a range of visual tasks. Experimental results indicate that visual backbone networks (ResNet-50/101) using the SASE attention module achieved the best performance compared to those using the current state-of-the-art attention modules. Codes are included in the supplementary material, and they will be made public later.

翻译：过去几年中，通道注意力和空间注意力模块作为深度神经网络的补充模块被广泛采用，在引入较低复杂度的同时增强了网络的表征能力。大多数注意力模块遵循挤压-激励范式。然而，设计此类注意力模块需要进行大量实验并消耗大量计算资源。与此同时，神经架构搜索能够自动化神经网络的设计过程，省去了为获得最优架构所需的大量实验。这促使我们设计一种能够通过神经架构搜索自动找到近似最优注意力模块的搜索架构。我们提出SASE——一种面向挤压与激励操作的搜索架构，通过在特定搜索空间内进行搜索来构建即插即用的注意力模块。该搜索空间被划分为四个不同的集合，分别对应通道维度或空间维度上的挤压或激励操作。此外，搜索集合不仅包含现有注意力模块，还包括此前未在注意力机制中使用的其他操作。据我们所知，SASE是首次尝试细分注意力搜索空间并探索超越当前已知注意力模块的架构。通过在一系列视觉任务上的大量实验对所搜索的注意力模块进行了测试。实验结果表明，使用SASE注意力模块的视觉骨干网络在性能上优于采用当前最先进注意力模块的同类网络。代码已包含在补充材料中，并将于后续公开。