Designing an effective channel attention mechanism implores one to find a lossy-compression method allowing for optimal feature representation. Despite recent progress in the area, it remains an open problem. FcaNet, the current state-of-the-art channel attention mechanism, attempted to find such an information-rich compression using Discrete Cosine Transforms (DCTs). One drawback of FcaNet is that there is no natural choice of the DCT frequencies. To circumvent this issue, FcaNet experimented on ImageNet to find optimal frequencies. We hypothesize that the choice of frequency plays only a supporting role and the primary driving force for the effectiveness of their attention filters is the orthogonality of the DCT kernels. To test this hypothesis, we construct an attention mechanism using randomly initialized orthogonal filters. Integrating this mechanism into ResNet, we create OrthoNet. We compare OrthoNet to FcaNet (and other attention mechanisms) on Birds, MS-COCO, and Places356 and show superior performance. On the ImageNet dataset, our method competes with or surpasses the current state-of-the-art. Our results imply that an optimal choice of filter is elusive and generalization can be achieved with a sufficiently large number of orthogonal filters. We further investigate other general principles for implementing channel attention, such as its position in the network and channel groupings.

翻译：设计有效的通道注意力机制需要找到一种有损压缩方法，以实现最优特征表示。尽管该领域近期取得了进展，这仍是一个未解难题。目前最先进的通道注意力机制FcaNet试图利用离散余弦变换（DCT）寻找此类信息丰富的压缩方法。FcaNet的一个缺陷在于缺乏自然选择DCT频率的依据。为解决此问题，FcaNet在ImageNet上通过实验寻找最优频率。我们假设频率选择仅起辅助作用，而注意力滤波器有效性的主要驱动力来自DCT核的正交性。为验证此假设，我们利用随机初始化的正交滤波器构建注意力机制，并将其集成到ResNet中创建OrthoNet。我们在Birds、MS-COCO和Places356数据集上将OrthoNet与FcaNet（及其他注意力机制）进行对比，展示了更优性能。在ImageNet数据集上，我们的方法可与当前最先进技术竞争甚至超越。结果表明，最优滤波器的选择难以实现，而通过足够数量的正交滤波器即可获得泛化能力。我们进一步探究了实现通道注意力的其他通用原则，如其在网络中的位置及通道分组方式。