In the rapidly evolving landscape of deep learning, the quest for models that balance expressivity with computational efficiency has never been more critical. This paper introduces Orchid, a novel architecture that reimagines sequence modeling by incorporating a new data-dependent convolution mechanism. Orchid is designed to address the inherent limitations of traditional attention mechanisms, particularly their quadratic complexity, without compromising the ability to capture long-range dependencies and in-context learning. At the core of Orchid lies the data-dependent convolution layer, which dynamically adjusts its kernel conditioned on input data using a dedicated conditioning neural network. We design two simple conditioning networks that maintain shift equivariance in the adaptive convolution operation. The dynamic nature of data-dependent convolution kernel, coupled with gating operations, grants Orchid high expressivity while maintaining efficiency and quasilinear scalability for long sequences. We rigorously evaluate Orchid across multiple domains, including language modeling and image classification, to showcase its performance and generality. Our experiments demonstrate that Orchid architecture not only outperforms traditional attention-based architectures such as BERT and Vision Transformers with smaller model sizes, but also extends the feasible sequence length beyond the limitations of the dense attention layers. This achievement represents a significant step towards more efficient and scalable deep learning models for sequence modeling.

翻译：摘要：在深度学习快速发展的背景下，对兼顾表达力与计算效率的模型的追求从未如此关键。本文提出Orchid，一种通过融入新型数据相关卷积机制来重新构想序列建模的架构。Orchid旨在解决传统注意力机制的内在局限性，特别是其二次复杂度，同时不损害捕捉长程依赖和上下文学习的能力。Orchid的核心是数据相关卷积层，该层通过专用的条件神经网络根据输入数据动态调整其卷积核。我们设计了两种简单的条件网络，以在自适应卷积操作中保持平移等变性。数据相关卷积核的动态特性，结合门控操作，使得Orchid在维持高效性和长序列准线性可扩展性的同时，拥有高表达力。我们在包括语言建模和图像分类在内的多个领域对Orchid进行了严格评估，以展示其性能与通用性。实验表明，Orchid架构不仅以更小的模型尺寸超越了传统基于注意力的架构（如BERT和Vision Transformer），还将可行序列长度扩展到了密集注意力层的限制之外。这一成果代表了迈向更高效、可扩展的序列建模深度学习模型的重要一步。