The quadratic complexity of attention mechanisms poses a critical bottleneck for large language models processing long contexts. While dynamic sparse attention methods offer input-adaptive efficiency, they face fundamental trade-offs: requiring preprocessing, lacking global evaluation, violating query independence, or incurring high computational overhead. We present RRAttention, a novel dynamic sparse attention method that simultaneously achieves all desirable properties through a head \underline{r}ound-\underline{r}obin (RR) sampling strategy. By rotating query sampling positions across attention heads within each stride, RRAttention maintains query independence while enabling efficient global pattern discovery with stride-level aggregation. Our method reduces complexity from $O(L^2)$ to $O(L^2/S^2)$ and employs adaptive Top-$τ$ selection for optimal sparsity. Extensive experiments on natural language understanding (HELMET) and multimodal video comprehension (Video-MME) demonstrate that RRAttention recovers over 99\% of full attention performance while computing only half of the attention blocks, achieving 2.4$\times$ speedup at 128K context length and outperforming existing dynamic sparse attention methods.

翻译：注意力机制的二次复杂度是大型语言模型处理长上下文的关键瓶颈。尽管动态稀疏注意力方法提供了输入自适应的效率，但它们面临根本性的权衡：需要预处理、缺乏全局评估、违反查询独立性或产生高计算开销。我们提出了RRAttention，一种新颖的动态稀疏注意力方法，它通过一种注意力头循环采样策略，同时实现了所有理想特性。该方法在每个步幅内轮转不同注意力头的查询采样位置，从而在保持查询独立性的同时，通过步幅级聚合实现高效的全局模式发现。我们的方法将复杂度从$O(L^2)$降低到$O(L^2/S^2)$，并采用自适应Top-$τ$选择以实现最优稀疏性。在自然语言理解（HELMET）和多模态视频理解（Video-MME）上的大量实验表明，RRAttention仅计算一半的注意力块即可恢复超过99%的全注意力性能，在128K上下文长度下实现了2.4倍的加速，并优于现有的动态稀疏注意力方法。