Block-sparse attention is promising for accelerating long-context LLM pre-filling, yet identifying relevant blocks efficiently remains a bottleneck. Existing methods typically employ coarse-grained attention as a proxy for block importance estimation, but often resort to expensive token-level searching or scoring, resulting in significant selection overhead. In this work, we trace the inaccuracy of standard coarse-grained attention via mean pooling to a theoretical root cause: the interaction between mean pooling and Rotary Positional Embeddings (RoPE). We prove that mean pooling acts as a low-pass filter that induces destructive interference in high-frequency dimensions, effectively creating a "blind spot" for local positional information (e.g., slash patterns). To address this, we introduce Prism, a training-free spectral-aware approach that decomposes block selection into high-frequency and low-frequency branches. By applying energy-based temperature calibration, Prism restores the attenuated positional signals directly from pooled representations, enabling block importance estimation using purely block-level operations, thereby improving efficiency. Extensive evaluations confirm that Prism maintains accuracy parity with full attention while delivering up to $\mathbf{5.1\times}$ speedup.

翻译：块稀疏注意力在加速长上下文LLM预填充方面前景广阔，但高效识别相关块仍是瓶颈。现有方法通常采用粗粒度注意力作为块重要性估计的代理，但往往依赖昂贵的词元级搜索或评分，导致显著的选择开销。本工作中，我们通过均值池化追溯标准粗粒度注意力不准确性的理论根源：均值池化与旋转位置编码（RoPE）之间的相互作用。我们证明均值池化作为低通滤波器，会在高频维度引发相消干涉，从而对局部位置信息（如斜线模式）形成“盲区”。为解决此问题，我们提出棱镜（Prism），一种无需训练的频谱感知方法，将块选择分解为高频与低频分支。通过基于能量的温度校准，棱镜直接从池化表示中恢复衰减的位置信号，实现纯块级操作的块重要性估计，从而提升效率。大量实验验证表明，棱镜在保持与全注意力精度相当的同时，实现了高达$\mathbf{5.1\times}$的加速比。