Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of natural language processing tasks. These capabilities stem primarily from the self-attention mechanism, which enables modeling of long-range dependencies. However, the quadratic complexity of self-attention with respect to sequence length poses significant computational and memory challenges, especially as sequence length extends to extremes. While various sparse attention and KV cache compression methods have been proposed to improve efficiency, they often suffer from limitations such as reliance on fixed patterns, inability to handle both prefilling and decoding stages, or the requirement for additional training. In this paper, we propose Training-free Context-adaptive Attention (TCA-Attention), a training-free sparse attention mechanism that selectively attends to only the informative tokens for efficient long-context inference. Our method consists of two lightweight phases: i) an offline calibration phase that determines head-specific sparsity budgets via a single forward pass, and ii) an online token selection phase that adaptively retains core context tokens using a lightweight redundancy metric. TCA-Attention provides a unified solution that accelerates both prefilling and decoding while reducing KV cache memory footprint, without requiring parameter updates or architectural changes. Theoretical analysis shows that our approach maintains bounded approximation error. Extensive experiments demonstrate that TCA-Attention achieves a 2.8$\times$ speedup and reduces KV cache by 61% at 128K context length while maintaining performance comparable to full attention across various benchmarks, offering a practical plug-and-play solution for efficient long-context inference.

翻译：大语言模型（LLM）已在广泛的自然语言处理任务中展现出卓越能力。这些能力主要源于自注意力机制，该机制能够建模长程依赖关系。然而，自注意力相对于序列长度的二次复杂度带来了显著的计算与内存挑战，尤其在序列长度达到极长规模时。尽管已有多种稀疏注意力与KV缓存压缩方法被提出以提升效率，但这些方法常存在依赖固定模式、无法同时处理预填充与解码阶段、或需要额外训练等局限。本文提出无需训练的情境自适应注意力机制（TCA-Attention），这是一种无需训练的稀疏注意力机制，通过选择性关注信息量高的令牌来实现高效的长上下文推理。我们的方法包含两个轻量级阶段：i）离线校准阶段：通过单次前向传播确定各注意力头特定的稀疏度预算；ii）在线令牌选择阶段：使用轻量级冗余度量自适应保留核心上下文令牌。TCA-Attention 提供了统一解决方案，可同时加速预填充与解码过程，并降低KV缓存内存占用，且无需参数更新或架构修改。理论分析表明该方法保持有界近似误差。大量实验证明，在128K上下文长度下，TCA-Attention 实现了2.8倍的加速，并将KV缓存减少61%，同时在各类基准测试中保持与完整注意力相当的性能，为高效长上下文推理提供了即插即用的实用解决方案。