Large Language Models (LLMs) have demonstrated remarkable capabilities in handling long context inputs, but this comes at the cost of increased computational resources and latency. Our research introduces a novel approach for the long context bottleneck to accelerate LLM inference and reduce GPU memory consumption. Our research demonstrates that LLMs can identify relevant tokens in the early layers before generating answers to a query. Leveraging this insight, we propose an algorithm that uses early layers of an LLM as filters to select and compress input tokens, significantly reducing the context length for subsequent processing. Our method, GemFilter, demonstrates substantial improvements in both speed and memory efficiency compared to existing techniques, such as standard attention and SnapKV/H2O. Notably, it achieves a 2.4$\times$ speedup and 30\% reduction in GPU memory usage compared to SOTA methods. Evaluation on the Needle in a Haystack task shows that GemFilter significantly outperforms standard attention, SnapKV and demonstrates comparable performance on the LongBench challenge. GemFilter is simple, training-free, and broadly applicable across different LLMs. Crucially, it provides interpretability by allowing humans to inspect the selected input sequence. These findings not only offer practical benefits for LLM deployment, but also enhance our understanding of LLM internal mechanisms, paving the way for further optimizations in LLM design and inference. Our code is available at \url{https://github.com/SalesforceAIResearch/GemFilter}.

翻译：大语言模型在处理长上下文输入方面展现出卓越能力，但这是以增加计算资源和延迟为代价的。本研究针对长上下文瓶颈提出了一种新颖方法，以加速大语言模型推理并降低GPU内存消耗。我们的研究表明，大语言模型在生成查询答案前能够在早期层中识别相关令牌。基于这一发现，我们提出一种算法，利用大语言模型的早期层作为过滤器来选择和压缩输入令牌，从而显著减少后续处理的上下文长度。我们的方法GemFilter相较于现有技术（如标准注意力机制和SnapKV/H2O）在速度和内存效率方面均展现出显著提升。值得注意的是，与最先进方法相比，它实现了2.4倍的加速和30%的GPU内存使用降低。在“大海捞针”任务上的评估表明，GemFilter显著优于标准注意力机制和SnapKV，并在LongBench挑战中展现出相当的性能。GemFilter具有简单性、无需训练的特点，并广泛适用于不同的大语言模型。关键的是，它通过允许人类检查所选输入序列来提供可解释性。这些发现不仅为大语言模型部署提供了实际效益，还增强了对大语言模型内部机制的理解，为进一步优化大语言模型设计和推理铺平了道路。我们的代码发布于\url{https://github.com/SalesforceAIResearch/GemFilter}。