Large language models with long context windows can answer complex questions directly from full-length academic, technical, and policy documents, but passing entire documents is often costly, slow, and can degrade answer quality while increasing the risk of unnecessary data leakage. This paper targets the common setting of answering many heterogeneous questions over long document(s), where fixed position heuristics and standard retrieval-augmented generation (RAG) can fail due to document structure variability and weak query-chunk semantic similarity, which often requires task- and domain-specific tuning of embedding retrievers. We propose {Selective Attention-Guided Extraction} (\ourmethod), a training-free, plug-and-play context reduction framework that uses a lightweight local LLM to perform a single prefilling pass and convert language model attention signals into a query-specific relevance heatmap at configurable granularities. \ourmethod\ further introduces \emph{differential attention} strategies to better isolate question-relevant evidence, then selects the top-scoring units under a user-defined token budget and forwards only this reduced context to a downstream LLM for answer generation. \ourmethod\ surpasses traditional reduction techniques across multiple long-document QA benchmarks, notably securing a top-4 rank on QuALITY-hard while constrained to a 10\% context budget. This enables a 90\% reduction in tokens with competitive accuracy, without the need for model fine-tuning or complex calibration.

翻译：长上下文窗口的大语言模型能够直接从完整的学术、技术及政策文档中回答复杂问题，但传递完整文档往往成本高昂、速度缓慢，且可能降低回答质量并增加不必要的数据泄露风险。本文针对在长篇文档上回答大量异质化问题的常见场景——在该场景中，由于文档结构多样性与查询-文本块语义相似性不足，固定位置启发式方法和标准检索增强生成（RAG）常会失效，且通常需要对嵌入检索器进行任务特定和领域特定调优。我们提出选择性注意力引导抽取方法（Selective Attention-Guided Extraction，简称SAGE），这是一种免训练、即插即用的上下文缩减框架，通过轻量级本地大语言模型执行单次预填充，将语言模型注意力信号转化为可配置粒度下的查询相关热力图。SAGE进一步引入差分注意力策略以更好隔离问题相关证据，在用户定义的标记预算下选择得分最高的单元，仅向下游大语言模型传递缩减后的上下文以生成答案。在多个长文档问答基准测试中，SAGE超越传统缩减技术，尤其在QuALITY-hard数据集上以10%上下文预算取得前四排名，在保持竞争性准确率的同时实现90%的标记缩减，且无需模型微调或复杂校准。