Query-focused summarization (QFS) aims to provide a summary of a single document/multi documents that can satisfy the information needs of a given query. It is useful for various real-world applications, such as abstractive snippet generation or more recent retrieval augmented generation (RAG). A prototypical QFS pipeline consists of a retriever (sparse or dense retrieval) and a generator (usually a large language model). However, applying large language models (LLM) potentially leads to hallucinations, especially when the evidence contradicts the prior belief of LLMs. There has been growing interest in developing new decoding methods to improve generation quality and reduce hallucination. In this work, we conduct a large-scale reproducibility study on one recently proposed decoding method -- Context-aware Decoding (CAD). In addition to replicating CAD's experiments on news summarization datasets, we include experiments on QFS datasets, and conduct more rigorous analysis on computational complexity and hyperparameter sensitivity. Experiments with eight different language models show that performance-wise, CAD improves QFS quality by (1) reducing factuality errors/hallucinations while (2) mostly retaining the match of lexical patterns, measured by ROUGE scores, while also at a cost of increased inference-time FLOPs and reduced decoding speed. The code implementation based on Huggingface Library is made available https://github.com/zhichaoxu-shufe/context-aware-decoding-qfs

翻译：查询聚焦摘要旨在提供满足给定查询信息需求的单文档或多文档摘要。该技术广泛应用于实际场景，如抽象式片段生成或近期兴起的检索增强生成。典型的查询聚焦摘要流程包含检索器（稀疏或稠密检索）与生成器（通常为大语言模型）。然而，大语言模型的应用可能导致幻觉现象，尤其在证据与模型先验信念相矛盾时。近年来，学界日益关注新型解码方法的开发以提升生成质量并减少幻觉。本研究对近期提出的解码方法——上下文感知解码——进行了大规模可重复性研究。除在新闻摘要数据集上复现上下文感知解码实验外，我们还在查询聚焦摘要数据集上开展实验，并对计算复杂度与超参数敏感性进行了更严谨的分析。基于八种不同语言模型的实验表明：在性能方面，上下文感知解码通过（1）减少事实性错误/幻觉同时（2）基本保留ROUGE分数衡量的词汇模式匹配，提升了查询聚焦摘要质量，但代价是推理时浮点运算次数增加及解码速度降低。基于Huggingface库的代码实现已开放于https://github.com/zhichaoxu-shufe/context-aware-decoding-qfs