Summarizing book-length documents (>100K tokens) that exceed the context window size of large language models (LLMs) requires first breaking the input document into smaller chunks and then prompting an LLM to merge, update, and compress chunk-level summaries. Despite the complexity and importance of this task, it has yet to be meaningfully studied due to the challenges of evaluation: existing book-length summarization datasets (e.g., BookSum) are in the pretraining data of most public LLMs, and existing evaluation methods struggle to capture errors made by modern LLM summarizers. In this paper, we present the first study of the coherence of LLM-based book-length summarizers implemented via two prompting workflows: (1) hierarchically merging chunk-level summaries, and (2) incrementally updating a running summary. We obtain 1193 fine-grained human annotations on GPT-4 generated summaries of 100 recently-published books and identify eight common types of coherence errors made by LLMs. Because human evaluation is expensive and time-consuming, we develop an automatic metric, BooookScore, that measures the proportion of sentences in a summary that do not contain any of the identified error types. BooookScore has high agreement with human annotations and allows us to systematically evaluate the impact of many other critical parameters (e.g., chunk size, base LLM) while saving $15K and 500 hours in human evaluation costs. We find that closed-source LLMs such as GPT-4 and Claude 2 produce summaries with higher BooookScore than the oft-repetitive ones generated by LLaMA 2. Incremental updating yields lower BooookScore but higher level of detail than hierarchical merging, a trade-off sometimes preferred by human annotators. We release code and annotations after blind review to spur more principled research on book-length summarization.

翻译：摘要： 总结超过大语言模型（LLM）上下文窗口大小的书籍长度文档（>10万词元）需要先将输入文档分割成较小的块，然后提示LLM合并、更新和压缩块级摘要。尽管这项任务复杂且重要，但由于评估挑战，尚未得到有意义的研究：现有书籍长度摘要数据集（例如，BookSum）包含在大多数公共LLM的预训练数据中，且现有评估方法难以捕捉现代LLM摘要器生成的错误。在本文中，我们首次研究了通过两种提示工作流实现的基于LLM的书籍长度摘要的连贯性：（1）分层合并块级摘要，以及（2）增量更新运行摘要。我们对GPT-4生成的100本近期出版书籍的摘要获取了1193份细粒度人工标注，并识别出LLM生成的八种常见连贯性错误类型。由于人工评估既昂贵又耗时，我们开发了一种自动指标——书名评分（BooookScore），用于衡量摘要中不包含任何已识别错误类型的句子比例。书名评分与人工标注高度一致，使我们能够系统评估许多其他关键参数（例如，块大小、基础LLM）的影响，同时节省了1.5万美元和500小时的人工评估成本。我们发现，像GPT-4和Claude 2这样的闭源LLM生成的摘要具有比LLaMA 2生成的常常重复的摘要更高的书名评分。与分层合并相比，增量更新产生较低的书名评分但更详细，这种权衡有时被人工标注者偏好。我们在盲审后发布代码和标注，以促进对书籍长度摘要更理论化的研究。