Emerging events, such as the COVID pandemic and the Ukraine Crisis, require a time-sensitive comprehensive understanding of the situation to allow for appropriate decision-making and effective action response. Automated generation of situation reports can significantly reduce the time, effort, and cost for domain experts when preparing their official human-curated reports. However, AI research toward this goal has been very limited, and no successful trials have yet been conducted to automate such report generation. We propose SmartBook, a novel task formulation targeting situation report generation, which consumes large volumes of news data to produce a structured situation report with multiple hypotheses (claims) summarized and grounded with rich links to factual evidence. We realize SmartBook for the Ukraine-Russia crisis by automatically generating intelligence analysis reports to assist expert analysts. The machine-generated reports are structured in the form of timelines, with each timeline organized by major events (or chapters), corresponding strategic questions (or sections) and their grounded summaries (or section content). Our proposed framework automatically detects real-time event-related strategic questions, which are more directed than manually-crafted analyst questions, which tend to be too complex, hard to parse, vague and high-level. Results from thorough qualitative evaluations show that roughly 82% of the questions in Smartbook have strategic importance, with at least 93% of the sections in the report being tactically useful. Further, experiments show that expert analysts tend to add more information into the SmartBook reports, with only 2.3% of the existing tokens being deleted, meaning SmartBook can serve as a useful foundation for analysts to build upon when creating intelligence reports.
翻译:突发性事件,如新冠疫情和乌克兰危机,需要时效性强的全面态势理解,以便做出适当决策并采取有效应对行动。自动生成态势报告可显著减少领域专家在准备其官方人工编撰报告时的时间、精力和成本。然而,针对这一目标的AI研究非常有限,目前尚未有成功实现此类报告自动生成的试验案例。我们提出智能书(SmartBook),这是一种面向态势报告生成的新型任务框架,能够利用海量新闻数据生成结构化的态势报告,其中包含多个假设(主张)的总结,并通过丰富的链接将事实证据作为依据。我们针对乌克兰-俄罗斯危机实现了智能书,通过自动生成情报分析报告来辅助专家分析师。机器生成的报告以时间线的形式组织,每条时间线按重大事件(或章节)编排,对应战略问题(或小节)及其基于证据的总结(或小节内容)。我们提出的框架能自动检测实时事件相关的战略问题,这些问题比手动设计的分析师问题更具针对性——后者往往过于复杂、难以解析、模糊且层次过高。全面定性评估结果表明,智能书中约82%的问题具有战略重要性,报告中至少93%的小节具有战术实用性。此外,实验显示专家分析师倾向于在智能书报告中补充更多信息,仅删除了2.3%的现有词汇,这表明智能书可作为分析师编写情报报告时的有效基础框架。