It is widely accepted that so-called facts can be checked by searching for information on the Internet. This process requires a fact-checker to formulate a search query based on the fact and to present it to a search engine. Then, relevant and believable passages need to be identified in the search results before a decision is made. This process is carried out by sub-editors at many news and media organisations on a daily basis. Here, we ask the question as to whether it is possible to automate the first step, that of query generation. Can we automatically formulate search queries based on factual statements which are similar to those formulated by human experts? Here, we consider similarity both in terms of textual similarity and with respect to relevant documents being returned by a search engine. First, we introduce a moderate-sized evidence collection dataset which includes 390 factual statements together with associated human-generated search queries and search results. Then, we investigate generating queries using a number of rule-based and automatic text generation methods based on pre-trained large language models (LLMs). We show that these methods have different merits and propose a hybrid approach which has superior performance in practice.
翻译:人们普遍认为,所谓的“事实”可以通过在互联网上搜索信息来验证。这一过程要求事实核查员基于待查事实制定搜索查询,并将其提交给搜索引擎。随后,需要在搜索结果中识别相关且可信的段落,才能做出判断。许多新闻和媒体机构的副编辑每天都会执行这一流程。在此,我们探讨一个问题:是否可能自动化第一步——查询生成?我们能否基于事实陈述自动生成与人类专家所制定的查询相似的搜索查询?在此,我们从文本相似性以及搜索引擎返回相关文档的角度来定义相似性。首先,我们引入一个中等规模的证据收集数据集,包含390条事实陈述及其对应的人工生成的搜索查询和搜索结果。接着,我们研究了基于预训练大语言模型(LLMs)的多种规则驱动和自动文本生成方法。结果表明,这些方法各有优劣,我们提出了一种在实践中性能更优的混合方法。