Mining large corpora can generate useful discoveries but is time-consuming for humans. We formulate a new task, D5, that automatically discovers differences between two large corpora in a goal-driven way. The task input is a problem comprising a research goal "$\textit{comparing the side effects of drug A and drug B}$" and a corpus pair (two large collections of patients' self-reported reactions after taking each drug). The output is a language description (discovery) of how these corpora differ (patients taking drug A "$\textit{mention feelings of paranoia}$" more often). We build a D5 system, and to quantitatively measure its performance, we 1) contribute a meta-dataset, OpenD5, aggregating 675 open-ended problems ranging across business, social sciences, humanities, machine learning, and health, and 2) propose a set of unified evaluation metrics: validity, relevance, novelty, and significance. With the dataset and the unified metrics, we confirm that language models can use the goals to propose more relevant, novel, and significant candidate discoveries. Finally, our system produces discoveries previously unknown to the authors on a wide range of applications in OpenD5, including temporal and demographic differences in discussion topics, political stances and stereotypes in speech, insights in commercial reviews, and error patterns in NLP models.

翻译：挖掘大规模语料库可生成有价值的发现，但对人类而言耗时巨大。本文提出新任务D5，以目标驱动方式自动发现两个大规模语料库间的差异。任务输入包含研究目标（如"比较药物A与药物B的副作用"）与语料对（两组患者服用不同药物后的自我报告反应集合），输出则为描述这些语料差异的语言表述（如服用药物A的患者"更频繁提及偏执感"）。我们构建了D5系统，并通过以下方式定量评估其性能：1）构建元数据集OpenD5，聚合675个涵盖商业、社会科学、人文学科、机器学习及健康领域的开放式问题；2）提出统一评价指标集：有效性、相关性、新颖性与重要性。借助该数据集与统一指标，我们证实语言模型能基于目标提出更相关、新颖且重要的候选发现。最终，我们的系统在OpenD5的广泛应用中发现了作者此前未知的结论，包括讨论主题的时间与人口统计差异、演讲中的政治立场与刻板印象、商业评论中的洞见，以及NLP模型中的错误模式。