Recently proposed systems for open-domain question answering (OpenQA) require large amounts of training data to achieve state-of-the-art performance. However, data annotation is known to be time-consuming and therefore expensive to acquire. As a result, the appropriate datasets are available only for a handful of languages (mainly English and Chinese). In this work, we introduce and publicly release PolQA, the first Polish dataset for OpenQA. It consists of 7,000 questions, 87,525 manually labeled evidence passages, and a corpus of over 7,097,322 candidate passages. Each question is classified according to its formulation, type, as well as entity type of the answer. This resource allows us to evaluate the impact of different annotation choices on the performance of the QA system and propose an efficient annotation strategy that increases the passage retrieval accuracy@10 by 10.55 p.p. while reducing the annotation cost by 82%.

翻译：近期提出的开放域问答系统需要大量训练数据才能达到最优性能，但数据标注耗时且成本高昂。因此，当前适用的问答数据集仅覆盖少数语言（主要为英语和中文）。本研究提出并公开发布PolQA——首个面向开放域问答的波兰语数据集。该数据集包含7,000个问题、87,525条人工标注的证据段落，以及超过7,097,322条候选段落的语料库。每个问题均根据其表述形式、类型及答案实体类别进行分类标注。本资源使我们能够评估不同标注策略对问答系统性能的影响，并提出一种高效标注方法，该方法可将段落检索准确率（@10）提升10.55个百分点，同时降低82%的标注成本。