In scientific research, the ability to effectively retrieve relevant documents based on complex, multifaceted queries is critical. Existing evaluation datasets for this task are limited, primarily due to the high cost and effort required to annotate resources that effectively represent complex queries. To address this, we propose a novel task, Scientific DOcument Retrieval using Multi-level Aspect-based quEries (DORIS-MAE), which is designed to handle the complex nature of user queries in scientific research. We developed a benchmark dataset within the field of computer science, consisting of 100 human-authored complex query cases. For each complex query, we assembled a collection of 100 relevant documents and produced annotated relevance scores for ranking them. Recognizing the significant labor of expert annotation, we also introduce Anno-GPT, a scalable framework for validating the performance of Large Language Models (LLMs) on expert-level dataset annotation tasks. LLM annotation of the DORIS-MAE dataset resulted in a 500x reduction in cost, without compromising quality. Furthermore, due to the multi-tiered structure of these complex queries, the DORIS-MAE dataset can be extended to over 4,000 sub-query test cases without requiring additional annotation. We evaluated 17 recent retrieval methods on DORIS-MAE, observing notable performance drops compared to traditional datasets. This highlights the need for better approaches to handle complex, multifaceted queries in scientific research. Our dataset and codebase are available at https://github.com/Real-Doris-Mae/Doris-Mae-Dataset.

翻译：在科学研究中，基于复杂、多层面的查询高效检索相关文档的能力至关重要。现有用于该任务的评估数据集十分有限，主要原因是能够有效表征复杂查询的资源需要高昂的标注成本与人力投入。为此，我们提出了一项新任务——基于多层次方面查询的科学文档检索（DORIS-MAE），旨在处理科学研究中用户查询的复杂特性。我们在计算机科学领域构建了一个基准数据集，包含100个人工编写的复杂查询案例。针对每个复杂查询，我们收集了100篇相关文档，并生成了用于排序的标注相关性分数。考虑到专家标注的高昂人力成本，我们还引入了Anno-GPT这一可扩展框架，用于验证大型语言模型（LLM）在专家级数据集标注任务中的性能。利用LLM对DORIS-MAE数据集进行标注，使成本降低了500倍，且未牺牲标注质量。此外，由于这些复杂查询的多层结构，DORIS-MAE数据集可扩展至超过4000个子查询测试案例，无需额外标注。我们在DORIS-MAE上评估了17种最新检索方法，发现与传统数据集相比，这些方法的表现出现显著下降。这凸显了在处理科学研究中复杂、多方面查询时，亟需更优方法。我们的数据集与代码库已开源，地址为https://github.com/Real-Doris-Mae/Doris-Mae-Dataset。