BEIR is a benchmark dataset for zero-shot evaluation of information retrieval models across 18 different domain/task combinations. In recent years, we have witnessed the growing popularity of a representation learning approach to building retrieval models, typically using pretrained transformers in a supervised setting. This naturally begs the question: How effective are these models when presented with queries and documents that differ from the training data? Examples include searching in different domains (e.g., medical or legal text) and with different types of queries (e.g., keywords vs. well-formed questions). While BEIR was designed to answer these questions, our work addresses two shortcomings that prevent the benchmark from achieving its full potential: First, the sophistication of modern neural methods and the complexity of current software infrastructure create barriers to entry for newcomers. To this end, we provide reproducible reference implementations that cover the two main classes of approaches: learned dense and sparse models. Second, there does not exist a single authoritative nexus for reporting the effectiveness of different models on BEIR, which has led to difficulty in comparing different methods. To remedy this, we present an official self-service BEIR leaderboard that provides fair and consistent comparisons of retrieval models. By addressing both shortcomings, our work facilitates future explorations in a range of interesting research questions that BEIR enables.

翻译：BEIR是一个用于零样本评估信息检索模型的基准数据集，涵盖18个不同的领域/任务组合。近年来，我们见证了基于表示学习的检索模型构建方法日益普及，通常采用预训练Transformer在监督环境下进行。这自然引出一个问题：当面对与训练数据不同的查询和文档时，这些模型的效果如何？示例包括跨域搜索（如医学或法律文本）以及使用不同类型的查询（如关键词与结构良好的问题）。尽管BEIR旨在回答这些问题，但我们的工作针对阻碍该基准充分发挥潜力的两个缺陷：首先，现代神经方法的复杂性和当前软件基础设施的复杂性给新入门者设置了障碍。为此，我们提供了涵盖两类主要方法（学习型密集模型与稀疏模型）的可复现参考实现。其次，目前缺乏一个统一的权威平台来报告不同模型在BEIR上的效果，这导致不同方法之间的比较存在困难。为解决这一问题，我们推出了一个官方的自助式BEIR排行榜，为检索模型提供公平且一致的对比。通过弥补这两个缺陷，我们的工作为BEIR所支持的广泛有趣研究问题中的未来探索提供了便利。