Quantifying bias in retrieval functions through document retrievability scores is vital for assessing recall-oriented retrieval systems. However, many studies investigating retrieval model bias lack validation of their query generation methods as accurate representations of retrievability for real users and their queries. This limitation results from the absence of established criteria for query generation in retrievability assessments. Typically, researchers resort to using frequent collocations from document corpora when no query log is available. In this study, we address the issue of reproducibility and seek to validate query generation methods by comparing retrievability scores generated from artificially generated queries to those derived from query logs. Our findings demonstrate a minimal or negligible correlation between retrievability scores from artificial queries and those from query logs. This suggests that artificially generated queries may not accurately reflect retrievability scores as derived from query logs. We further explore alternative query generation techniques, uncovering a variation that exhibits the highest correlation. This alternative approach holds promise for improving reproducibility when query logs are unavailable.

翻译：通过文档可检索性分数量化检索函数中的偏差，对于评估面向召回的检索系统至关重要。然而，许多研究在探究检索模型偏差时，未能验证其查询生成方法能否准确反映真实用户及其查询的可检索性。这一局限源于可检索性评估中缺乏既定的查询生成标准。通常，在无查询日志可用时，研究者会采用文档语料库中的高频搭配词。本研究旨在解决可重复性问题，并通过对比人工生成查询与查询日志导出的可检索性分数，验证查询生成方法。我们的研究发现，人工查询与查询日志的可检索性分数之间相关性极低甚至可忽略不计，表明人工生成的查询可能无法准确反映基于查询日志的可检索性分数。我们进一步探索了其他查询生成技术，发现一种具有最高相关性的变体方法。当无法获取查询日志时，这一替代方法有望提高研究的可重复性。