Assessing the validity of user simulators when used for the evaluation of information retrieval systems remains an open question, constraining their effective use and the reliability of simulation-based results. To address this issue, we conduct a comprehensive literature review with a particular focus on methods for the validation of simulated user queries with regard to real queries. Based on the review, we develop a taxonomy that structures the current landscape of available measures. We empirically corroborate the taxonomy by analyzing the relationships between the different measures applied to four different datasets representing diverse search scenarios. Finally, we provide concrete recommendations on which measures or combinations of measures should be considered when validating user simulation in different contexts. Furthermore, we release a dedicated library with the most commonly used measures to facilitate future research.

翻译：评估用户模拟器在信息检索系统评估中的有效性仍是一个悬而未决的问题，这限制了其有效使用以及基于模拟的结果的可靠性。为解决这一问题，我们进行了全面的文献综述，特别关注模拟用户查询相对于真实查询的验证方法。基于此综述，我们构建了一个分类体系，用以梳理当前可用度量指标的全貌。我们通过分析应用于四个代表不同搜索场景的数据集时各度量指标间的关系，从经验上验证了该分类体系。最后，我们针对不同情境下验证用户模拟时应考虑哪些度量指标或其组合，提供了具体建议。此外，我们发布了一个包含最常用度量指标的专用库，以促进未来研究。