User simulators are increasingly central to interactive information retrieval, yet the community lacks standardized evaluation tools. Simulators serve two objectives, behavioral realism (matching real user behavior) and tester reliability (producing valid system rankings), and these are often conflated despite being distinct and sometimes conflicting. We present SimEval-IR, an open-source toolkit and benchmark suite that makes this distinction measurable. SimEval-IR provides: (1) a canonical session schema unifying session search and conversational interactions, with validated dataset adapters and explicit loss accounting; (2) three executable benchmarks covering behavioral realism, tester reliability with RATE-style estimation, and an analysis linking the two; and (3) baseline results across four real datasets in two languages and four simulator families. Our key finding: the classifier-discriminator ''human-likeness'' check, the dominant realism test in the literature, has essentially no pooled predictive power for system-ranking validity ($r{=}{+}0.09$, $n{=}48$), while marginal click-depth distance and Fréchet distance over session embeddings give a much stronger signal ($|r|{=}0.43$ and $0.40$, $p{\leq}0.005$). SimEval-IR is released with all configurations and scripts to reproduce the reported analysis.

翻译：用户模拟器在交互式信息检索中日益核心，但该领域缺乏标准化的评估工具。模拟器服务于两个目标：行为真实性（匹配真实用户行为）和测试可靠性（生成有效的系统排名），尽管这两个目标截然不同且有时相互冲突，却常被混淆。我们提出SimEval-IR——一个开源工具包和基准测试集，使这一区别可量化。SimEval-IR提供：(1) 规范化的会话模式，统一会话式搜索与对话交互，附带经过验证的数据集适配器和显式损失核算；(2) 三个可执行的基准测试，涵盖行为真实性、基于RATE式估计的测试可靠性，以及连接两者的分析；(3) 跨四个真实数据集（两种语言）和四个模拟器系列的基线结果。我们的关键发现：文献中主要的真实性测试——分类器判别器“类人性”检验，对系统排名有效性几乎没有聚合预测能力（$r{=}{+}0.09$，$n{=}48$），而边际点击深度距离和会话嵌入上的弗雷歇距离则提供更强信号（$|r|{=}0.43$和$0.40$，$p{\leq}0.005$）。SimEval-IR随附所有配置与脚本发布，可复现所述分析。