How should an agent's performance in a multiagent environment be evaluated when there is a limited sample size or a high cost of running a trial? The AIVAT family of variance reduction techniques was proposed to address this challenge by introducing unbiased low-variance estimators of agents' expected payoffs. An important component of AIVAT is a heuristic value function that discriminates between potentially low- and high-value counterfactual histories. A notable gap in the literature is that there is little to no constraint or guideline on how the heuristic value function should be chosen or how uncertainty in its output should be handled. In our first contribution, we parameterize the heuristic value function to highlight AIVAT's potential vulnerabilities: a) the sample variance can be set pathologically low by directly applying gradient descent on the sample variance, and b) one can p-hack to draw a desired statistical conclusion via gradient descent/ascent on the test statistic. The main takeaway is that the heuristic value function should be fixed prior to observing the evaluation data! In our second contribution, we show how the heuristic uncertainty can be propagated to quantify the uncertainty of AIVAT estimates. It is then possible to further reduce the variance using inverse-variance weighted averaging, but AIVAT's unbiasedness guarantee may have to be sacrificed. In our experiments, we use a dataset of 10,000 poker hands to demonstrate our heuristic pathology and uncertainty results, with the latter yielding a 43.0% reduction in the number of samples (poker hands) needed to draw statistical conclusions.

翻译：在多智能体环境中，当样本量有限或试验成本高昂时，应如何评估智能体的表现？AIVAT方差缩减技术家族通过引入无偏低方差估计量来评估智能体的期望收益，从而应对这一挑战。AIVAT的一个关键组件是启发式价值函数，它能区分潜在低价值与高价值的反事实历史。文献中的一个显著空白在于，关于如何选择启发式价值函数或如何处理其输出的不确定性，几乎没有约束或指导。在本文的第一项贡献中，我们对启发式价值函数进行参数化，以揭示AIVAT的潜在脆弱性：a) 通过对样本方差直接应用梯度下降，可病态性地降低样本方差；b) 通过对检验统计量进行梯度下降/上升，可进行p-hacking以获得期望的统计结论。主要启示是：启发式价值函数应在观察评估数据之前固定下来！在第二项贡献中，我们展示了如何传播启发式不确定性，以量化AIVAT估计的不确定性。随后可通过逆方差加权平均进一步降低方差，但可能需要牺牲AIVAT的无偏性保证。在实验中，我们使用10,000手扑克牌的数据集，验证了启发式病态与不确定性结果，后者将得出统计结论所需的样本（扑克牌手数）减少了43.0%。