When resources are scarce, an allocation policy is needed to decide who receives a resource. This problem occurs, for instance, when allocating scarce medical resources and is often solved using modern ML methods. This paper introduces methods to evaluate index-based allocation policies -- that allocate a fixed number of resources to those who need them the most -- by using data from a randomized control trial. Such policies create dependencies between agents, which render the assumptions behind standard statistical tests invalid and limit the effectiveness of estimators. Addressing these challenges, we translate and extend recent ideas from the statistics literature to present an efficient estimator and methods for computing asymptotically correct confidence intervals. This enables us to effectively draw valid statistical conclusions, a critical gap in previous work. Our extensive experiments validate our methodology in practical settings, while also showcasing its statistical power. We conclude by proposing and empirically verifying extensions of our methodology that enable us to reevaluate a past randomized control trial to evaluate different ML allocation policies in the context of a mHealth program, drawing previously invisible conclusions.

翻译：当资源稀缺时，需要一种分配策略来决定谁获得资源。例如，在分配稀缺医疗资源时会出现此问题，通常通过现代机器学习方法解决。本文介绍了利用随机对照试验数据评估基于指数的分配策略——即向最需要资源者分配固定数量资源的方法。此类策略在个体间产生依赖关系，导致标准统计检验的假设失效并限制了估计量的有效性。针对这些挑战，我们转化并扩展统计学文献中的最新思想，提出了一个高效估计量以及计算渐近正确置信区间的方法。这使我们能够有效得出有效的统计结论，弥补了先前研究中的关键空白。我们的广泛实验验证了该方法在实际场景中的有效性，同时展示了其统计效力。最后，我们提出并通过实验验证了方法的扩展，使我们能够重新评估一项过去的随机对照试验，以评估移动健康项目中不同机器学习分配策略，得出了此前未被发现的结论。