Formal model evaluation methods typically certify that a model satisfies a prescribed target key performance indicator (KPI) level. However, in many applications, the relevant target KPI level may not be known a priori, and the user may instead wish to compare candidate models by analyzing the full trade-offs between performance and reliability achievable at test time by the models. This task, requiring the reliable estimate of the test-time KPI distributions, is made more complicated by the fact that the same data must often be used both to pre-select a subset of candidate models and to estimate their KPI distributions, causing a potential post-selection bias. In this work, we introduce post-selection distributional model evaluation (PS-DME), a general framework for statistically valid distributional model assessment after arbitrary data-dependent model pre-selection. Building on e-values, PS-DME controls post-selection false coverage rate (FCR) for the distributional KPI estimates and we establish explicit conditions under which it is provably more sample efficient than a baseline method based on sample splitting. Experiments on synthetic data, text-to-SQL decoding with large language models, and telecom network performance evaluation demonstrate that PS-DME enables reliable comparison of candidate configurations across a range of reliability levels, supporting the statistically reliable exploration of performance--reliability trade-offs.

翻译：正式的模型评估方法通常确保模型满足预设的目标关键性能指标（KPI）水平。然而，在许多应用中，相关的目标KPI水平可能事先未知，用户可能更希望通过分析模型在测试时能达到的性能与可靠性之间的完整权衡来比较候选模型。这一任务需要可靠地估计测试时KPI分布，但由于同一数据往往既要用于预选候选模型子集，又要用于估计其KPI分布，可能导致潜在的后选择偏差，因此变得更加复杂。在本工作中，我们提出了后选择分布模型评估（PS-DME），一个用于在任意依赖数据的模型预选后进行统计有效的分布模型评估的通用框架。基于e-values，PS-DME控制分布KPI估计的后选择错误覆盖率（FCR），我们建立了显式条件，证明其在样本效率上优于基于样本分割的基线方法。在合成数据、基于大语言模型的文本到SQL解码以及电信网络性能评估上的实验表明，PS-DME能够在不同可靠性水平上实现候选配置的可靠比较，支持对性能-可靠性权衡进行统计可靠的探索。