Formal model evaluation methods typically certify that a model satisfies a prescribed target key performance indicator (KPI) level. However, in many applications, the relevant target KPI level may not be known a priori, and the user may instead wish to compare candidate models by analyzing the full trade-offs between performance and reliability achievable at test time by the models. This task, requiring the reliable estimate of the test-time KPI distributions, is made more complicated by the fact that the same data must often be used both to pre-select a subset of candidate models and to estimate their KPI distributions, causing a potential post-selection bias. In this work, we introduce post-selection distributional model evaluation (PS-DME), a general framework for statistically valid distributional model assessment after arbitrary data-dependent model pre-selection. Building on e-values, PS-DME controls post-selection false coverage rate (FCR) for the distributional KPI estimates and is proved to be more sample efficient than a baseline method based on sample splitting. Experiments on synthetic data, text-to-SQL decoding with large language models, and telecom network performance evaluation demonstrate that PS-DME enables reliable comparison of candidate configurations across a range of reliability levels, supporting the statistically reliable exploration of performance--reliability trade-offs.

翻译：正式模型评估方法通常验证模型是否满足预设的关键性能指标（KPI）水平。然而，在许多应用中，相关的目标KPI水平可能预先未知，用户可能更希望通过分析模型在测试时性能与可靠性之间的完整权衡来比较候选模型。这一任务需要可靠估计测试时的KPI分布，但常因需用同一数据集既预选候选模型子集又估计其KPI分布而变得复杂，从而引发潜在的后选择偏差。本文提出后选择分布性模型评估（PS-DME），一个在任意数据依赖的模型预选择后实现统计有效的分布性模型评估通用框架。基于e值，PS-DME控制分布性KPI估计的后选择错误覆盖率（FCR），并被证明比基于样本分割的基线方法具有更高的样本效率。在合成数据、大语言模型的文本到SQL解码以及电信网络性能评估上的实验表明，PS-DME能够在不同可靠性水平下可靠比较候选配置，支持对性能-可靠性权衡的统计可靠探索。