Formal model evaluation methods typically certify that a model satisfies a prescribed target key performance indicator (KPI) level. However, in many applications, the relevant target KPI level may not be known a priori, and the user may instead wish to compare candidate models by analyzing the full trade-offs between performance and reliability achievable at test time by the models. This task, requiring the reliable estimate of the test-time KPI distributions, is made more complicated by the fact that the same data must often be used both to pre-select a subset of candidate models and to estimate their KPI distributions, causing a potential post-selection bias. In this work, we introduce post-selection distributional model evaluation (PS-DME), a general framework for statistically valid distributional model assessment after arbitrary data-dependent model pre-selection. Building on e-values, PS-DME controls post-selection false coverage rate (FCR) for the distributional KPI estimates and is proved to be more sample efficient than a baseline method based on sample splitting. Experiments on synthetic data, text-to-SQL decoding with large language models, and telecom network performance evaluation demonstrate that PS-DME enables reliable comparison of candidate configurations across a range of reliability levels, supporting the statistically reliable exploration of performance--reliability trade-offs.

翻译：正式的模型评估方法通常需验证模型满足预设的关键性能指标（KPI）水平。然而，在许多应用中，目标KPI水平可能事先未知，用户更希望分析模型在测试时实现的性能与可靠性之间的完整权衡，从而比较候选模型。该任务需要可靠估计测试时的KPI分布，但由于同一数据通常需同时用于预选候选模型子集并估计其KPI分布，这可能导致后验选择偏差，使问题复杂化。本文提出后验选择分布模型评估（PS-DME），一个通用的统计有效框架，用于在任意数据依赖模型预选后进行分布型模型评估。基于e值构建，PS-DME能控制分布型KPI估计的后验选择错误覆盖率（FCR），并证明其比基于样本分割的基线方法更具样本效率。在合成数据、大语言模型文本到SQL解码及电信网络性能评估上的实验表明，PS-DME能在不同可靠性水平下实现候选配置的可靠比较，为性能-可靠性权衡的统计可信探索提供支持。