Suppose that one can construct a valid $(1-\delta)$-confidence interval (CI) for each of $K$ parameters of potential interest. If a data analyst uses an arbitrary data-dependent criterion to select some subset $S$ of parameters, then the aforementioned CIs for the selected parameters are no longer valid due to selection bias. We design a new method to adjust the intervals in order to control the false coverage rate (FCR). The main established method is the "BY procedure" by Benjamini and Yekutieli (JASA, 2005). The BY guarantees require certain restrictions on the selection criterion and on the dependence between the CIs. We propose a new simple method which, in contrast, is valid under any dependence structure between the original CIs, and any (unknown) selection criterion, but which only applies to a special, yet broad, class of CIs that we call e-CIs. To elaborate, our procedure simply reports $(1-\delta|S|/K)$-CIs for the selected parameters, and we prove that it controls the FCR at $\delta$ for confidence intervals that implicitly invert e-values; examples include those constructed via supermartingale methods, via universal inference, or via Chernoff-style bounds, among others. The e-BY procedure is admissible, and recovers the BY procedure as a special case via a particular calibrator. Our work also has implications for post-selection inference in sequential settings, since it applies at stopping times, to continuously-monitored confidence sequences, and under bandit sampling. We demonstrate the efficacy of our procedure using numerical simulations and real A/B testing data from Twitter.
翻译:假设我们可以为$K$个潜在感兴趣参数中的每一个构造有效的$(1-\delta)$置信区间。如果数据分析师使用任意依赖于数据的标准选择某个参数子集$S$,则由于选择偏差,所选参数的上述置信区间不再有效。我们设计了一种新方法来调整这些区间,以控制错误覆盖率(FCR)。目前已建立的主要方法是Benjamini和Yekutieli(JASA,2005)提出的“BY程序”。BY保证要求对选择标准以及置信区间之间的依赖性施加某些限制。我们提出了一种新的简单方法,与之相反,该方法对原始置信区间之间的任意依赖结构以及任意(未知)选择标准均有效,但仅适用于一类特殊且广泛的置信区间,我们称之为e-CIs。具体来说,我们的程序只需报告所选参数的$(1-\delta|S|/K)$置信区间,我们证明该程序对于隐式反转e值的置信区间(例如通过超鞅方法、通用推理或切诺夫界等构造的置信区间)在$\delta$水平上控制了FCR。e-BY程序是可采纳的,并通过特定校准器将BY程序作为特例恢复。我们的工作对序贯设置中的选择后推断也有意义,因为它适用于停止时间、连续监测的置信序列以及赌博机采样。我们通过数值模拟和来自Twitter的真实A/B测试数据证明了我们程序的有效性。