In subgroup analysis, testing the existence of a subgroup with a differential treatment effect serves as protection against spurious subgroup discovery. Despite its importance, this hypothesis testing possesses a complicated nature: parameter characterizing subgroup classification is not identified under the null hypothesis of no subgroup. Due to this irregularity, the existing methods have the following two limitations. First, the asymptotic null distribution of test statistics often takes an intractable form, which necessitates computationally demanding resampling methods to calculate the critical value. Second, the dimension of personal attributes characterizing subgroup membership is not allowed to be of high dimension. To solve these two problems simultaneously, this study develops a novel shrinkage likelihood ratio test for the existence of a subgroup using a logistic-normal mixture model. The proposed test statistics are built on a modified likelihood function that shrinks possibly high-dimensional unidentified parameters toward zero under the null hypothesis while retaining power under the alternative. This shrinkage helps handle the irregularity and restore the simple chi-square-type asymptotics even under the high-dimensional regime.

翻译：在子群分析中，检验是否存在具有差异化治疗效应的子群，是防止虚假子群发现的重要保护机制。尽管其意义重大，但该假设检验存在一个复杂特性：在无子群的原假设下，刻画子群分类的参数无法被识别。由于这一非正则性，现有方法存在以下两个局限：首先，检验统计量的渐近零分布往往形式复杂，需要借助计算量庞大的重抽样方法来确定临界值；其次，不允许刻画子群成员属性的个人特征维度过高。为同时解决这两个问题，本研究基于逻辑正态混合模型，提出了一种新颖的收缩似然比检验方法用于检验子群的存在性。所提出的检验统计量基于修正的似然函数构建，该函数在原假设下将可能高维的未识别参数向零收缩，同时保持备择假设下的检验功效。这种收缩有助于处理非正则性，并能在高维场景下恢复简单的卡方型渐近分布特性。