Maximum likelihood estimation in logistic regression with mixed effects is known to often result in estimates on the boundary of the parameter space. Such estimates, which include infinite values for fixed effects and singular or infinite variance components, can cause havoc to numerical estimation procedures and inference. We introduce an appropriately scaled additive penalty to the log-likelihood function, or an approximation thereof, which penalizes the fixed effects by the Jeffreys' invariant prior for the model with no random effects and the variance components by a composition of negative Huber loss functions. The resulting maximum penalized likelihood estimates are shown to lie in the interior of the parameter space. Appropriate scaling of the penalty guarantees that the penalization is soft enough to preserve the optimal asymptotic properties expected by the maximum likelihood estimator, namely consistency, asymptotic normality, and Cram\'er-Rao efficiency. Our choice of penalties and scaling factor preserves equivariance of the fixed effects estimates under linear transformation of the model parameters, such as contrasts. Maximum softly-penalized likelihood is compared to competing approaches on two real-data examples, and through comprehensive simulation studies that illustrate its superior finite sample performance.

翻译：混合效应逻辑回归中的最大似然估计常导致参数空间边界上的估计值。此类估计（包括固定效应的无穷值以及奇异或无穷方差分量）会对数值估计过程和统计推断造成严重干扰。我们提出在似然函数或其近似函数中添加适当缩放的加性惩罚项：针对固定效应采用无随机效应模型下的Jeffreys不变先验进行惩罚，针对方差分量则采用负Huber损失函数的复合形式进行惩罚。经证明，所得最大惩罚似然估计位于参数空间内部。适当的惩罚缩放能确保惩罚力度足够温和，从而保留最大似然估计期望的最优渐近性质——即相合性、渐近正态性和Cramér-Rao有效性。我们选择的惩罚函数与缩放因子还可确保固定效应估计在模型参数线性变换（如对比变换）下的等变性。通过两个真实数据案例及综合模拟研究，将最大软惩罚似然估计与竞争方法进行比较，结果表明该方法在小样本场景下具有更优性能。