Even though dropout is a popular regularization technique, its theoretical properties are not fully understood. In this paper we study dropout regularization in extended generalized linear models based on double exponential families, for which the dispersion parameter can vary with the features. A theoretical analysis shows that dropout regularization prefers rare but important features in both the mean and dispersion, generalizing an earlier result for conventional generalized linear models. Training is performed using stochastic gradient descent with adaptive learning rate. To illustrate, we apply dropout to adaptive smoothing with B-splines, where both the mean and dispersion parameters are modelled flexibly. The important B-spline basis functions can be thought of as rare features, and we confirm in experiments that dropout is an effective form of regularization for mean and dispersion parameters that improves on a penalized maximum likelihood approach with an explicit smoothness penalty.

翻译：尽管Dropout是一种流行的正则化技术，但其理论性质尚未被完全理解。本文研究基于双指数族的扩展广义线性模型中的Dropout正则化，其中离散参数可随特征变化。理论分析表明，Dropout正则化在均值和离散度中均偏好罕见但重要的特征，这推广了传统广义线性模型的早期结论。训练采用自适应学习率的随机梯度下降方法。为说明起见，我们将Dropout应用于B样条的自适应平滑，其中均值和离散度参数均被灵活建模。重要的B样条基函数可视为罕见特征，实验证实Dropout是对均值和离散度参数的有效正则化形式，其性能优于带有显式平滑惩罚的惩罚最大似然方法。