Regularized optimal transport (OT) has received much attention in recent years starting from Cuturi's paper with Kullback-Leibler (KL) divergence regularized OT. In this paper, we propose to regularize the OT problem using the family of $\alpha$-R\'enyi divergences for $\alpha \in (0, 1)$. R\'enyi divergences are neither $f$-divergences nor Bregman distances, but they recover the KL divergence in the limit $\alpha \nearrow 1$. The advantage of introducing the additional parameter $\alpha$ is that for $\alpha \searrow 0$ we obtain convergence to the unregularized OT problem. For the KL regularized OT problem, this was achieved by letting the regularization parameter tend to zero, which causes numerical instabilities. We present two different ways to obtain premetrics on probability measures, namely by R\'enyi divergence constraints and penalization. The latter premetric interpolates between the unregularized and KL regularized OT problem with weak convergence of the minimizer, generalizing the interpolating property of KL regularized OT. We use a nested mirror descent algorithm for solving the primal formulation. Both on real and synthetic data sets R\'enyi regularized OT plans outperform their KL and Tsallis counterparts in terms of being closer to the unregularized transport plans and recovering the ground truth in inference tasks better.

翻译：正则化最优传输（OT）自Cuturi提出库尔贝克-莱布勒（KL）散度正则化OT的论文以来，近年来受到了广泛关注。本文提出使用$\alpha \in (0, 1)$的$\alpha$-雷尼散度族来正则化OT问题。雷尼散度既非$f$-散度也非布雷格曼距离，但在极限$\alpha \nearrow 1$时恢复为KL散度。引入额外参数$\alpha$的优势在于，当$\alpha \searrow 0$时，我们能够得到无正则化OT问题的收敛。对于KL正则化OT问题，此前是通过令正则化参数趋于零来实现的，但这会引发数值不稳定性。我们提出了两种在概率测度上获得预度量的方法，即通过雷尼散度约束和惩罚项。后者得到的预度量在弱收敛意义下插值了无正则化与KL正则化OT问题，推广了KL正则化OT的插值性质。我们采用嵌套镜像下降算法求解原始形式。在真实数据集和合成数据集上，雷尼正则化OT方案在更接近无正则化传输方案以及推断任务中更准确恢复真实情况方面，均优于KL和Tsallis正则化方案。