We consider the fundamental problem of estimating a discrete distribution on a domain of size $K$ with high probability in Kullback-Leibler divergence. We provide upper and lower bounds on the minimax estimation rate, which show that the optimal rate is between $\big(K + \ln(K)\ln(1/δ)\big) /n$ and $\big(K\ln\ln(K) + \ln(K)\ln(1/δ)\big) /n$ at error probability $δ$ and sample size $n$, which pins down the rate up to the doubly logarithmic factor $\ln \ln K$ that multiplies $K$. Our upper bound uses techniques from online learning to construct a novel estimator via online-to-batch conversion. Perhaps surprisingly, the tail behavior of the minimax rate is worse than for the squared total variation and squared Hellinger distance, for which it is $\big(K + \ln(1/δ)\big) /n$, i.e. without the $\ln K$ multiplying $\ln (1/δ)$. As a consequence, we cannot obtain a fully tight lower bound from the usual reduction to these smaller distances. Moreover, we show that this lower bound cannot be achieved by the standard lower bound approach based on a reduction to hypothesis testing, and instead we need to introduce a new reduction to what we call weak hypothesis testing. We investigate the source of the gap with other divergences further in refined results, which show that the total variation rate is achievable for Kullback-Leibler divergence after all (in fact by he maximum likelihood estimator) if we rule out outcome probabilities smaller than $O(\ln(K/δ) / n)$, which is a vanishing set as $n$ increases for fixed $K$ and $δ$. This explains why minimax Kullback-Leibler estimation is more difficult than asymptotic estimation.

翻译：我们考虑在Kullback-Leibler散度意义下以高概率估计定义域大小为$K$的离散分布这一基本问题。我们给出了极小极大估计速率的上界与下界，结果表明在误差概率为$δ$、样本量为$n$时，最优速率介于$\big(K + \ln(K)\ln(1/δ)\big) /n$与$\big(K\ln\ln(K) + \ln(K)\ln(1/δ)\big) /n$之间，从而将速率确定至乘以$K$的双对数因子$\ln \ln K$的精度。我们的上界采用在线学习技术，通过在线到批处理的转换构建了一种新颖的估计器。可能令人惊讶的是，该极小极大速率的尾部行为比平方总变差和平方Hellinger距离更差——后两者的速率为$\big(K + \ln(1/δ)\big) /n$，即不包含$\ln K$与$\ln (1/δ)$的乘积项。因此，我们无法通过常规的到这些较小距离的归约得到完全紧致的下界。此外，我们证明基于假设检验归约的标准下界方法无法达到该下界，为此需要引入一种新的归约方法，即我们提出的弱假设检验归约。我们通过更精细的结果进一步探究了与其他散度产生差异的根源：结果表明，若排除小于$O(\ln(K/δ) / n)$的结果概率（当$K$和$δ$固定而$n$增大时，这类概率构成一个趋于零的集合），则总变差速率对于Kullback-Leibler散度同样可达（实际上可通过极大似然估计实现）。这解释了为何极小极大Kullback-Leibler估计比渐近估计更为困难。