Noise-contrastive estimation (NCE) is a statistically consistent method for learning unnormalized probabilistic models. It has been empirically observed that the choice of the noise distribution is crucial for NCE's performance. However, such observations have never been made formal or quantitative. In fact, it is not even clear whether the difficulties arising from a poorly chosen noise distribution are statistical or algorithmic in nature. In this work, we formally pinpoint reasons for NCE's poor performance when an inappropriate noise distribution is used. Namely, we prove these challenges arise due to an ill-behaved (more precisely, flat) loss landscape. To address this, we introduce a variant of NCE called "eNCE" which uses an exponential loss and for which normalized gradient descent addresses the landscape issues provably when the target and noise distributions are in a given exponential family.

翻译：在统计学上,噪声-扰动估计(NCE)是一种一致的方法,用于学习不规范的概率模型。从经验上看,选择噪音分布对于NCE的表现至关重要。然而,这种观察从未正式或量化。事实上,甚至还不清楚选择不当的噪音分布引起的困难是否属于统计或算法性质。在这项工作中,我们正式确定在使用不适当的噪音分布时NCE表现不佳的原因。也就是说,我们证明这些挑战是由于行为不良(更确切地说,是平坦的)损失环境造成的。为了解决这个问题,我们引入了称为“ENCE”的NCE变种,即使用指数损失,在目标与噪音分布在一个特定指数式家庭时,平坦度梯度梯度下降能够解决地貌问题。