This article describes an efficient method to learn distributed representations, also known as embeddings. This is accomplished minimizing an objective function similar to the one introduced in the Word2Vec algorithm and later adopted in several works. The optimization computational bottleneck is the calculation of the softmax normalization constants for which a number of operations scaling quadratically with the sample size is required. This complexity is unsuited for large datasets and negative sampling is a popular workaround, allowing one to obtain distributed representations in linear time with respect to the sample size. Negative sampling consists, however, in a change of the loss function and hence solves a different optimization problem from the one originally proposed. Our contribution is to show that the sotfmax normalization constants can be estimated in linear time, allowing us to design an efficient optimization strategy to learn distributed representations. We test our approximation on two popular applications related to word and node embeddings. The results evidence competing performance in terms of accuracy with respect to negative sampling with a remarkably lower computational time.

翻译：本文提出了一种学习分布式表示（即嵌入表示）的高效方法。该方法通过最小化与Word2Vec算法中引入并被后续多项研究采用的目标函数来实现。优化过程中的计算瓶颈在于softmax归一化常数的计算，其运算复杂度与样本量呈二次方关系。这种复杂度不适合大规模数据集，而负采样作为一种流行的规避方案，能够在与样本量呈线性关系的时间内获得分布式表示。然而，负采样本质上改变了损失函数，因此求解的是与原始目标不同的优化问题。我们的贡献在于证明softmax归一化常数可以在线性时间内估计，从而设计出高效的学习分布式表示的优化策略。我们在词嵌入和节点嵌入两个热门应用上验证了该近似方法的性能。实验结果表明，在保持与负采样相当准确性的同时，该方法显著降低了计算时间。