Distributed representations of words encode lexical semantic information, but what type of information is encoded, and how? Focusing on the skip-gram with negative-sampling method, we found that the squared norm of static word embedding encodes the information gain conveyed by the word; the information gain is defined by the Kullback-Leibler divergence of the co-occurrence distribution of the word to the unigram distribution of the corpus. Our findings are explained by the theoretical framework of the exponential family of probability distributions and confirmed through precise experiments that remove spurious correlations arising from word frequency. We demonstrate that both the KL divergence and the squared norm of embedding provide a useful metric of a word's informativeness in tasks such as keyword extraction, part-of-speech discrimination, and hypernym classification.

翻译：分布式词表示编码了词汇语义信息，但编码的信息类型及其机制尚待阐明。聚焦于负采样跳元模型，我们发现静态词嵌入的平方范数编码了词汇所传达的信息增益；该信息增益由词汇共现分布与语料库单字分布之间的库尔贝克-莱布勒散度定义。这一发现可通过指数族概率分布理论框架进行解释，并通过消除词频引发的伪相关性的精确实验得到验证。我们证明，KL散度与词嵌入平方范数可作为衡量词汇信息性的有效指标，应用于关键词提取、词性判别及上位词分类等任务中。