Models such as Word2Vec and GloVe construct word embeddings based on the co-occurrence probability $P(i,j)$ of words $i$ and $j$ in text corpora. The resulting vectors $W_i$ not only group semantically similar words but also exhibit a striking linear analogy structure -- for example, $W_{\text{king}} - W_{\text{man}} + W_{\text{woman}} \approx W_{\text{queen}}$ -- whose theoretical origin remains unclear. Previous observations indicate that this analogy structure: (i) already emerges in the top eigenvectors of the matrix $M(i,j) = P(i,j)/P(i)P(j)$, (ii) strengthens and then saturates as more eigenvectors of $M (i, j)$, which controls the dimension of the embeddings, are included, (iii) is enhanced when using $\log M(i,j)$ rather than $M(i,j)$, and (iv) persists even when all word pairs involved in a specific analogy relation (e.g., king-queen, man-woman) are removed from the corpus. To explain these phenomena, we introduce a theoretical generative model in which words are defined by binary semantic attributes, and co-occurrence probabilities are derived from attribute-based interactions. This model analytically reproduces the emergence of linear analogy structure and naturally accounts for properties (i)-(iv). It can be viewed as giving fine-grained resolution into the role of each additional embedding dimension. It is robust to various forms of noise and agrees well with co-occurrence statistics measured on Wikipedia and the analogy benchmark introduced by Mikolov et al.

翻译：诸如 Word2Vec 和 GloVe 等模型基于文本语料库中词语 $i$ 和 $j$ 的共现概率 $P(i,j)$ 来构建词嵌入。所得的向量 $W_i$ 不仅将语义相似的词语聚集在一起，还展现出一种引人注目的线性类比结构——例如，$W_{\text{king}} - W_{\text{man}} + W_{\text{woman}} \approx W_{\text{queen}}$——其理论起源尚不明确。先前的观察表明，这种类比结构：(i) 已在矩阵 $M(i,j) = P(i,j)/P(i)P(j)$ 的顶部特征向量中涌现；(ii) 随着控制嵌入维度的 $M(i,j)$ 的更多特征向量被纳入，该结构先增强后趋于饱和；(iii) 当使用 $\log M(i,j)$ 而非 $M(i,j)$ 时得到增强；(iv) 即使将特定类比关系（例如，国王-女王，男人-女人）涉及的所有词对从语料库中移除，该结构依然存在。为解释这些现象，我们引入了一个理论生成模型，其中词语由二元语义属性定义，共现概率则源于基于属性的交互作用。该模型解析地再现了线性类比结构的涌现，并自然地解释了性质 (i)-(iv)。它可以被视为对每个新增嵌入维度作用的精细解析。该模型对多种形式的噪声具有鲁棒性，并且与在维基百科上测量的共现统计数据以及 Mikolov 等人引入的类比基准测试结果吻合良好。