We propose semantic smoothing, a smoothing method for language models that uses embeddings to share statistical observations across semantically similar contexts. The starting point is a decomposition of log-perplexity that motivates smoothing as a collection of distribution-estimation problems under Kullback-Leibler (KL) loss. We then show that, under a Lipschitz-logit model for embedding-based language generation, proximity of context embeddings implies proximity of the corresponding next-word distributions in KL divergence. Combining these observations, we formulate semantic smoothing as distribution estimation in KL loss with KL-proximity side information. For $n$ samples on a $d$-symbol alphabet with a side-information distribution at KL distance $Δ$, we give an interpolation estimator with worst-case KL risk $O(\min\{Δ,d/n\})$, and prove a matching-order lower bound for uniform side information. We extend the estimator to multiple and empirically estimated synonymous distributions. Experiments on synthetic Markov data and WikiText-103 bigram models using Word2Vec, GloVe, and GPT-2 embeddings show that semantic smoothing consistently reduces test perplexity when applied to add-constant and Kneser-Ney estimates.

翻译：我们提出语义平滑，一种利用嵌入在语义相似上下文间共享统计观测的语言模型平滑方法。其出发点是对数困惑度的分解，将平滑问题转化为在Kullback-Leibler（KL）损失下的分布估计问题集合。随后我们证明，在基于嵌入的语言生成的Lipschitz-logit模型假设下，上下文嵌入的邻近性意味着对应下一个词分布在KL散度上的邻近性。结合这些观察，我们将语义平滑形式化为一种带有KL邻近性辅助信息的KL损失分布估计问题。针对包含$n$个样本的$d$符号字母表，给定KL距离为$Δ$的辅助信息分布，我们提出一种插值估计器，其最坏情况下KL风险为$O(\min\{Δ,d/n\})$，并针对均匀辅助信息证明匹配阶的下界。我们将该估计器扩展至多个同义分布及其经验估计情形。在合成马尔可夫数据和Wikitext-103二元语法模型上的实验表明，采用Word2Vec、GloVe和GPT-2嵌入时，语义平滑在应用于加常数估计和Kneser-Ney估计后，能持续降低测试困惑度。