Today's probabilistic language generators fall short when it comes to producing coherent and fluent text despite the fact that the underlying models perform well under standard metrics, e.g., perplexity. This discrepancy has puzzled the language generation community for the last few years. In this work, we posit that the abstraction of natural language generation as a discrete stochastic process--which allows for an information-theoretic analysis--can provide new insights into the behavior of probabilistic language generators, e.g., why high-probability texts can be dull or repetitive. Humans use language as a means of communicating information, aiming to do so in a simultaneously efficient and error-minimizing manner; in fact, psycholinguistics research suggests humans choose each word in a string with this subconscious goal in mind. We formally define the set of strings that meet this criterion: those for which each word has an information content close to the expected information content, i.e., the conditional entropy of our model. We then propose a simple and efficient procedure for enforcing this criterion when generating from probabilistic models, which we call locally typical sampling. Automatic and human evaluations show that, in comparison to nucleus and top-k sampling, locally typical sampling offers competitive performance (in both abstractive summarization and story generation) in terms of quality while consistently reducing degenerate repetitions.

翻译：当今的概率语言生成器在生成连贯流畅文本方面表现欠佳，尽管其底层模型在标准指标（如困惑度）上表现良好。这一矛盾在过去几年中一直困扰着语言生成领域。本文中，我们提出将自然语言生成抽象为离散随机过程（这一视角允许进行信息论分析），可为概率语言生成器的行为提供新见解，例如为何高概率文本可能单调或重复。人类将语言作为信息交流的媒介，致力于在高效性与错误最小化之间取得平衡；事实上，心理语言学研究提示，人类在语句中选择每个词语时都潜在地以此为目标。我们正式定义了满足该标准的字符串集合：其中每个词语的信息含量接近预期信息含量（即模型的条件熵）。进而提出一种简单高效的程序——称为局部典型采样，用于在从概率模型生成时强制执行这一标准。自动评估与人工评估表明，与核采样和top-k采样相比，局部典型采样在质量上具有竞争力的表现（在抽象摘要和故事生成任务中），同时持续减少退化性重复。