Generating text with autoregressive language models (LMs) is of great importance to many natural language processing (NLP) applications. Previous solutions for this task often produce text that contains degenerative expressions or lacks semantic consistency. Recently, Su et al. introduced a new decoding method, contrastive search, based on the isotropic representation space of the language model and obtained new state of the art on various benchmarks. Additionally, Su et al. argued that the representations of autoregressive LMs (e.g. GPT-2) are intrinsically anisotropic which is also shared by previous studies. Therefore, to ensure the language model follows an isotropic distribution, Su et al. proposed a contrastive learning scheme, SimCTG, which calibrates the language model's representations through additional training. In this study, we first answer the question: "Are autoregressive LMs really anisotropic?". To this end, we extensively evaluate the isotropy of LMs across 16 major languages. Surprisingly, we find that the anisotropic problem only exists in the two specific English GPT-2-small/medium models. On the other hand, all other evaluated LMs are naturally isotropic which is in contrast to the conclusion drawn by previous studies. Based on our findings, we further assess the contrastive search decoding method using off-the-shelf LMs on four generation tasks across 16 languages. Our experimental results demonstrate that contrastive search significantly outperforms previous decoding methods without any additional training. More notably, on 12 out of the 16 evaluated languages, contrastive search performs comparably with human-level performances as judged by human evaluations. Our code and other related resources are publicly available at https://github.com/yxuansu/Contrastive_Search_Is_What_You_Need.

翻译：基于自回归语言模型（LMs）的文本生成对许多自然语言处理（NLP）应用至关重要。此前针对该任务的解决方案常产生包含退化表达或缺乏语义一致性的文本。近期，Su等人提出了一种基于语言模型各向同性表示空间的新解码方法——对比搜索，并在多项基准测试上取得了最新最优结果。此外，Su等人认为自回归语言模型（如GPT-2）的表示本质上是各向异性的，这一观点与先前研究一致。因此，为确保语言模型遵循各向同性分布，Su等人提出了一种对比学习方案SimCTG，通过额外训练校准语言模型的表示空间。在本研究中，我们首先回答了一个关键问题："自回归语言模型真的是各向异性的吗？"为此，我们系统评估了涵盖16种主流语言的多种语言模型的各向同性特性。令人惊讶的是，我们发现各向异性问题仅存在于两个特定英语模型（GPT-2-small/medium）中。与此前研究结论相反，其他所有被评估的语言模型天然具备各向同性特征。基于该发现，我们进一步在16种语言的四项生成任务中，使用现成语言模型评估对比搜索解码方法。实验结果表明，对比搜索无需任何额外训练即可显著优于此前解码方法。更值得关注的是，在16种评估语言中，有12种语言的对比搜索表现达到了与人类水平相当的性能（经人工评估验证）。我们的代码及相关资源已在https://github.com/yxuansu/Contrastive_Search_Is_What_You_Need 公开。