Languages are not created randomly but rather to communicate information. There is a strong association between languages and their underlying meanings, resulting in a sparse joint distribution that is heavily peaked according to their correlations. Moreover, these peak values happen to match with the marginal distribution of languages due to the sparsity. With the advent of LLMs trained on big data and large models, we can now precisely assess the marginal distribution of languages, providing a convenient means of exploring the sparse structures in the joint distribution for effective inferences. In this paper, we categorize languages as either unambiguous or {\epsilon}-ambiguous and present quantitative results to demonstrate that the emergent abilities of LLMs, such as language understanding, in-context learning, chain-of-thought prompting, and effective instruction fine-tuning, can all be attributed to Bayesian inference on the sparse joint distribution of languages.

翻译：语言并非随机产生，而是旨在传递信息。语言与其潜在含义之间存在强关联，由此形成一种稀疏的联合分布，该分布因相关性而呈现显著的尖峰形态。此外，由于稀疏性，这些峰值恰好与语言的边际分布相匹配。随着基于大数据和大模型训练的LLM（大语言模型）的出现，我们现在能够精确评估语言的边际分布，从而为探索联合分布中的稀疏结构以进行有效推断提供了便捷途径。在本文中，我们将语言分类为无歧义语言和{\epsilon}-歧义语言，并给出定量结果，证明LLM的涌现能力——如语言理解、上下文学习、思维链提示和有效的指令微调——均可归因于对语言稀疏联合分布的贝叶斯推断。