Languages are not created randomly but rather to communicate information. There is a strong association between languages and their underlying meanings, resulting in a sparse joint distribution that is heavily peaked according to their correlations. Moreover, these peak values happen to match with the marginal distribution of languages due to the sparsity. With the advent of LLMs trained on big data and large models, we can now precisely assess the marginal distribution of languages, providing a convenient means of exploring the sparse structures in the joint distribution for effective inferences. In this paper, we categorize languages as either unambiguous or {\epsilon}-ambiguous and present quantitative results to demonstrate that the emergent abilities of LLMs, such as language understanding, in-context learning, chain-of-thought prompting, and effective instruction fine-tuning, can all be attributed to Bayesian inference on the sparse joint distribution of languages.

翻译：语言并非随机产生，而是为了传递信息。语言与其潜在含义之间存在强关联，导致其联合分布稀疏，并因相关性而呈现显著峰值。此外，由于稀疏性，这些峰值恰好与语言的边缘分布相匹配。随着基于大数据和大模型训练的LLM的出现，我们如今能够精确评估语言的边缘分布，从而提供一种便捷手段来探索联合分布中的稀疏结构以实现有效推理。本文将语言分为无歧义语言与ε-歧义语言两类，并通过定量结果证明：LLM的涌现能力（如语言理解、上下文学习、思维链提示及高效指令微调）均可归因于对语言稀疏联合分布的贝叶斯推理。