Languages are not created randomly but rather to communicate information. There is a strong association between languages and their underlying meanings, resulting in a sparse joint distribution that is heavily peaked according to their correlations. Moreover, these peak values happen to match with the marginal distribution of languages due to the sparsity. With the advent of LLMs trained on big data and large models, we can now precisely assess the marginal distribution of languages, providing a convenient means of exploring the sparse structures in the joint distribution for effective inferences. In this paper, we categorize languages as either unambiguous or {\epsilon}-ambiguous and present quantitative results to demonstrate that the emergent abilities of LLMs, such as language understanding, in-context learning, chain-of-thought prompting, and effective instruction fine-tuning, can all be attributed to Bayesian inference on the sparse joint distribution of languages.

翻译：语言并非随机生成，而是用于交流信息。语言与其潜在含义之间存在强关联，导致其联合分布稀疏且根据相关性呈现显著峰值。由于这种稀疏性，这些峰值恰好与语言的边缘分布相匹配。随着基于大数据和大模型训练的LLM的出现，我们现在能够精确评估语言的边缘分布，从而为有效推理提供了一种便捷方法来探索联合分布中的稀疏结构。本文将语言分为非歧义类和{\epsilon}-歧义类，并通过定量结果表明：LLM的涌现能力（如语言理解、上下文学习、思维链提示及高效指令微调）均可归结为对语言稀疏联合分布的贝叶斯推理。