What are the root causes of hallucinations in large language models (LLMs)? We use Communication Complexity to prove that the Transformer layer is incapable of composing functions (e.g., identify a grandparent of a person in a genealogy) if the domains of the functions are large enough; we show through examples that this inability is already empirically present when the domains are quite small. We also point out that several mathematical tasks that are at the core of the so-called compositional tasks thought to be hard for LLMs are unlikely to be solvable by Transformers, for large enough instances and assuming that certain well accepted conjectures in the field of Computational Complexity are true.

翻译：大语言模型（LLM）产生幻觉的根本原因是什么？我们利用通信复杂性理论证明：当函数定义域足够大时，Transformer层无法实现函数复合（例如在族谱中识别某人的祖父母）。通过实例表明，即使在定义域相当小的情况下，这种能力缺失在经验层面已然存在。我们还指出，在足够大的实例规模下，且假设计算复杂性领域某些公认猜想成立的情况下，那些被视为LLM难以解决的所谓组合任务核心中的若干数学问题，Transformer很可能无法求解。