A common practice in large language model (LLM) usage for complex analytical tasks such as code generation, is to sample a solution for the entire task within the model's context window. Previous works have shown that subtask decomposition within the model's context (chain of thought), is beneficial for solving such tasks. In this work, we point a limitation of LLMs' ability to perform several sub-tasks within the same context window - an in-context hardness of composition, pointing to an advantage for distributing a decomposed problem in a multi-agent system of LLMs. The hardness of composition is quantified by a generation complexity metric, i.e., the number of LLM generations required to sample at least one correct solution. We find a gap between the generation complexity of solving a compositional problem within the same context relative to distributing it among multiple agents, that increases exponentially with the solution's length. We prove our results theoretically and demonstrate them empirically.

翻译：在大型语言模型（LLM）用于复杂分析任务（如代码生成）时，一种常见做法是在模型的上下文窗口内为整个任务采样一个解决方案。先前的研究表明，在模型上下文内进行子任务分解（思维链）有助于解决此类任务。在这项工作中，我们指出了LLM在同一上下文窗口中执行多个子任务的能力存在一个局限——即上下文内的组合困难，这凸显了在LLM多智能体系统中分配分解后问题的优势。组合困难通过一个生成复杂度指标进行量化，即至少采样到一个正确解决方案所需的LLM生成次数。我们发现，在同一上下文中解决组合问题与将其分配给多个智能体这两种方式的生成复杂度之间存在差距，且该差距随解决方案长度的增加呈指数级增长。我们从理论上证明了这一结果，并通过实验进行了验证。