We initiate a formal investigation into the design and analysis of LLM-based algorithms, i.e. algorithms that contain one or multiple calls of large language models (LLMs) as sub-routines and critically rely on the capabilities of LLMs. While LLM-based algorithms, ranging from basic LLM calls with prompt engineering to complicated LLM-powered agent systems and compound AI systems, have achieved remarkable empirical success, the design and optimization of them have mostly relied on heuristics and trial-and-errors, which is largely due to a lack of formal and analytical study for these algorithms. To fill this gap, we start by identifying the computational-graph representation of LLM-based algorithms, the design principle of task decomposition, and some key abstractions, which then facilitate our formal analysis for the accuracy and efficiency of LLM-based algorithms, despite the black-box nature of LLMs. Through extensive analytical and empirical investigation in a series of case studies, we demonstrate that the proposed framework is broadly applicable to a wide range of scenarios and diverse patterns of LLM-based algorithms, such as parallel, hierarchical and recursive task decomposition. Our proposed framework holds promise for advancing LLM-based algorithms, by revealing the reasons behind curious empirical phenomena, guiding the choices of hyperparameters, predicting the empirical performance of algorithms, and inspiring new algorithm design. To promote further study of LLM-based algorithms, we release our source code at https://github.com/modelscope/agentscope/tree/main/examples/paper_llm_based_algorithm.

翻译：本文对大语言模型（LLM）算法的设计与分析展开系统性研究。这类算法包含一个或多个大语言模型调用作为子程序，并高度依赖大语言模型的能力。尽管从基于提示工程的基础大语言模型调用，到复杂的大语言模型驱动智能体系统及复合人工智能系统等各类大语言模型算法已取得显著实证成果，但其设计与优化仍主要依赖启发式方法与试错过程，这很大程度上源于缺乏对此类算法的形式化与解析研究。为填补这一空白，我们首先确立了大语言模型算法的计算图表示、任务分解设计原则及关键抽象概念，这些要素为我们在面对大语言模型黑箱特性的情况下，对其算法精度与效率进行形式化分析提供了基础。通过一系列案例研究中的广泛解析与实证分析，我们证明所提出的框架适用于多种场景及多样化的大语言模型算法模式，例如并行、层次化与递归任务分解。该框架通过揭示经验现象背后的原理、指导超参数选择、预测算法实证表现以及启发新算法设计，展现出推动大语言模型算法发展的潜力。为促进大语言模型算法的后续研究，我们在 https://github.com/modelscope/agentscope/tree/main/examples/paper_llm_based_algorithm 公开了源代码。