Recent work demonstrated great promise in the idea of orchestrating collaborations between LLMs, human input, and various tools to address the inherent limitations of LLMs. We propose a novel perspective called semantic decoding, which frames these collaborative processes as optimization procedures in semantic space. Specifically, we conceptualize LLMs as semantic processors that manipulate meaningful pieces of information that we call semantic tokens (known thoughts). LLMs are among a large pool of other semantic processors, including humans and tools, such as search engines or code executors. Collectively, semantic processors engage in dynamic exchanges of semantic tokens to progressively construct high-utility outputs. We refer to these orchestrated interactions among semantic processors, optimizing and searching in semantic space, as semantic decoding algorithms. This concept draws a direct parallel to the well-studied problem of syntactic decoding, which involves crafting algorithms to best exploit auto-regressive language models for extracting high-utility sequences of syntactic tokens. By focusing on the semantic level and disregarding syntactic details, we gain a fresh perspective on the engineering of AI systems, enabling us to imagine systems with much greater complexity and capabilities. In this position paper, we formalize the transition from syntactic to semantic tokens as well as the analogy between syntactic and semantic decoding. Subsequently, we explore the possibilities of optimizing within the space of semantic tokens via semantic decoding algorithms. We conclude with a list of research opportunities and questions arising from this fresh perspective. The semantic decoding perspective offers a powerful abstraction for search and optimization directly in the space of meaningful concepts, with semantic tokens as the fundamental units of a new type of computation.

翻译：近期研究表明，通过协调大语言模型、人类输入与各类工具之间的协作，能够有效解决大语言模型固有的局限性。本文提出一种名为"语义解码"的新视角，将这类协作过程视为语义空间中的优化流程。具体而言，我们将大语言模型概念化为语义处理器，用于操控我们称为"语义标记"（即已知思想）的有意义信息单元。大语言模型与包括人类及搜索引擎、代码执行器等工具在内的众多语义处理器共同构成一个庞大集群。这些语义处理器通过动态交换语义标记，逐步构建出高实用性的输出结果。我们将这种语义处理器之间协作交互、在语义空间中优化搜索的过程定义为"语义解码算法"。该概念与深入研究过的"句法解码"问题形成直接类比——后者旨在设计算法以最优方式利用自回归语言模型提取高实用性的句法标记序列。通过聚焦语义层次并忽略句法细节，我们获得了工程化人工智能系统的新视角，得以构想具有更高复杂性和更强能力的系统。在本立场论文中，我们首先形式化从句法标记到语义标记的转变，以及句法解码与语义解码之间的类比关系。随后探讨通过语义解码算法在语义标记空间进行优化的可能性。最后，我们总结了这一新视角催生的研究机遇与待解问题。语义解码视角为在意义概念空间中直接进行搜索与优化提供了强大的抽象框架，其中语义标记构成了新型计算的基本单元。