Textual Large Language Models (LLMs) provide a simple and familiar interface: a string of text is used for both input and output. However, the information conveyed to an LLM often has a richer structure and semantics, which is not conveyed in a string. For example, most prompts contain both instructions ("Summarize this paper into a paragraph") and data (the paper to summarize), but these are usually not distinguished when passed to the model. This can lead to model confusion and security risks, such as prompt injection attacks. This work addresses this shortcoming by introducing an LLM-native mark-up language, LLMON (LLM Object Notation, pronounced "Lemon"), that enables the structure and semantic metadata of the text to be communicated in a natural way to an LLM. This information can then be used during model training, model prompting, and inference implementation, leading to improvements in model accuracy, safety, and security. This is analogous to how programming language types can be used for many purposes, such as static checking, code generation, dynamic checking, and IDE highlighting. We discuss the general design requirements of an LLM-native markup language, introduce the LLMON markup language and show how it meets these design requirements, describe how the information contained in a LLMON artifact can benefit model training and inference implementation, and provide some preliminary empirical evidence of its value for both of these use cases. We also discuss broader issues and research opportunities that are enabled with an LLM-native approach.

翻译：文本型大语言模型（LLM）提供了一种简单且熟悉的接口：输入和输出均采用文本字符串形式。然而，传递给LLM的信息通常具有更丰富的结构和语义，这些在字符串中无法体现。例如，大多数提示包含指令（"将该论文总结为一段"）和数据（待总结的论文本身），但传递给模型时两者通常未被区分。这可能导致模型混淆和安全风险，如提示注入攻击。本研究通过引入一种LLM原生标记语言LLMON（LLM对象表示法，读作"Lemon"），解决这一缺陷，该语言能够以自然方式向LLM传递文本的结构和语义元数据。此类信息可用于模型训练、提示设计及推理实现，从而提升模型准确性、安全性和可靠性。这与编程语言类型可应用于静态检查、代码生成、动态检查及IDE高亮等多重用途的方式类似。我们讨论了LLM原生标记语言的通用设计要求，介绍了LLMON标记语言并展示其如何满足这些要求，描述了LLMON工件包含的信息如何惠及模型训练与推理实现，并提供了初步实证证据证明其在这两种用例中的价值。此外，我们还探讨了通过LLM原生方法实现更广泛的研究问题与机遇。