Textual Large Language Models (LLMs) provide a simple and familiar interface: a string of text is used for both input and output. However, the information conveyed to an LLM often has a richer structure and semantics, which is not conveyed in a string. For example, most prompts contain both instructions ("Summarize this paper into a paragraph") and data (the paper to summarize), but these are usually not distinguished when passed to the model. This can lead to model confusion and security risks, such as prompt injection attacks. This work addresses this shortcoming by introducing an LLM-native mark-up language, LLMON (LLM Object Notation, pronounced "Lemon"), that enables the structure and semantic metadata of the text to be communicated in a natural way to an LLM. This information can then be used during model training, model prompting, and inference implementation, leading to improvements in model accuracy, safety, and security. This is analogous to how programming language types can be used for many purposes, such as static checking, code generation, dynamic checking, and IDE highlighting. We discuss the general design requirements of an LLM-native markup language, introduce the LLMON markup language and show how it meets these design requirements, describe how the information contained in a LLMON artifact can benefit model training and inference implementation, and provide some preliminary empirical evidence of its value for both of these use cases. We also discuss broader issues and research opportunities that are enabled with an LLM-native approach.

翻译：文本型大型语言模型（LLM）提供了一个简单且熟悉的接口：输入和输出均采用字符串形式。然而，传递给LLM的信息往往具有更丰富的结构与语义，而这些无法在字符串中体现。例如，大多数提示既包含指令（如“将本文总结为一段”），也包含数据（需要总结的论文），但传递给模型时这两者通常未被区分。这可能导致模型混淆及安全风险，例如提示注入攻击。为解决这一缺陷，本文引入了一种LLM原生标记语言——LLMON（LLM对象表示法，发音为“Lemon”），该语言能够以自然方式向LLM传递文本的结构与语义元数据。这些信息随后可用于模型训练、提示设计及推理实现，从而提升模型的准确性、安全性与可靠性。这与编程语言类型可用于静态检查、代码生成、动态检查及IDE高亮等多重用途类似。我们讨论了LLM原生标记语言的通用设计需求，介绍了LLMON标记语言并展示其如何满足这些设计需求，描述了LLMON构件所含信息如何助力模型训练与推理实现，并提供了初步经验证据证明其在两类应用场景中的价值。此外，我们还探讨了基于LLM原生方法所能实现的更广泛问题与研究机遇。