The rise of capabilities expressed by large language models has been quickly followed by the integration of the same complex systems into application level logic. Algorithms, programs, systems, and companies are built around structured prompting to black box models where the majority of the design and implementation lies in capturing and quantifying the `agent mode'. The standard way to shape a closed language model is to prime it for a specific task with a tailored prompt, often initially handwritten by a human. The textual prompts co-evolve with the codebase, taking shape over the course of project life as artifacts which must be reviewed and maintained, just as the traditional code files might be. Unlike traditional code, we find that prompts do not receive effective static testing and linting to prevent runtime issues. In this work, we present a novel dataset called PromptSet, with more than 61,000 unique developer prompts used in open source Python programs. We perform analysis on this dataset and introduce the notion of a static linter for prompts. Released with this publication is a HuggingFace dataset and a Github repository to recreate collection and processing efforts, both under the name \texttt{pisterlabs/promptset}.

翻译：大语言模型所展现出的能力提升迅速推动了将其复杂系统集成至应用层逻辑的进程。围绕对黑箱模型的结构化提示构建算法、程序、系统乃至公司已成为常态，其中大部分设计与实现工作在于捕获并量化"智能体模式"。塑造封闭式语言模型的标准方法是通过量身定制的提示词（通常最初由人工撰写）使其适配特定任务。文本提示词与代码库协同演化，在项目生命周期中逐渐形成须经审查和维护的工件——正如传统代码文件一般。然而与传统代码不同，我们发现提示词缺乏有效的静态测试与代码检查机制用以预防运行时问题。本研究提出名为PromptSet的新型数据集，包含开源Python程序中逾61000条独特的开发者提示词。我们对数据集进行分析，并引入提示词静态检查器的概念。随本文发布的有HuggingFace数据集与用于重现数据收集和处理流程的GitHub仓库，两者均以\texttt{pisterlabs/promptset}命名。