Operationalizing large language models (LLMs) for custom, repetitive data pipelines is challenging, particularly due to their unpredictable and potentially catastrophic failures. Acknowledging the inevitability of these errors, we focus on identifying when LLMs may be generating incorrect responses when used repeatedly as part of data generation pipelines. We present SPADE, a method for automatically synthesizing assertions that identify bad LLM outputs. SPADE analyzes prompt version histories to create candidate assertion functions and then selects a minimal set that fulfills both coverage and accuracy requirements. In testing across nine different real-world LLM pipelines, SPADE efficiently reduces the number of assertions by 14% and decreases false failures by 21% when compared to simpler baselines.

翻译：随着大语言模型在定制化、重复性数据管道中的常态化应用，其不可预测且可能引发灾难性故障的特点带来了严峻挑战。鉴于此类错误的不可避免性，我们聚焦于识别大语言模型在重复用于数据生成管道时可能产生错误响应的情形。本文提出SPADE方法，通过自动合成断言机制来甄别大语言模型的错误输出。该方法分析提示词版本历史以创建候选断言函数，进而筛选出满足覆盖率和准确率要求的最小断言集。在九个真实世界的大语言模型管道测试中，与简单基线方法相比，SPADE将断言数量缩减14%，同时将误报率降低21%。