By simply composing prompts, developers can prototype novel generative applications with Large Language Models (LLMs). To refine prototypes into products, however, developers must iteratively revise prompts by evaluating outputs to diagnose weaknesses. Formative interviews (N=8) revealed that developers invest significant effort in manually evaluating outputs as they assess context-specific and subjective criteria. We present EvalLM, an interactive system for iteratively refining prompts by evaluating multiple outputs on user-defined criteria. By describing criteria in natural language, users can employ the system's LLM-based evaluator to get an overview of where prompts excel or fail, and improve these based on the evaluator's feedback. A comparative study (N=12) showed that EvalLM, when compared to manual evaluation, helped participants compose more diverse criteria, examine twice as many outputs, and reach satisfactory prompts with 59% fewer revisions. Beyond prompts, our work can be extended to augment model evaluation and alignment in specific application contexts.

翻译：通过简单地编写提示，开发者可以快速原型化基于大语言模型（LLM）的新颖生成式应用。然而，为了将原型完善为产品，开发者必须通过评估输出来诊断弱点，从而迭代式地修订提示。格式化访谈（N=8）揭示，开发者在评估上下文特定且主观的标准时，需投入大量精力手动评估输出。我们提出EvalLM——一种交互式系统，通过评估多个输出是否符合用户定义的标准来迭代优化提示。通过用自然语言描述标准，用户可利用系统基于LLM的评估器获取提示优势与不足的概览，并根据评估器反馈加以改进。对比研究（N=12）表明，与手动评估相比，EvalLM帮助参与者构建更多样化的标准，检查两倍数量的输出，并以减少59%的修订次数达到满意的提示。除提示优化外，我们的工作可扩展至特定应用场景中的模型评估与对齐增强。