By simply composing prompts, developers can prototype novel generative applications with Large Language Models (LLMs). To refine prototypes into products, however, developers must iteratively revise prompts by evaluating outputs to diagnose weaknesses. Formative interviews (N=8) revealed that developers invest significant effort in manually evaluating outputs as they assess context-specific and subjective criteria. We present EvalLM, an interactive system for iteratively refining prompts by evaluating multiple outputs on user-defined criteria. By describing criteria in natural language, users can employ the system's LLM-based evaluator to get an overview of where prompts excel or fail, and improve these based on the evaluator's feedback. A comparative study (N=12) showed that EvalLM, when compared to manual evaluation, helped participants compose more diverse criteria, examine twice as many outputs, and reach satisfactory prompts with 59% fewer revisions. Beyond prompts, our work can be extended to augment model evaluation and alignment in specific application contexts.

翻译：通过简单编写提示，开发者即可用大型语言模型快速原型化新型生成式应用。然而，要将原型精炼为产品，开发者必须通过评估输出来诊断薄弱环节，从而迭代式地修订提示。形成性访谈（N=8）揭示，开发者在评估情境化与主观性标准时，需投入大量精力进行人工输出评估。我们提出EvalLM，一个通过基于用户定义标准评估多个输出来迭代优化提示的交互式系统。用户用自然语言描述标准后，可借助系统的LLM评估器获得提示优缺点的全局视图，并依据评估反馈加以改进。对比研究（N=12）表明：与人工评估相比，EvalLM帮助参与者制定更多样化的标准，检查两倍数量的输出，并以减少59%的修订次数达到满意的提示。除提示优化外，我们的工作可扩展至特定应用场景中的模型评估与对齐增强。