In many text-generation problems, users may prefer not only a single response, but a diverse range of high-quality outputs from which to choose. Quality-diversity (QD) search algorithms aim at such outcomes, by continually improving and diversifying a population of candidates. However, the applicability of QD to qualitative domains, like creative writing, has been limited by the difficulty of algorithmically specifying measures of quality and diversity. Interestingly, recent developments in language models (LMs) have enabled guiding search through AI feedback, wherein LMs are prompted in natural language to evaluate qualitative aspects of text. Leveraging this development, we introduce Quality-Diversity through AI Feedback (QDAIF), wherein an evolutionary algorithm applies LMs to both generate variation and evaluate the quality and diversity of candidate text. When assessed on creative writing domains, QDAIF covers more of a specified search space with high-quality samples than do non-QD controls. Further, human evaluation of QDAIF-generated creative texts validates reasonable agreement between AI and human evaluation. Our results thus highlight the potential of AI feedback to guide open-ended search for creative and original solutions, providing a recipe that seemingly generalizes to many domains and modalities. In this way, QDAIF is a step towards AI systems that can independently search, diversify, evaluate, and improve, which are among the core skills underlying human society's capacity for innovation.

翻译：在许多文本生成问题中，用户不仅期望获得单一回答，更希望从多样化的高质量输出中进行选择。质量-多样性（QD）搜索算法正是为实现这一目标而设计，通过持续改进和扩充候选种群来提升输出效果。然而，QD算法在创意写作等定性领域的应用长期受限于难以用算法化方式定义质量与多样性的度量标准。值得关注的是，语言模型（LM）领域的最新进展使得通过AI反馈引导搜索成为可能——研究者可通过自然语言提示LM对文本的定性特征进行评估。基于这一突破，我们提出通过AI反馈实现质量-多样性优化（QDAIF）方法：该方法采用进化算法，利用LM同时生成文本变体并评估候选文本的质量与多样性。在创意写作领域的评估中，相较于非QD对照组，QDAIF能在指定搜索空间中以更高采样质量覆盖更广范围。进一步的人类评估验证了QDAIF生成创意文本时，AI评估与人类评估之间存在合理的一致性。本研究结果表明，AI反馈在引导面向创意与原创解决方案的开放式搜索方面具有巨大潜力，该方法有望推广至多个领域与模态。由此，QDAIF标志着向具备自主搜索、多样化生成、评估与改进能力的AI系统迈出关键一步——这些能力正是人类社会创新能力的核心要素。