In many text-generation problems, users may prefer not only a single response, but a diverse range of high-quality outputs from which to choose. Quality-diversity (QD) search algorithms aim at such outcomes, by continually improving and diversifying a population of candidates. However, the applicability of QD to qualitative domains, like creative writing, has been limited by the difficulty of algorithmically specifying measures of quality and diversity. Interestingly, recent developments in language models (LMs) have enabled guiding search through AI feedback, wherein LMs are prompted in natural language to evaluate qualitative aspects of text. Leveraging this development, we introduce Quality-Diversity through AI Feedback (QDAIF), wherein an evolutionary algorithm applies LMs to both generate variation and evaluate the quality and diversity of candidate text. When assessed on creative writing domains, QDAIF covers more of a specified search space with high-quality samples than do non-QD controls. Further, human evaluation of QDAIF-generated creative texts validates reasonable agreement between AI and human evaluation. Our results thus highlight the potential of AI feedback to guide open-ended search for creative and original solutions, providing a recipe that seemingly generalizes to many domains and modalities. In this way, QDAIF is a step towards AI systems that can independently search, diversify, evaluate, and improve, which are among the core skills underlying human society's capacity for innovation.

翻译：在许多文本生成问题中，用户可能不仅希望获得单一回答，更期望从一系列高质量且多样化的输出中进行选择。质量多样性（QD）搜索算法正是为此目标而设计，通过不断改进和多样化候选种群来实现这一目标。然而，QD在定性领域（如创意写作）的应用一直受到算法化定义质量与多样性指标的难题限制。有趣的是，语言模型（LM）的最新进展已能通过人工智能反馈引导搜索，即利用自然语言提示LM评估文本的定性方面。基于这一突破，我们提出基于人工智能反馈的质量多样性（QDAIF）方法，其中进化算法利用LM同时生成变体并评估候选文本的质量与多样性。在创意写作领域的评估中，QDAIF通过高质量样本覆盖了指定搜索空间中更广泛的范围，优于非QD对照组。进一步对QDAIF生成的创意文本进行人类评估，验证了AI评估与人类评估之间具有合理的一致性。我们的研究结果凸显了人工智能反馈在引导开放式搜索以产生创意和原始解决方案方面的潜力，为跨领域、跨模态的通用化方案提供了实践路径。由此，QDAIF标志着向具备自主搜索、多样化、评估与改进能力的AI系统迈出了一步——这些能力正是人类社会创新能力的核心基础。