Large Language Models have consistently demonstrated a lack of creativity and diversity across tasks. Prior work has focused on addressing whether models are capable of generating creative outputs. Here, we aim to consider novelty and investigate what makes model-generated content novel or not novel in a task-specific manner. We propose a fine-grained evaluation metric GENIE to measure the novelty of responses along task-specific features with respect to a population of responses. We show that unlike GENIE, holistic metrics struggle to capture the high-dimensionality of novelty and do not provide insight on which properties they target. Finally, we use GENIE to measure the effectiveness of mitigation methods that address creativity to better understand where these methods can improve novelty.

翻译：大语言模型在各类任务中持续表现出创造性与多样性的匮乏。既有研究主要聚焦于模型能否生成创造性输出。本文旨在探讨新颖性这一概念，并以任务特异性方式研究模型生成内容的新颖性判定机制。我们提出了一种细粒度评估指标GENIE，该指标能够基于任务特定特征，在群体响应中衡量每条响应的新颖性程度。实验表明，相较于GENIE，整体性评估指标难以捕捉新颖性的高维特性，且无法揭示其评估目标的具体属性。最后，我们运用GENIE测度各类创造性缓解策略的有效性，以深入理解这些方法在提升新颖性方面的作用边界。