This paper proposes dialogue as a method for evaluating generative AI tools for culturally-situated creative practice, that recognizes the socially situated nature of art. Drawing on sociologist Howard Becker's concept of Art Worlds, this method expands the scope of traditional AI and creativity evaluations beyond benchmarks, user studies with crowd-workers, or focus groups conducted with artists. Our method involves two mutually informed dialogues: 1) 'dialogues with art worlds' placing artists in conversation with experts such as art historians, curators, and archivists, and 2)'dialogues with the machine,' facilitated through structured artist- and critic-led experimentation with state-of-the-art generative AI tools. We demonstrate the value of this method through a case study with artists and experts steeped in non-western art worlds, specifically the Persian Gulf. We trace how these dialogues help create culturally rich and situated forms of evaluation for representational possibilities of generative AI that mimic the reception of generative artwork in the broader art ecosystem. Putting artists in conversation with commentators also allow artists to shift their use of the tools to respond to their cultural and creative context. Our study can provide generative AI researchers an understanding of the complex dynamics of technology, human creativity and the socio-politics of art worlds, to build more inclusive machines for diverse art worlds.

翻译：本文提出将对话作为一种评估生成式AI工具在文化情境化创作实践中应用的方法，该方法承认艺术的社会情境性。借鉴社会学家霍华德·贝克尔的"艺术界"概念，该方法将传统AI与创造力评估的范围扩展到基准测试、众包工作者用户研究或艺术家焦点小组之外。我们的方法包含两个相互关联的对话：1）"与艺术界对话"——让艺术家与艺术史学家、策展人和档案管理员等专家进行交流；2）"与机器对话"——通过结构化的艺术家主导和评论家引导的实验，利用最先进的生成式AI工具进行。我们通过一个案例研究展示了该方法的价值，该案例涉及沉浸于非西方艺术界（特别是波斯湾地区）的艺术家和专家。我们追踪了这些对话如何帮助建立文化丰富且情境化的评估形式，以评估生成式AI的表征可能性，这些评估形式模仿了生成式艺术作品在更广泛艺术生态系统中的接受过程。让艺术家与评论家对话也使艺术家能够调整工具的使用方式，以回应当地的文化和创作背景。我们的研究可为生成式AI研究者提供对技术复杂性、人类创造力以及艺术界社会政治动态的理解，从而为多元艺术界构建更具包容性的机器系统。