Generative AI systems across modalities, ranging from text, image, audio, and video, have broad social impacts, but there exists no official standard for means of evaluating those impacts and which impacts should be evaluated. We move toward a standard approach in evaluating a generative AI system for any modality, in two overarching categories: what is able to be evaluated in a base system that has no predetermined application and what is able to be evaluated in society. We describe specific social impact categories and how to approach and conduct evaluations in the base technical system, then in people and society. Our framework for a base system defines seven categories of social impact: bias, stereotypes, and representational harms; cultural values and sensitive content; disparate performance; privacy and data protection; financial costs; environmental costs; and data and content moderation labor costs. Suggested methods for evaluation apply to all modalities and analyses of the limitations of existing evaluations serve as a starting point for necessary investment in future evaluations. We offer five overarching categories for what is able to be evaluated in society, each with their own subcategories: trustworthiness and autonomy; inequality, marginalization, and violence; concentration of authority; labor and creativity; and ecosystem and environment. Each subcategory includes recommendations for mitigating harm. We are concurrently crafting an evaluation repository for the AI research community to contribute existing evaluations along the given categories. This version will be updated following a CRAFT session at ACM FAccT 2023.

翻译：跨模态（文本、图像、音频和视频）的生成式人工智能系统具有广泛的社会影响，但目前尚无官方标准来评估这些影响以及应评估哪些影响。我们致力于建立一种标准方法，用于评估任意模态的生成式人工智能系统，主要分为两大类别：可在无预定应用的基础系统中评估的内容，以及可在社会中评估的内容。我们描述了具体的社会影响类别，以及如何在基础技术系统中、继而如何在人与社会层面开展和进行评估。我们的基础系统框架定义了七类社会影响：偏见、刻板印象与表征性危害；文化价值观与敏感内容；性能差异；隐私与数据保护；财务成本；环境成本；以及数据与内容审核劳动力成本。建议的评估方法适用于所有模态，对现有评估局限性的分析可作为未来评估必要投资的起点。我们提出了五大社会评估类别，每类包含子类别：可信度与自主性；不平等、边缘化与暴力；权力集中；劳动与创造力；以及生态系统与环境。每个子类别均包含减轻危害的建议。我们正同步为人工智能研究社区构建一个评估资源库，用于按给定类别提交现有评估。本版本将根据ACM FAccT 2023的CRAFT会议结果进行更新。