The generative AI revolution in recent years has been spurred by an expansion in compute power and data quantity, which together enable extensive pre-training of powerful text-to-image (T2I) models. With their greater capabilities to generate realistic and creative content, these T2I models like DALL-E, MidJourney, Imagen or Stable Diffusion are reaching ever wider audiences. Any unsafe behaviors inherited from pretraining on uncurated internet-scraped datasets thus have the potential to cause wide-reaching harm, for example, through generated images which are violent, sexually explicit, or contain biased and derogatory stereotypes. Despite this risk of harm, we lack systematic and structured evaluation datasets to scrutinize model behavior, especially adversarial attacks that bypass existing safety filters. A typical bottleneck in safety evaluation is achieving a wide coverage of different types of challenging examples in the evaluation set, i.e., identifying 'unknown unknowns' or long-tail problems. To address this need, we introduce the Adversarial Nibbler challenge. The goal of this challenge is to crowdsource a diverse set of failure modes and reward challenge participants for successfully finding safety vulnerabilities in current state-of-the-art T2I models. Ultimately, we aim to provide greater awareness of these issues and assist developers in improving the future safety and reliability of generative AI models. Adversarial Nibbler is a data-centric challenge, part of the DataPerf challenge suite, organized and supported by Kaggle and MLCommons.

翻译：近年来，生成式人工智能的革新得益于计算能力和数据量的扩张，这两者共同推动了强大的文本到图像（T2I）模型的大规模预训练。随着DALL-E、MidJourney、Imagen或Stable Diffusion等T2I模型在生成逼真且富有创意的内容方面能力的增强，它们正被越来越广泛的受众所使用。这些模型因在未经筛选的网络爬取数据集上进行预训练而继承的任何不安全行为，都有可能造成广泛危害，例如生成包含暴力、色情内容或带有偏见和贬低刻板印象的图像。尽管存在这种危害风险，我们仍缺乏系统化和结构化的评估数据集来审视模型行为，特别是针对能够绕开现有安全过滤器的对抗性攻击。安全评估中的一个典型瓶颈是，在评估集中难以广泛覆盖不同类型的具有挑战性的样本，即识别“未知的未知”或长尾问题。为满足这一需求，我们引入了“对抗性掰断”（Adversarial Nibbler）挑战。该挑战旨在通过众包方式收集多样化的失败模式，并奖励那些成功发现当前最先进T2I模型中安全漏洞的参与者。最终，我们旨在提高对这些问题认识，并协助开发者改进生成式AI模型未来的安全性和可靠性。“对抗性掰断”是一项以数据为中心的挑战，是DataPerf挑战套件的一部分，由Kaggle和MLCommons组织和支持。