Data collected from the real world tends to be biased, unbalanced, and at risk of exposing sensitive and private information. This reality has given rise to the idea of creating synthetic datasets to alleviate risk, bias, harm, and privacy concerns inherent in the real data. This concept relies on Generative AI models to produce unbiased, privacy-preserving synthetic data while being true to the real data. In this new paradigm, how can we tell if this approach delivers on its promises? We present an auditing framework that offers a holistic assessment of synthetic datasets and AI models trained on them, centered around bias and discrimination prevention, fidelity to the real data, utility, robustness, and privacy preservation. We showcase our framework by auditing multiple generative models on diverse use cases, including education, healthcare, banking, human resources, and across different modalities, from tabular, to time-series, to natural language. Our use cases demonstrate the importance of a holistic assessment in order to ensure compliance with socio-technical safeguards that regulators and policymakers are increasingly enforcing. For this purpose, we introduce the trust index that ranks multiple synthetic datasets based on their prescribed safeguards and their desired trade-offs. Moreover, we devise a trust-index-driven model selection and cross-validation procedure via auditing in the training loop that we showcase on a class of transformer models that we dub TrustFormers, across different modalities. This trust-driven model selection allows for controllable trust trade-offs in the resulting synthetic data. We instrument our auditing framework with workflows that connect different stakeholders from model development to audit and certification via a synthetic data auditing report.

翻译：从现实世界收集的数据往往存在偏差、不平衡，并存在暴露敏感和私有信息的风险。这一现状催生了创建合成数据集以减轻真实数据中固有风险、偏差、伤害和隐私问题的想法。这一概念依赖生成式AI模型来生成无偏、保护隐私且忠实于真实数据的合成数据。在这种新范式下，我们如何判断该方法能否兑现其承诺？我们提出一个审计框架，该框架围绕偏差与歧视预防、对真实数据的保真度、实用性、鲁棒性和隐私保护，对合成数据集及其上训练的AI模型进行整体评估。我们通过审计多个生成模型，在包括教育、医疗、银行、人力资源在内的不同用例以及从表格数据、时间序列到自然语言的不同模态中展示该框架。我们的用例证明了进行整体评估的重要性，以确保符合监管机构和政策制定者日益强制实施的社会技术保障措施。为此，我们引入信任指数，该指数根据预设保障措施及其期望的权衡对多个合成数据集进行排序。此外，我们设计了一种通过训练循环审计实现的、由信任指数驱动的模型选择与交叉验证程序，并以我们称为TrustFormers的一类跨不同模态的Transformer模型为例进行展示。这种信任驱动的模型选择允许在生成的合成数据中实现可控的信任权衡。我们通过合成数据审计报告，为我们的审计框架配置了连接从模型开发到审计与认证的不同利益相关者的工作流。