Data collected from the real world tends to be biased, unbalanced, and at risk of exposing sensitive and private information. This reality has given rise to the idea of creating synthetic datasets to alleviate risk, bias, harm, and privacy concerns inherent in the real data. This concept relies on Generative AI models to produce unbiased, privacy-preserving synthetic data while being true to the real data. In this new paradigm, how can we tell if this approach delivers on its promises? We present an auditing framework that offers a holistic assessment of synthetic datasets and AI models trained on them, centered around bias and discrimination prevention, fidelity to the real data, utility, robustness, and privacy preservation. We showcase our framework by auditing multiple generative models on diverse use cases, including education, healthcare, banking, human resources, and across different modalities, from tabular, to time-series, to natural language. Our use cases demonstrate the importance of a holistic assessment in order to ensure compliance with socio-technical safeguards that regulators and policymakers are increasingly enforcing. For this purpose, we introduce the trust index that ranks multiple synthetic datasets based on their prescribed safeguards and their desired trade-offs. Moreover, we devise a trust-index-driven model selection and cross-validation procedure via auditing in the training loop that we showcase on a class of transformer models that we dub TrustFormers, across different modalities. This trust-driven model selection allows for controllable trust trade-offs in the resulting synthetic data. We instrument our auditing framework with workflows that connect different stakeholders from model development to audit and certification via a synthetic data auditing report.

翻译：现实世界收集的数据往往存在偏差、不平衡，并面临暴露敏感和隐私信息的风险。这一现实催生了创建合成数据集以缓解真实数据中固有风险、偏差、损害和隐私问题的理念。该概念依赖生成式AI模型在忠实于真实数据的同时，生成无偏差且保护隐私的合成数据。在这一新范式下，我们如何判断该方法是否兑现其承诺？本文提出一个审计框架，该框架围绕偏差与歧视预防、对真实数据的保真度、实用性、鲁棒性及隐私保护，对合成数据集及基于其训练的AI模型进行整体评估。我们通过审计多个生成模型来展示该框架，案例涵盖教育、医疗、银行、人力资源等不同领域，以及表格数据、时间序列、自然语言等不同模态。我们的案例证明了整体评估的重要性，以确保符合监管者和政策制定者日益强化的社会技术保障措施。为此，我们引入了信任指数，该指数根据预设保障措施及所需权衡对多个合成数据集进行排序。此外，我们设计了一种通过训练循环中审计实现的、基于信任指数的模型选择与交叉验证流程，并在不同模态下（针对我们称为TrustFormers的一类Transformer模型）进行了展示。这种信任驱动的模型选择使得最终合成数据中的信任权衡变得可控。我们通过工作流将审计框架与从模型开发到审计及认证的不同利益相关者连接起来，并生成合成数据审计报告。