Evaluating the utility of synthetic data is critical for measuring the effectiveness and efficiency of synthetic algorithms. Existing results focus on empirical evaluations of the utility of synthetic data, whereas the theoretical understanding of how utility is affected by synthetic data algorithms remains largely unexplored. This paper establishes utility theory from a statistical perspective, aiming to quantitatively assess the utility of synthetic algorithms based on a general metric. The metric is defined as the absolute difference in generalization between models trained on synthetic and original datasets. We establish analytical bounds for this utility metric to investigate critical conditions for the metric to converge. An intriguing result is that the synthetic feature distribution is not necessarily identical to the original one for the convergence of the utility metric as long as the model specification in downstream learning tasks is correct. Another important utility metric is model comparison based on synthetic data. Specifically, we establish sufficient conditions for synthetic data algorithms so that the ranking of generalization performances of models trained on the synthetic data is consistent with that from the original data. Finally, we conduct extensive experiments using non-parametric models and deep neural networks to validate our theoretical findings.

翻译：评估合成数据的效用对于衡量合成算法的有效性和效率至关重要。现有研究主要关注合成数据效用的实证评估，而理论层面关于效用如何受合成数据算法影响的问题仍鲜有探索。本文从统计视角建立效用理论，旨在基于一般性度量对合成算法的效用进行定量评估。该度量定义为在合成数据集与原始数据集上训练模型泛化性能的绝对差异。我们通过推导该效用度量的解析界，研究其收敛的关键条件。一个有趣的发现是：当下游学习任务中的模型设定正确时，合成特征分布无需与原始特征分布完全相同即可实现效用度量的收敛。另一个重要的效用度量是基于合成数据的模型比较。具体而言，我们建立了合成数据算法的充分条件，使得基于合成数据训练的模型泛化性能排序与基于原始数据的结果保持一致。最后，我们通过非参数模型和深度神经网络开展大量实验验证理论发现。