Synthetic data algorithms are widely employed in industries to generate artificial data for downstream learning tasks. While existing research primarily focuses on empirically evaluating utility of synthetic data, its theoretical understanding is largely lacking. This paper bridges the practice-theory gap by establishing relevant utility theory in a statistical learning framework. It considers two utility metrics: generalization and ranking of models trained on synthetic data. The former is defined as the generalization difference between models trained on synthetic and on real data. By deriving analytical bounds for this utility metric, we demonstrate that the synthetic feature distribution does not need to be similar as that of real data for ensuring comparable generalization of synthetic models, provided proper model specifications in downstream learning tasks. The latter utility metric studies the relative performance of models trained on synthetic data. In particular, we discover that the distribution of synthetic data is not necessarily similar as the real one to ensure consistent model comparison. Interestingly, consistent model comparison is still achievable even when synthetic responses are not well generated, as long as downstream models are separable by a generalization gap. Finally, extensive experiments on non-parametric models and deep neural networks have been conducted to validate these theoretical findings.

翻译：合成数据算法广泛应用于工业界，用于为下游学习任务生成人工数据。现有研究主要关注合成数据效用的实证评估，但其理论理解仍较为匮乏。本文通过在统计学习框架中建立相关的效用理论，弥合了实践与理论之间的差距。本文考虑了两种效用度量指标：基于合成数据训练的模型的泛化能力和排序能力。前者定义为合成数据与真实数据上训练的模型之间的泛化差异。通过推导该效用度量的解析边界，我们证明：若下游学习任务中模型设定适当，合成特征分布无需与真实数据分布相似，即可保证合成模型的泛化能力与真实模型相当。后者研究了合成数据训练的模型的相对性能。特别地，我们发现，为确保一致的模型比较，合成数据分布无需与真实数据分布相似。有趣的是，即使合成响应变量生成质量不佳，只要下游模型可通过泛化差距进行区分，仍可实现一致的模型比较。最后，通过在非参数模型与深度神经网络上开展大量实验，验证了上述理论发现。