In generative modeling, numerous successful approaches leverage a low-dimensional latent space, e.g., Stable Diffusion models the latent space induced by an encoder and generates images through a paired decoder. Although the selection of the latent space is empirically pivotal, determining the optimal choice and the process of identifying it remain unclear. In this study, we aim to shed light on this under-explored topic by rethinking the latent space from the perspective of model complexity. Our investigation starts with the classic generative adversarial networks (GANs). Inspired by the GAN training objective, we propose a novel "distance" between the latent and data distributions, whose minimization coincides with that of the generator complexity. The minimizer of this distance is characterized as the optimal data-dependent latent that most effectively capitalizes on the generator's capacity. Then, we consider parameterizing such a latent distribution by an encoder network and propose a two-stage training strategy called Decoupled Autoencoder (DAE), where the encoder is only updated in the first stage with an auxiliary decoder and then frozen in the second stage while the actual decoder is being trained. DAE can improve the latent distribution and as a result, improve the generative performance. Our theoretical analyses are corroborated by comprehensive experiments on various models such as VQGAN and Diffusion Transformer, where our modifications yield significant improvements in sample quality with decreased model complexity.

翻译：在生成建模中，众多成功方法利用低维潜在空间，例如稳定扩散模型对编码器诱导的潜在空间进行建模，并通过配对的解码器生成图像。尽管潜在空间的选择在经验上至关重要，但确定最优选择及其识别过程仍不明确。在本研究中，我们旨在通过从模型复杂性的角度重新思考潜在空间，来阐明这一尚未充分探索的话题。我们的研究始于经典的生成对抗网络（GANs）。受GAN训练目标的启发，我们提出了潜在分布与数据分布之间的一种新型“距离”，其最小化与生成器复杂性的最小化一致。该距离的最小化器被表征为最有效利用生成器能力的最优数据依赖潜在表示。接着，我们考虑通过编码器网络参数化这种潜在分布，并提出一种名为解耦自编码器（DAE）的两阶段训练策略：在第一阶段，编码器仅与辅助解码器一起更新；在第二阶段，编码器被冻结，同时训练实际解码器。DAE能够改善潜在分布，从而提升生成性能。我们的理论分析得到了对多种模型（如VQGAN和扩散变换器）的全面实验验证，其中我们的改进在降低模型复杂性的同时显著提高了样本质量。