Diffusion models have attained remarkable breakthroughs in the real-world super-resolution (SR) task, albeit at slow inference and high demand on devices. To accelerate inference, recent works like GenDR adopt step distillation to minimize the step number to one. However, the memory boundary still restricts the maximum processing size, necessitating tile-by-tile restoration of high-resolution images. Through profiling the pipeline, we pinpoint that the variational auto-encoder (VAE) is the bottleneck of latency and memory. To completely solve the problem, we leverage pixel-(un)shuffle operations to eliminate the VAE, reversing the latent-based GenDR to pixel-space GenDR-Pix. However, upscale with x8 pixelshuffle may induce artifacts of repeated patterns. To alleviate the distortion, we propose a multi-stage adversarial distillation to progressively remove the encoder and decoder. Specifically, we utilize generative features from the previous stage models to guide adversarial discrimination. Moreover, we propose random padding to augment generative features and avoid discriminator collapse. We also introduce a masked Fourier space loss to penalize the outliers of amplitude. To improve inference performance, we empirically integrate a padding-based self-ensemble with classifier-free guidance to improve inference scaling. Experimental results show that GenDR-Pix performs 2.8x acceleration and 60% memory-saving compared to GenDR with negligible visual degradation, surpassing other one-step diffusion SR. Against all odds, GenDR-Pix can restore 4K image in only 1 second and 6GB.

翻译：扩散模型在真实世界超分辨率任务中取得了显著突破，但其推理速度缓慢且对设备要求较高。为加速推理，近期工作如GenDR采用步数蒸馏将步骤数最小化为一步。然而，内存限制仍制约着最大处理尺寸，导致高分辨率图像需通过分块方式进行恢复。通过对流程进行性能剖析，我们确定变分自编码器是延迟与内存的瓶颈所在。为彻底解决该问题，我们利用像素重排与反重排操作消除VAE，将基于潜空间的GenDR逆转为像素空间的GenDR-Pix。然而，使用8倍像素重排进行上采样可能引发重复图案伪影。为减轻失真，我们提出多阶段对抗蒸馏以逐步移除编码器和解码器。具体而言，我们利用前一阶段模型的生成特征来引导对抗判别过程。此外，我们提出随机填充策略以增强生成特征并避免判别器坍缩。同时引入掩码傅里叶空间损失以惩罚幅值异常点。为提升推理性能，我们通过经验性方法将基于填充的自集成与无分类器引导相结合以改进推理缩放。实验结果表明，与GenDR相比，GenDR-Pix在视觉质量几乎无损的情况下实现了2.8倍加速与60%内存节省，性能超越其他单步扩散超分辨率方法。尤为突出的是，GenDR-Pix仅需1秒和6GB内存即可完成4K图像恢复。