Realistic image super-resolution (Real-ISR) aims to reproduce perceptually realistic image details from a low-quality input. The commonly used adversarial training based Real-ISR methods often introduce unnatural visual artifacts and fail to generate realistic textures for natural scene images. The recently developed generative stable diffusion models provide a potential solution to Real-ISR with pre-learned strong image priors. However, the existing methods along this line either fail to keep faithful pixel-wise image structures or resort to extra skipped connections to reproduce details, which requires additional training in image space and limits their extension to other related tasks in latent space such as image stylization. In this work, we propose a pixel-aware stable diffusion (PASD) network to achieve robust Real-ISR as well as personalized stylization. In specific, a pixel-aware cross attention module is introduced to enable diffusion models perceiving image local structures in pixel-wise level, while a degradation removal module is used to extract degradation insensitive features to guide the diffusion process together with image high level information. By simply replacing the base diffusion model with a personalized one, our method can generate diverse stylized images without the need to collect pairwise training data. PASD can be easily integrated into existing diffusion models such as Stable Diffusion. Experiments on Real-ISR and personalized stylization demonstrate the effectiveness of our proposed approach. The source code and models can be found at \url{https://github.com/yangxy/PASD}.

翻译：逼真图像超分辨率（Real-ISR）旨在从低质量输入中重现感知上逼真的图像细节。常用的基于对抗训练的Real-ISR方法通常引入不自然的视觉伪影，且难以对自然场景图像生成逼真的纹理。近期发展的生成式稳定扩散模型凭借预训练的强图像先验，为Real-ISR提供了潜在解决方案。然而，现有方法要么无法保持像素级图像结构的保真度，要么依赖额外的跳跃连接来重建细节，这需要在图像空间中进行额外训练，并限制了其在潜在空间其他相关任务（如图像风格化）中的扩展。本研究提出一种像素感知稳定扩散（PASD）网络，以实现鲁棒的Real-ISR及个性化风格化。具体而言，引入像素感知交叉注意力模块使扩散模型能够在像素级感知图像局部结构，同时利用退化移除模块提取对退化不敏感的特征，结合图像高层信息引导扩散过程。通过简单地将基础扩散模型替换为个性化模型，我们的方法无需收集配对训练数据即可生成多样化的风格化图像。PASD可轻松集成到现有扩散模型（如Stable Diffusion）中。在Real-ISR和个性化风格化上的实验证明了所提方法的有效性。源代码与模型见\url{https://github.com/yangxy/PASD}。