We explore the "hidden" ability of large-scale pre-trained image generation models, such as Stable Diffusion and Imagen, in non-visible light domains, taking Synthetic Aperture Radar (SAR) data for a case study. Due to the inherent challenges in capturing satellite data, acquiring ample SAR training samples is infeasible. For instance, for a particular category of ship in the open sea, we can collect only few-shot SAR images which are too limited to derive effective ship recognition models. If large-scale models pre-trained with regular images can be adapted to generating novel SAR images, the problem is solved. In preliminary study, we found that fine-tuning these models with few-shot SAR images is not working, as the models can not capture the two primary differences between SAR and regular images: structure and modality. To address this, we propose a 2-stage low-rank adaptation method, and we call it 2LoRA. In the first stage, the model is adapted using aerial-view regular image data (whose structure matches SAR), followed by the second stage where the base model from the first stage is further adapted using SAR modality data. Particularly in the second stage, we introduce a novel prototype LoRA (pLoRA), as an improved version of 2LoRA, to resolve the class imbalance problem in SAR datasets. For evaluation, we employ the resulting generation model to synthesize additional SAR data. This augmentation, when integrated into the training process of SAR classification as well as segmentation models, yields notably improved performance for minor classes

翻译：我们探索了大规模预训练图像生成模型（如Stable Diffusion和Imagen）在非可见光领域的“隐藏”能力，并以合成孔径雷达（SAR）数据为案例进行研究。由于卫星数据捕获固有的挑战，获取充足的SAR训练样本不可行。例如，对于公海中的特定船只类别，我们只能收集到少量SAR图像，这些图像过于有限，无法导出有效的船只识别模型。如果能够将使用常规图像预训练的大规模模型调整为生成新型SAR图像，问题将得到解决。初步研究发现，使用少量SAR图像对这些模型进行微调效果不佳，因为这些模型无法捕捉SAR图像与常规图像之间的两个主要差异：结构和模态。为解决此问题，我们提出了一种两阶段低秩适应方法，称为2LoRA。在第一阶段，使用航拍常规图像数据（其结构与SAR匹配）对模型进行适应；随后在第二阶段，利用SAR模态数据对第一阶段的基础模型进一步适应。特别是在第二阶段，我们引入了一种新型原型LoRA（pLoRA）作为2LoRA的改进版本，以解决SAR数据集中的类别不平衡问题。为进行评估，我们使用生成的模型合成额外的SAR数据。将这种数据增强集成到SAR分类和分割模型的训练过程中后，显著提升了少数类别的性能。