Diffusion-based models for text-to-image generation have gained immense popularity due to recent advancements in efficiency, accessibility, and quality. Although it is becoming increasingly feasible to perform inference with these systems using consumer-grade GPUs, training them from scratch still requires access to large datasets and significant computational resources. In the case of medical image generation, the availability of large, publicly accessible datasets that include text reports is limited due to legal and ethical concerns. While training a diffusion model on a private dataset may address this issue, it is not always feasible for institutions lacking the necessary computational resources. This work demonstrates that pre-trained Stable Diffusion models, originally trained on natural images, can be adapted to various medical imaging modalities by training text embeddings with textual inversion. In this study, we conducted experiments using medical datasets comprising only 100 samples from three medical modalities. Embeddings were trained in a matter of hours, while still retaining diagnostic relevance in image generation. Experiments were designed to achieve several objectives. Firstly, we fine-tuned the training and inference processes of textual inversion, revealing that larger embeddings and more examples are required. Secondly, we validated our approach by demonstrating a 2\% increase in the diagnostic accuracy (AUC) for detecting prostate cancer on MRI, which is a challenging multi-modal imaging modality, from 0.78 to 0.80. Thirdly, we performed simulations by interpolating between healthy and diseased states, combining multiple pathologies, and inpainting to show embedding flexibility and control of disease appearance. Finally, the embeddings trained in this study are small (less than 1 MB), which facilitates easy sharing of medical data with reduced privacy concerns.

翻译：基于扩散的文本到图像生成模型因近期在效率、可访问性和质量方面的进步而广受青睐。尽管使用消费级GPU进行推理日益可行，但从零开始训练这些模型仍需访问大规模数据集和大量计算资源。在医学图像生成领域，由于法律和伦理问题，包含文本报告的大规模公开数据集的可获取性十分有限。虽然在私有数据集上训练扩散模型可解决这一问题，但对于缺乏必要计算资源的机构而言，这并不总是可行的。本研究表明，通过文本反演训练文本嵌入，可将原本基于自然图像训练的预训练Stable Diffusion模型适配到多种医学成像模态。在本研究中，我们使用了来自三种医学模态、仅含100个样本的医学数据集进行实验。嵌入训练耗时仅数小时，同时仍能在图像生成中保留诊断相关性。实验旨在实现多个目标。首先，我们微调了文本反演的训练与推理过程，发现需要更大的嵌入和更多样本。其次，我们通过将前列腺癌MRI检测（一种具有挑战性的多模态成像模态）的诊断准确性指标（AUC）从0.78提升至0.80（提高2%），验证了所提出方法的有效性。第三，我们通过在健康与疾病状态之间插值、组合多种病理特征以及图像修复进行模拟，展示了嵌入的灵活性和对疾病外观的控制能力。最后，本研究中训练的嵌入体积极小（小于1 MB），便于在降低隐私问题的前提下轻松共享医学数据。