Recent research arXiv:2410.15027 has explored the use of diffusion transformers (DiTs) for task-agnostic image generation by simply concatenating attention tokens across images. However, despite substantial computational resources, the fidelity of the generated images remains suboptimal. In this study, we reevaluate and streamline this framework by hypothesizing that text-to-image DiTs inherently possess in-context generation capabilities, requiring only minimal tuning to activate them. Through diverse task experiments, we qualitatively demonstrate that existing text-to-image DiTs can effectively perform in-context generation without any tuning. Building on this insight, we propose a remarkably simple pipeline to leverage the in-context abilities of DiTs: (1) concatenate images instead of tokens, (2) perform joint captioning of multiple images, and (3) apply task-specific LoRA tuning using small datasets (e.g., $20\sim 100$ samples) instead of full-parameter tuning with large datasets. We name our models In-Context LoRA (IC-LoRA). This approach requires no modifications to the original DiT models, only changes to the training data. Remarkably, our pipeline generates high-fidelity image sets that better adhere to prompts. While task-specific in terms of tuning data, our framework remains task-agnostic in architecture and pipeline, offering a powerful tool for the community and providing valuable insights for further research on product-level task-agnostic generation systems. We release our code, data, and models at https://github.com/ali-vilab/In-Context-LoRA

翻译：近期研究arXiv:2410.15027探索了通过简单拼接图像间注意力标记来实现任务无关图像生成的扩散Transformer（DiTs）。然而，尽管投入了大量计算资源，生成图像的保真度仍不理想。本研究重新评估并简化了该框架，提出以下假设：文本到图像DiT本质上具备上下文生成能力，仅需极少量调优即可激活。通过多任务实验，我们定性证明了现有文本到图像DiT无需任何调优即可有效执行上下文生成。基于此发现，我们提出了一个极其简洁的流程来利用DiT的上下文能力：（1）拼接图像而非标记，（2）执行多图像联合描述生成，（3）使用小规模数据集（例如$20\sim 100$个样本）进行任务特定的LoRA调优，而非采用大规模数据集的全参数调优。我们将该模型命名为上下文LoRA（IC-LoRA）。该方法无需修改原始DiT模型，仅需调整训练数据。值得注意的是，我们的流程能生成更贴合提示的高保真图像集。虽然在调优数据层面具有任务特定性，但我们的框架在架构和流程上仍保持任务无关性，为学界提供了强大工具，并为产品级任务无关生成系统的进一步研究提供了宝贵见解。我们在https://github.com/ali-vilab/In-Context-LoRA 发布了代码、数据与模型。