Beyond scaling base models with more data or parameters, fine-tuned adapters provide an alternative way to generate high fidelity, custom images at reduced costs. As such, adapters have been widely adopted by open-source communities, accumulating a database of over 100K adapters-most of which are highly customized with insufficient descriptions. This paper explores the problem of matching the prompt to a set of relevant adapters, built on recent work that highlight the performance gains of composing adapters. We introduce Stylus, which efficiently selects and automatically composes task-specific adapters based on a prompt's keywords. Stylus outlines a three-stage approach that first summarizes adapters with improved descriptions and embeddings, retrieves relevant adapters, and then further assembles adapters based on prompts' keywords by checking how well they fit the prompt. To evaluate Stylus, we developed StylusDocs, a curated dataset featuring 75K adapters with pre-computed adapter embeddings. In our evaluation on popular Stable Diffusion checkpoints, Stylus achieves greater CLIP-FID Pareto efficiency and is twice as preferred, with humans and multimodal models as evaluators, over the base model. See stylus-diffusion.github.io for more.

翻译：除通过增加数据或参数来扩展基础模型外，微调适配器提供了一种以更低成本生成高保真定制图像的替代方案。因此，适配器已被开源社区广泛采用，积累了超过10万个适配器的数据库——其中大部分高度定制且缺乏充分描述。本文基于近期强调适配器组合性能提升的研究，探索了如何将提示词与一组相关适配器进行匹配的问题。我们提出Stylus方法，能够根据提示词的关键词高效选择并自动组合任务特定适配器。Stylus采用三阶段策略：首先通过改进的描述与嵌入对适配器进行摘要总结，其次检索相关适配器，最后基于提示词的关键词通过评估适配器与提示词的契合度进行组合。为评估Stylus，我们构建了包含75,000个适配器及预计算适配器嵌入的StylusDocs数据集。在主流Stable Diffusion检查点上的评估表明，Stylus实现了更高的CLIP-FID帕累托效率，且以人类与多模态模型作为评估者时，其偏好度是基础模型的两倍。更多信息请见stylus-diffusion.github.io。