Various stuff and things in visual data possess specific traits, which can be learned by deep neural networks and are implicitly represented as the visual prior, e.g., object location and shape, in the model. Such prior potentially impacts many vision tasks. For example, in conditional image synthesis, spatial conditions failing to adhere to the prior can result in visually inaccurate synthetic results. This work aims to explicitly learn the visual prior and enable the customization of sampling. Inspired by advances in language modeling, we propose to learn Visual prior via Generative Pre-Training, dubbed VisorGPT. By discretizing visual locations of objects, e.g., bounding boxes, human pose, and instance masks, into sequences, VisorGPT can model visual prior through likelihood maximization. Besides, prompt engineering is investigated to unify various visual locations and enable customized sampling of sequential outputs from the learned prior. Experimental results demonstrate that VisorGPT can effectively model the visual prior, which can be employed for many vision tasks, such as customizing accurate human pose for conditional image synthesis models like ControlNet. Code will be released at https://github.com/Sierkinhane/VisorGPT.

翻译：视觉数据中的各种物体和材质具有特定属性，这些属性可由深度神经网络学习并隐式表示为模型中的视觉先验（如目标位置与形状）。此类先验知识对众多视觉任务具有潜在影响，例如在条件图像合成中，不符合先验的空间条件可能导致视觉上不准确的合成结果。本工作旨在显式学习视觉先验并支持采样的定制化。受语言建模进展的启发，我们提出通过生成式预训练学习视觉先验，称为VisorGPT。通过将目标的视觉位置（如边界框、人体姿态、实例掩码）离散化为序列，VisorGPT可通过最大化似然对视觉先验进行建模。此外，我们研究了提示工程以统一各类视觉位置，并支持从学习到的先验中对序列输出进行定制化采样。实验结果表明，VisorGPT能有效建模视觉先验，可应用于众多视觉任务，例如为ControlNet等条件图像合成模型定制精确的人体姿态。代码将发布在https://github.com/Sierkinhane/VisorGPT。