In this work, we investigate the problem of sketch-based object localization on natural images, where given a crude hand-drawn sketch of an object, the goal is to localize all the instances of the same object on the target image. This problem proves difficult due to the abstract nature of hand-drawn sketches, variations in the style and quality of sketches, and the large domain gap existing between the sketches and the natural images. To mitigate these challenges, existing works proposed attention-based frameworks to incorporate query information into the image features. However, in these works, the query features are incorporated after the image features have already been independently learned, leading to inadequate alignment. In contrast, we propose a sketch-guided vision transformer encoder that uses cross-attention after each block of the transformer-based image encoder to learn query-conditioned image features leading to stronger alignment with the query sketch. Further, at the output of the decoder, the object and the sketch features are refined to bring the representation of relevant objects closer to the sketch query and thereby improve the localization. The proposed model also generalizes to the object categories not seen during training, as the target image features learned by our method are query-aware. Our localization framework can also utilize multiple sketch queries via a trainable novel sketch fusion strategy. The model is evaluated on the images from the public object detection benchmark, namely MS-COCO, using the sketch queries from QuickDraw! and Sketchy datasets. Compared with existing localization methods, the proposed approach gives a $6.6\%$ and $8.0\%$ improvement in mAP for seen objects using sketch queries from QuickDraw! and Sketchy datasets, respectively, and a $12.2\%$ improvement in AP@50 for large objects that are `unseen' during training.

翻译：本研究探讨了自然图像中基于草图的物体定位问题：给定一个粗略的手绘物体草图，目标是定位目标图像中所有同类物体实例。该问题因手绘草图的抽象性、草图风格与质量的多样性以及草图与自然图像之间巨大的域差异而颇具挑战性。为解决这些困难，现有工作提出基于注意力机制的框架，将查询信息融入图像特征。然而，这些方法在图像特征独立学习完成后才引入查询特征，导致对齐不充分。对此，我们提出一种草图引导的视觉Transformer编码器，在基于Transformer的图像编码器每个模块后使用交叉注意力，以学习查询条件化的图像特征，从而实现与查询草图更强的对齐。此外，在解码器输出端，我们进一步优化物体特征与草图特征，使相关物体的表征更趋近于草图查询，从而提升定位效果。由于所提方法学习到的目标图像特征具有查询感知能力，模型还能泛化至训练中未见的物体类别。我们的定位框架还可通过一种可训练的草图融合策略，利用多个草图查询。模型在MS-COCO公共目标检测基准数据集上，使用QuickDraw!和Sketchy数据集的草图查询进行评估。与现有定位方法相比，所提方法在QuickDraw!和Sketchy数据集上对已知物体的mAP分别提升6.6%和8.0%，对训练中"未见"的大物体在AP@50上提升12.2%。