Sketches are highly expressive, inherently capturing subjective and fine-grained visual cues. The exploration of such innate properties of human sketches has, however, been limited to that of image retrieval. In this paper, for the first time, we cultivate the expressiveness of sketches but for the fundamental vision task of object detection. The end result is a sketch-enabled object detection framework that detects based on what \textit{you} sketch -- \textit{that} ``zebra'' (e.g., one that is eating the grass) in a herd of zebras (instance-aware detection), and only the \textit{part} (e.g., ``head" of a ``zebra") that you desire (part-aware detection). We further dictate that our model works without (i) knowing which category to expect at testing (zero-shot) and (ii) not requiring additional bounding boxes (as per fully supervised) and class labels (as per weakly supervised). Instead of devising a model from the ground up, we show an intuitive synergy between foundation models (e.g., CLIP) and existing sketch models build for sketch-based image retrieval (SBIR), which can already elegantly solve the task -- CLIP to provide model generalisation, and SBIR to bridge the (sketch$\rightarrow$photo) gap. In particular, we first perform independent prompting on both sketch and photo branches of an SBIR model to build highly generalisable sketch and photo encoders on the back of the generalisation ability of CLIP. We then devise a training paradigm to adapt the learned encoders for object detection, such that the region embeddings of detected boxes are aligned with the sketch and photo embeddings from SBIR. Evaluating our framework on standard object detection datasets like PASCAL-VOC and MS-COCO outperforms both supervised (SOD) and weakly-supervised object detectors (WSOD) on zero-shot setups. Project Page: \url{https://pinakinathc.github.io/sketch-detect}

翻译：素描具有高度表现力，天然能够捕捉主观且精细的视觉线索。然而，人类素描这种固有属性的探索此前仅限于图像检索领域。本文首次将素描的表现力应用于基础视觉任务——目标检测。最终成果是一个基于素描的目标检测框架，能够根据您所画的草图进行检测：从一群斑马中检测出"正在吃草的那只斑马"（实例感知检测），或仅检测您想要的"斑马头部"等部件（部件感知检测）。我们进一步要求模型在满足以下条件时仍能工作：（i）测试时无需预知类别（零样本）；（ii）无需额外边界框（全监督所需）和类别标签（弱监督所需）。我们并未从头设计模型，而是展示了基础模型（如CLIP）与现有素描检索模型（SBIR）之间的直觉协同——CLIP提供模型泛化能力，SBIR弥合素描与照片之间的鸿沟。具体而言，我们首先对SBIR模型的素描分支和照片分支进行独立提示学习，借助CLIP的泛化能力构建高度可泛化的素描编码器和照片编码器。随后设计训练范式，将学习到的编码器适配至目标检测任务，使检测框的区域嵌入与SBIR中的素描嵌入和照片嵌入对齐。在PASCAL-VOC和MS-COCO等标准目标检测数据集上的评估表明，本框架在零样本设置下优于全监督目标检测器（SOD）和弱监督目标检测器（WSOD）。项目页面：\url{https://pinakinathc.github.io/sketch-detect}