This paper deals with the problem of localizing objects in image and video datasets from visual exemplars. In particular, we focus on the challenging problem of egocentric visual query localization. We first identify grave implicit biases in current query-conditioned model design and visual query datasets. Then, we directly tackle such biases at both frame and object set levels. Concretely, our method solves these issues by expanding limited annotations and dynamically dropping object proposals during training. Additionally, we propose a novel transformer-based module that allows for object-proposal set context to be considered while incorporating query information. We name our module Conditioned Contextual Transformer or CocoFormer. Our experiments show the proposed adaptations improve egocentric query detection, leading to a better visual query localization system in both 2D and 3D configurations. Thus, we are able to improve frame-level detection performance from 26.28% to 31.26 in AP, which correspondingly improves the VQ2D and VQ3D localization scores by significant margins. Our improved context-aware query object detector ranked first and second in the VQ2D and VQ3D tasks in the 2nd Ego4D challenge. In addition to this, we showcase the relevance of our proposed model in the Few-Shot Detection (FSD) task, where we also achieve SOTA results. Our code is available at https://github.com/facebookresearch/vq2d_cvpr.

翻译：本文研究基于视觉示例在图像和视频数据集中定位目标的问题，具体聚焦于具有挑战性的自我中心视觉查询定位任务。我们首先指出现有查询条件模型设计与视觉查询数据集中存在的严重隐性偏差，继而从帧级和目标集层面直接解决这些偏差问题。具体而言，我们的方法通过扩展有限标注和动态丢弃训练过程中的目标提案来消除上述偏差。此外，我们提出了一种基于Transformer的新型模块，该模块可在融合查询信息的同时考虑目标提案集的上下文特征，并将其命名为条件上下文Transformer（CocoFormer）。实验表明，所提出的改进方法提升了自我中心查询检测性能，在2D和3D配置下均实现了更优的视觉查询定位系统。最终，我们将帧级检测性能从26.28% AP提升至31.26% AP，并显著提高了VQ2D与VQ3D定位评分。在第二届Ego4D挑战赛中，我们提出的上下文感知查询目标检测器分别获得VQ2D与VQ3D任务第一名及第二名。此外，我们在小样本检测（FSD）任务中验证了所提模型的有效性，并取得了当前最优结果。代码开源地址：https://github.com/facebookresearch/vq2d_cvpr。