We present a novel and effective method calibrating cross-modal features for text-based person search. Our method is cost-effective and can easily retrieve specific persons with textual captions. Specifically, its architecture is only a dual-encoder and a detachable cross-modal decoder. Without extra multi-level branches or complex interaction modules as the neck following the backbone, our model makes a high-speed inference only based on the dual-encoder. Besides, our method consists of two novel losses to provide fine-grained cross-modal features. A Sew loss takes the quality of textual captions as guidance and aligns features between image and text modalities. A Masking Caption Modeling (MCM) loss uses a masked captions prediction task to establish detailed and generic relationships between textual and visual parts. We show the top results in three popular benchmarks, including CUHK-PEDES, ICFG-PEDES, and RSTPReID. In particular, our method achieves 73.81% Rank@1, 74.25% Rank@1 and 57.35% Rank@1 on them, respectively. In addition, we also validate each component of our method with extensive experiments. We hope our powerful and scalable paradigm will serve as a solid baseline and help ease future research in text-based person search.
翻译:我们提出了一种新颖且有效的方法,用于校准基于文本的行人搜索中的跨模态特征。该方法成本低廉,能够轻松地通过文本描述检索特定行人。具体而言,其架构仅由一个双编码器和一个可分离的跨模态解码器组成。无需在主干网络后添加额外的多层级分支或复杂交互模块作为颈部,我们的模型仅基于双编码器即可实现高速推理。此外,该方法包含两种新颖的损失函数,以提供细粒度的跨模态特征。一种“缝损失”(Sew loss)以文本描述的质量为指导,对齐图像与文本模态之间的特征;另一种“掩码标题建模(MCM)损失”利用掩码标题预测任务,建立文本与视觉部分之间详细且通用的关系。我们在三个主流基准数据集(包括CUHK-PEDES、ICFG-PEDES和RSTPReID)上展示了顶尖结果。具体而言,我们的方法在这些数据集上分别达到了73.81%的Rank@1、74.25%的Rank@1和57.35%的Rank@1。此外,我们还通过大量实验验证了该方法中每个组件的有效性。我们希望这一强大且可扩展的范式能够作为坚实的基线,助力未来基于文本的行人搜索研究。