Most existing cross-modal retrieval methods employ two-stream encoders with different architectures for images and texts, \textit{e.g.}, CNN for images and RNN/Transformer for texts. Such discrepancy in architectures may induce different semantic distribution spaces and limit the interactions between images and texts, and further result in inferior alignment between images and texts. To fill this research gap, inspired by recent advances of Transformers in vision tasks, we propose to unify the encoder architectures with Transformers for both modalities. Specifically, we design a cross-modal retrieval framework purely based on two-stream Transformers, dubbed \textbf{Hierarchical Alignment Transformers (HAT)}, which consists of an image Transformer, a text Transformer, and a hierarchical alignment module. With such identical architectures, the encoders could produce representations with more similar characteristics for images and texts, and make the interactions and alignments between them much easier. Besides, to leverage the rich semantics, we devise a hierarchical alignment scheme to explore multi-level correspondences of different layers between images and texts. To evaluate the effectiveness of the proposed HAT, we conduct extensive experiments on two benchmark datasets, MSCOCO and Flickr30K. Experimental results demonstrate that HAT outperforms SOTA baselines by a large margin. Specifically, on two key tasks, \textit{i.e.}, image-to-text and text-to-image retrieval, HAT achieves 7.6\% and 16.7\% relative score improvement of Recall@1 on MSCOCO, and 4.4\% and 11.6\% on Flickr30k respectively. The code is available at \url{https://github.com/LuminosityX/HAT}.

翻译：现有跨模态检索方法大多采用不同架构的双流编码器处理图像和文本（如图像使用CNN，文本使用RNN/Transformer）。这种架构差异可能导致语义分布空间的不一致，限制图像与文本间的交互，进而造成模态间对齐效果不佳。为填补这一研究空白，受Transformer在视觉任务中最新进展的启发，我们提出采用Transformer统一两种模态的编码器架构。具体而言，我们设计了一个基于双流Transformer的跨模态检索框架，名为**层次对齐Transformer（HAT）**，包含图像Transformer、文本Transformer及层次对齐模块。通过采用相同架构，编码器可为图像和文本生成特征更相似的表示，使模态间交互与对齐更加高效。此外，为充分利用丰富语义，我们设计了层次对齐方案，探索图像与文本不同层间的多级对应关系。为评估所提HAT的有效性，我们在MSCOCO和Flickr30K两个基准数据集上进行了大量实验。结果表明，HAT显著优于现有最优基线方法。具体而言，在图像到文本检索和文本到图像检索两项关键任务中，HAT的Recall@1在MSCOCO上分别提升7.6%和16.7%，在Flickr30K上分别提升4.4%和11.6%。代码已开源至\url{https://github.com/LuminosityX/HAT}。