While deep learning based image retrieval is reported to be vulnerable to adversarial attacks, existing works are mainly on image-to-image retrieval with their attacks performed at the front end via query modification. By contrast, we present in this paper the first study about a threat that occurs at the back end of a text-to-image retrieval (T2IR) system. Our study is motivated by the fact that the image collection indexed by the system will be regularly updated due to the arrival of new images from various sources such as web crawlers and advertisers. With malicious images indexed, it is possible for an attacker to indirectly interfere with the retrieval process, letting users see certain images that are completely irrelevant w.r.t. their queries. We put this thought into practice by proposing a novel Trojan-horse attack (THA). In particular, we construct a set of Trojan-horse images by first embedding word-specific adversarial information into a QR code and then putting the code on benign advertising images. A proof-of-concept evaluation, conducted on two popular T2IR datasets (Flickr30k and MS-COCO), shows the effectiveness of the proposed THA in a white-box mode.

翻译：尽管基于深度学习的图像检索被报道易受对抗性攻击，但现有研究主要集中在图像到图像检索，其攻击通过查询修改在前端进行。相比之下，本文首次研究了文本到图像检索系统后端面临的威胁。我们的研究基于如下动机：由于网络爬虫和广告商等各类来源不断带来新图像，系统索引的图像集将定期更新。当恶意图像被索引后，攻击者有可能间接干扰检索过程，使用户看到与其查询完全无关的特定图像。我们通过提出一种新型特洛伊木马攻击将这一设想付诸实践。具体而言，我们首先将单词特定对抗信息嵌入二维码，再将二维码附加于良性广告图像上，从而构建特洛伊木马图像集。在两个主流T2IR数据集（Flickr30k和MS-COCO）上进行的概念验证评估表明，所提出的THA在白盒模式下具有有效性。