While deep learning based image retrieval is reported to be vulnerable to adversarial attacks, existing works are mainly on image-to-image retrieval with their attacks performed at the front end via query modification. By contrast, we present in this paper the first study about a threat that occurs at the back end of a text-to-image retrieval (T2IR) system. Our study is motivated by the fact that the image collection indexed by the system will be regularly updated due to the arrival of new images from various sources such as web crawlers and advertisers. With malicious images indexed, it is possible for an attacker to indirectly interfere with the retrieval process, letting users see certain images that are completely irrelevant w.r.t. their queries. We put this thought into practice by proposing a novel Trojan-horse attack (THA). In particular, we construct a set of Trojan-horse images by first embedding word-specific adversarial information into a QR code and then putting the code on benign advertising images. A proof-of-concept evaluation, conducted on two popular T2IR datasets (Flickr30k and MS-COCO), shows the effectiveness of the proposed THA in a white-box mode.

翻译：尽管基于深度学习的图像检索已被证实易受对抗性攻击影响，现有研究主要聚焦于图像到图像检索，且攻击多通过前端查询修改实现。与此不同，本文首次研究了一种针对文本到图像检索（T2IR）系统后端的安全威胁。我们的研究源于以下事实：由于网络爬虫和广告商等多种来源的新图像不断涌入，系统索引的图像集合需定期更新。当恶意图像被索引后，攻击者可能间接干扰检索过程，使用户看到与查询内容完全无关的特定图像。我们将这一设想付诸实践，提出了一种新型木马攻击（THA）：首先将单词特异性对抗信息嵌入二维码，再将二维码叠加于良性广告图像上，以此构建一组木马图像。基于两个主流T2IR数据集（Flickr30k和MS-COCO）的概念验证评估表明，所提出的THA在白盒模式下具有有效性。