Visual-Semantic Embedding (VSE) networks can help search engines better understand the meaning behind visual content and associate it with relevant textual information, leading to more accurate search results. VSE networks can be used in cross-modal search engines to embed image and textual descriptions in a shared space, enabling image-to-text and text-to-image retrieval tasks. However, the full potential of VSE networks for search engines has yet to be fully explored. This paper presents Boon, a novel cross-modal search engine that combines two state-of-the-art networks: the GPT-3.5-turbo large language model, and the VSE network VITR (VIsion Transformers with Relation-focused learning) to enhance the engine's capabilities in extracting and reasoning with regional relationships in images. VITR employs encoders from CLIP that were trained with 400 million image-description pairs and it was fine-turned on the RefCOCOg dataset. Boon's neural-based components serve as its main functionalities: 1) a 'cross-modal search engine' that enables end-users to perform image-to-text and text-to-image retrieval. 2) a 'multi-lingual conversational AI' component that enables the end-user to converse about one or more images selected by the end-user. Such a feature makes the search engine accessible to a wide audience, including those with visual impairments. 3) Boon is multi-lingual and can take queries and handle conversations about images in multiple languages. Boon was implemented using the Django and PyTorch frameworks. The interface and capabilities of the Boon search engine are demonstrated using the RefCOCOg dataset, and the engine's ability to search for multimedia through the web is facilitated by Google's API.

翻译：视觉语义嵌入（VSE）网络能够帮助搜索引擎更深入地理解视觉内容所蕴含的意义，并将其与相关文本信息关联，从而提升检索结果的准确性。VSE网络可应用于跨模态搜索引擎，通过将图像与文本描述嵌入到同一共享空间中，实现图像到文本及文本到图像的检索任务。然而，VSE网络在搜索引擎中的全面潜力尚未被充分挖掘。本文提出Boon——一种新型跨模态搜索引擎，它融合了两种前沿网络：GPT-3.5-turbo大语言模型与VSE网络VITR（基于关系聚焦学习的视觉Transformer），以增强引擎在图像区域关系提取与推理方面的能力。VITR采用在4亿图像-描述对数据集上预训练的CLIP编码器，并在RefCOCOg数据集上进行了微调。Boon的神经组件构成了其核心功能：1）"跨模态搜索引擎"支持终端用户执行图像到文本及文本到图像的检索；2）"多语言对话式AI"组件允许终端用户围绕一张或多张选定图像进行对话，该功能使搜索引擎可服务于包括视障人士在内的广泛用户群体；3）Boon支持多语言查询与图像对话。Boon基于Django和PyTorch框架实现，其界面与检索能力通过RefCOCOg数据集进行演示，并通过谷歌API实现网络多媒体资源的搜索功能。