Visual-Semantic Embedding (VSE) networks can help search engines better understand the meaning behind visual content and associate it with relevant textual information, leading to more accurate search results. VSE networks can be used in cross-modal search engines to embed image and textual descriptions in a shared space, enabling image-to-text and text-to-image retrieval tasks. However, the full potential of VSE networks for search engines has yet to be fully explored. This paper presents Boon, a novel cross-modal search engine that combines two state-of-the-art networks: the GPT-3.5-turbo large language model, and the VSE network VITR (VIsion Transformers with Relation-focused learning) to enhance the engine's capabilities in extracting and reasoning with regional relationships in images. VITR employs encoders from CLIP that were trained with 400 million image-description pairs and it was fine-turned on the RefCOCOg dataset. Boon's neural-based components serve as its main functionalities: 1) a 'cross-modal search engine' that enables end-users to perform image-to-text and text-to-image retrieval. 2) a 'multi-lingual conversational AI' component that enables the end-user to converse about one or more images selected by the end-user. Such a feature makes the search engine accessible to a wide audience, including those with visual impairments. 3) Boon is multi-lingual and can take queries and handle conversations about images in multiple languages. Boon was implemented using the Django and PyTorch frameworks. The interface and capabilities of the Boon search engine are demonstrated using the RefCOCOg dataset, and the engine's ability to search for multimedia through the web is facilitated by Google's API.

翻译：视觉语义嵌入（VSE）网络能够帮助搜索引擎更深入地理解视觉内容背后的含义，并将其与相关文本信息关联，从而提升搜索结果的准确性。VSE网络可用于跨模态搜索引擎，在共享空间中嵌入图像和文本描述，支持图像到文本及文本到图像的检索任务。然而，VSE网络在搜索引擎中的潜力尚未被充分挖掘。本文提出Boon——一种新颖的跨模态搜索引擎，它结合了两种最先进的网络：GPT-3.5-turbo大语言模型和VSE网络VITR（基于关系学习的视觉变换器），以增强引擎在提取和推理图像区域关系方面的能力。VITR采用CLIP的编码器（该编码器基于4亿个图像-描述对训练），并在RefCOCOg数据集上进行了微调。Boon的神经组件构成其核心功能：1) “跨模态搜索引擎”使终端用户能够实现图像到文本及文本到图像的检索；2) “多语言对话式AI”组件允许终端用户就所选择的一张或多张图像进行对话，这一特性使搜索引擎可惠及广泛用户群体，包括视障人士；3) Boon支持多语言，能接受多语言查询并处理关于图像的多语言对话。Boon基于Django和PyTorch框架实现。本文通过RefCOCOg数据集展示Boon搜索引擎的界面与能力，并借助Google API实现其通过网页搜索多媒体的功能。