Multi-modal visual understanding of images with prompts involves using various visual and textual cues to enhance the semantic understanding of images. This approach combines both vision and language processing to generate more accurate predictions and recognition of images. By utilizing prompt-based techniques, models can learn to focus on certain features of an image to extract useful information for downstream tasks. Additionally, multi-modal understanding can improve upon single modality models by providing more robust representations of images. Overall, the combination of visual and textual information is a promising area of research for advancing image recognition and understanding. In this paper we will try an amount of prompt design methods and propose a new method for better extraction of semantic information

翻译：多模态视觉理解结合视觉与文本线索来增强图像的语义理解。该方法融合视觉与语言处理技术，以生成更准确的图像预测与识别结果。通过利用基于提示的技术，模型能够聚焦于图像的特定特征，从而提取对下游任务有用的信息。此外，相比于单模态模型，多模态理解通过提供更鲁棒的图像表征来提升性能。总体而言，视觉与文本信息的结合是推动图像识别与理解研究的前沿领域。本文尝试了多种提示设计方法，并提出了一种更优的语义信息提取新方法。