Recent advancements enlarge the capabilities of large language models (LLMs) in zero-shot image-to-text generation and understanding by integrating multi-modal inputs. However, such success is typically limited to English scenarios due to the lack of large-scale and high-quality non-English multi-modal resources, making it extremely difficult to establish competitive counterparts in other languages. In this paper, we introduce the Ziya-Visual series, a set of bilingual large-scale vision-language models (LVLMs) designed to incorporate visual semantics into LLM for multi-modal dialogue. Composed of Ziya-Visual-Base and Ziya-Visual-Chat, our models adopt the Querying Transformer from BLIP-2, further exploring the assistance of optimization schemes such as instruction tuning, multi-stage training and low-rank adaptation module for visual-language alignment. In addition, we stimulate the understanding ability of GPT-4 in multi-modal scenarios, translating our gathered English image-text datasets into Chinese and generating instruction-response through the in-context learning method. The experiment results demonstrate that compared to the existing LVLMs, Ziya-Visual achieves competitive performance across a wide range of English-only tasks including zero-shot image-text retrieval, image captioning, and visual question answering. The evaluation leaderboard accessed by GPT-4 also indicates that our models possess satisfactory image-text understanding and generation capabilities in Chinese multi-modal scenario dialogues. Code, demo and models are available at ~\url{https://huggingface.co/IDEA-CCNL/Ziya-BLIP2-14B-Visual-v1}.

翻译：近期研究通过整合多模态输入，增强了大型语言模型在零样本图像到文本生成与理解方面的能力。然而，由于缺乏大规模、高质量的非英语多模态资源，这种成功通常局限于英语场景，使得在其他语言中建立具有竞争力的模型极为困难。本文介绍Ziya-Visual系列，这是一组双语大规模视觉语言模型，旨在将视觉语义融入大型语言模型以实现多模态对话。该系列包含Ziya-Visual-Base和Ziya-Visual-Chat，采用BLIP-2中的查询转换器，并进一步探索了指令微调、多阶段训练和低秩适配模块等优化方案对视觉-语言对齐的辅助作用。此外，我们激发GPT-4在多模态场景中的理解能力，将收集的英文图像-文本数据集翻译为中文，并通过上下文学习方法生成指令-响应对。实验结果表明，与现有视觉语言模型相比，Ziya-Visual在多种纯英语任务（包括零样本图像-文本检索、图像描述和视觉问答）上取得了具有竞争力的性能。通过GPT-4评估的排行榜也表明，我们的模型在中文多模态场景对话中具备令人满意的图像-文本理解与生成能力。代码、演示和模型可在~\url{https://huggingface.co/IDEA-CCNL/Ziya-BLIP2-14B-Visual-v1}获取。