The recent success of ChatGPT and GPT-4 has drawn widespread attention to multimodal dialogue systems. However, there is a lack of datasets in the academic community that can effectively evaluate the multimodal generation capabilities of Visual Language Models (VLMs) in textual-visual chat tasks. In this paper, we address this gap by introducing two novel multimodal datasets: the synthetic CLEVR-ATVC dataset (620K) and the manually pictured Fruit-ATVC dataset (50K). These datasets incorporate both visual and text-based inputs and outputs. Furthermore, to facilitate the accountability of multimodal systems in rejecting human requests, similar to language-based ChatGPT conversations, we introduce specific rules as supervisory signals within the datasets. This allows the trained VLM to provide a yes or no answer after engaging in visual and textual reasoning, accompanied by a language explanation to clarify the reasons behind the inability to execute the given human instruction. Our proposed method involves a two-stage training procedure, which includes training the image auto-encoder and the auto-regressive transformer from scratch. The first stage employs a discrete variational autoencoder (dVAE) to compress each image into concise tokens, which are then combined with text tokens into a single data stream. This stream is subsequently fed into the decoder-based transformer to generate visual re-creations and textual feedback in the second stage. We conduct comprehensive analyses of experimental results, focusing on re-created image quality, answer accuracy, and the model's behavior when faced with uncertainty and imperfect user queries. Through our explorations and findings, we aim to contribute valuable insights into the accountability of textual-visual generative models.

翻译：ChatGPT和GPT-4的最新成功引发了学术界对多模态对话系统的广泛关注。然而，当前学术社区仍缺乏能有效评估视觉语言模型（VLM）在文本-视觉对话任务中多模态生成能力的数据集。本文通过引入两个新型多模态数据集——合成数据CLEVR-ATVC（62万样本）和人工标注图像数据Fruit-ATVC（5万样本）——填补了这一空白。这两个数据集同时包含视觉与文本模态的输入和输出。此外，为实现多模态系统在拒绝用户请求时具有可解释性（类似基于语言的ChatGPT对话），我们在数据集中引入特定规则作为监督信号，使训练后的VLM在执行视觉与文本推理后能输出"是/否"答案，并通过语言解释阐明无法执行给定指令的原因。我们提出的方法采用两阶段训练流程：从零开始训练图像自编码器和自回归Transformer。第一阶段使用离散变分自编码器（dVAE）将每张图像压缩为紧凑的离散令牌，并与文本令牌融合成统一数据流；第二阶段将该数据流输入解码器型Transformer，生成视觉再创作结果与文本反馈。我们系统分析了实验结果，重点关注再创作图像质量、答案准确性，以及模型面对不确定性和不完美用户查询时的行为表现。通过探索与发现，本研究旨在为提升文本-视觉生成模型的可解释性提供重要洞见。