Bootstrapping from pre-trained language models has been proven to be an efficient approach for building vision-language models (VLM) for tasks such as image captioning or visual question answering. However, outputs of these models rarely align with user's rationales for specific answers. In order to improve this alignment and reinforce commonsense reasons, we propose a tuning paradigm based on human interactions with machine-generated data. Our ILLUME executes the following loop: Given an image-question-answer prompt, the VLM samples multiple candidate rationales, and a human critic provides feedback via preference selection, used for fine-tuning. This loop increases the training data and gradually carves out the VLM's rationalization capabilities that are aligned with human intent. Our exhaustive experiments demonstrate that ILLUME is competitive with standard supervised finetuning while using significantly fewer training data and only requiring minimal feedback.

翻译：从预训练语言模型中引导出视觉语言模型（VLM），已被证明是构建图像描述或视觉问答等任务的一种高效方法。然而，这些模型的输出往往与用户针对特定答案的推理理由不一致。为改善这种对齐并增强常识推理，我们提出一种基于人类与机器生成数据交互的微调范式。我们的ILLUME执行如下循环：给定图像-问题-答案提示，VLM采样多个候选推理理由，人类评审员通过偏好选择提供反馈以进行微调。该循环增加了训练数据，并逐步塑造出与人类意图对齐的VLM推理能力。详尽的实验表明，ILLUME在显著减少训练数据且仅需最少反馈的情况下达到了与标准监督微调相当的性能。