Utilizing large language models (LLMs) to compose off-the-shelf visual tools represents a promising avenue of research for developing robust visual assistants capable of addressing diverse visual tasks. However, these methods often overlook the potential for continual learning, typically by freezing the utilized tools, thus limiting their adaptation to environments requiring new knowledge. To tackle this challenge, we propose CLOVA, a Closed-Loop Visual Assistant, which operates within a framework encompassing inference, reflection, and learning phases. During the inference phase, LLMs generate programs and execute corresponding tools to complete assigned tasks. In the reflection phase, a multimodal global-local reflection scheme analyzes human feedback to determine which tools require updating. Lastly, the learning phase employs three flexible approaches to automatically gather training data and introduces a novel prompt tuning scheme to update the tools, allowing CLOVA to efficiently acquire new knowledge. Experimental findings demonstrate that CLOVA surpasses existing tool-usage methods by 5% in visual question answering and multiple-image reasoning, by 10% in knowledge tagging, and by 20% in image editing. These results underscore the significance of the continual learning capability in general visual assistants.

翻译：摘要：利用大型语言模型（LLMs）编排现成的视觉工具，是开发能够处理多样化视觉任务的稳健视觉助手的一个有前景的研究方向。然而，这些方法常忽视持续学习的潜力，通常采用冻结所用工具的方式，从而限制了其对需要新知识环境的适应能力。为应对这一挑战，我们提出CLOVA，一种闭环视觉助手，其运行框架包含推理、反思与学习三个阶段。在推理阶段，LLMs生成程序并执行相应工具以完成指定任务；在反思阶段，一种多模态全局-局部反思方案分析人类反馈，以判断哪些工具需要更新；最后，学习阶段采用三种灵活方法自动收集训练数据，并引入一种新颖的提示调优方案来更新工具，使CLOVA能够高效获取新知识。实验结果表明，CLOVA在视觉问答与多图像推理任务中性能超越现有工具使用方法5%，在知识标注任务中提升10%，在图像编辑任务中提升20%。这些结果突显了通用视觉助手中持续学习能力的重要性。