Recent research on Large Language Models (LLMs) has led to remarkable advancements in general NLP AI assistants. Some studies have further explored the use of LLMs for planning and invoking models or APIs to address more general multi-modal user queries. Despite this progress, complex visual-based tasks still remain challenging due to the diverse nature of visual tasks. This diversity is reflected in two aspects: 1) Reasoning paths. For many real-life applications, it is hard to accurately decompose a query simply by examining the query itself. Planning based on the specific visual content and the results of each step is usually required. 2) Flexible inputs and intermediate results. Input forms could be flexible for in-the-wild cases, and involves not only a single image or video but a mixture of videos and images, e.g., a user-view image with some reference videos. Besides, a complex reasoning process will also generate diverse multimodal intermediate results, e.g., video narrations, segmented video clips, etc. To address such general cases, we propose a multi-modal AI assistant, AssistGPT, with an interleaved code and language reasoning approach called Plan, Execute, Inspect, and Learn (PEIL) to integrate LLMs with various tools. Specifically, the Planner is capable of using natural language to plan which tool in Executor should do next based on the current reasoning progress. Inspector is an efficient memory manager to assist the Planner to feed proper visual information into a specific tool. Finally, since the entire reasoning process is complex and flexible, a Learner is designed to enable the model to autonomously explore and discover the optimal solution. We conducted experiments on A-OKVQA and NExT-QA benchmarks, achieving state-of-the-art results. Moreover, showcases demonstrate the ability of our system to handle questions far more complex than those found in the benchmarks.

翻译：论文摘要：近年来，针对大型语言模型（LLMs）的研究在通用自然语言处理人工智能助手领域取得了显著进展。部分研究进一步探索利用LLMs进行规划并调用模型或应用程序接口（API），以应对更通用的多模态用户查询。然而，由于视觉任务的多样性，复杂的视觉基础任务仍具挑战性。这种多样性体现在两方面：1）推理路径。在实际应用中，仅通过分析查询本身往往难以准确分解任务，通常需要依据具体视觉内容及每一步的执行结果进行规划。2）灵活输入与中间结果。在开放场景下，输入形式可能具有灵活性，不仅涉及单张图像或视频，还可能包含视频与图像的混合输入，例如用户视角的图像配合若干参考视频。此外，复杂的推理过程还会生成多样化的多模态中间结果，如视频旁白、分割后的视频片段等。为解决此类通用场景，我们提出了一种多模态人工智能助手AssistGPT，采用名为“规划、执行、检查与学习”（PEIL）的交错代码与语言推理方法，将LLMs与多种工具集成。具体而言，规划器（Planner）能够根据当前推理进度，使用自然语言规划执行器（Executor）中下一步应调用的工具。检查器（Inspector）作为高效的内存管理器，协助规划器将正确的视觉信息输入特定工具。最后，鉴于整个推理过程的复杂性与灵活性，我们设计了学习者（Learner），使模型能够自主探索并发现最优解。我们在A-OKVQA和NExT-QA基准测试上进行了实验，取得了最先进的性能。此外，实际案例展示了我们的系统在处理远超基准测试复杂度的查询时具备的能力。