Solving complex visual tasks such as "Who invented the musical instrument on the right?" involves a composition of skills: understanding space, recognizing instruments, and also retrieving prior knowledge. Recent work shows promise by decomposing such tasks using a large language model (LLM) into an executable program that invokes specialized vision models. However, generated programs are error-prone: they omit necessary steps, include spurious ones, and are unable to recover when the specialized models give incorrect outputs. Moreover, they require loading multiple models, incurring high latency and computation costs. We propose Visual Program Distillation (VPD), an instruction tuning framework that produces a vision-language model (VLM) capable of solving complex visual tasks with a single forward pass. VPD distills the reasoning ability of LLMs by using them to sample multiple candidate programs, which are then executed and verified to identify a correct one. It translates each correct program into a language description of the reasoning steps, which are then distilled into a VLM. Extensive experiments show that VPD improves the VLM's ability to count, understand spatial relations, and reason compositionally. Our VPD-trained PaLI-X outperforms all prior VLMs, achieving state-of-the-art performance across complex vision tasks, including MMBench, OK-VQA, A-OKVQA, TallyQA, POPE, and Hateful Memes. An evaluation with human annotators also confirms that VPD improves model response factuality and consistency. Finally, experiments on content moderation demonstrate that VPD is also helpful for adaptation to real-world applications with limited data.

翻译：解决"右侧乐器是谁发明的？"这类复杂视觉任务需要综合多项技能：理解空间关系、识别乐器以及检索先验知识。近期研究表明，通过大语言模型（LLM）将此类任务分解为可执行的程序（该程序调用专门视觉模型）展现出良好前景。然而，生成的程序存在易错性：遗漏必要步骤、包含无关步骤，且当专用模型输出错误结果时无法恢复。此外，此类方法需要加载多个模型，导致高延迟和计算成本。我们提出视觉程序蒸馏（VPD），这是一种指令微调框架，可生成仅需单次前向传播就能解决复杂视觉任务的视觉语言模型（VLM）。VPD利用LLM采样多个候选程序，通过执行和验证筛选正确程序，从而蒸馏LLM的推理能力。它将每个正确程序转化为推理步骤的语言描述，再蒸馏到VLM中。大量实验表明，VPD显著提升了VLM在计数、空间关系理解和组合推理方面的能力。经VPD训练的PaLI-X模型在所有现有VLM中表现最优，在MMBench、OK-VQA、A-OKVQA、TallyQA、POPE和Hateful Memes等复杂视觉任务中均达到最先进水平。人类评估者的评价也证实VPD提升了模型响应的真实性和一致性。最后，内容审核实验表明，VPD在数据有限的实际应用场景中同样具有适配性。