Visual programs are executable code generated by large language models to address visual reasoning problems. They decompose complex questions into multiple reasoning steps and invoke specialized models for each step to solve the problems. However, these programs are prone to logic errors, with our preliminary evaluation showing that 58% of the total errors are caused by program logic errors. Debugging complex visual programs remains a major bottleneck for visual reasoning. To address this, we introduce VDebugger, a novel critic-refiner framework trained to localize and debug visual programs by tracking execution step by step. VDebugger identifies and corrects program errors leveraging detailed execution feedback, improving interpretability and accuracy. The training data is generated through an automated pipeline that injects errors into correct visual programs using a novel mask-best decoding technique. Evaluations on six datasets demonstrate VDebugger's effectiveness, showing performance improvements of up to 3.2% in downstream task accuracy. Further studies show VDebugger's ability to generalize to unseen tasks, bringing a notable improvement of 2.3% on the unseen COVR task. Code, data and models are made publicly available at https://github.com/shirley-wu/vdebugger/

翻译：视觉程序是由大型语言模型生成的用于解决视觉推理问题的可执行代码。它们将复杂问题分解为多个推理步骤，并调用专用模型逐步解决问题。然而，这些程序容易产生逻辑错误，我们的初步评估表明总错误中58%由程序逻辑错误导致。调试复杂视觉程序仍是视觉推理的主要瓶颈。为此，我们提出VDebugger——一种新颖的批评-精炼框架，通过逐步骤跟踪执行来定位和调试视觉程序。VDebugger利用详细的执行反馈识别并修正程序错误，提升可解释性与准确性。训练数据通过自动化流程生成，该流程采用新颖的掩码最优解码技术向正确视觉程序中注入错误。在六个数据集上的评估证明了VDebugger的有效性，下游任务准确率最高提升3.2%。进一步研究表明VDebugger能泛化至未见任务，在未见COVR任务上带来2.3%的显著提升。代码、数据与模型已公开于https://github.com/shirley-wu/vdebugger/