Recent works have shown that Large Language Models (LLMs) could empower traditional neuro-symbolic models via programming capabilities to translate language into module descriptions, thus achieving strong visual reasoning results while maintaining the model's transparency and efficiency. However, these models usually exhaustively generate the entire code snippet given each new instance of a task, which is extremely ineffective. We propose generative neuro-symbolic visual reasoning by growing and reusing modules. Specifically, our model consists of three unique stages, module initialization, module generation, and module execution. First, given a vision-language task, we adopt LLMs to examine whether we could reuse and grow over established modules to handle this new task. If not, we initialize a new module needed by the task and specify the inputs and outputs of this new module. After that, the new module is created by querying LLMs to generate corresponding code snippets that match the requirements. In order to get a better sense of the new module's ability, we treat few-shot training examples as test cases to see if our new module could pass these cases. If yes, the new module is added to the module library for future reuse. Finally, we evaluate the performance of our model on the testing set by executing the parsed programs with the newly made visual modules to get the results. We find the proposed model possesses several advantages. First, it performs competitively on standard tasks like visual question answering and referring expression comprehension; Second, the modules learned from one task can be seamlessly transferred to new tasks; Last but not least, it is able to adapt to new visual reasoning tasks by observing a few training examples and reusing modules.

翻译：近期研究表明，大语言模型（LLMs）可通过编程能力赋能传统神经符号模型，将语言转化为模块描述，从而在保持模型透明性与高效性的同时实现强大的视觉推理效果。然而，现有模型通常针对每个新任务实例完整生成整个代码片段，导致效率极低。我们提出通过模块生成与复用实现生成式神经符号视觉推理。具体而言，本模型包含三个独特阶段：模块初始化、模块生成与模块执行。首先，给定视觉-语言任务，我们利用LLMs评估是否可复用并扩展已有模块以处理新任务。若不可行，则初始化任务所需的新模块，并指定其输入输出。随后，通过查询LLMs生成符合要求的相应代码片段来创建新模块。为更准确评估新模块能力，我们将少样本训练实例作为测试用例，判断新模块能否通过验证。若通过，则将新模块添加至模块库以供未来复用。最终，我们通过执行包含新构建视觉模块的解析程序，在测试集上评估模型性能。研究发现本模型具有三重优势：其一，在视觉问答、指代表达理解等标准任务上表现优异；其二，从单一任务习得的模块可无缝迁移至新任务；其三，通过观察少量训练样例并复用模块，模型即可适应新的视觉推理任务。