Humans tend to decompose a sentence into different parts like \textsc{sth do sth at someplace} and then fill each part with certain content. Inspired by this, we follow the \textit{principle of modular design} to propose a novel image captioner: learning to Collocate Visual-Linguistic Neural Modules (CVLNM). Unlike the \re{widely used} neural module networks in VQA, where the language (\ie, question) is fully observable, \re{the task of collocating visual-linguistic modules is more challenging.} This is because the language is only partially observable, for which we need to dynamically collocate the modules during the process of image captioning. To sum up, we make the following technical contributions to design and train our CVLNM: 1) \textit{distinguishable module design} -- \re{four modules in the encoder} including one linguistic module for function words and three visual modules for different content words (\ie, noun, adjective, and verb) and another linguistic one in the decoder for commonsense reasoning, 2) a self-attention based \textit{module controller} for robustifying the visual reasoning, 3) a part-of-speech based \textit{syntax loss} imposed on the module controller for further regularizing the training of our CVLNM. Extensive experiments on the MS-COCO dataset show that our CVLNM is more effective, \eg, achieving a new state-of-the-art 129.5 CIDEr-D, and more robust, \eg, being less likely to overfit to dataset bias and suffering less when fewer training samples are available. Codes are available at \url{https://github.com/GCYZSL/CVLMN}

翻译：人类倾向于将句子分解为不同部分，例如“某人在某地做某事”，然后用特定内容填充每个部分。受此启发，我们遵循模块化设计原理，提出了一种新颖的图像描述器：学习组合视觉-语言神经模块（CVLNM）。与VQA中广泛使用的神经模块网络不同（其中语言（即问题）是完全可观察的），组合视觉-语言模块的任务更具挑战性。这是因为语言仅部分可观察，我们需在图像描述过程中动态组合模块。总结而言，我们在设计和训练CVLNM时做出以下技术贡献：1）可区分的模块设计——编码器中的四个模块，包括一个用于功能词的 linguistic 模块和三个用于不同内容词（即名词、形容词和动词）的视觉模块，以及解码器中另一个用于常识推理的 linguistic 模块；2）基于自注意力的模块控制器，用于增强视觉推理的鲁棒性；3）基于词性的句法损失，施加于模块控制器以进一步正则化CVLNM的训练。在MS-COCO数据集上的大量实验表明，我们的CVLNM更有效（例如，实现了129.5 CIDEr-D的最新结果）且更鲁棒（例如，不易过拟合数据集偏差，且在训练样本较少时受影响更小）。代码可在https://github.com/GCYZSL/CVLMN获取。