We investigate the use of multimodal information contained in images as an effective method for enhancing the commonsense of Transformer models for text generation. We perform experiments using BART and T5 on concept-to-text generation, specifically the task of generative commonsense reasoning, or CommonGen. We call our approach VisCTG: Visually Grounded Concept-to-Text Generation. VisCTG involves captioning images representing appropriate everyday scenarios, and using these captions to enrich and steer the generation process. Comprehensive evaluation and analysis demonstrate that VisCTG noticeably improves model performance while successfully addressing several issues of the baseline generations, including poor commonsense, fluency, and specificity.

翻译：我们调查利用图像中所含多式联运信息作为提高文本生成变异模型常识的有效方法。我们利用BART和T5进行概念到文字生成实验,特别是基因常识推理任务,或CommonGen。我们称之为我们的方法:VisCTG:视觉化概念到文字生成。VisCTG涉及说明描述适当的日常情景,并使用这些字幕丰富和引导生成过程。全面评价和分析表明,VisCTG显著改进了模型的性能,同时成功地解决了包括不良常识、流利和特殊性在内的几代基线人的问题。