In recent years, there has been a wide interest in designing deep neural network-based models that automate downstream software engineering tasks on source code, such as code document generation, code search, and program repair. Although the main objective of these studies is to improve the effectiveness of the downstream task, many studies only attempt to employ the next best neural network model, without a proper in-depth analysis of why a particular solution works or does not, on particular tasks or scenarios. In this paper, using an example eXplainable AI (XAI) method (attention mechanism), we study two recent large language models (LLMs) for code (CodeBERT and GraphCodeBERT) on a set of software engineering downstream tasks: code document generation (CDG), code refinement (CR), and code translation (CT). Through quantitative and qualitative studies, we identify what CodeBERT and GraphCodeBERT learn (put the highest attention on, in terms of source code token types), on these tasks. We also show some of the common patterns when the model does not work as expected (performs poorly even on easy problems) and suggest recommendations that may alleviate the observed challenges.

翻译：近年来，设计基于深度神经网络的模型以自动化源代码上的下游软件工程任务（如代码文档生成、代码搜索和程序修复）引起了广泛关注。尽管这些研究的主要目标是提升下游任务的有效性，但许多研究仅尝试应用最新的神经网络模型，而缺乏对特定解决方案在特定任务或场景下有效或失效原因的深入分析。本文以可解释AI（XAI）方法（注意力机制）为例，研究了两类近期用于代码的大型语言模型（CodeBERT和GraphCodeBERT）在多项软件工程下游任务中的表现：代码文档生成、代码优化和代码翻译。通过定量与定性研究，我们揭示了CodeBERT和GraphCodeBERT在这些任务中注意力的最高聚焦点（以源代码令牌类型衡量），并展示了模型在未按预期工作（即使在简单问题上表现不佳）时的常见模式，同时提出了缓解上述挑战的建议。