Large Language Models (LLM) are a new class of computation engines, "programmed" via prompt engineering. We are still learning how to best "program" these LLMs to help developers. We start with the intuition that developers tend to consciously and unconsciously have a collection of semantics facts in mind when working on coding tasks. Mostly these are shallow, simple facts arising from a quick read. For a function, examples of facts might include parameter and local variable names, return expressions, simple pre- and post-conditions, and basic control and data flow, etc. One might assume that the powerful multi-layer architecture of transformer-style LLMs makes them inherently capable of doing this simple level of "code analysis" and extracting such information, implicitly, while processing code: but are they, really? If they aren't, could explicitly adding this information help? Our goal here is to investigate this question, using the code summarization task and evaluate whether automatically augmenting an LLM's prompt with semantic facts explicitly, actually helps. Prior work shows that LLM performance on code summarization benefits from few-shot samples drawn either from the same-project or from examples found via information retrieval methods (such as BM25). While summarization performance has steadily increased since the early days, there is still room for improvement: LLM performance on code summarization still lags its performance on natural-language tasks like translation and text summarization. We find that adding semantic facts actually does help! This approach improves performance in several different settings suggested by prior work, including for two different Large Language Models. In most cases, improvement nears or exceeds 2 BLEU; for the PHP language in the challenging CodeSearchNet dataset, this augmentation actually yields performance surpassing 30 BLEU.

翻译：大型语言模型（LLM）是一种新型计算引擎，通过提示工程进行“编程”。我们仍在探索如何最佳地“编程”这些LLM以帮助开发者。我们基于一个直觉：开发者在处理编码任务时，会有意或无意地在脑海中保留一系列语义事实。这些大多是快速阅读产生的浅层、简单事实。以函数为例，这些事实可能包括参数和局部变量名称、返回表达式、简单的前置和后置条件、基本控制流和数据流等。人们可能会假设，Transformer风格LLM的强大多层架构使其天生具备处理这种简单“代码分析”并隐式提取此类信息的能力，但事实果真如此吗？如果并非如此，显式添加这些信息是否有帮助？本文旨在通过代码摘要任务探究这一问题，评估将语义事实显式自动增强到LLM提示中是否真正有效。先前研究表明，LLM在代码摘要任务中的性能受益于从同一项目中抽取的少样本示例，或通过信息检索方法（如BM25）找到的样本。虽然自早期以来摘要性能稳步提升，但仍存在改进空间：LLM在代码摘要上的性能仍落后于其在翻译和文本摘要等自然语言任务上的表现。我们发现添加语义事实确实有效！该方法在先前研究建议的多种不同设置下均能提升性能，包括针对两种不同的大型语言模型。在大多数情况下，提升幅度接近或超过2个BLEU；对于具有挑战性的CodeSearchNet数据集中的PHP语言，这种增强甚至使性能超过30 BLEU。