Large Language Models (LLM) are a new class of computation engines, "programmed" via prompt engineering. We are still learning how to best "program" these LLMs to help developers. We start with the intuition that developers tend to consciously and unconsciously have a collection of semantics facts in mind when working on coding tasks. Mostly these are shallow, simple facts arising from a quick read. For a function, examples of facts might include parameter and local variable names, return expressions, simple pre- and post-conditions, and basic control and data flow, etc. One might assume that the powerful multi-layer architecture of transformer-style LLMs makes them inherently capable of doing this simple level of "code analysis" and extracting such information, implicitly, while processing code: but are they, really? If they aren't, could explicitly adding this information help? Our goal here is to investigate this question, using the code summarization task and evaluate whether automatically augmenting an LLM's prompt with semantic facts explicitly, actually helps. Prior work shows that LLM performance on code summarization benefits from few-shot samples drawn either from the same-project or from examples found via information retrieval methods (such as BM25). While summarization performance has steadily increased since the early days, there is still room for improvement: LLM performance on code summarization still lags its performance on natural-language tasks like translation and text summarization. We find that adding semantic facts actually does help! This approach improves performance in several different settings suggested by prior work, including for two different Large Language Models. In most cases, improvement nears or exceeds 2 BLEU; for the PHP language in the challenging CodeSearchNet dataset, this augmentation actually yields performance surpassing 30 BLEU.

翻译：大型语言模型（LLM）是一类新型计算引擎，通过提示工程进行“编程”。我们仍在探索如何最佳地“编程”这些LLM以帮助开发者。我们基于一个直觉：开发者在处理编码任务时，会有意识或无意识地在脑海中积累一系列语义事实。这些大多是源自快速阅读的浅显简单事实。对于函数而言，事实示例可能包括参数和局部变量名称、返回表达式、简单的前置和后置条件、基本控制流和数据流等。人们可能假设Transformer风格LLM强大的多层架构使其天生具备这种简单的“代码分析”能力，能在处理代码时隐式提取此类信息——但事实果真如此吗？若并非如此，显式添加这些信息是否有帮助？本研究旨在通过代码摘要任务探究此问题，评估自动用语义事实增强LLM提示是否真的有助益。先前研究表明，LLM在代码摘要任务上的性能受益于从同项目或通过信息检索方法（如BM25）找到的样本中抽取的少样本示例。尽管摘要性能自早期以来稳步提升，但仍有改进空间：LLM在代码摘要上的性能仍落后于其在翻译、文本摘要等自然语言任务上的表现。我们发现添加语义事实确实有效！该方法在先前工作提出的多种不同设置（包括两种不同的大型语言模型）中均提升了性能。在大多数情况下，性能提升接近或超过2个BLEU值；对于具有挑战性的CodeSearchNet数据集中的PHP语言，该增强方法实际使得性能超过了30 BLEU。