Prompting with natural language instructions has recently emerged as a popular method of harnessing the capabilities of large language models. Given the inherent ambiguity present in natural language, it is intuitive to consider the possible advantages of prompting with less ambiguous prompt styles, such as the use of pseudo-code. In this paper we explore if prompting via pseudo-code instructions helps improve the performance of pre-trained language models. We manually create a dataset of pseudo-code prompts for 132 different tasks spanning classification, QA and generative language tasks, sourced from the Super-NaturalInstructions dataset. Using these prompts along with their counterparts in natural language, we study their performance on two LLM families - BLOOM and CodeGen. Our experiments show that using pseudo-code instructions leads to better results, with an average increase (absolute) of 7-16 points in F1 scores for classification tasks and an improvement (relative) of 12-38% in aggregate ROUGE scores across all tasks. We include detailed ablation studies which indicate that code comments, docstrings, and the structural clues encoded in pseudo-code all contribute towards the improvement in performance. To the best of our knowledge our work is the first to demonstrate how pseudo-code prompts can be helpful in improving the performance of pre-trained LMs.

翻译：近年来，使用自然语言指令进行提示已成为挖掘大语言模型能力的一种流行方法。鉴于自然语言本身存在固有的歧义性，我们自然可以想到采用歧义更少的提示风格（例如伪代码）可能具有优势。本文探究通过伪代码指令进行提示是否有助于提升预训练语言模型的性能。我们基于Super-NaturalInstructions数据集，手动构建了覆盖132个不同任务（包括分类、问答和生成式语言任务）的伪代码提示数据集。利用这些提示及其对应的自然语言提示，我们在两个大语言模型系列——BLOOM和CodeGen——上研究了它们的性能。实验结果表明，使用伪代码指令可带来更优结果：在分类任务中，F1分数平均提升（绝对值）7-16个点；在所有任务中，ROUGE分数总和相对提升12-38%。我们进行的详细消融研究表明，代码注释、文档字符串以及伪代码中编码的结构线索均对性能提升有所贡献。据我们所知，我们的工作是首次证明伪代码提示如何有助于提升预训练语言模型性能的研究。