Prompting with natural language instructions has recently emerged as a popular method of harnessing the capabilities of large language models. Given the inherent ambiguity present in natural language, it is intuitive to consider the possible advantages of prompting with less ambiguous prompt styles, such as the use of pseudo-code. In this paper we explore if prompting via pseudo-code instructions helps improve the performance of pre-trained language models. We manually create a dataset of pseudo-code prompts for 132 different tasks spanning classification, QA and generative language tasks, sourced from the Super-NaturalInstructions dataset. Using these prompts along with their counterparts in natural language, we study their performance on two LLM families - BLOOM and CodeGen. Our experiments show that using pseudo-code instructions leads to better results, with an average increase (absolute) of 7-16 points in F1 scores for classification tasks and an improvement (relative) of 12-38% in aggregate ROUGE-L scores across all tasks. We include detailed ablation studies which indicate that code comments, docstrings, and the structural clues encoded in pseudo-code all contribute towards the improvement in performance. To the best of our knowledge our work is the first to demonstrate how pseudo-code prompts can be helpful in improving the performance of pre-trained LMs.

翻译：通过自然语言指令进行提示已成为利用大型语言模型能力的流行方法。鉴于自然语言固有的歧义性，直觉上考虑使用歧义性较低的提示风格（如伪代码）可能具有优势。本文探索了通过伪代码指令提示是否有助于提升预训练语言模型的性能。我们手动构建了一个涵盖132个任务的伪代码提示数据集，这些任务来自Super-NaturalInstructions数据集，涉及分类、问答和生成类语言任务。通过将这些提示及其对应的自然语言提示进行对比，我们研究了它们在两个LLM系列（BLOOM和CodeGen）上的表现。实验表明，使用伪代码指令能带来更优结果：分类任务的F1分数平均提升（绝对值）7-16分，所有任务的ROUGE-L总分平均相对提升12-38%。我们进行了详细的消融研究，结果表明代码注释、文档字符串以及伪代码中编码的结构线索都对性能提升有所贡献。据我们所知，我们的研究首次证明了伪代码提示如何有助于提升预训练语言模型的性能。