Intermediate step methodologies like chain of thoughts (COT) have demonstrated effectiveness in enhancing the performance of Large Language Models (LLMs) on code generation. This study explores the utilization of intermediate languages, including various programming languages, natural language solutions, and pseudo-code, and systematically evaluates their impact on the performance of LLMs in code generation tasks. Our experiments encompass eleven models across the CodeLlama, GPT, and Mistral families, as well as newly released smaller models. Our findings reveal that intermediate languages generally exhibit greater efficacy in larger models that have not yet achieved state-of-the-art performance. Natural language consistently emerges as the most effective intermediate representation across all target languages. However, we observe no universally effective intermediate formal language across different models and target languages. Furthermore, we uncover a weak correlation between the correctness of intermediate solutions and final generation, suggesting that improvements may stem from the chain-of-thought effect rather than language-specific transfer. Interestingly, we discover that for GPT family models, prompting multiple times without explicit self-correction instructions yields performance gains across the studied languages.

翻译：诸如思维链（COT）等中间步骤方法已被证明能有效提升大语言模型（LLM）在代码生成任务上的性能。本研究探讨了中间语言（包括多种编程语言、自然语言解决方案和伪代码）的运用，并系统评估了它们对LLM代码生成任务表现的影响。我们的实验涵盖了CodeLlama、GPT和Mistral系列中的十一个模型，以及新近发布的较小规模模型。研究结果表明，中间语言通常在尚未达到最先进性能水平的大型模型中表现出更高的效能。在所有目标语言中，自然语言始终是最有效的中间表示形式。然而，我们并未发现存在一种能在不同模型与目标语言间普遍有效的中间形式语言。此外，我们揭示了中间解决方案的正确性与最终生成结果之间的弱相关性，这表明性能提升可能源于思维链效应而非特定语言的迁移。有趣的是，我们发现对于GPT系列模型，在没有显式自我修正指令的情况下进行多次提示，能在所研究的语言中带来性能增益。