The emergence of large language models (LLMs) has significantly promoted the development of code generation task, sparking a surge in pertinent literature. Current research is hindered by redundant generation results and a tendency to overfit local patterns in the short term. Although existing studies attempt to alleviate the issue by adopting a multi-token prediction strategy, there remains limited focus on choosing the appropriate processing length for generations. By analyzing the attention between tokens during the generation process of LLMs, it can be observed that the high spikes of the attention scores typically appear at the end of lines. This insight suggests that it is reasonable to treat each line of code as a fundamental processing unit and generate them sequentially. Inspired by this, we propose the LSR-MCTS algorithm, which leverages MCTS to determine the code line-by-line and select the optimal path. Further, we integrate a self-refine mechanism at each node to enhance diversity and generate higher-quality programs through error correction. Extensive experiments and comprehensive analyses on three public coding benchmarks demonstrate that our method outperforms the state-of-the-art performance approaches.

翻译：大型语言模型（LLMs）的出现显著推动了代码生成任务的发展，引发了相关文献的激增。当前研究受限于冗余的生成结果以及在短期内过度拟合局部模式的倾向。尽管现有研究尝试通过采用多标记预测策略来缓解该问题，但在为生成内容选择合适的处理长度方面关注仍然有限。通过分析LLMs生成过程中标记之间的注意力机制，可以观察到注意力分数的高峰通常出现在代码行的末尾。这一发现表明，将每行代码视为基本处理单元并顺序生成是合理的。受此启发，我们提出了LSR-MCTS算法，该算法利用MCTS逐行确定代码并选择最优路径。此外，我们在每个节点集成了自优化机制，以增强多样性并通过纠错生成更高质量的程序。在三个公共编码基准上进行的广泛实验和综合分析表明，我们的方法优于当前最先进的性能方案。