The rapid development of large language models (LLMs) has led to significant advancements in code completion tasks. While larger models have higher accuracy, they also cost much more to run. Meanwhile, model cascading has been proven effective to conserve computational resources while enhancing accuracy in LLMs on natural language generation tasks. It generates output with the smallest model in a set, and only queries the larger models when it fails to meet predefined quality criteria. However, this strategy has not been used in code completion tasks, primarily because assessing the quality of code completions differs substantially from assessing natural language, where the former relies heavily on the functional correctness. To address this, we propose letting each model generate and execute a set of test cases for their solutions, and use the test results as the cascading threshold. We show that our model cascading strategy reduces computational costs while increases accuracy compared to generating the output with a single model. We also introduce a heuristics to determine the optimal combination of the number of solutions, test cases, and test lines each model should generate, based on the budget. Compared to speculative decoding, our method works on black-box models, having the same level of cost-accuracy trade-off, yet providing much more choices based on the server's budget. Ours is the first work to optimize cost-accuracy trade-off for LLM code generation with model cascading.

翻译：大型语言模型（LLM）的快速发展显著推动了代码补全任务的进步。虽然更大规模的模型具有更高的准确性，但其运行成本也显著更高。与此同时，模型级联已被证明在自然语言生成任务中能够有效节约计算资源并提升LLM的准确性。该策略通过集合中最小的模型生成输出，仅当输出无法满足预设质量标准时才查询更大的模型。然而，这一策略尚未应用于代码补全任务，主要原因在于评估代码补全质量与评估自然语言存在本质差异，前者高度依赖于功能正确性。为解决此问题，我们提出让每个模型为其生成的解决方案创建并执行一组测试用例，并将测试结果作为级联阈值。实验表明，与使用单一模型生成输出相比，我们的模型级联策略在降低计算成本的同时提高了准确性。我们还引入了一种启发式方法，可根据预算确定各模型应生成的解决方案数量、测试用例数量及测试代码行的最优组合。相较于推测解码，我们的方法适用于黑盒模型，在保持相同成本-准确性权衡水平的同时，能够根据服务器预算提供更多选择。本研究是首个通过模型级联优化LLM代码生成成本-准确性权衡的工作。