Large language models (LLMs) excel at implementing code from functionality descriptions, but struggle with algorithmic problems that require not only implementation but also identification of the suitable algorithm. Moreover, LLM-generated programs lack guaranteed correctness and require human verification. To address these challenges, we propose ALGO, a framework that synthesizes Algorithmic programs with LLM-Generated Oracles to guide the creation and verify their correctness. ALGO first generates a probably correct but possibly slow reference oracle by prompting an LLM to exhaustively enumerate all the combinations of relevant variables. This oracle is then utilized to guide an arbitrary search strategy in exploring the algorithm space and to verify the algorithms synthesized. Our study shows that the LLM-generated oracles are correct for 88% of the cases. With the oracles as verifiers, ALGO can be integrated with any existing code generation model in a model-agnostic manner to enhance its performance. Experiments show that when equipped with ALGO, we achieve an 8x better one-submission pass rate over the Codex model and a 2.6x better one-submission pass rate over CodeT, the current state-of-the-art model on CodeContests. We can also get 1.3x better pass rate over the ChatGPT Code Interpreter on unseen problems.

翻译：摘要：大型语言模型（LLMs）在根据功能描述实现代码方面表现出色，但在处理不仅需要实现还需识别合适算法的算法问题时则表现欠佳。此外，LLM生成的程序缺乏正确性保证，需要人工验证。为解决这些挑战，我们提出ALGO框架，该框架通过LLM生成的预言机来合成算法程序，以指导算法创建并验证其正确性。ALGO首先通过提示LLM穷举所有相关变量组合，生成一个可能正确但速度较慢的参考预言机。该预言机随后用于指导任意搜索策略探索算法空间，并验证所合成算法的正确性。研究表明，LLM生成的预言机在88%的案例中是正确的。借助预言机作为验证器，ALGO能以模型无关的方式与任何现有代码生成模型集成，从而提升其性能。实验表明，配备ALGO后，我们在CodeContests数据集上的一次提交通过率相较于Codex模型提升8倍，相较于当前最先进的CodeT模型提升2.6倍。对于未见问题，相较于ChatGPT代码解释器，我们的通过率也提升1.3倍。