The hallucination of code generation models hinders their applicability to systems requiring higher safety standards. One critical bottleneck in addressing code hallucination is the difficulty of identifying the functional correctness of generated code, due to its unnatural form. We address this core bottleneck by automatically generating unit tests using dynamic code analysis tools, leveraging the \emph{executable nature} of code. Accordingly, we propose a \emph{selective code generator} that abstains from uncertain generations -- based on the functional correctness evaluated by generated unit tests -- to theoretically control the correctness among non-abstained answers, \ie the false discovery rate. Finally, we propose to use generated unit tests in evaluation as well as in learning for precise code evaluation, calling this paradigm \emph{FuzzEval}. We demonstrate the efficacy of our method along with the controllability of code hallucination and reasonable selection efficiency.

翻译：代码生成模型的“幻觉”问题限制了其在需要高安全标准的系统中的应用。解决代码幻觉的一个关键瓶颈在于，由于生成代码的非自然形式，难以识别其功能正确性。为攻克这一核心瓶颈，我们利用代码的**可执行特性**，通过动态代码分析工具自动生成单元测试。据此，我们提出一种**选择性代码生成器**，该生成器基于生成的单元测试评估的功能正确性，对不确定的生成结果进行弃权（即拒绝生成），从而在理论上控制未弃权答案的正确性，即错误发现率。最后，我们提出在评估和训练过程中均使用生成的单元测试以实现精确的代码评估，并将这一范式命名为**FuzzEval**。我们通过实验证明了该方法的有效性、对代码幻觉的可控性以及合理的筛选效率。