While language models are increasingly more proficient at code generation, they still frequently generate incorrect programs. Many of these programs are obviously wrong, but others are more subtle and pass weaker correctness checks such as being able to compile. In this work, we focus on these counterfeit samples: programs sampled from a language model that 1) have a high enough log-probability to be generated at a moderate temperature and 2) pass weak correctness checks. Overall, we discover that most models have a very shallow understanding of counterfeits through three clear failure modes. First, models mistakenly classify them as correct. Second, models are worse at reasoning about the execution behaviour of counterfeits and often predict their execution results as if they were correct. Third, when asking models to fix counterfeits, the likelihood of a model successfully repairing a counterfeit is often even lower than that of sampling a correct program from scratch. Counterfeits also have very unexpected properties: first, counterfeit programs for problems that are easier for a model to solve are not necessarily easier to detect and only slightly easier to execute and repair. Second, counterfeits from a given model are just as confusing to the model itself as they are to other models. Finally, both strong and weak models are able to generate counterfeit samples that equally challenge all models. In light of our findings, we recommend that care and caution be taken when relying on models to understand their own samples, especially when no external feedback is incorporated.

翻译：尽管语言模型在代码生成方面日益精通，但它们仍频繁生成错误程序。其中许多程序明显有误，但另一些更为隐蔽，能通过较弱的正确性检查（如通过编译）。本研究聚焦于这些"假象样本"：从语言模型中采样的、满足以下条件的程序：1）具有足够高的对数概率，可在中等温度下生成；2）能通过较弱的正确性检查。总体而言，我们通过三种明确的故障模式发现，大多数模型对假象样本的理解非常浅薄。首先，模型会错误地将它们归类为正确。其次，模型在推理假象样本的执行行为方面表现更差，且经常预测其执行结果如同正确程序一般。第三，当要求模型修复假象样本时，模型成功修复假象样本的可能性通常甚至低于从头采样一个正确程序。假象样本还具有非常出人意料的特性：首先，对于模型更易解决的问题，其假象程序不一定更易检测，且仅略微更易执行和修复。其次，给定模型产生的假象样本对该模型本身造成的困惑程度，与其他模型所面临的困惑程度相当。最后，强大和弱小的模型均能生成同等挑战所有模型的假象样本。基于研究发现，我们建议在依赖模型理解自身样本时务必谨慎，尤其是在未引入外部反馈的情况下。