Although the synthesis of programs encoding policies often carries the promise of interpretability, systematic evaluations to assess the interpretability of these policies were never performed, likely because of the complexity of such an evaluation. In this paper, we introduce a novel metric that uses large-language models (LLM) to assess the interpretability of programmatic policies. For our metric, an LLM is given both a program and a description of its associated programming language. The LLM then formulates a natural language explanation of the program. This explanation is subsequently fed into a second LLM, which tries to reconstruct the program from the natural language explanation. Our metric measures the behavioral similarity between the reconstructed program and the original. We validate our approach using obfuscated programs that are used to solve classic programming problems. We also assess our metric with programmatic policies synthesized for playing a real-time strategy game, comparing the interpretability scores of programmatic policies synthesized by an existing system to lightly obfuscated versions of the same programs. Our LLM-based interpretability score consistently ranks less interpretable programs lower and more interpretable ones higher. These findings suggest that our metric could serve as a reliable and inexpensive tool for evaluating the interpretability of programmatic policies.

翻译：尽管编码策略的程序合成通常承诺具有可解释性，但针对这些策略可解释性的系统评估此前从未开展，究其原因在于此类评估的复杂性。本文提出一种基于大型语言模型（LLM）的新型度量方法，用于评估程序化策略的可解释性。在该度量中，首先向LLM提供一段程序及其关联编程语言的描述，由LLM生成该程序的自然语言解释；随后将该解释输入第二个LLM，由其尝试根据自然语言解释重建原始程序。我们的度量通过计算重建程序与原始程序的行为相似度来实现。我们采用用于解决经典编程问题的混淆程序验证了该方法，并针对一个实时策略游戏合成的程序化策略进行评估，将现有系统合成的程序化策略可解释性得分与其轻度混淆版本进行对比。基于LLM的可解释性得分持续将可解释性较差的程序评为较低分，而将可解释性较好的程序评为较高分。研究结果表明，该度量可作为评估程序化策略可解释性的可靠且低成本工具。