Recent breakthroughs in pre-trained code models, such as CodeBERT and Codex, have shown their superior performance in various downstream tasks. The correctness and unambiguity of API usage among these code models are crucial for achieving desirable program functionalities, requiring them to learn various API fully qualified names structurally and semantically. Recent studies reveal that even state-of-the-art pre-trained code models struggle with suggesting the correct APIs during code generation. However, the reasons for such poor API usage performance are barely investigated. To address this challenge, we propose using knowledge probing as a means of interpreting code models, which uses cloze-style tests to measure the knowledge stored in models. Our comprehensive study examines a code model's capability of understanding API fully qualified names from two different perspectives: API call and API import. Specifically, we reveal that current code models struggle with understanding API names, with pre-training strategies significantly affecting the quality of API name learning. We demonstrate that natural language context can assist code models in locating Python API names and generalize Python API name knowledge to unseen data. Our findings provide insights into the limitations and capabilities of current pre-trained code models, and suggest that incorporating API structure into the pre-training process can improve automated API usage and code representations. This work provides significance for advancing code intelligence practices and direction for future studies. All experiment results, data and source code used in this work are available at \url{https://doi.org/10.5281/zenodo.7902072}.

翻译：近期，CodeBERT和Codex等预训练代码模型在各类下游任务中展现了卓越性能。这些模型对API使用的正确性和无歧义性对于实现理想程序功能至关重要，要求它们从结构和语义层面掌握各类API的完整限定名。最新研究表明，即便最先进的预训练代码模型在代码生成时也难以推荐正确的API。然而，此类API使用性能欠佳的原因尚鲜有探究。为应对这一挑战，我们提出采用知识探测作为解释代码模型的手段，通过完形填空测试衡量模型中存储的知识。本综合研究从API调用和API导入两个不同维度，审视了代码模型理解API完整限定名的能力。具体而言，我们揭示了当前代码模型在理解API名称方面存在困难，而预训练策略显著影响API名称学习的质量。我们证明自然语言上下文可辅助代码模型定位Python API名称，并将Python API名称知识泛化至未见数据。本研究结论阐明了当前预训练代码模型的局限性与能力，并建议在预训练过程中融入API结构以提升自动化API使用与代码表征。此工作对推进代码智能实践具有重要价值，并为后续研究指明方向。本研究所用实验成果、数据及源代码均公开于\url{https://doi.org/10.5281/zenodo.7902072}。