Neural Code Completion Tools (NCCTs) have reshaped the field of software development, which accurately suggest contextually-relevant code snippets benefiting from language modeling techniques. However, language models may emit the training data verbatim during inference with appropriate prompts. This memorization property raises privacy concerns of commercial NCCTs about the hard-coded credential leakage, leading to unauthorized access to systems. Therefore, to answer whether NCCTs will inadvertently emit the hard-coded credential, we propose an evaluation tool called Hard-coded Credential Revealer (HCR). HCR effectively constructs test prompts from GitHub code files with credentials to trigger memorization phenomenon of commercial NCCTs. Then, HCR extracts credentials with pre-defined format from the responses by four designed filters. We apply HCR to evaluate two representative commercial NCCTs: GitHub Copilot and Amazon CodeWhisperer and successfully extracted 2,702 hard-coded credentials from Copilot and 129 secrets from CodeWhisper under the black-box setting, among which at least 3.6% and 5.4% secrets are real strings from GitHub repositories. Moreover, two operational credentials were identified. The experimental results raise the severe privacy concern of the potential leakage of hard-coded credentials in the training data of commercial NCCTs.

翻译：神经代码补全工具（NCCTs）已重塑软件开发领域，其通过语言建模技术准确建议上下文相关的代码片段。然而，语言模型在推理过程中可能通过适当提示逐字输出训练数据。这种记忆特性引发了关于商用NCCTs中硬编码凭证泄露的隐私担忧，进而导致对系统的未授权访问。因此，为探究NCCTs是否会无意中泄露硬编码凭证，我们提出了一种名为硬编码凭证揭示器（HCR）的评估工具。HCR从包含凭证的GitHub代码文件中有效构建测试提示，以触发商用NCCTs的记忆现象。随后，HCR通过四种设计的过滤器从响应中提取预先定义格式的凭证。我们采用HCR评估了两种代表性商用NCCTs：GitHub Copilot和Amazon CodeWhisperer，在黑盒设置下成功从Copilot中提取了2,702个硬编码凭证，从CodeWhisper中提取了129个秘密，其中至少3.6%和5.4%的秘密是来自GitHub仓库的真实字符串。此外，还识别出两个操作凭证。实验结果表明，商用NCCTs的训练数据中硬编码凭证存在潜在泄露风险，引发了严重的隐私担忧。