Artificial Intelligence and Machine Learning have witnessed rapid, significant improvements in Natural Language Processing (NLP) tasks. Utilizing Deep Learning, researchers have taken advantage of repository comments in Software Engineering to produce accurate methods for detecting Self-Admitted Technical Debt (SATD) from 20 open-source Java projects' code. In this work, we improve SATD detection with a novel approach that leverages the Bidirectional Encoder Representations from Transformers (BERT) architecture. For comparison, we re-evaluated previous deep learning methods and applied stratified 10-fold cross-validation to report reliable F$_1$-scores. We examine our model in both cross-project and intra-project contexts. For each context, we use re-sampling and duplication as augmentation strategies to account for data imbalance. We find that our trained BERT model improves over the best performance of all previous methods in 19 of the 20 projects in cross-project scenarios. However, the data augmentation techniques were not sufficient to overcome the lack of data present in the intra-project scenarios, and existing methods still perform better. Future research will look into ways to diversify SATD datasets in order to maximize the latent power in large BERT models.

翻译：人工智能和机器学习在自然语言处理（NLP）任务中经历了快速、显著的改进。利用深度学习，研究者通过软件工程中的仓库注释，从20个开源Java项目的代码中开发出用于检测自我承认技术债务（SATD）的精确方法。在本研究中，我们提出了一种新颖方法，利用双向编码器表示 Transformer（BERT）架构来改进 SATD 检测。为进行比较，我们重新评估了以往的深度学习方法，并应用分层10折交叉验证报告可靠的 F$_1$ 分数。我们在跨项目与项目内两种情境下检验模型。针对每种情境，我们使用重采样和复制作为增强策略以应对数据不平衡问题。我们发现，在跨项目场景中，我们训练的 BERT 模型在20个项目中的19个上超越了所有以往方法的最佳表现。然而，数据增强技术不足以克服项目内场景中数据不足的问题，现有方法仍表现更优。未来研究将探索多样化 SATD 数据集的方式，以最大化大型 BERT 模型的潜在能力。