Natural language understanding (NLU) has two core tasks: intent classification and slot filling. The success of pre-training language models resulted in a significant breakthrough in the two tasks. One of the promising solutions called BERT can jointly optimize the two tasks. We note that BERT-based models convert each complex token into multiple sub-tokens by wordpiece algorithm, which generates a mismatch between the lengths of the tokens and the labels. This leads to BERT-based models do not do well in label prediction which limits model performance improvement. Many existing models can be compatible with this issue but some hidden semantic information is discarded in the fine-tuning process. We address the problem by introducing a novel joint method on top of BERT which explicitly models the multiple sub-tokens features after wordpiece tokenization, thereby contributing to the two tasks. Our method can well extract the contextual features from complex tokens by the proposed sub-words attention adapter (SAA), which preserves overall utterance information. Additionally, we propose an intent attention adapter (IAA) to obtain the full sentence features to aid users to predict intent. Experimental results confirm that our proposed model is significantly improved on two public benchmark datasets. In particular, the slot filling F1 score is improved from 96.1 to 98.2 (2.1% absolute) on the Airline Travel Information Systems (ATIS) dataset.

翻译：自然语言理解（NLU）包含两个核心任务：意图分类和槽填充。预训练语言模型的成功在这两个任务中取得了重大突破。其中一种名为BERT的解决方案能够联合优化这两个任务。我们注意到，基于BERT的模型通过词片段算法将每个复杂词元转换为多个子词元，这导致词元与标签之间的长度不匹配。这一问题使得基于BERT的模型在标签预测方面表现不佳，从而限制了模型性能的提升。许多现有模型虽能兼容此问题，但在微调过程中会丢弃部分隐含语义信息。为了解决这一问题，我们在BERT基础上引入了一种新的联合方法，该方法在词片段分词后显式建模多个子词元的特征，从而对两个任务均有所贡献。通过提出的子词注意力适配器（SAA），我们的方法能够有效提取复杂词元的上下文特征，同时保留完整的语句信息。此外，我们提出了意图注意力适配器（IAA）以获取完整句子特征，从而辅助用户进行意图预测。实验结果证明，我们的模型在两个公开基准数据集上取得了显著提升。特别是在航空旅行信息系统（ATIS）数据集中，槽填充的F1分数从96.1提升至98.2（绝对提升2.1%）。