Knowledge base construction entails acquiring structured information to create a knowledge base of factual and relational data, facilitating question answering, information retrieval, and semantic understanding. The challenge called "Knowledge Base Construction from Pretrained Language Models" at International Semantic Web Conference 2023 defines tasks focused on constructing knowledge base using language model. Our focus was on Track 1 of the challenge, where the parameters are constrained to a maximum of 1 billion, and the inclusion of entity descriptions within the prompt is prohibited. Although the masked language model offers sufficient flexibility to extend its vocabulary, it is not inherently designed for multi-token prediction. To address this, we present Vocabulary Expandable BERT for knowledge base construction, which expand the language model's vocabulary while preserving semantic embeddings for newly added words. We adopt task-specific re-pre-training on masked language model to further enhance the language model. Through experimentation, the results show the effectiveness of our approaches. Our framework achieves F1 score of 0.323 on the hidden test set and 0.362 on the validation set, both data set is provided by the challenge. Notably, our framework adopts a lightweight language model (BERT-base, 0.13 billion parameters) and surpasses the model using prompts directly on large language model (Chatgpt-3, 175 billion parameters). Besides, Token-Recode achieves comparable performances as Re-pretrain. This research advances language understanding models by enabling the direct embedding of multi-token entities, signifying a substantial step forward in link prediction task in knowledge graph and metadata completion in data management.

翻译：知识库构建涉及获取结构化信息以创建包含事实和关系数据的知识库，从而支持问答、信息检索和语义理解。国际语义网会议2023年举办的"基于预训练语言模型的知识库构建"挑战赛定义了相关任务，重点研究利用语言模型构建知识库。我们聚焦于该挑战赛的第一赛道，其约束条件为参数规模不超过10亿，且禁止在提示中包含实体描述。尽管掩码语言模型具备扩展词表的充分灵活性，但其本质上并非为多令牌预测设计。为此，我们提出用于知识库构建的词汇可扩展BERT模型，在扩展语言模型词表的同时，为新添加词语保留语义嵌入。我们采用针对特定任务的掩码语言模型重新预训练来进一步增强语言模型。实验结果表明了所提方法的有效性。我们的框架在隐藏测试集上取得0.323的F1分数，在验证集上取得0.362的F1分数，两个数据集均由挑战赛提供。值得注意的是，本框架采用轻量级语言模型（BERT-base，1.3亿参数），其性能超越了直接在大语言模型（ChatGPT-3，1750亿参数）上使用提示的方法。此外，Token-Recode方法取得了与重新预训练相当的性能。本研究通过支持多令牌实体的直接嵌入，推动了语言理解模型的发展，标志着知识图谱链接预测任务与数据管理中的元数据补全领域迈出了实质性的一步。