Named Entity Recognition (NER) is a Natural Language Processing technique for extracting information from textual documents. However, much of the existing research on NER has been centered around English-language documents, leaving a gap in the availability of datasets tailored to the financial domain in Portuguese. This study addresses the need for NER within the financial domain, focusing on Portuguese-language texts extracted from earnings call transcriptions of Brazilian banks. By curating a comprehensive dataset comprising 384 transcriptions and leveraging weak supervision techniques for annotation, we evaluate the performance of monolingual models trained on Portuguese (BERTimbau and PTT5) and multilingual models (mBERT and mT5). Notably, we introduce a novel approach that reframes the token classification task as a text generation problem, enabling fine-tuning and evaluation of T5 models. Following the fine-tuning of the models, we conduct an evaluation on the test dataset, employing performance and error metrics. Our findings reveal that BERT-based models consistently outperform T5-based models. Furthermore, while the multilingual models exhibit comparable macro F1-scores, BERTimbau demonstrates superior performance over PTT5. A manual analysis of sentences generated by PTT5 and mT5 unveils a degree of similarity ranging from 0.89 to 1.0, between the original and generated sentences. However, critical errors emerge as both models exhibit discrepancies, such as alterations to monetary and percentage values, underscoring the importance of accuracy and consistency in the financial domain. Despite these challenges, PTT5 and mT5 achieve impressive macro F1-scores of 98.52% and 98.85%, respectively, with our proposed approach. Furthermore, our study sheds light on notable disparities in memory and time consumption for inference across the models.
翻译:命名实体识别(NER)是一种从文本文档中提取信息的自然语言处理技术。然而,现有NER研究大多集中于英文文档,导致面向葡萄牙语金融领域的数据集存在空白。本研究针对金融领域的NER需求,聚焦于从巴西银行财报电话会议转录文本中提取的葡萄牙语文本。通过构建包含384份转录文本的综合数据集,并利用弱监督技术进行标注,我们评估了基于葡萄牙语训练的单语模型(BERTimbau和PTT5)与多语言模型(mBERT和mT5)的性能。值得注意的是,我们提出了一种新颖方法,将令牌分类任务重新定义为文本生成问题,从而实现了T5模型的微调与评估。在模型微调后,我们采用性能指标和错误指标在测试数据集上进行了评估。研究结果表明,基于BERT的模型始终优于基于T5的模型。此外,尽管多语言模型展现出相当的宏F1分数,但BERTimbau的性能优于PTT5。对PTT5和mT5生成句子的手动分析显示,原始句子与生成句子之间的相似度范围为0.89至1.0。然而,两个模型均出现关键性错误,例如对货币和百分比数值的更改,这凸显了金融领域准确性与一致性的重要性。尽管存在这些挑战,PTT5和mT5在我们提出的方法下仍分别达到了98.52%和98.85%的宏F1分数。此外,本研究揭示了各模型在推理时的内存与时间消耗方面的显著差异。