Pre-trained language models have shown impressive performance on a variety of tasks and domains. Previous research on financial language models usually employs a generic training scheme to train standard model architectures, without completely leveraging the richness of the financial data. We propose a novel domain specific Financial LANGuage model (FLANG) which uses financial keywords and phrases for better masking, together with span boundary objective and in-filing objective. Additionally, the evaluation benchmarks in the field have been limited. To this end, we contribute the Financial Language Understanding Evaluation (FLUE), an open-source comprehensive suite of benchmarks for the financial domain. These include new benchmarks across 5 NLP tasks in financial domain as well as common benchmarks used in the previous research. Experiments on these benchmarks suggest that our model outperforms those in prior literature on a variety of NLP tasks. Our models, code and benchmark data are publicly available on Github and Huggingface.

翻译：以往关于金融语言模式的研究通常采用通用培训计划来培训标准模式架构,而没有完全利用金融数据的丰富性。我们提出了一个新的具体领域金融局域网管理模式(FLANG),该模式使用财务关键词和短语来更好地遮掩,同时跨越边界目标和完成目标。此外,实地的评价基准有限。为此,我们提供了财务语言理解评价(FLUE),这是财务领域的一套开放源的全面基准。其中包括金融领域5项国家劳工局任务的新基准以及以往研究中使用的共同基准。对这些基准的实验表明,我们的模型比先前文献中的各种国家劳工局任务要好。我们的模型、代码和基准数据在Github和Huggingface上公开提供。