Financial forecasting has been an important and active area of machine learning research, as even the most modest advantage in predictive accuracy can be parlayed into significant financial gains. Recent advances in natural language processing (NLP) bring the opportunity to leverage textual data, such as earnings reports of publicly traded companies, to predict the return rate for an asset. However, when dealing with such a sensitive task, the consistency of models -- their invariance under meaning-preserving alternations in input -- is a crucial property for building user trust. Despite this, current financial forecasting methods do not consider consistency. To address this problem, we propose FinTrust, an evaluation tool that assesses logical consistency in financial text. Using FinTrust, we show that the consistency of state-of-the-art NLP models for financial forecasting is poor. Our analysis of the performance degradation caused by meaning-preserving alternations suggests that current text-based methods are not suitable for robustly predicting market information. All resources are available on GitHub.

翻译：金融预测一直是机器学习研究的重要且活跃的领域，因为即使是最微小的预测精度提升，也能转化为显著的金融收益。自然语言处理（NLP）的最新进展为利用公开交易公司财报等文本数据预测资产收益率带来了机遇。然而，在处理这类敏感任务时，模型的一致性——即在保持语义不变的输入变化下模型的不变性——是建立用户信任的关键属性。尽管如此，当前的金融预测方法并未考虑一致性。为解决这一问题，我们提出了FinTrust，一个评估金融文本逻辑一致性的工具。利用FinTrust，我们发现当前最先进的NLP模型在金融预测中的一致性较差。我们对由语义保持变化导致的性能下降的分析表明，当前的基于文本的方法并不适合稳健地预测市场信息。所有资源均已在GitHub上发布。