Causality is fundamental in human cognition and has drawn attention in diverse research fields. With growing volumes of textual data, discerning causalities within text data is crucial, and causal text mining plays a pivotal role in extracting meaningful patterns. This study conducts comprehensive evaluations of ChatGPT's causal text mining capabilities. Firstly, we introduce a benchmark that extends beyond general English datasets, including domain-specific and non-English datasets. We also provide an evaluation framework to ensure fair comparisons between ChatGPT and previous approaches. Finally, our analysis outlines the limitations and future challenges in employing ChatGPT for causal text mining. Specifically, our analysis reveals that ChatGPT serves as a good starting point for various datasets. However, when equipped with a sufficient amount of training data, previous models still surpass ChatGPT's performance. Additionally, ChatGPT suffers from the tendency to falsely recognize non-causal sequences as causal sequences. These issues become even more pronounced with advanced versions of the model, such as GPT-4. In addition, we highlight the constraints of ChatGPT in handling complex causality types, including both intra/inter-sentential and implicit causality. The model also faces challenges with effectively leveraging in-context learning and domain adaptation. Our code is available on \url{https://github.com/retarfi/gemcausal}

翻译：因果关系是人类认知的基础，并已引起多个研究领域的关注。随着文本数据量的不断增长，识别文本数据中的因果关系至关重要，而因果文本挖掘在提取有意义模式方面发挥着关键作用。本研究对ChatGPT的因果文本挖掘能力进行了全面评估。首先，我们引入了一个基准数据集，其范围超越通用英语数据集，涵盖特定领域和非英语数据集。我们还提供了一个评估框架，以确保ChatGPT与先前方法之间的公平比较。最后，我们的分析概述了在因果文本挖掘中应用ChatGPT的局限性与未来挑战。具体而言，分析表明，ChatGPT可作为多种数据集的良好起点。然而，当配备足够数量的训练数据时，先前的模型仍能超越ChatGPT的性能。此外，ChatGPT存在将非因果序列错误识别为因果序列的倾向，这一问题在GPT-4等高级版本模型中更为显著。同时，我们强调了ChatGPT在处理复杂因果关系类型（包括句内/句间因果关系和隐含因果关系）方面的局限性，该模型在有效利用上下文学习和领域自适应方面也面临挑战。我们的代码已开源在\url{https://github.com/retarfi/gemcausal}。