Detecting sarcasm remains a challenging task in the areas of Natural Language Processing (NLP) despite recent advances in neural network approaches. Currently, Pre-trained Language Models (PLMs) and Large Language Models (LLMs) are the preferred approach for sarcasm detection. However, the complexity of sarcastic text, combined with linguistic diversity and cultural variation across communities, has made the task more difficult even for PLMs and LLMs. Beyond that, those models also exhibit unreliable detection of words or tokens that require extra grounding for analysis. Building on a state-of-the-art prompting method in LLMs for sarcasm detection called Pragmatic Metacognitive Prompting (PMP), we introduce a retrieval-aware approach that incorporates retrieved contextual information for each target text. Our pipeline explores two complementary ways to provide context: adding non-parametric knowledge using web-based retrieval when the model lacks necessary background, and eliciting the model's own internal knowledge for a self-knowledge awareness strategy. We evaluated our approach with three datasets, such as Twitter Indonesia Sarcastic, SemEval-2018 Task 3, and MUStARD. Non-parametric retrieval resulted in a significant 9.87% macro-F1 improvement on Twitter Indonesia Sarcastic compared to the original PMP method. Self-knowledge retrieval improves macro-F1 by 3.29% on Semeval and by 4.08% on MUStARD. These findings highlight the importance of context in enhancing LLMs performance in sarcasm detection task, particularly the involvement of culturally specific slang, references, or unknown terms to the LLMs. Future work will focus on optimizing the retrieval of relevant contextual information and examining how retrieval quality affects performance. The experiment code is available at: https://github.com/wllchrst/sarcasm-detection_pmp_knowledge-base.
翻译:讽刺检测仍然是自然语言处理(NLP)领域中的一项挑战性任务,尽管近年来神经网络方法取得了进展。目前,预训练语言模型(PLMs)和大语言模型(LLMs)是讽刺检测的首选方法。然而,讽刺文本的复杂性,加上不同社区间的语言多样性和文化差异,使得即使对于PLMs和LLMs来说,该任务也变得更为困难。此外,这些模型在检测需要额外基础进行分析的词语或标记时也表现出不可靠性。基于一种用于讽刺检测的先进LLMs提示方法——实用元认知提示(PMP),我们引入了一种检索感知方法,该方法为每个目标文本整合了检索到的上下文信息。我们的流程探索了两种互补的提供上下文的方式:当模型缺乏必要背景时,使用基于网络的检索添加非参数知识;以及激发模型自身的内部知识,以实现自我知识感知策略。我们在三个数据集上评估了我们的方法,例如Twitter Indonesia Sarcastic、SemEval-2018 Task 3和MUStARD。与原始PMP方法相比,非参数检索在Twitter Indonesia Sarcastic数据集上实现了显著的9.87%宏F1提升。自我知识检索在Semeval数据集上提高了3.29%的宏F1,在MUStARD数据集上提高了4.08%。这些发现突显了上下文在提升LLMs讽刺检测任务性能中的重要性,特别是涉及LLMs不熟悉的文化特定俚语、引用或未知术语时。未来的工作将侧重于优化相关上下文信息的检索,并研究检索质量如何影响性能。实验代码可在以下网址获取:https://github.com/wllchrst/sarcasm-detection_pmp_knowledge-base。