Current benchmarks for evaluating large language models (LLMs) in social media moderation completely overlook a serious threat: covert advertisements, which disguise themselves as regular posts to deceive and mislead consumers into making purchases, leading to significant ethical and legal concerns. In this paper, we present the CHASM, a first-of-its-kind dataset designed to evaluate the capability of Multimodal Large Language Models (MLLMs) in detecting covert advertisements on social media. CHASM is a high-quality, anonymized, manually curated dataset consisting of 4,992 instances, based on real-world scenarios from the Chinese social media platform Rednote. The dataset was collected and annotated under strict privacy protection and quality control protocols. It includes many product experience sharing posts that closely resemble covert advertisements, making the dataset particularly challenging.The results show that under both zero-shot and in-context learning settings, none of the current MLLMs are sufficiently reliable for detecting covert advertisements.Our further experiments revealed that fine-tuning open-source MLLMs on our dataset yielded noticeable performance gains. However, significant challenges persist, such as detecting subtle cues in comments and differences in visual and textual structures.We provide in-depth error analysis and outline future research directions. We hope our study can serve as a call for the research community and platform moderators to develop more precise defenses against this emerging threat.

翻译：当前用于评估社交媒体内容审核中大语言模型（LLMs）的标准基准完全忽视了一个严重威胁：隐性广告。这类广告伪装成常规帖子，诱骗和误导消费者进行购买，引发重大伦理与法律问题。本文首次提出数据集CHASM，旨在评估多模态大语言模型（MLLMs）检测社交媒体隐性广告的能力。CHASM是一个高质量、匿名化且经人工整理的基准数据集，包含基于中文社交平台“小红书”真实场景的4,992个实例。该数据集在严格的隐私保护与质量控制协议下收集和标注，涵盖大量与隐性广告高度相似的产品体验分享帖，使数据集极具挑战性。结果表明，在零样本和上下文学习设置下，当前所有MLLMs均无法可靠检测隐性广告。进一步实验发现，在CHASM上微调开源MLLMs可带来显著的性能提升，但仍面临重大挑战，例如检测评论中的微妙线索以及视觉与文本结构的差异。我们提供了深入的错误分析，并展望了未来研究方向，期望本研究能呼吁学术界和平台审核者针对这一新兴威胁制定更精确的防御策略。