Phishing and spam emails remain a major cybersecurity threat, with attackers increasingly leveraging Large Language Models (LLMs) to craft highly deceptive content. This study presents a comprehensive email dataset containing phishing, spam, and legitimate messages, explicitly distinguishing between human- and LLM-generated content. Each email is annotated with its category, emotional appeal (e.g., urgency, fear, authority), and underlying motivation (e.g., link-following, credential theft, financial fraud). We benchmark multiple LLMs on their ability to identify these emotional and motivational cues and select the most reliable model to annotate the full dataset. To evaluate classification robustness, emails were also rephrased using several LLMs while preserving meaning and intent. A state-of-the-art LLM was then assessed on its performance across both original and rephrased emails using expert-labeled ground truth. The results highlight strong phishing detection capabilities but reveal persistent challenges in distinguishing spam from legitimate emails. Our dataset and evaluation framework contribute to improving AI-assisted email security systems. To support open science, all code, templates, and resources are available on our project site.

翻译：网络钓鱼与垃圾邮件仍是网络安全的主要威胁，攻击者日益利用大型语言模型（LLMs）生成极具欺骗性的内容。本研究提出了一个包含钓鱼邮件、垃圾邮件及正常邮件的综合性邮件数据集，明确区分了人工生成与LLM生成的内容。每封邮件均标注了其类别、情感诉求（如紧迫感、恐惧感、权威性）及潜在动机（如诱导点击链接、窃取凭证、金融诈骗）。我们对多种LLM识别这些情感与动机线索的能力进行了基准测试，并选取最可靠的模型完成全数据集的标注。为评估分类鲁棒性，研究还使用多个LLM对邮件进行语义与意图保持的复述改写。基于专家标注的真实数据，我们进一步评估了前沿LLM在原始邮件与改写邮件上的性能表现。结果表明，模型在钓鱼邮件检测方面表现优异，但在区分垃圾邮件与正常邮件时仍面临持续挑战。本数据集及评估框架有助于改进AI辅助的邮件安全系统。为支持开放科学，所有代码、模板及相关资源均已在项目网站公开。