Phishing and spam emails remain a major cybersecurity threat, with attackers increasingly leveraging Large Language Models (LLMs) to craft highly deceptive content. This study presents a comprehensive email dataset containing phishing, spam, and legitimate messages, explicitly distinguishing between human- and LLM-generated content. Each email is annotated with its category, emotional appeal (e.g., urgency, fear, authority), and underlying motivation (e.g., link-following, credential theft, financial fraud). We benchmark multiple LLMs on their ability to identify these emotional and motivational cues and select the most reliable model to annotate the full dataset. To evaluate classification robustness, emails were also rephrased using several LLMs while preserving meaning and intent. A state-of-the-art LLM was then assessed on its performance across both original and rephrased emails using expert-labeled ground truth. The results highlight strong phishing detection capabilities but reveal persistent challenges in distinguishing spam from legitimate emails. Our dataset and evaluation framework contribute to improving AI-assisted email security systems. To support open science, all code, templates, and resources are available on our project site.

翻译：网络钓鱼与垃圾邮件仍是网络安全的主要威胁，攻击者日益利用大语言模型（LLMs）生成极具欺骗性的内容。本研究提出了一个包含钓鱼邮件、垃圾邮件及正常邮件的综合性邮件数据集，并明确区分了人工生成与LLM生成的内容。每封邮件均标注了其类别、情感诉求（如紧迫性、恐惧感、权威性）及潜在动机（如诱导点击链接、窃取凭证、金融诈骗）。我们以多种LLMs为基准，评估其识别这些情感与动机线索的能力，并选择最可靠的模型对完整数据集进行标注。为评估分类鲁棒性，研究还使用多个LLMs对邮件进行语义与意图保持的改写。随后基于专家标注的真实数据，评估了前沿LLM在原始邮件与改写邮件上的性能表现。结果表明模型具备较强的钓鱼邮件检测能力，但在区分垃圾邮件与正常邮件方面仍存在持续挑战。本数据集与评估框架有助于改进AI辅助的邮件安全系统。为支持开放科学，所有代码、模板及相关资源均已发布于项目网站。