Phishing continues to be one of the most prevalent attack vectors, making accurate classification of phishing URLs essential. Recently, large language models (LLMs) have demonstrated promising results in phishing URL detection. However, their reasoning capabilities that enabled such performance remain underexplored. To this end, in this paper, we propose a Least-to-Most prompting framework for phishing URL detection. In particular, we introduce an "answer sensitivity" mechanism that guides Least-to-Most's iterative approach to enhance reasoning and yield higher prediction accuracy. We evaluate our framework using three URL datasets and four state-of-the-art LLMs, comparing against a one-shot approach and a supervised model. We demonstrate that our framework outperforms the one-shot baseline while achieving performance comparable to that of the supervised model, despite requiring significantly less training data. Furthermore, our in-depth analysis highlights how the iterative reasoning enabled by Least-to-Most, and reinforced by our answer sensitivity mechanism, drives these performance gains. Overall, we show that this simple yet powerful prompting strategy consistently outperforms both one-shot and supervised approaches, despite requiring minimal training or few-shot guidance. Our experimental setup can be found in our Github repository github.sydney.edu.au/htri0928/least-to-most-phishing-detection.

翻译：钓鱼攻击依然是最普遍的威胁载体之一，这使得钓鱼URL的精准分类至关重要。近年来，大型语言模型在钓鱼URL检测中展现出有前景的结果。然而，促成此类性能的推理能力仍未得到充分探索。为此，本文提出一种用于钓鱼URL检测的最少到最多提示框架。具体而言，我们引入一种"答案敏感性"机制，该机制引导最少到最多框架的迭代方法以增强推理能力并产生更高的预测准确率。我们使用三个URL数据集和四种先进的大型语言模型评估所提框架，并与单样本方法及监督模型进行对比。实验表明，我们的框架显著优于单样本基线，同时达到与监督模型相当的性能，尽管所需训练数据量大幅减少。此外，我们的深入分析揭示了最少到最多框架所实现的迭代推理，以及通过答案敏感性机制强化的过程，如何驱动这些性能提升。总体而言，我们证明这种简洁而强大的提示策略在仅需极少训练或少量样本指导的情况下，始终优于单样本和监督方法。我们的实验设置可在GitHub仓库github.sydney.edu.au/htri0928/least-to-most-phishing-detection中查看。