Knowledge sharing about emerging threats is crucial in the rapidly advancing field of cybersecurity and forms the foundation of Cyber Threat Intelligence (CTI). In this context, Large Language Models are becoming increasingly significant in the field of cybersecurity, presenting a wide range of opportunities. This study surveys the performance of ChatGPT, GPT4all, Dolly, Stanford Alpaca, Alpaca-LoRA, Falcon, and Vicuna chatbots in binary classification and Named Entity Recognition (NER) tasks performed using Open Source INTelligence (OSINT). We utilize well-established data collected in previous research from Twitter to assess the competitiveness of these chatbots when compared to specialized models trained for those tasks. In binary classification experiments, Chatbot GPT-4 as a commercial model achieved an acceptable F1 score of 0.94, and the open-source GPT4all model achieved an F1 score of 0.90. However, concerning cybersecurity entity recognition, all evaluated chatbots have limitations and are less effective. This study demonstrates the capability of chatbots for OSINT binary classification and shows that they require further improvement in NER to effectively replace specially trained models. Our results shed light on the limitations of the LLM chatbots when compared to specialized models, and can help researchers improve chatbots technology with the objective to reduce the required effort to integrate machine learning in OSINT-based CTI tools.

翻译：新兴威胁的知识共享在快速发展的网络安全领域至关重要，是网络威胁情报（CTI）的基础。在此背景下，大语言模型在网络安全领域日益重要，展现出广泛的应用前景。本研究调查了ChatGPT、GPT4all、Dolly、Stanford Alpaca、Alpaca-LoRA、Falcon和Vicuna聊天机器人在使用开源情报（OSINT）执行二分类和命名实体识别（NER）任务中的性能。我们利用先前研究中从Twitter收集的成熟数据，评估这些聊天机器人与针对这些任务训练的专门模型相比的竞争力。在二分类实验中，商业模型Chatbot GPT-4取得了0.94的可接受F1分数，开源模型GPT4all取得了0.90的F1分数。然而，在网络安全实体识别方面，所有评估的聊天机器人都存在局限性且效果较差。本研究展示了聊天机器人在OSINT二分类中的能力，并指出它们在NER方面需要进一步改进才能有效替代专门训练的模型。我们的结果揭示了LLM聊天机器人与专门模型相比存在的局限性，有助于研究人员改进聊天机器人技术，从而减少将机器学习集成到基于OSINT的CTI工具中所需的工作量。