Tactics, Techniques, and Procedures (TTPs) outline the methods attackers use to exploit vulnerabilities. The interpretation of TTPs in the MITRE ATT&CK framework can be challenging for cybersecurity practitioners due to presumed expertise and complex dependencies. Meanwhile, advancements with Large Language Models (LLMs) have led to recent surge in studies exploring its uses in cybersecurity operations. It is, however, unclear how LLMs can be used in an efficient and proper way to provide accurate responses for critical domains such as cybersecurity. This leads us to investigate how to better use two types of LLMs: small-scale encoder-only (e.g., RoBERTa) and larger decoder-only (e.g., GPT-3.5) LLMs to comprehend and summarize TTPs with the intended purposes (i.e., tactics) of a cyberattack procedure. This work studies and compares the uses of supervised fine-tuning (SFT) of encoder-only LLMs vs. Retrieval Augmented Generation (RAG) for decoder-only LLMs (without fine-tuning). Both SFT and RAG techniques presumably enhance the LLMs with relevant contexts for each cyberattack procedure. Our studies show decoder-only LLMs with RAG achieves better performance than encoder-only models with SFT, particularly when directly relevant context is extracted by RAG. The decoder-only results could suffer low `Precision' while achieving high `Recall'. Our findings further highlight a counter-intuitive observation that more generic prompts tend to yield better predictions of cyberattack tactics than those that are more specifically tailored.

翻译：战术、技术与程序（TTPs）概述了攻击者利用漏洞的方法。由于需要预设的专业知识和复杂的依赖关系，网络安全从业者在解读MITRE ATT&CK框架中的TTPs时可能面临挑战。与此同时，大型语言模型（LLMs）的进步引发了近期对其在网络安全运营中应用探索的研究热潮。然而，如何以高效且恰当的方式利用LLMs为网络安全等关键领域提供准确响应尚不明确。这促使我们研究如何更好地利用两类LLMs：小规模仅编码器模型（如RoBERTa）和更大规模的仅解码器模型（如GPT-3.5），以理解并总结网络攻击流程中TTPs的预期目的（即战术）。本研究比较了仅编码器LLMs的监督微调（SFT）与仅解码器LLMs的检索增强生成（RAG，无需微调）的应用。SFT和RAG技术均旨在通过为每个网络攻击流程提供相关上下文来增强LLMs。我们的研究表明，采用RAG的仅解码器LLMs比采用SFT的仅编码器模型表现更优，尤其是在RAG提取出直接相关上下文时。仅解码器模型的结果可能在实现高`召回率`的同时出现低`精确率`。我们的发现进一步揭示了一个反直觉的现象：相较于高度定制的提示，更通用的提示往往能对网络攻击战术做出更好的预测。