Modern security operation centers (SOCs) rely on operators and a tapestry of logging and alerting tools with large scale collection and query abilities. SOC investigations are tedious as they rely on manual efforts to query diverse data sources, overlay related logs, and correlate the data into information and then document results in a ticketing system. Security orchestration, automation, and response (SOAR) tools are a new technology that promise to collect, filter, and display needed data; automate common tasks that require SOC analysts' time; facilitate SOC collaboration; and, improve both efficiency and consistency of SOCs. SOAR tools have never been tested in practice to evaluate their effect and understand them in use. In this paper, we design and administer the first hands-on user study of SOAR tools, involving 24 participants and 6 commercial SOAR tools. Our contributions include the experimental design, itemizing six characteristics of SOAR tools and a methodology for testing them. We describe configuration of the test environment in a cyber range, including network, user, and threat emulation; a full SOC tool suite; and creation of artifacts allowing multiple representative investigation scenarios to permit testing. We present the first research results on SOAR tools. We found that SOAR configuration is critical, as it involves creative design for data display and automation. We found that SOAR tools increased efficiency and reduced context switching during investigations, although ticket accuracy and completeness (indicating investigation quality) decreased with SOAR use. Our findings indicated that user preferences are slightly negatively correlated with their performance with the tool; overautomation was a concern of senior analysts, and SOAR tools that balanced automation with assisting a user to make decisions were preferred.

翻译：现代安全操作中心(SOCs)依靠操作者以及大量收集和查询能力的记录和警报工具的挂毯和操作者。SOC的调查是乏味的,因为它们依靠人工工作来查询各种数据来源,叠加相关日志,并将数据与信息联系起来,然后在售票系统中记录结果。安全管弦、自动化和反应工具是一种新技术,有可能收集、过滤和显示所需数据;将需要SOC分析员时间的共同任务自动化;促进SOC的合作;以及提高SOCs的效率和一致性。SOAR工具在实践中从未被测试过,以评价其效果并了解其使用情况。在本文件中,我们设计和管理SOAR工具的第一次亲手用户研究,涉及24名参与者和6个商业SOAR工具。我们的贡献包括实验设计,将SOAR工具的6个特性逐项列出,以及测试数据的方法。我们用网络、用户和威胁模拟等平衡范围来描述测试环境的配置情况;SOCSOC工具全套套;以及用具有多种代表性的调查情景来评估它们的效果,同时测试SOAR工具的升级。我们发现SAR工具是SOAA的升级。