Modern security operation centers (SOCs) rely on operators and a tapestry of logging and alerting tools with large scale collection and query abilities. SOC investigations are tedious as they rely on manual efforts to query diverse data sources, overlay related logs, and correlate the data into information and then document results in a ticketing system. Security orchestration, automation, and response (SOAR) tools are a new technology that promise to collect, filter, and display needed data; automate common tasks that require SOC analysts' time; facilitate SOC collaboration; and, improve both efficiency and consistency of SOCs. SOAR tools have never been tested in practice to evaluate their effect and understand them in use. In this paper, we design and administer the first hands-on user study of SOAR tools, involving 24 participants and 6 commercial SOAR tools. Our contributions include the experimental design, itemizing six characteristics of SOAR tools and a methodology for testing them. We describe configuration of the test environment in a cyber range, including network, user, and threat emulation; a full SOC tool suite; and creation of artifacts allowing multiple representative investigation scenarios to permit testing. We present the first research results on SOAR tools. We found that SOAR configuration is critical, as it involves creative design for data display and automation. We found that SOAR tools increased efficiency and reduced context switching during investigations, although ticket accuracy and completeness (indicating investigation quality) decreased with SOAR use. Our findings indicated that user preferences are slightly negatively correlated with their performance with the tool; overautomation was a concern of senior analysts, and SOAR tools that balanced automation with assisting a user to make decisions were preferred.

翻译：现代安全运营中心（SOC）依赖操作人员以及具备大规模数据采集与查询能力的日志记录和告警工具。由于需要人工查询多种数据源、叠加相关日志、将数据关联转化为信息，并在工单系统中记录结果，SOC的调查过程繁琐耗时。安全编排、自动化与响应（SOAR）工具作为一项新兴技术，能够收集、过滤并展示所需数据；自动化通常需要安全分析师投入时间的常规任务；促进SOC团队协作；并提升SOC的效率与一致性。然而，目前尚无针对SOAR工具在实际应用中的效果评估与使用理解的实证研究。本文设计并开展了首次针对SOAR工具的实操用户研究，涉及24名参与者和6款商业SOAR工具。我们的贡献包括实验设计、归纳SOAR工具的六项特征以及相应的测试方法。我们描述了在网络靶场中配置测试环境的过程，涵盖网络、用户及威胁模拟；一套完整的SOC工具套件；以及创建可供多种代表性调查场景测试的工件。我们首次呈现了关于SOAR工具的研究结果。研究发现，SOAR工具的配置至关重要，因为它涉及数据展示与自动化的创造性设计。此外，SOAR工具虽能提升调查效率并减少上下文切换，但使用SOAR后工单的准确性与完整性（反映调查质量）有所下降。研究结果表明，用户偏好与其工具使用绩效呈轻微负相关；高级分析师对过度自动化表示担忧，而能够平衡自动化与辅助用户决策的SOAR工具更受青睐。