AIOps (Artificial Intelligence for IT Operations) solutions leverage the massive data produced during the operations of large-scale systems and machine learning models to assist software engineers in their system operations. As operation data produced in the field are subject to constant evolution from factors like the changing operational environment and user base, the models in AIOps solutions need to be constantly maintained after deployment. While prior works focus on innovative modeling techniques to improve the performance of AIOps models before releasing them into the field, when and how to maintain AIOps models remain an under-investigated topic. In this work, we performed a case study on three large-scale public operation data to assess different model maintenance approaches regarding their performance, maintenance cost, and stability. We observed that active model maintenance approaches achieve better and more stable performance than a stationary approach. Particularly, applying sophisticated model maintenance approaches (e.g., concept drift detection, time-based ensembles, or online learning approaches) could provide better performance, efficiency, and stability than simply retraining AIOps models periodically. In addition, we observed that, although some maintenance approaches (e.g., time-based ensemble and online learning) can save model training time, they significantly sacrifice model testing time, which could hinder their applications in AIOps solutions where the operation data arrive at high speed and volume and where instant predictions are required. Our findings highlight that practitioners should consider the evolution of operation data and actively maintain AIOps models over time. Our observations can also guide researchers and practitioners to investigate more efficient and effective model maintenance techniques that fit in the context of AIOps.

翻译：AIOps（人工智能 for IT运维）解决方案利用大规模系统运维过程中产生的海量数据与机器学习模型，协助软件工程师进行系统运维。由于实际运维数据会因运行环境变化、用户群体演变等因素持续演化，AIOps解决方案中的模型在部署后需持续进行维护。现有研究主要聚焦于在模型发布前通过创新建模技术提升性能，但关于AIOps模型何时维护及如何维护的问题仍鲜有探讨。本研究通过三个大规模公开运维数据的案例研究，评估不同模型维护方法在性能、维护成本及稳定性方面的表现。我们发现，主动维护方法相比静态方法能实现更优且更稳定的性能。特别地，应用复杂模型维护技术（例如概念漂移检测、基于时间的集成学习或在线学习方法）相比周期性重训练AIOps模型，可提供更佳的性能、效率与稳定性。此外，研究还发现某些维护方法（如基于时间的集成学习与在线学习）虽能缩短模型训练时间，但会显著增加模型测试时间，可能限制其在高速高流量运维数据场景及需即时预测的AIOps解决方案中的应用。研究结果强调，从业者应关注运维数据的演化特性，并对AIOps模型进行主动维护。本研究的发现也可指导研究人员与从业者探索更高效、更适应AIOps场景的模型维护技术。