Complex software systems, e.g., Cyber-Physical Systems (CPSs), interact with the real world; thus, they often behave unexpectedly in uncertain environments. Testing such systems is challenging due to limited resources, time, complex testing infrastructure setup, and the inherent uncertainties in their operating environment. Devising uncertainty-aware testing solutions supported with test optimization techniques can be considered as a mandate for tackling this challenge. This paper proposes an uncertainty-aware and time-aware test case prioritization approach, named UncerPrio, for optimizing a sequence of tests to execute with a multi-objective search. To guide the prioritization with uncertainty, we identify four uncertainty measures: uncertainty measurement (AUM), uncertainty space (PUS), the number of uncertainties (ANU), and uncertainty coverage (PUU). Based on these measures and their combinations, we proposed 10 uncertainty-aware and multi-objective test case prioritization problems, and each problem was additionally defined with one cost objective (execution cost, PET) to be minimized and one effective measure (model coverage, PTR) to be maximized. Moreover, considering time constraints for test executions (i.e., time-aware), we defined 10 time budgets for all the 10 problems for identifying the best strategy in solving uncertainty-aware test prioritization. In our empirical study, we employed four well-known Multi-Objective Search Algorithms (MuOSAs): NSGA-II, MOCell, SPEA2, and CellDE with five use cases from two industrial CPS subject systems, and used Random Algorithm (RS) as the comparison baseline. Results show that all the MuOSAs significantly outperformed RS. The strategy of Prob.6 f(PET,PTR,AUM,ANU) (i.e., the problem with uncertainty measures AUM and ANU combined) achieved the overall best performance in observing uncertainty when using 100% time budget.

翻译：复杂软件系统（如信息物理系统CPSs）与现实世界交互，因此常在不确定环境中产生意外行为。受限于资源、时间、复杂的测试基础设施搭建及其运行环境的固有不确定性，对此类系统进行测试颇具挑战性。设计基于测试优化技术的不确定性感知测试方案，可视为应对该挑战的必要举措。本文提出一种名为UncerPrio的不确定性感知与时间感知测试用例优先级排序方法，通过多目标搜索优化测试执行序列。为引导不确定性导向的优先级排序，我们识别出四种不确定性度量指标：不确定性度量（AUM）、不确定性空间（PUS）、不确定性数量（ANU）及不确定性覆盖率（PUU）。基于这些指标及其组合，我们提出了10个不确定性感知的多目标测试用例优先级排序问题，每个问题额外定义了一个需最小化的成本目标（执行成本PET）和一个需最大化的有效度量指标（模型覆盖率PTR）。此外，考虑测试执行的时间约束（即时间感知），我们为全部10个问题定义了10个时间预算，以确定解决不确定性感知优先级排序问题的最佳策略。实证研究中，我们采用四种主流多目标搜索算法（MuOSAs）：NSGA-II、MOCell、SPEA2和CellDE，并使用两个工业CPS被测系统的五个用例，以随机算法（RS）作为比较基线。结果表明，所有多目标搜索算法均显著优于RS。当使用100%时间预算时，采用策略Prob.6 f(PET,PTR,AUM,ANU)（即组合不确定性度量AUM与ANU的问题）在观测不确定性方面取得了整体最优性能。