Regression testing for LLM-based systems poses unique challenges because individual regression instances provide limited information about system-level regressions. A failure in a single instance does not necessarily indicate a meaningful regression or provide sufficient information to diagnose affected behaviors. Conversely, detecting regressions based only on overall system performance changes is too coarse-grained, as it does not identify which behaviors are affected. This motivates analyzing regression instances at an intermediate level through test suite slices. To address this challenge, we propose CASPER, a change-aware slice prioritization framework for efficient regression testing of prompt-level and model-level changes in LLM-based systems. CASPER first identifies regression slices containing semantically related instances with consistent performance characteristics using an evolutionary slice identification approach. Given a change to an LLM-based application, CASPER prioritizes slices according to their likelihood of regression using behavioral information extracted from execution logs. We instantiate CASPER in the software issue resolution domain and evaluate slice identification against clustering-based baselines and regressed slice prioritization against a random ranking baseline. Results show that CASPER generates more consistent slices while maintaining comparable or better semantic coherence and improves regressed slice prioritization across different LLM-based system changes and testing budgets.

翻译：暂无翻译