Background: A good search strategy is essential for a successful systematic literature study. Historically, database searches have been the norm, which has later been complemented with snowball searches. Our conjecture is that we can perform even better searches if combining the two search approaches, referred to as a hybrid search strategy. Objective: Our main objective was to compare and evaluate a hybrid search strategy. Furthermore, we compared some alternative hybrid search strategies to assess whether it was possible to identify more cost-efficient ways of searching for relevant primary studies. Method: To compare and evaluate the hybrid search strategy, we replicated an SLR on industry-academia collaboration in software engineering. The SLR used a more traditional approach to searching for relevant articles for an SLR, while the replication was conducted using a hybrid search strategy. Results: In our evaluation, the hybrid search strategy was superior in identifying relevant primary studies. It identified 30 percent more primary studies and even more when focusing only on peer-reviewed articles. To embrace individual viewpoints when assessing research articles and minimise the risk of missing primary studies, we introduced two new concepts, wild cards and borderline articles, when conducting systematic literature studies. Conclusions: The hybrid search strategy is a strong contender for being used when conducting systematic literature studies. Furthermore, alternative hybrid search strategies may be viable if selected wisely in relation to the start set for snowballing. Finally, the two new concepts were judged as essential to cater for different individual judgements and to minimise the risk of excluding primary studies that ought to be included.
翻译:背景:良好的搜索策略对于成功的系统文献研究至关重要。历史上,数据库搜索一直是主流做法,随后滚雪球搜索作为补充。我们的假设是,若将这两种搜索方法结合(称为混合搜索策略),能够实现更优的搜索效果。目标:主要目标是比较和评估混合搜索策略。此外,我们比较了几种备选混合搜索策略,以判断是否存在更高效的识别相关原始研究的方式。方法:为比较和评估混合搜索策略,我们重复了一项关于软件工程产学合作的系统文献综述(SLR)。原SLR采用传统方法搜索相关文献,而重复实验则使用混合搜索策略。结果:在评估中,混合搜索策略在识别相关原始研究方面表现更优。它识别出的原始研究数量增加了30%,若仅聚焦同行评审文章则提升更多。为纳入研究者评估论文时的个体观点,并降低遗漏原始研究的风险,我们在系统文献研究中引入了两个新概念:"wild cards"和"borderline articles"。结论:混合搜索策略是开展系统文献研究时的有力选择。此外,若根据滚雪球初始集合理选择,替代性混合搜索策略也可能可行。最后,这两个新概念被认为对于考虑不同个体判断、并最小化排除应纳入的原始研究的风险至关重要。