Reproducibility is a major concern across scientific fields. Human-Computer Interaction (HCI), in particular, is subject to diverse reproducibility challenges due to the wide range of research methodologies employed. In this article, we explore how the increasing adoption of Large Language Models (LLMs) across all user experience (UX) design and research activities impacts reproducibility in HCI. In particular, we review upcoming reproducibility challenges through the lenses of analogies from past to future (mis)practices like p-hacking and prompt-hacking, general bias, support in data analysis, documentation and education requirements, and possible pressure on the community. We discuss the risks and chances for each of these lenses with the expectation that a more comprehensive discussion will help shape best practices and contribute to valid and reproducible practices around using LLMs in HCI research.

翻译：【论文摘要】可重复性是各科学领域关注的核心问题。由于采用多样化的研究方法论，人机交互（HCI）领域尤其面临独特的可重复性挑战。本文探讨了大型语言模型（LLMs）在用户体验（UX）设计与研究活动中的广泛应用，将如何影响HCI领域的可重复性。具体而言，我们通过类比分析从过去到未来可能出现的（不当）实践（如p值操纵与提示词操纵）、普遍性偏差、数据分析支持、文档记录和教育需求，以及对学术社群可能造成的压力等维度，综述了即将面临的可重复性挑战。针对每个维度，我们深入讨论了其潜在风险与机遇，期望通过更全面的探讨推动最佳实践的形成，为HCI研究中合理运用LLMs建立有效且可重复的研究规范。