Human-in-the-loop optimization utilizes human expertise to guide machine optimizers iteratively and search for an optimal solution in a solution space. While prior empirical studies mainly investigated novices, we analyzed the impact of the levels of expertise on the outcome quality and corresponding subjective satisfaction. We conducted a study (N=60) in text, photo, and 3D mesh optimization contexts. We found that novices can achieve an expert level of quality performance, but participants with higher expertise led to more optimization iteration with more explicit preference while keeping satisfaction low. In contrast, novices were more easily satisfied and terminated faster. Therefore, we identified that experts seek more diverse outcomes while the machine reaches optimal results, and the observed behavior can be used as a performance indicator for human-in-the-loop system designers to improve underlying models. We inform future research to be cautious about the impact of user expertise when designing human-in-the-loop systems.

翻译：人机循环优化利用人类专业知识迭代引导机器优化器，在解空间中搜索最优解。先前实证研究主要关注新手群体，而本研究分析了专业水平差异对最终质量及相应主观满意度的影响。我们在文本、照片和三维网格优化场景中开展了一项实验（N=60），发现新手能够达到专家级别的质量表现，但专业水平更高的参与者会进行更多优化迭代，提出更明确的偏好，同时保持较低的满意度。相比之下，新手更容易满足且终止速度更快。因此我们确认，当机器达到最优结果时，专家会寻求更多样化的输出，这种可观测行为可作为人机循环系统设计者改进底层模型的性能指标。我们建议未来研究在设计人机循环系统时需谨慎考虑用户专业水平带来的影响。