Two studies of a human-AI collaborative design tool were carried out in order to understand the influence design recommendations have on the design process. The tool investigated is based on an evolutionary algorithm attempting to design a virtual car to travel as far as possible in a fixed time. Participants were able to design their own cars, make recommendations to the algorithm and view sets of recommendations from the algorithm. The algorithm-recommended sets were designs which had been previously tested; some sets were simply randomly picked and other sets were picked using MAP-Elites. In the first study 808 design sessions were recorded as part of a science outreach program, each with analytical data of how each participant used the tool. To provide context to this quantitative data, a smaller double-blind lab study was also carried out with 12 participants. In the lab study the same quantitative data from the large scale study was collected alongside responses to interview questions. Although there is some evidence that the MAP-Elites provide higher-quality individual recommendations, neither study provides convincing evidence that these recommendations have a more positive influence on the design process than simply a random selection of designs. In fact, it seems that providing a combination of MAP-Elites and randomly selected recommendations is beneficial to the process. Furthermore, simply viewing recommendations from the MAP-Elites had a positive influence on engagement in the design task and the quality of the final design produced. Our findings are significant both for researchers designing new mixed-initiative tools, and those who wish to evaluate existing tools. Most significantly, we found that metrics researchers currently use to evaluate the success of human-AI collaborative algorithms do not measure the full influence these algorithms have on the design process.

翻译：为了解设计建议对设计过程的影响，我们开展了两项针对人机协同设计工具的研究。该工具基于进化算法，旨在设计一辆能在固定时间内行驶最远距离的虚拟汽车。参与者可以自主设计汽车、向算法提供建议，并查看算法给出的建议集。算法推荐的设计集均经过预先测试：部分集为随机选取，其余集则通过MAP-Elites方法选取。在第一项研究中，作为科学推广项目的一部分，我们记录了808次设计会话，并获取每位参与者使用工具的完整分析数据。为解释这些定量数据，我们还开展了一项包含12名参与者的双盲小型实验室研究。实验室研究在收集与大规模研究相同定量数据的同时，还记录了受访者对访谈问题的回复。尽管有证据表明MAP-Elites能提供更优质的个体建议，但两项研究均未有力证明这些建议比随机选取的设计对设计过程具有更积极的影响。事实上，结合使用MAP-Elites与随机选取的建议反而更有利于设计过程。此外，仅查看MAP-Elites的建议便能提升参与者对设计任务的参与度以及最终设计质量。我们的发现对设计新型混合主动工具的研究人员及评估现有工具的学者均具重要意义。尤为关键的是，我们观察到当前用于评估人机协同算法成功度的指标，并未全面衡量这些算法对设计过程的影响。