Generative surveying -- where collections of LLM-based personas provide feedback on messages -- has emerged as a cheap and scalable alternative to traditional market research. However, LLMs are sensitive to small variations in prompt design and conclusions drawn from generative surveys may depend on arbitrary phrasing choices. Controlling for this sensitivity requires including semantically equivalent perturbations in the analysis. In this paper, we show that standard hypothesis tests, including the sign test and Wilcoxon signed-rank test, are invalid under a statistical model for generative surveying that includes realistic perturbation structure. We propose a permutation test that is valid under this model and formally characterize the conditions under which standard tests fail. Applying our framework to a simple generative surveying problem, we estimate relevant parameters, characterize the power of the permutation test under realistic conditions, and provide practical guidance on budget allocation across personas, perturbations, and replicates. Finally, we show that both the magnitude and direction of the estimated effect are sensitive to the choice of model, even within the same model family.

翻译：生成式调研——即利用基于大语言模型的用户画像集合对信息提供反馈——已成为传统市场调研的一种廉价且可扩展的替代方案。然而，大语言模型对提示设计中的微小变化高度敏感，从生成式调研中得出的结论可能取决于任意的措辞选择。为控制这种敏感性，需在分析中纳入语义等价的扰动。本文证明，在包含现实扰动结构的生成式调研统计模型下，包括符号检验和Wilcoxon符号秩检验在内的标准假设检验均失效。我们提出一种在该模型下有效的置换检验，并严格刻画了标准检验失效的条件。将所提框架应用于简单的生成式调研问题，我们估计了相关参数，刻画了置换检验在现实条件下的统计效能，并就如何在用户画像、扰动和重复试验之间分配预算提供了实践指导。最后，我们证明：即使在同一模型族内，估计效应的幅度与方向均对模型选择敏感。