Large Language Models can generate synthetic survey responses at low cost, but their accuracy varies unpredictably across questions. We study the design problem of allocating a fixed budget of human respondents across estimation tasks when cheap LLM predictions are available for every task. Our framework combines three components. First, building on Prediction-Powered Inference, we characterize a question-specific rectification difficulty that governs how quickly the estimator's variance decreases with human sample size. Second, we derive a closed-form optimal allocation rule that directs more human labels to tasks where the LLM is least reliable. Third, since rectification difficulty depends on unobserved human responses for new surveys, we propose a meta-learning approach, trained on historical data, that predicts it for entirely new tasks without pilot data. The framework extends to general M-estimation, covering regression coefficients and multinomial logit partworths for conjoint analysis. We validate the framework on two datasets spanning different domains, question types, and LLMs, showing that our approach captures 61-79% of the theoretically attainable efficiency gains, achieving 11.4% and 10.5% MSE reductions without requiring any pilot human data for the target survey.

翻译：大型语言模型能以低成本生成合成调查响应，但其在不同问题上的准确性变化难以预测。我们研究了当每个任务均可获取廉价LLM预测时，如何在估计任务间分配固定数量的人类受访者预算这一设计问题。本文框架包含三个组成部分：首先，基于预测驱动推断，我们定义了问题特有的校正难度，它描述了估计量方差随人类样本量减少的速度；其次，我们推导出闭合形式的最优分配规则，将更多人工标注导向LLM最不可靠的任务；第三，由于校正难度依赖于新调查中未观测的人类响应，我们提出一种基于历史数据训练的元学习方法，无需试点数据即可为全新任务预测该指标。该框架可推广至一般的M估计场景，涵盖回归系数及联合分析中的多项逻辑部分价值函数。我们在涵盖不同领域、问题类型与LLM的两个数据集上验证了框架有效性，结果显示我们的方法能捕获理论可达效率增益的61%-79%，在无需目标调查任何试点人类数据的情况下，实现11.4%和10.5%的均方误差降低。