Fine-tuning large language models (LLMs) to aggregate multiple preferences has attracted considerable research attention. With aggregation algorithms advancing, a potential economic scenario arises where fine-tuning services are provided to agents with different preferences. In this context, agents may benefit from strategically misreporting their preferences, but this could harm the aggregation performance. This paper addresses such incentive issues by framing it as a mechanism design problem: an LLM provider determines the fine-tuning objective (training rule) and the pricing scheme (payment rule) for agents. We primarily focus on training rules that maximize social welfare subject to certain regularizations, referred to as SW-Max rules. First, we show that under most circumstances, truthful reporting is sub-optimal with simply a SW-Max rule, thereby highlighting the necessity of payments. Second, we extend the VCG payment to implement SW-Max rules in dominant-strategy incentive compatibility (DSIC). We characterize sufficient conditions for payment equivalence and derive the necessary conditions for a payment rule to implement a SW-Max rule in DSIC and other principles. Third, we demonstrate that our mechanism is approximately DSIC with perturbed input, showcasing its robustness against the inevitable errors in real-world applications. Experiments on real LLM training results further confirm the practical implications of our results.

翻译：通过微调大语言模型（LLMs）以聚合多种偏好已引起广泛的研究关注。随着聚合算法的进步，一种潜在的经济场景随之出现：微调服务被提供给具有不同偏好的智能体。在此背景下，智能体可能通过策略性地误报其偏好而获益，但这可能会损害聚合性能。本文通过将其构建为一个机制设计问题来解决此类激励问题：LLM提供者需为智能体确定微调目标（训练规则）和定价方案（支付规则）。我们主要关注在特定正则化约束下最大化社会福利的训练规则，称之为SW-Max规则。首先，我们证明在大多数情况下，仅使用SW-Max规则时如实报告是次优的，从而凸显了支付机制的必要性。其次，我们扩展了VCG支付，以在占优策略激励相容（DSIC）下实现SW-Max规则。我们刻画了支付等价性的充分条件，并推导了支付规则在DSIC及其他原则下实现SW-Max规则的必要条件。第三，我们证明了在输入存在扰动的情况下，我们的机制是近似DSIC的，这展示了其在实际应用中对抗不可避免误差的鲁棒性。基于真实LLM训练结果的实验进一步证实了我们结论的实际意义。