We study LLM policy synthesis: using a language model to iteratively generate programmatic agent policies for multi-agent environments. Rather than training neural policies via reinforcement learning, our framework prompts an LLM to produce Python policy functions, evaluates them in self-play, and refines them using performance feedback across iterations. We investigate feedback engineering (the design of what evaluation information is shown to the LLM during refinement) comparing sparse feedback (scalar reward only) against dense feedback (reward plus social metrics: efficiency, equality, sustainability, peace). Across two canonical Sequential Social Dilemmas (Gathering and Cleanup) and two frontier LLMs (Claude Sonnet 4.6, Gemini 3.1 Pro), dense feedback consistently matches or exceeds sparse feedback on all metrics. We explain the asymmetry through feedback aliasing: when scalar reward alone maps distinct failure modes to the same value (e.g., under- vs. over-cleaning), social metrics break the alias and let the LLM diagnose which corrective direction to take. Social metrics thus function as a coordination signal rather than a distraction, yielding strategies such as Voronoi territory partitioning and waste-adaptive cleaner schedules. Code at https://github.com/vicgalle/llm-policies-social-dilemmas.

翻译：我们研究大语言模型策略生成：利用语言模型为多智能体环境迭代式地生成程序化智能体策略。我们的框架不采用强化学习来训练神经策略，而是通过提示大语言模型生成Python策略函数，在自博弈中评估这些函数，并依据跨迭代的性能反馈进行优化。我们研究了反馈工程（即设计在优化过程中向大语言模型展示何种评估信息），比较了稀疏反馈（仅基于标量奖励）与密集反馈（奖励加社会指标：效率、平等、可持续性、和平）。在两个经典序贯社会困境（Gathering和Cleanup）及两个前沿大语言模型（Claude Sonnet 4.6、Gemini 3.1 Pro）上，密集反馈在所有指标上一致达到或超过稀疏反馈的性能。我们通过反馈混叠解释这一不对称现象：当标量奖励将不同失败模式映射至相同数值时（例如清理不足与清理过度），社会指标能打破混叠，使大语言模型能够诊断应采取的校正方向。因此，社会指标起到协调信号而非干扰作用，产生了Voronoi区域划分和废弃物自适应清理调度等策略。代码地址：https://github.com/vicgalle/llm-policies-social-dilemmas。