In-Context Learning has shown great potential for aligning Large Language Models (LLMs) with human values, helping reduce harmful outputs and accommodate diverse preferences without costly post-training, known as In-Context Alignment (ICA). However, LLMs' comprehension of input prompts remains agnostic, limiting ICA's ability to address value tensions--human values are inherently pluralistic, often imposing conflicting demands, e.g., stimulation vs. tradition. Current ICA methods therefore face the Instruction Bottleneck challenge, where LLMs struggle to reconcile multiple intended values within a single prompt, leading to incomplete or biased alignment. To address this, we propose PICACO, a novel pluralistic ICA method. Without fine-tuning, PICACO optimizes a meta-instruction that incorporates multiple values to better elicit LLMs' understanding of them and improve alignment. This is achieved by maximizing the total correlation between specified values and LLM responses, which theoretically reinforces value conformity and reduces distractive noise, resulting in more effective instructions. Extensive experiments on five value sets show that PICACO works well with both black-box and open-source LLMs, outperforms several recent strong baselines, and achieves a better balance across up to 8 distinct values.

翻译：情境学习在使大型语言模型与人类价值观对齐方面展现出巨大潜力，有助于减少有害输出并适应多样化偏好，而无需昂贵的后训练，这一过程被称为情境对齐。然而，LLMs对输入提示的理解仍不明确，限制了ICA处理价值张力（即人类价值观固有的多元性，常引发相互冲突的需求，例如刺激与传统的对立）的能力。当前ICA方法因此面临"指令瓶颈"挑战：LLMs难以在单一提示中协调多重预期价值，导致不完整或有偏颇的对齐。为解决此问题，我们提出PICACO——一种新颖的多元ICA方法。无需微调，PICACO通过优化整合多重价值的元指令，更有效地激发LLMs对这些价值的理解并改善对齐。该方法通过最大化指定价值与LLM响应之间的总相关性来实现，理论上有助于强化价值一致性并减少干扰噪声，从而生成更有效的指令。在五组价值集上的广泛实验表明，PICACO能良好适配黑盒与开源LLMs，超越多个近期强基线模型，并在多达8种不同价值间实现更优平衡。